I would need 2x scaling for my 2k display.

Dark-Ice is larger than most. Edit the default.theme file.
Adjust MenuIconSize, SmallIconSize, LargeIconSize, HugeIconSize.
Also adjust the font sizes.
Share your best efforts here for discussion and for others to improve upon.

@gijsbers The icons in the window title bar are still very small, any idea how to cope with that?

Yes. Consider Fedora and AntiX. They provide IceWM themes for HiDPI displays.
Copy a large theme you like to your ~/.icewm/themes/ directory and cd into the theme directory.
Identify the icons which are on the title bar. Say close*.xpm, mini*.xpm and maxi*.xpm.
Make sure you have convert from ImageMagick. Then do something like this:

identify close*.xpm mini*.xpm maxi*.xpm
for f in close*.xpm mini*.xpm maxi*.xpm; do convert $f -resize 200% $f; done
identify close*.xpm mini*.xpm maxi*.xpm

Then adjust TitleBarHeight= in the default.theme file to the appropriate size.

I think the height in pixels of the menu button controls the height of the taskbar. In my new antiX-Magic theme, I had to make each size of the theme separate so that the taskbar would resize properly. In the theme, which I originally copied from the Materia theme, I reorganized the lines so that all of the heights and font sizes that needed to change for the different sizes were at the top to make it easy to clone it to a new size, like my antiX-Magic-Huge4K theme. This means that my sub-themes will be for differences on that size of the theme, using the same building blocks. The thread with the links to the files at the end is here: https://www.antixforum.com/forums/topic/new-theme-for-antix-icewm/page/3/ I created Small, Medium, Large, Huge4K, and Thin (for super efficinent space usage) themes, basically all the same except for sizing. Here is a pic of the Huge4K one in 3840x2160 resolution...

Running icewm from Ubuntu 22 gdm the DPI is scaled for 4k, even on small screens. I can't figure out how to get it back, just lost an hour of my life to this. If I start with startx, all is well.

Looks like a GDM problem then. IceWM doesn't know much about DPI. Let alone how to scale it.

Solved with this ~/.Xresources file:

Xft.dpi: 96

This page came in handy: https://wiki.archlinux.org/title/HiDPI#Xorg

Not sure if it could've been this that did the job though, I did both before last login:

gsettings set org.gnome.settings-daemon.plugins.xsettings overrides "{'Xft/DPI': <98304>}"