Hi,
I thought so, as it usually happens with mem usage around 1,7-1,8GB. So I guess you forgot to check return value of some malloc()-Alike functions (VirtualAlloc, HeapAlloc, ..), and so...
;-)
As for the Win7 system, I only booted it an started up the game, so no other applications taking up memory. Pagefile is turned on, so OS should be able to do paging. I once accidentally started ROTT on Windows XP machine twice and there I was in paging hell...
I saw that your executable is large-address-aware:
dumpbin /headers RotT-Win32.exe
.... Application can handle large (>2GB) addresses
32 bit word machine
But I don't use the /PAE and /3GB switches in boot.ini (or its equivalent in Win7, if there is something like that), so could that be something to try? I think it was causing some troubles for me or my drivers and also it's not really recommended to be used, so that I didn't enable it. Without PAE and 3GB switch, 2GB per-process limit is still kicking in, even if executable is LAAW. Still wondering what is sucking up so much memory, though... Paging won't help if 2GB barrier gets hit somehow.
To reproduce the error, maybe you can try to use https://github.com/lowleveldesign/process-governor and enforce a tighter memory limit on the process on your machine?