Okay it *might* be a "me" issue or something, since a friend of mine got it to work just fine (although it used 5 GB, which makes me think something is being 32-bit in the code somewhere on my end)
He's running an RX 570 with the mesa drivers
On the old renderer, he got 1323/52.9/15.2, 128/5.1/0.5, and 0/NaN/? (It wouldn't respond to click or anything, but closed immediately when he clicked the x of the window)
On the new renderer, he got 1308/52.3/22.7, 996/39.8/21.7, 1051/42.0/20.4, so it seems this new render helps a lot!