The die seemed really slippery like it was on ice. It seemed like it's movement treated it as a sphere, rather than a 20 sided die in how it interacted with the world. Maybe having a bumpy movement from that wouldn't work though.
when I had a collision that was more accurate to the 20 sided die the player would keep on getting launched into the sky, I should have experimented more with it to make it feel more like a 20 sided die