If anyone else has the same issue like me when I run a game that hammers my GPU after forcing DX12 with this "-force-d3d12", that issue went away.
And I limit the app to 60fps too. After that it was smooth all the way as before it would start to slowdown heavy.