what about with game maker?
Viewing post in Screenshot effect question.
If you create your polygon shapes in advance (e.g. using window_shape_create_polygon_from_path) and use window_shape_set_nc, the underlying Windows functionality is pretty performant - in my test with flipping between two different dinosaur-based shapes the operation takes between 0.5-1ms on average, which is more than enough to do this at 60fps.