As far as I’m aware, no one has made a DLL to do this in GameMaker yet, but I could do one if you’ve got $20 or so.
Viewing post in Screenshot effect question.
If you create your polygon shapes in advance (e.g. using window_shape_create_polygon_from_path) and use window_shape_set_nc, the underlying Windows functionality is pretty performant - in my test with flipping between two different dinosaur-based shapes the operation takes between 0.5-1ms on average, which is more than enough to do this at 60fps.