So finally got to test this, and it only does this when you run the jar file using the built-in jdk of macOS aka it's a problem with the jdk. Using a macOS version of jre (8u221) in the *.app/Contents/Resources folder did not crash it when toggling fullscreen. Turns out the jre inside the app is the Windows version which is why you have to replace it to make it work
Here's how to fix the Mac build
https://itch.io/t/1069538/doesnt-work-on-mac