It's pretty tricky adding mod support, as I'm using paid addons for all the physics simulations. It might need a lot of work. But I am thinking about it!
For physics you can try to look into cocos creator or cocos2d-x. There free and open source game engines. With cocos2d-x you can change the code to your liking. On top of that cocos is royalty free.
Edit: it supports .fbx files natively to the engine.
