Thanks for the feedback!
The physics were a bit of a fight. Even after disabling the character's ability to push and interact with physics objects, The engine would constantly bounce them around. I ended up increasing the weight of objects to counter it, but I guess it didn't work to full effect.
The portal should not move, They were locked in place. Maybe I overlooked it.
Again, Thank you for playing!