Using a framework is going to badly hurt performance no matter what, and it sounds like you're already well on the way to making your game. Stay the course. :)
I will do my best, thanks for the support :)