In the end it always comes down to what you think is best. Like No Time To Play already mentioned, using a framework, or( pre-made game engine for that matter) will always lead to a slowdown... It can also limit your possibilities. If you don't mind that and think the development will be easier that way and think that would turn out better despite the setbacks, it could be the right way to go, but if you already well-versed in working more low-level and you think you can do it without any framework, why not stick with that? After all, if you pick a framework after the development has already begun you also risk that your own code is too much different from what the framework requires and that you may end up having to redo the entire game from scratch.... Nothing works more demotivating than that, I tell ya.
Another downside of existing frameworks is that you are bound to what the framework designers want. I quit some frameworks myself in the past because of decisions by the developers that were in my opinion beyond stupidity. At one framework I even requested an account deletion and everything I ever posted there (this due to a combo of stupid decisions and a community I was fed up with anyway). The advantage of using your own code and your own frameworks is that you are not reliant at all. And sometimes decisions are so drastic they can cost you your project. From what I see, the discontinuation of "UnityScript", and older versions of Unity used to support "Boo", but that too was discontinued... I guess too little people used Boo, but if you were one of those few, you're screwed.
Not that I want to scare you away from using existing frameworks, as it can also have many benefits (like taking a lot of trouble out of your hands), but it is definitely something to think about.
In the end, you da boss... Overall I think it's unwise to ask what tool to use, or to use a framework or go for low level programming. In the end, it always comes down to the kind of programmer you are. To a lot of people their programming language is like a religion... Criticize it and be prepared for very rude and sometimes even threatening behavior. I do remember getting yelled at after I said I hate COBOL and that by doing so I'd be neglecting programming history (since COBOL is the oldest compiler still in use), however that is quite harsh if you know that I once wrote an article on Game Jolt in which I praised Major Grace Hopper, the inventor of the first compiler and also the inventor of COBOL. It just shows how passionate people are. And also a bit arrogant about their language, if you don't mind me saying it. I use C++ myself, because I value my self-reliance, but no I don't love the language... I think it's crap... I use it because of lack of alternatives at the present time which fulfill all my needs. C++ doesn't fulfill them either, but at least I can use C++ to come as close as possible... at least now... I don't see why using C++ makes me better than others... I also use C# at times, or Python, or Lua, and (if I really have to) JavaScript.... And well more languages... It's just what I think is best at the time... If there's anything at all that could make me a better coder it's not my choice of language... I think it would rather be my over 35 years of experience in this field... 😁