Good question. In my case, for HR, I wanted to go with the self-insert approach. When making him, I was wondering how to do this properly. The best answer I could reach was to make him devoid of a strong personality. He has many traits that clashes with each other. He can be gentle, or sadistic. He can be proactive or timid. Impulsive or indecisive. Perverted or serious. Etc.
He being "passive" like you say, it's on purpose. The girls will comment on this sometimes, wishing he could be more aggressive. And that's the thing, he can be hotblooded too.
And that bring us to the girls. Yes, they get interested in him way too quickly (with some exceptions). That's on purpose too. Otherwise it would go against the general genre and setting of this game. While each girl has their own personality and problems you need to take care of, the main goal is to fill the house of cute girls and bang them. This is a harem eroge after all.
That being said, I'd like to believe it's not as bad as some JP eroge I usually play, where you get to bang the girl in the first hour or two of meeting her.
In any case, this is the logic I went with. But in reality I might just be a bad writer. But I can only improve if people play my stuff, so~
I always welcome and value any kind of criticism. You're welcome to join the Discord server and discuss further if you like.