What program do you use for modeling it doesn't look like Koikatsu or anything else I have seen in H games.
It's Koikatsu, but I went away from the default look (and especially head shape) everyone uses
The shading style is a lot different also how did you set it up?
I've played around a lot with different combinations of settings until I had found one I was happy with.
Which set up are you asking about specifically?