I think why I, and a lot of others it seems looking at the comments, struggled with the controls was partially because the 'flight' didn't match our expectations.
At least, when I imagine someone from the Touhou series 'flying', I imagine it as a lot snappier and more responsive than this. Like, in the Bullet hell games, you can stop, hover, and instantly speed away in a different direction. In this however, flight is more momentum based; you can't turn 180 and expect to keep your speed.
Still, after a few minutes I kind of got the hang of it and it was pretty fun. One thing I think its missing is sound; if you could add like subtle sounds of wind moving past and water moving I think it'd really add to the immersion and make it feel like you were really flying around Gensokyo.