Hmm, normally you have to blow him 4 times for it to trigger. That's the last thing I can think of that prevent you from meeting him.
Have you update the save? Also, you can access the gym in the outside.
Let me know if it still doesn't work.
If you can't even see the option for Gym from the outside, it is most likely that you have an old version of the game downloaded.
For a period of time, I forgot to removed the v0.3 from the itch download section, so you may have mistakenly downloaded it. The latest public version is v0.4d.
Let me know if you're able to resolve the problem by doing this.