I could be wrong, but it has to be at night I think, you can pass time in your room Before packing things up
I Keep passing time over and over yet he won't appear. Nanami even says she should go say goodbye. Is he in a different place
u should bring akemi as well and he will appear in the usual spot