I think you misunderstood me. Sorry if the way I worded it wasn't super clear.
The door doesn't hide itself if you miss something. It only shows up after you walk through it.
Since you can't walk through it from the outside, you need to walk through it from inside the tower.
After you've walked through it from inside, then it'll show up outside.