I'm in a castle area after getting the ghost ability, and I can't find the boss key. I checked a walkthrough and I literally opened the chest already, all I can do is dig under the boss gate which feels like cheating. I can also see a chest in the wall but it's surrounded by blocks I can't dig through.
I think digging under the gate is intended to be an option. When the game doesn't want you to be able to do that, it sticks anti-magic fields by the gate (although with stamina upgrades this may not actually stop you), just like the switch-operated gate you saw in the quadrant that was supposed to have the boss key.