Thanks for the feedback!
There's actually no interact button. Switches and secret walls are activated either by running into them, or hitting them.
Garden Shuffle is actually procedurally generated. The red and yellow keys with both spawn in their own hedge maze area, opposite to the red/yellow door respectively. But I'll agree I should probably mark them on the map.