This was fun! Really great work on the art. I got myself stuck in a situation at one point where what looked like an inventory menu was open, but I couldn't figure out how to close or interact with it and ended up dying :( Nice work overall, though!
Yeah, the controls are admittedly a bit confusing. The panel you were stuck on was the building menu. You can cycle through the options (of which I only managed to implement two) by pressing tab, and cancel building with the X option.