That prompt is not automatic, but it happens when you click on the campfire icon (the quickslot in the Dungeon). I think the controller is somehow highlighting and triggering it OR the campfire key is mapped to the same key.
While controller is supported (tested with mine without any issue), it can be a bit buggy due to my inexperience back when I handled the input system.
It would help to know exactly why it's happening (same mapped key or highlighting by mistake) so I can attempt to fix it, but in general I suggest using Keyboard and Mouse for the best experience.
Regardless, I'll note the issue down so I can consider the buttons from highlighting since they're mapped to the gamepad anyway. Not sure if that'll fix the issue though, just my theory of what's happening.