Hold down the buttons instead of just pressing them. X + UP
just tried it again, it closes only once out of ten times (and even then it immediately opens back up again)
I have changed opening and closing the inventory from the pause menu.
