Edit: I figured it out. Pressing Z opens the inventory, pressing it again closes it without selecting the new item(although it is still indicated in the bottom left and it looks selected). I needed to select the item/close the inventory with X for it to work as intended :D.