Incredibly, I have accidentally figured it out upon reaching the sword ghost! Seems there is indeed a separation between magical and physical attacks in this game
all the attacks just do 2 damage, so the first two hp upgrades essentially dont do anything since you start at 0 h
