Nono, I think that makes sense and is clear.
And I think you might be right. It's just very jarring to go from "everything is one hit" to "things have hit points". Most players won't have this experience so maybe it's fine. :)
