Great suggestion!
To be honest, the post-jam full version is going to be on mobile, so wasn't planning on updating the mouse handling... that said, I think accessibility concerns like this are worth including in an update to the desktop / web versions too! Will add it to the list :)