Has this always happened? What OS are you running?
"binds.cfg" in your config dir is where the key bindings are defined, you could have a look at that file and see if anything looks out of the ordinary. i think the behavior of the mouse buttons might be hardcoded though.
I've thought about this a lot over the years since it's such an obviously slow process, would lend itself extremely well to a multi-core and/or GPU accelerated approach, but it's never quite been clear how I'd pull it off - I'm not a particularly knowledgeable or powerful programmer in the directions that would make it easy to try those approaches. I'll let you know if I ever figure anything out.
Some of my old notes about it are on the playscii trello board, which I haven't touched in some time:
