I don't think it will make it annoying.. this game has a LOT of potential!
Well, I was thinking more of adding some puzzles to it, and additional (ranged) enemies. I think if I want to do that - player would have to have more control. But - that random movement thing could be perfect as additional mechanics after you do something (like - gotten hit by a goblin or drink some leftover beer to regain... Ekhm... Health).
But that's something I would have to test for :).