Allow and Deny buttons have shortcuts! It's Space Bar for Allow and Shift for Deny.
About the randomness, yeah, I felt that too... Sadly I was out most of the weekend and could not improve it but thought it was enough for the Game Jam.
Thanks for the comment !