There's a bug with the attack speed. It seems that the higher my attack/s get, the slower the towers actually attack. When I stopped believing what the info said and threw the attack speed upgrade out of my deck the game suddenly got a lot easier.