I think the basic concept works, but the ramp-up is too sudden. Maybe start out with only 3 outlawed items at the start? It would also help to initially take out or reduce the point penalty for ‘kicked’ blue and green bugs.
Thinking further about it: Maybe it’s not that these things are too difficult, but that swapping out the wooden plates takes slightly too much time (relatively speaking).
I like the way the signs feel like physical objects just because of exactly the right sound effect.
( P.S. Also … is the player the bad guy? Keeping out bugs based on their color? :-p )