I think that that would make the game much harder in later levels. ( Even harder than they already are ). I can understand having a harder time to remember which color is which, but I think that that can add to the desired difficulty. If I were to make more than 20 levels, and really expand outside of the jam, I would hope to design each level with the idea of introducing new colors as the levels progressed. I would try to hard wire the player's brain into remembering which color was which key, by getting them to use 2 to 4 colors a lot in the earlier levels.