Taken on it's own, I like the concept as a game.
My one criticism is that from a Game Jam perspective the use of "Only One" color feels more like a hindrance to the game than an asset. I think the most interesting potential for puzzles would stem from using multiple colors at a time, or even combining two colors to make new colors. Just my two cents.
If you keep developing this you could end up with something great!