Interesting mechanic, I've not seen this before! However, it felt like I could just keep the goal colour until the end and then I'd be okay. As others have said, it might be good if there's a limit on time or perhaps the number of colour swaps or something.