Using tools other than Octo to develop a game is entirely valid. Many participants in the past have made their own forks with new debugging features. Some of these ultimately made their way into mainline. In other cases, participants made their own assemblers from scratch.
The main goal of the jam, though, is to work within the existing creative constraints of the CHIP-8/SCHIP/XO-CHIP specifications. I'd strongly recommend trying to stick to them. For example, you could achieve something like what you're proposing by building your levels around using only pairs of colors from a carefully-chosen XO-Chip four-color palette.
It would also be possible to write what you want in such a way that it works properly (without palette swaps) on Octo's reference runtime, but swaps palettes in a specially-prepared runtime. (edit: you pointed this out yourself while I was composing my reply!) If you were to add such a feature it is essential that you make this clear on your submission page, out of fairness for other participants.