itch.io

You can use ANY colors in the world, as long as there's only 4 of them on the screen. If you mean on a really technical level, the shades were determined by the hardware it was running on, so I can't really answer that.

I don't believe that there was anything stopping the display of the Gameboy to be any colour, however they would likely have to be monochromatic. On a purely technical level I believe that the palettes that the hardware could display were purely monochromatic, with the proper Gameboy displaying a 4 colours green palette and specific variations having a 4 colours grey scale (this is from a very limited knowledge so feel free to correct me if I'm wrong), no idea the exact reason for this but is likely something to do with the technical capabilities. So, for a pallette that at least seems that the game boy could run, I would suggest that you could simply get a 4 colour monochromatic palette, such as the typical Gameboy palette, and hue shift it to the colour range you want. This will hopefully provide an authentic Gameboy feel. Another tip for keeping your colours Gameboy-like would be having a slight desaturation of the colours as I don't believe that the original game boy was able to display highly saturated colours. There's no right or wrong way of doing this and you can use any colours you want in order to complete this project.

Tldr:

While there wasn't any colour restriction on the hardware that I'm aware of, if you want your palette to be Gameboy-like you should pick one colour and make a monochromatic palette based off of that.