The gameboy actually has a considerably higher resolution at 160×144 as opposed to the jam's 64x64 limitation. So just using GBStudio would not be low enough of a resolution. As long as you limit the engine to displaying only 64x64, it's just like any other engine and can be used though I'm sure. I would suggest having a windows or html5 build of it to submit to the jam. If you only give a gameboy rom image, I have a feeling it wouldn't be downloaded/played/rated as much.
EDIT: Note that I am not sure if it's possible to limit gbstudio to a different resolution.