can I make a 3d game in ue5 or ue4

Of course you can, but not for this competition.

no, this jam is for retro homebrew on the Nintendo Game Boy

More specifically:

• The final deliverable must consist of a GB or GBC ROM file.
• GBC features are allowed (entries may be GBC only)
• SGB features are allowed (but don't rely on them)
• You can use RGBDS (ASM), GBDK (C), ZGB (C), GB Studio, GBSDK(C+RGBDS ASM) or anything that produces a valid ROM.