I feel like there are so many things that could be done.
Play with the evolution of Game Boy. Start with a standard and have the world be black and white and then into color with a Gameboo Color. Have a lack of backlight so you need to be around lit areas in the 3D World to see the screen. Maybe upgrade to a backlit model down the road.
Could play with the DS Idea of Pictochat. Have chat messages pop up when in range of potential danger.
Using the Gameboy as a way to equip items/handle inventory much like an old JRPG or Pokemon would. Handling all the gamey sort of things with the handheld would create a fairly unique atmosphere IMO.
Seeing something or someone in first person that looks innocent but swapping to handheld mode would show behind them due to the 2D Nature potentially revealing danger.
I could go on and on. A lot of these ideas probably wouldn't work or have some strong problems in terms of designing but I just think playing around with the concept could present so many interesting gameplay opportunities. It's definitely a mechanic that an entire game can be built around.