Not a dumb question! Since it's a binary choice, a really strict reading might leave you wondering how to roll sometimes when facing something that isn't literally the Wizard or a Mortal, so I intentionally left it a bit open to interpretation.
My preferred approach is to make the Wizard's minions a mix of mortals convinced to help them for one reason or another and various magical creatures that wouldn't exist if not for the Wizard, and to give the Mortals some problems that come from the Wizard and some that come from other Mortals. That way, the SkelJWs could have a mix of roll high and roll low when both trying to help Mortals and trying to go after the Wizard.
Environmental risks, traps, and other dangers that don't directly come from Mortals or the Wizard can be assessed based on who caused/initiated them, what purpose they caused the danger for, and/or who is aided by the danger's threat being realized.