My idea would be to make a game in pico-8, with Fennel, but pico-8 by itself is proprietary.

As long as your game code and assets use FOSS/free culture licenses then your game meets the jam requirements. Some players will be okay with using pico-8 to try your game, others might skip it because of the proprietary license.