There is no size limit for the jam itself. Itch has a size limit of 1GB which can be raised by request. https://itch.io/docs/creators/faq#what-limits-apply-to-projects-and-accounts
Also, keep in mind that larger downloads may dissuade other people from trying your game.