Well, I think it’s best to go back to the idea of the jam.
It’s meant to be a time-limited focus on the project, and you don’t just start out before the jam starts – also because that would be disrespectful and unfair towards the other participants.
That being said, where to draw the line between project focused work and general work is pretty subjective. You can say you create an engine, potentially for this project, but it's a general purpose engine that you may use for other stuff as well, or is a learning process independent from this project.
If the engine is specifically for this project, maybe it's just too big of a scope for a game jam with such a short time limit?