Normal mode only starts lagging at around stage 90 in Endless mode. Next time I play I'll try see if it's only certain reliquaries or the mode in general.
Also you might've considered already but the endless amount of aspects for that one guy is broken. I've almost (barring some misplays) gotten him to oneshot 100+ endless waves. It's surprisingly good for entertainment though since almost anything can pivot. Maybe make that unit cost 20 for symbolism.