Haha, it's only frustrating because the game does a good job of getting you gripped into the narrative / presentation. I actually ate the "!" but it didn't appear to have any effect I could notice on my playthrough.
Idea for how to implement check points:
Perhaps the author removes a page off the typewriter at a certain point, and you always start at the most recent page. This would fit thematically, and you could even have the author write about how strange it is that the centipede seems to appear even when a fresh page is inserted into the typewriter. Maybe the author is afraid of what might happen if they ever stop typing...maybe it will crawl off the page and do nefarious things. :D