I understand what you mean - numbers are a tool used to represent things, so many aspects of a game can be replaced by them. In my opinion, taking a game idea and replacing some of its components with numeric representations (for example, replacing blocks with a number indicating their height) or simply numbering game elements just doesn't quite hit the theme. That's probably what made it much harder (personally, also my perfectionism)
That's why I chose to make a game that uses numbers as a main element of a puzzle. However, later I found a way to remake my game without the use of numbers... ¯\_(ツ)_/¯