The game literally tells you how to do this. If you get a number higher than 10 then you would just put the number in the ones place as your answer, so 11 would be 1 and 12 would be 2, so on and so forth. you will never have a negative either because regardless of what the math says you always take the bigger number minus the smaller one.
Viewing post in Voices of the Void 0.8.1 feedback
Bruh, lmao. The exact words in the tutorial are "simply remove the tenths column." In hindsight I understand this, but the meaning of the words specifically is unclear, especially in the use of the word "column." This led me to think it was referring to one of the columns on the screen, as that's designed in rows and columns similar to a spreadsheet. The spreadsheet comparison is made stronger by the use of MOD(), which, unless it has some secondary meaning, is nonsensical.
As for the second point, you got the correct answer, but the way you're doing it isn't mathematically correct. It's the result of an unnecessary simplification without an understanding of the operations. It also flies directly in the face of the tutorial text, which specifically provides a negative result as an example. This is also another point that's unclear, as it doesn't explain that it expects the absolute value, I didn't remember that 1-8=-7 but |1-8|=7, and many people won't know that an equation surrounded by pipes means absolute value. On top of that, the include of MOD() around the equation threw me off even more because I was trying to figure out how a broken modulo operation fits in.
So, no, the game does not literally tell you how to do this. There are gaps in the explanation of what's wanted.