It's actually triggered by answering yes to there being a problem and then yes that you were joking. This will make the narrator mad at your dishonesty and cause the hard problem.
Divide all the terms by x^3 and then substitute infinity for x. Since you will be left with 3/infinity, the answer is zero because any finite number divided by infinity is zero.
The ending where you hear the passcode is called, "Bottom of the Barrel" Achieved by doing very poorly on the first part of the test. When the narrator goes to show you the end screen, he says his password as he's typing it.
