Post by trimute2 in SIC-1 comments

Viewing post in SIC-1 comments

since cycles and bytes are only counted for the first set of inputs optimization scores are really easy to abuse, its often possible to output the requirements for the first test above the actual programed solution. This is especially easy to do on the last level due to the nature of the levels challenge.

its been awhile since a played other zach-likes but if I remember correctly they usually use the score your solution scored the worst on. Though here the program doesn't reset between tests so I'm not sure if that's do-able.

jaredkrinke2 years ago

Thanks! There is some logic to filter out these kinds of submissions on the server, to avoid showing them to other players (the submitter will still see their scores, however, as in your screenshot).

Having said that, the current filtering mechanism is not perfect, so some determined player could certainly get their bogus scores to appear on those histograms (side question: if someone goes through that much trouble, should I be flattered?). I could add a heuristic that tests programs on random inputs to make sure they're similar to the fixed inputs.

trimute22 years ago

I don't know if this is possible with the way the game is set up, but what I would recommend is simply restarting the players solution on the second set of tests. The way I abused the system was simply hard coding the correct output for the first test above the actual solution, if the program where to run from the start for the second set of tests that hard coded output would cause the solution to fail.

jaredkrinke2 years ago

I haven't looked at the code in a while, but it's likely that that is exactly what I'm doing on the server side. It would make sense to do the same thing on the client side (I don't recall if there was a reason for the discrepancy), although that could potentially break some existing solutions. Regardless, thanks for flagging this for me!

jaredkrinke2 years ago

Just released an updated client that resets the simulation (and stats) when advancing to the next test set. This is similar to what is done server side, and should prevent this trick (except for the handful of "tutorial" assignments that don't validate solutions using random data).

jaredkrinke2 years ago

One additional note: prior to February 2020, it was possible to submit hard-coded output like this and make it onto the histograms. This is no longer possible, but I have to go back and clean up the histograms.

I'm also planning to see if I can, as you suggested, reset the machine for each set of tests (just need to ensure this doesn't break legitimate solutions).

itch.io

Viewing post in SIC-1 comments