Alright so after a lot of blind digging, I found that changing the pixel per unit of the sprite affected how the scale responded. I finally worked out that if the camera pixel per unit (which is not given by the framework currently!) is different from the sprite's you need to multiply the scaling factor by `spritePixelPerUnit / cameraPixelPerUnit` and it'll fit!
I think the camera pixel per unit ratio is something that should be added to the engine, otherwise such calculations are near impossible without some dirty hacks.