no sorry it is this that needs tweaking
#define round(X) (int)((X)+0.5
This is just a standard way to round to the nearest int, which needs to be done because you can't have a separation of 60.5 pixels, for example, because pixels come in wholes.
So the only things I can change that would affect the layers are DPI and mu (depth of field). There are combinations of these that give a better spread of separation, but it often turns out messing up the stereogram.
You have obviously chosen the depth of field and DPI values carefully, if you can cycle through all the values that give a good separation you may approximate a 3D scene
You could have a seperation of half a pixel, it's just this pixel takes on 2 values over 2 consecutive frames.........
To this end you could have any fraction of a pixel within reason of frame rate, bit depth and human perception...