Xbox rounds -0.0 to the negative range and 0.0 to the positive range. This
commit also restores RCC instruction clamping to be done on the output of
reciprocal calculation (which current Xemu release does) with fix for the
input=Infinity case.
If tPosition is a zero-vector, then invViewport matrix had no effect.
Bounding w-coordinate away from zero and infinity must be done before
applying invViewport (which is needed for OpenGL/Vulkan) to emulate
Xbox hardware behaviour properly.
z_perspective is true implies w-buffering and then the w-coordinate stored
in the depth buffer should also be interpolated in a perspective-correct
way. We do this by calculating w and setting gl_FragDepth in the fragment
shader.
Since enabling polygon offset and setting values using glPolygonOffset
won't have any effect when manually setting gl_FragDepth for w-buffering,
we introduce the depthOffset variable to obtain similar behaviour (but the
glPolygonOffset factor-argument is currently not emulated.) (Note that
glPolygonOffset is OpenGL implementation-dependent and it might be good to
use depthOffset for z-buffering as well, but this is not done here and we
still use OpenGL/Vulkan zbias functionality.)
This also implements depth clipping and clamping in the fragment shader.
If triangles are clipped, the shadows of the small rocks in Halo 2 Beaver
Creek map can have flickering horizontal lines. The shadows are drawn on
the ground in another pass with the same models as for the ground, but for
some reason with depth clamping enabled. The flickering happens if Xemu
clips the ground triangles, but the exact same shadow triangles are depth
clamped, so there are small differences in the coordinates. The shadows
are drawn with depth function GL_EQUAL so there is no tolerance for any
differences. Clipping in the fragment shader solves the problem because
the ground and shadow triangles remain exactly the same regardless of
depth clipping/clamping. For some performance gain, it might be a good
idea to cull triangles by depth in the geometry shader, but this is not
implemented here.
In the programmable vertex shader we always multiply position output by w
because this improves numerical stability in subsequent floating point
computations by modern GPUs. This usually means that the perspective
divide done by the vertex program gets undone.
The magic bounding constants 5.42101e-020 and 1.884467e+019 are replaced
by 5.421011e-20 and 1.8446744e19, i.e. more decimals added. This makes the
32-bit floating point numbers represent exactly 2^(-64) and 2^64 (raw bits
0x1f800000 and 0x5f800000) which seem more likely the correct values
although testing with hardware was not done to this precision.
Testing indicates that the same RCC instruction magic constants are also
applied to both fixed function and programmable vertex shader w-coordinate
output. This bounding replaces the special test for w==0.0 and abs(w)==inf
which used to set vtx_inv_w=1.0 (which did not match Xbox hardware
behaviour.)
