nv2a: color_binding || zeta_binding assertion failure

[RageXbox]

Title

https://xemu.app/titles/5451001a/#Nickelodeon-SpongeBob-SquarePants-in-Battle-for-Bikini-Bottom
https://xemu.app/titles/4d570005/#Area-51
https://xemu.app/titles/5553001c/#Batman-Rise-of-Sin-Tzu
https://xemu.app/titles/5553004b/#Cold-Fear
https://xemu.app/titles/55530018/#Beyond-Good-Evil

Bug Description

nv2a: color_binding || zeta_binding assertion failure from bubble moves.

Expected Behavior

It shouldn't crash from bubble moves.

xemu Version

v0.6.2-34-g62d04a3636
Was working in v.0.6.2-32-gd0fd0cb3e4

System Information

OS: Windows 10
CPU: Intel(R) Xeon(R) CPU(TM) E5-2678 v3 @ 2.50GHz
GPU: NVIDIA Quadro P5000
GPU Driver: 462.96 NVIDIA

Additional Context

No response

[re4thewin]

this error flips between line 2615 and 919 in pgraph.c every other assert

[re4thewin]

on the newest master (41) these lines have changed to 2630 and 934 in pgraph.c

[antangelo]

Combing through the compat reports, these other games run into this assertion:

• Area 51 (4d570005), occurs when picking up pistol
• Batman: Rise of Sin Tzu (5553001c), crashes immediately
• Cold Fear (5553004b), crashes when you try to open a door

[abaire]

Batman is tracked with #561 so one should probably be closed as a dup of the other

[antangelo]

I've been looking into 5451001a using nv2a-trace and it appears that neither the color nor zeta buffers are changed after the offending SET_BEGIN_END instructions on real hardware. Going into SET_BEGIN_END, the game sets the color mask to zero and disables the depth and stencil tests. In xemu, this (along with marking the surfaces as dirty) results in both buffers being unbound.

I'm not sure what the purpose of running SET_BEGIN_END with no color masks, depth or stencil testing enabled is. The game is sending vertices over, so it's not empty either.

[abaire]

I've been looking into 5451001a using nv2a-trace and it appears that neither the color nor zeta buffers are changed after the offending SET_BEGIN_END instructions on real hardware. Going into SET_BEGIN_END, the game sets the color mask to zero and disables the depth and stencil tests. In xemu, this (along with marking the surfaces as dirty) results in both buffers being unbound.

I'm not sure what the purpose of running SET_BEGIN_END with no color masks, depth or stencil testing enabled is. The game is sending vertices over, so it's not empty either.

Double checking: is it clearing both the color and depth masks (NV097_SET_COLOR_MASK && NV097_SET_DEPTH_MASK)?

[antangelo]

I've been looking into 5451001a using nv2a-trace and it appears that neither the color nor zeta buffers are changed after the offending SET_BEGIN_END instructions on real hardware. Going into SET_BEGIN_END, the game sets the color mask to zero and disables the depth and stencil tests. In xemu, this (along with marking the surfaces as dirty) results in both buffers being unbound.
I'm not sure what the purpose of running SET_BEGIN_END with no color masks, depth or stencil testing enabled is. The game is sending vertices over, so it's not empty either.

Double checking: is it clearing both the color and depth masks (NV097_SET_COLOR_MASK && NV097_SET_DEPTH_MASK)?

Both color and depth are cleared. The stencil mask isn't being cleared (or set at all during the nv2a-trace capture).

[abaire]

I've been looking into 5451001a using nv2a-trace and it appears that neither the color nor zeta buffers are changed after the offending SET_BEGIN_END instructions on real hardware. Going into SET_BEGIN_END, the game sets the color mask to zero and disables the depth and stencil tests. In xemu, this (along with marking the surfaces as dirty) results in both buffers being unbound.
I'm not sure what the purpose of running SET_BEGIN_END with no color masks, depth or stencil testing enabled is. The game is sending vertices over, so it's not empty either.

Double checking: is it clearing both the color and depth masks (NV097_SET_COLOR_MASK && NV097_SET_DEPTH_MASK)?

Both color and depth are cleared. The stencil mask isn't being cleared (or set at all during the nv2a-trace capture).

Interesting, is it rendering shadows, by any chance? I'm wondering if the stencil buffer is writable and it's intentionally not modifying the color/depth buffers. Do you happen to know if the depth buffer is set to d24s8?

[antangelo]

Double checking: is it clearing both the color and depth masks (NV097_SET_COLOR_MASK && NV097_SET_DEPTH_MASK)?

Both color and depth are cleared. The stencil mask isn't being cleared (or set at all during the nv2a-trace capture).

Interesting, is it rendering shadows, by any chance? I'm wondering if the stencil buffer is writable and it's intentionally not modifying the color/depth buffers. Do you happen to know if the depth buffer is set to d24s8?

The stencil buffer being writable was my thought too, which is partly why I wanted zeta dumps from a real xbox. I haven't checked into what vertices are being sent during the SET_BEGIN_END instruction, but given that the zeta buffer is unchanged it doesn't seem to be rendering anything.

The surface format is being set to 0x8080224, which should indeed correspond to NV097_SET_SURFACE_FORMAT_ZETA_Z24S8.

Editing to add that I had considered whether or not the stencil buffer could be written without the stencil test being enabled, and according to Khronos it seems that the stencil test is assumed to have passed if the test is disabled, at least in opengl. I haven't checked what the specific behavior is on Xbox, but in any case the game last sets the stencil func to 0x202, before a different SET_BEGIN_END.

[antangelo]

Took another look through the log and it doesn't seem to be setting any of the stencil operations explicitly. The last value NV097_SET_STENCIL_OP_ZPASS is set to is 0x1e00, which is GL_KEEP in xemu. Unless I'm missing something, this would mean that it can't be writing to any of the buffers even with the stencil mask on. This does, however, match the fact that the zeta buffers are unchanged from the previous draw operation.

[Triticum0]

@RageXbox Add the 3 games to the list so they appear on the compatibility list
https://xemu.app/titles/4d570005/#Area-51
https://xemu.app/titles/5553001c/#Batman-Rise-of-Sin-Tzu
https://xemu.app/titles/5553004b/#Cold-Fear

[RageXbox]

Done

[Triticum0]

Also affects Beyond Good & Evil @RageXbox please add to list
https://xemu.app/titles/55530018/#Beyond-Good-Evil

[abaire]

Took another look through the log and it doesn't seem to be setting any of the stencil operations explicitly. The last value NV097_SET_STENCIL_OP_ZPASS is set to is 0x1e00, which is GL_KEEP in xemu. Unless I'm missing something, this would mean that it can't be writing to any of the buffers even with the stencil mask on. This does, however, match the fact that the zeta buffers are unchanged from the previous draw operation.

It could just be that the game is buggy?

If I disable the assert and suppress the handling of the END_OP_END (by verifying that at least one of pg->color_binding || pg->zeta_binding is non-null) it looks like Spongebob works (this is on my work branch with other PRs, but it's an easy fix to isolate):

[antangelo]

Took another look through the log and it doesn't seem to be setting any of the stencil operations explicitly. The last value NV097_SET_STENCIL_OP_ZPASS is set to is 0x1e00, which is GL_KEEP in xemu. Unless I'm missing something, this would mean that it can't be writing to any of the buffers even with the stencil mask on. This does, however, match the fact that the zeta buffers are unchanged from the previous draw operation.

It could just be that the game is buggy?

If I disable the assert and suppress the handling of the END_OP_END (by verifying that at least one of pg->color_binding || pg->zeta_binding is non-null) it looks like Spongebob works (this is on my work branch with other PRs, but it's an easy fix to isolate

This is what I did as well in the past, and it is hardware accurate for Spongebob (it doesn't modify either buffer on hardware as you may expect). I haven't been able to confirm this on hardware for the other games, but ignoring draws will make Coldfear and Batman playable as well.

However, I wrote a naive test case and on real hardware it throws a color/zeta limit error, so I think there is something in the state that governs whether the draw op works or not. I haven' been able to figure out what it is though, so I've refrained from submitting a PR to address it.

[abaire]

Interesting. I have a test case written but am waiting on a multihour nv2a-trace to complete before I can run it. In my experience a color/zeta limit error usually means misconfigured surface format or DMA, will see if I can repro and figure out what's going on.

[antangelo]

This is the test case I had written, if it is of any use: https://github.com/antangelo/nxdk_pgraph_tests/blob/45b6a104d71c9d14f4f5e3d90366785fb5d1b4ab/src/tests/stencil_tests.cpp#L27-L29
The other cases worked as expected, notably the stencil buffer is not modified if only the depth test is enabled, so it would stand to reason that the correct behavior if both is disabled is to skip the draw call on that basis. It always seemed odd to me that simply disabling both tests is enough to trigger that error.

[abaire]

Confirmed that it does fail with a color/zeta limit error on HW, very interesting. I'll look at the pgraph a bit more and see if I can find the missing context.

The first interesting thing I found is that the error is only thrown if the draw isn't entirely discarded; I noticed that Spongebob is using the fixed function pipeline and changed my test to do the same while failing to use the correct transformation matrix. It rendered fine until I caught the bug and fixed it at which point the limit error was thrown again.

UPDATE: Did a bit more testing and it looks like the draw call that's done with writes disabled should render to the screen (I set up a trash FBO and captured the draw w/ renderdoc, mesh looks like it's probably the horned helmet that appears during the move). If I disable color masking and edit the pixel shader to force it to pure white opaque, I can see the helmet rendered in the lower left corner of my trash framebuffer. However, I also noticed that alpha discard is enabled for this draw, and alpha is below the discard threshold if I don't override it (the combiner multiplies everything with a combiner constant set to 0,0,0,0 in the last stage before output). I'll update the test to see if the HW is performing the alpha discard before hitting the color/zeta limit error (similar to the way it passes if the vertices are all clipped).

UPDATE: Alpha discard is not the solution. I am starting to suspect that there's some state change that happens early on in the program that allows the draw call to work. I started capturing the pgraph log about 3 frames before the crash and reproduced the state (aside from the transformation matrices and pixel shader) but it still dies on HW with the same problem. I'm going to try to do a full pgraph trace from the start of the program to get a picture of the entire state at the time of the offending draw.

UPDATE: Even with a full pgraph trace from bootup (walked backwards from the crash to try to piece the state back together) it still dies. I don't have the same mesh, I don't have the same transformation matrices, I don't use the same clip regions. I also modified my xemu hackup so that the trash framebuffer object is a 256x256 texture (matching the output of the bad draw, previously it just used the gl_display_buffer) and the mesh does not get clipped.
I do use a different mode to specify vertices, so I'll adjust that to rule it out as a cause, but this is very odd behavior. I guess it could be possible that there's some configuration going on outside of pgraph commands as well.

[RageXbox]

Also affects Beyond Good & Evil @RageXbox please add to list https://xemu.app/titles/55530018/#Beyond-Good-Evil
Done

[abaire]

@antangelo I'm pretty sure I found the issue.

I dumped all of the register writes to the pgraph area during a Spongebob game from start to the earliest place I could crash it and compared to my test. I found and tried applying a number of differences and eventually discovered that setting 0xFD400880 to 0x28c3ff (the value that Spongebob sets it to early on and doesn't change through the crash; I did not check to see if it sets it later. The default is 0x8cfff) prevents the exception.

I assume that 0x880 is some sort of control register that disables or bypasses the validation.

abaire/nxdk_pgraph_tests@f0bb744#diff-ecc0768980ac93f98ca7a964016bb20e0bb1b96d0933608607bbbdf04bcf8096R49 is the important bit.

Your tests are more thorough than mine, so it'd be great if you could verify that setting this register fixes your test and PR to the pgraph repo. You probably need to rebase as I only added the register base stuff when I added register comparing a week or so back.

I just reopened PR #989 with the assumption that you'll be able to repro this success case, in which case I think we should just support discarding nop draws (I did not notice any glitches due to missed side effects, if we want to be pedantic we could try comparing registers before and after the masked off draw, but I think we need to build up a list of ignorable registers).

[antangelo]

Writing to that register does indeed allow my tests to complete without encountering the limit error. I did some further probing, and it seems that bit 11 of register 0x880 is the one that controls whether it errors or not, see here. I am not sure what the other bits do, I only tested bits 10 and 11 (with 10 having no effect).

[abaire]

Nice! I updated the comment in my PR to capture your finding in case somebody wants to properly implement checking against this register rather than blindly ignoring nop draws. In my opinion this would only be worth enforcing if folks start using xemu as a development platform since we can be confident that any retail game would have to disable the check or it would've crashed and presumably not passed certification.