Test failure: GC/API/NoGCRegion/Callback/Callback.cmd

[v-wenyuxu]

Failed in: runtime-coreclr jitstress-random 20240922.1

Failed tests:

coreclr osx arm64 Checked jitstress_random_2 @ OSX.1200.ARM64.Open
    - GC/API/NoGCRegion/Callback/Callback.cmd
    - GC/API/NoGCRegion/Callback_Svr/Callback_Svr.cmd

Error message:

 Process terminated. Assertion failed.
status == StartNoGCRegionStatus.Succeeded
   at Test.TestSimple(Int32 oh)
   at Test.TestEntryPoint()
   at __GeneratedMainWrapper.Main()
waitpid() returned successfully (wstatus 00000000) WEXITSTATUS 0 WTERMSIG 0
/private/tmp/helix/working/AA550922/w/9A86087F/e/GC/GC/../API/NoGCRegion/Callback/Callback.sh: line 440:  8566 Abort trap: 6           (core dumped) $LAUNCHER $ExePath "${CLRTestExecutionArguments[@]}"

Return code:      1
Raw output file:      /tmp/helix/working/AA550922/w/9A86087F/uploads/API/NoGCRegion/Callback/output.txt
Raw output:
BEGIN EXECUTION
/tmp/helix/working/AA550922/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true Callback.dll ''
[createdump] Gathering state for process 8566 
[createdump] Crashing thread 558619 signal 6 (0006)
[createdump] Writing crash report to file /cores/coredump.8566.dmp.crashreport.json
[createdump] Crash report successfully written
[createdump] Writing minidump with heap to file /cores/coredump.8566.dmp
[createdump] Written 337035400 bytes (20571 pages) to core file
[createdump] Target process is alive
[createdump] Dump successfully written in 1152ms
Expected: 100
Actual: 134
END EXECUTION - FAILED
Test failed. Trying to see if dump file was created in /cores since 9/22/2024 12:29:30 PM
Processing /cores/coredump.8566.dmp.crashreport.json
Printing stacktrace from '/cores/coredump.8566.dmp.crashreport.json'
Invoking llvm-symbolizer --pretty-print
Errors while running llvm-symbolizer --pretty-print
System.ComponentModel.Win32Exception (2): An error occurred trying to start process 'llvm-symbolizer' with working directory '/private/tmp/helix/working/AA550922/w/9A86087F/e/GC/GC'. No such file or directory
   at System.Diagnostics.Process.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, String[] envp, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 499
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 456
   at CoreclrTestLib.CoreclrTestWrapperLib.TryPrintStackTraceFromCrashReport(String crashReportJsonFile, TextWriter outputWriter)
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.
> /private/tmp/helix/working/AA550922/w/9A86087F/e/GC/GC/../API/NoGCRegion/Callback/Callback.sh

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 146
   at Xunit.Assert.True(Boolean condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 128
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor67|0_68(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-diagnostics-process
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

[BruceForstall]

These tests are failing on all platforms on "normal" jitstress:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=814875&view=ms.vss-test-web.build-test-results-tab

[BruceForstall]

I tracked this down to #107831 -- @EgorBo

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[EgorBo]

@BruceForstall I've just validated that this is fixed by #108161 as well. It was an incorrect use of the jit-stress mode, when it's enabled the optimization allowed negative values too (which are never added to the new switch). I am fine merging revert if it's blocking now (isn't ran once a week?), my PR should be ready once CI is finished

[BruceForstall]

I would merge the revert #108169 since it passed all the pipeline jobs, but nobody approved it yet, so I can't.

[BruceForstall]

I am fine merging revert if it's blocking now (isn't ran once a week?),

Why does it matter if it's run once a week? We need it always clean so people can trigger it on PRs and expect it clean. And the weekend run might be hiding other failures.

Also, jitstress and libraries-jitstress are run every night.

[v-wenyuxu]

Failed in: runtime-coreclr jitstress 20240922.1

Failed tests:

coreclr osx x64 Checked jitstress1 @ OSX.1200.Amd64.Open
    - GC/API/NoGCRegion/Callback/Callback.cmd
    - GC/API/NoGCRegion/Callback_Svr/Callback_Svr.cmd
coreclr osx x64 Checked jitstress2 @ OSX.1200.Amd64.Open
    - GC/API/NoGCRegion/Callback/Callback.cmd
    - GC/API/NoGCRegion/Callback_Svr/Callback_Svr.cmd
coreclr osx arm64 Checked jitstress2 @ OSX.1200.ARM64.Open
    - GC/API/NoGCRegion/Callback/Callback.cmd
    - GC/API/NoGCRegion/Callback_Svr/Callback_Svr.cmd
coreclr osx arm64 Checked jitstress1 @ OSX.1200.ARM64.Open
    - GC/API/NoGCRegion/Callback/Callback.cmd
    - GC/API/NoGCRegion/Callback_Svr/Callback_Svr.cmd

Error message:

 Process terminated. Assertion failed.
status == StartNoGCRegionStatus.Succeeded
   at Test.TestCascade()
   at Test.TestEntryPoint()
   at __GeneratedMainWrapper.Main()
waitpid() returned successfully (wstatus 00000000) WEXITSTATUS 0 WTERMSIG 0
/private/tmp/helix/working/B4600958/w/B2A109A0/e/GC/GC/../API/NoGCRegion/Callback/Callback.sh: line 449: 24178 Abort trap: 6           (core dumped) $LAUNCHER $ExePath "${CLRTestExecutionArguments[@]}"

Return code:      1
Raw output file:      /tmp/helix/working/B4600958/w/B2A109A0/uploads/API/NoGCRegion/Callback/output.txt
Raw output:
BEGIN EXECUTION
/tmp/helix/working/B4600958/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true Callback.dll ''
[createdump] Gathering state for process 24178 
[createdump] Crashing thread 3f3977 signal 6 (0006)
[createdump] Writing crash report to file /cores/coredump.24178.dmp.crashreport.json
[createdump] Crash report successfully written
[createdump] Writing minidump with heap to file /cores/coredump.24178.dmp
[createdump] Written 405426312 bytes (98981 pages) to core file
[createdump] Target process is alive
[createdump] Dump successfully written in 1955ms
Expected: 100
Actual: 134
END EXECUTION - FAILED
Test failed. Trying to see if dump file was created in /cores since 9/23/2024 1:22:25 AM
Processing /cores/coredump.24178.dmp.crashreport.json
Printing stacktrace from '/cores/coredump.24178.dmp.crashreport.json'
Invoking llvm-symbolizer --pretty-print
Errors while running llvm-symbolizer --pretty-print
System.ComponentModel.Win32Exception (2): An error occurred trying to start process 'llvm-symbolizer' with working directory '/private/tmp/helix/working/B4600958/w/B2A109A0/e/GC/GC'. No such file or directory
   at System.Diagnostics.Process.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, String[] envp, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 499
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 456
   at System.Diagnostics.Process.Start() in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1284
   at CoreclrTestLib.CoreclrTestWrapperLib.TryPrintStackTraceFromCrashReport(String crashReportJsonFile, TextWriter outputWriter)
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.
> /private/tmp/helix/working/B4600958/w/B2A109A0/e/GC/GC/../API/NoGCRegion/Callback/Callback.sh

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 146
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor67|0_68(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

[EgorBo]

Why does it matter if it's run once a week? We need it always clean so people can trigger it on PRs and expect it clean. And the weekend run might be hiding other failures.

I just didn't see the urge of reverting since the fix is already filed and the issue seems to be just an incorrect use of jitstress flag. Since my fix hit an unexpected assert on CI, I am fine merging the revert