Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MSBuild crashing in the build #92290

Open
jkoritzinsky opened this issue Sep 19, 2023 · 8 comments
Open

MSBuild crashing in the build #92290

jkoritzinsky opened this issue Sep 19, 2023 · 8 comments
Labels
area-Infrastructure blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab
Milestone
10.0.0

Comments

@jkoritzinsky
Copy link
Member

jkoritzinsky commented Sep 19, 2023 •
edited by build-analysis bot
Loading

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=410579
Build error leg or test failing: Build / Mono Product Build windows x64 release / Build product
Pull request: #91768

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "Child Node \"[0-9]+\" exited prematurely.",
  "BuildRetry": true,
  "ExcludeConsoleLog": false
}

Report

Build Definition Step Name Console log Pull Request
813486 dotnet/runtime Build Log
808246 dotnet/runtime Build managed test components Log #106099
806715 dotnet/runtime Build product Log #107750
2536282 dotnet-runtime Build product Log
805432 dotnet/runtime Build product Log
800800 dotnet/runtime Build product Log
799712 dotnet/runtime Build product Log
799254 dotnet/runtime Build product Log
797989 dotnet/runtime Build product Log #106340
794507 dotnet/runtime Build product Log #107221
2528068 dotnet-runtime Build product Log #41316
793385 dotnet/runtime Build product Log #107170
792806 dotnet/runtime Build product Log #107151
791955 dotnet/runtime Build product Log #107061
790358 dotnet/runtime Build product Log
790355 dotnet/runtime Build product Log
789508 dotnet/runtime Build product Log #107023
2522700 dotnet-runtime Build product Log
786159 dotnet/runtime Build managed test components Log
785263 dotnet/runtime Build Tests Log #106766

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 1 20

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=410579
Error message validated: Child Node "[0-9]+" exited prematurely.
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 9/19/2023 6:06:29 PM UTC
@jkoritzinsky jkoritzinsky added blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab labels Sep 19, 2023
@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Sep 19, 2023
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Sep 19, 2023
@jkoritzinsky jkoritzinsky added area-Infrastructure and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Sep 19, 2023
@ghost
Copy link

ghost commented Sep 19, 2023

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=410579
Build error leg or test failing: Build / Mono Product Build windows x64 release / Build product
Pull request: #91768

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "Child Node \"[0-9]+\" exited prematurely.",
  "BuildRetry": true,
  "ExcludeConsoleLog": false
}
Author: jkoritzinsky
Assignees: -
Labels:

blocking-clean-ci, area-Infrastructure, untriaged, Known Build Error
Milestone: -

@MichalStrehovsky MichalStrehovsky mentioned this issue Oct 5, 2023
Closed
@ViktorHofer
Copy link
Member

ViktorHofer commented Oct 20, 2023 •
edited
Loading

190 hits per month

cc @rainersigwald @rokonec

@lewing
Copy link
Member

lewing commented Nov 16, 2023

reup for 97 hits per month

@agocke agocke added this to the 9.0.0 milestone Nov 28, 2023
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Nov 28, 2023
@MichalStrehovsky
Copy link
Member

We now have more useful information in the logs for these. Two of the available logs have this in it:

2024-02-19T21:13:25.8115210Z MSBUILD : error MSB4166: Child node "2" exited prematurely. Shutting down. Diagnostic information may be found in files in "/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/MSBuildTemprunner/" and will be named MSBuild_*.failure.txt. This location can be changed by setting the MSBUILDDEBUGPATH environment variable to a different directory.
2024-02-19T21:13:25.8126770Z MSBUILD : error MSB4166: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/MSBuildTemprunner/MSBuild_pid-50918_4ee39488d5ca4c7d9fb049f19f43c144.failure.txt:
2024-02-19T21:13:25.8128200Z MSBUILD : error MSB4166: UNHANDLED EXCEPTIONS FROM PROCESS 50918:
2024-02-19T21:13:25.8129250Z MSBUILD : error MSB4166: =====================
2024-02-19T21:13:25.8140670Z MSBUILD : error MSB4166: 2/19/2024 9:13:25 PM
2024-02-19T21:13:25.8241810Z MSBUILD : error MSB4166: System.IO.IOException: No buffer space available
2024-02-19T21:13:25.8260730Z MSBUILD : error MSB4166:  ---> System.Net.Sockets.SocketException (55): No buffer space available
2024-02-19T21:13:25.8363710Z MSBUILD : error MSB4166:    at System.IO.Pipes.PipeStream.WriteCore(ReadOnlySpan`1 buffer)
2024-02-19T21:13:25.8364490Z MSBUILD : error MSB4166:    --- End of inner exception stack trace ---
2024-02-19T21:13:25.8465500Z MSBUILD : error MSB4166:    at System.IO.Pipes.PipeStream.WriteCore(ReadOnlySpan`1 buffer)
2024-02-19T21:13:25.8569320Z MSBUILD : error MSB4166:    at System.IO.Pipes.PipeStream.Write(Byte[] buffer, Int32 offset, Int32 count)
2024-02-19T21:13:25.8670730Z MSBUILD : error MSB4166:    at Microsoft.Build.BackEnd.NodeEndpointOutOfProcBase.RunReadLoop(Stream localReadPipe, Stream localWritePipe, ConcurrentQueue`1 localPacketQueue, AutoResetEvent localPacketAvailable, AutoResetEvent localTerminatePacketPump)
2024-02-19T21:13:25.8773480Z MSBUILD : error MSB4166: ===================

@rainersigwald does this ring any bell?

@rainersigwald
Copy link
Member

All of the recent hits seem to be dotnet/msbuild#10342, which should be fixed in the preview7 GA build. Is there a timeline for updating to that? Anything my team can help with?

@rainersigwald
Copy link
Member 

2024-02-19T21:13:25.8115210Z MSBUILD : error MSB4166: Child node "2" exited prematurely. Shutting down. Diagnostic information may be found in files in "/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/MSBuildTemprunner/" and will be named MSBuild_*.failure.txt. This location can be changed by setting the MSBUILDDEBUGPATH environment variable to a different directory.
2024-02-19T21:13:25.8126770Z MSBUILD : error MSB4166: /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/MSBuildTemprunner/MSBuild_pid-50918_4ee39488d5ca4c7d9fb049f19f43c144.failure.txt:
2024-02-19T21:13:25.8128200Z MSBUILD : error MSB4166: UNHANDLED EXCEPTIONS FROM PROCESS 50918:
2024-02-19T21:13:25.8129250Z MSBUILD : error MSB4166: =====================
2024-02-19T21:13:25.8140670Z MSBUILD : error MSB4166: 2/19/2024 9:13:25 PM
2024-02-19T21:13:25.8241810Z MSBUILD : error MSB4166: System.IO.IOException: No buffer space available
2024-02-19T21:13:25.8260730Z MSBUILD : error MSB4166:  ---> System.Net.Sockets.SocketException (55): No buffer space available
2024-02-19T21:13:25.8363710Z MSBUILD : error MSB4166:    at System.IO.Pipes.PipeStream.WriteCore(ReadOnlySpan`1 buffer)
2024-02-19T21:13:25.8364490Z MSBUILD : error MSB4166:    --- End of inner exception stack trace ---
2024-02-19T21:13:25.8465500Z MSBUILD : error MSB4166:    at System.IO.Pipes.PipeStream.WriteCore(ReadOnlySpan`1 buffer)
2024-02-19T21:13:25.8569320Z MSBUILD : error MSB4166:    at System.IO.Pipes.PipeStream.Write(Byte[] buffer, Int32 offset, Int32 count)
2024-02-19T21:13:25.8670730Z MSBUILD : error MSB4166:    at Microsoft.Build.BackEnd.NodeEndpointOutOfProcBase.RunReadLoop(Stream localReadPipe, Stream localWritePipe, ConcurrentQueue`1 localPacketQueue, AutoResetEvent localPacketAvailable, AutoResetEvent localTerminatePacketPump)
2024-02-19T21:13:25.8773480Z MSBUILD : error MSB4166: ===================

this one I don't recognize, but I haven't seen it in the current links.

@rainersigwald
Copy link
Member

Is it possible to have multiple issues with known problems that match a signature with prioritization? I'd like to create one like at Microsoft.Build.BackEnd.Logging.LoggingContext.LogBuildEvent to bucket the dotnet/msbuild#10342 hits so we can use this one for "everything else that causes an MSBuild worker node crash".

@jkoritzinsky
Copy link
Member Author

You can have multiple known issues I think, but I don't think there's a prioritization mechanism.

I'd recommend having a separate issue to bucket the dotnet/msbuild#10342 hits and leave this one as a catch-all to trigger build retries when a node dies (and we can triage specific failure classes into different known issues if they happen with enough frequency).

@agocke agocke modified the milestones: 9.0.0, 10.0.0 Aug 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area-Infrastructure blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab
Projects
Runtime Infra
Status: No status
Milestone
10.0.0
Development

No branches or pull requests
6 participants
@lewing @agocke @jkoritzinsky @rainersigwald @ViktorHofer @MichalStrehovsky