Set names to runtime internal threads #75113

HJLeee · 2022-09-06T06:54:50Z

It gives names to internal threads so that they can be distinguished easier when debugging, triaging, and/or profiling.

(gdb) info threads
  Id   Target Id                                            Frame
* 1    Thread 0x7ffff7a4a3c0 (LWP 296950) "corerun"         __GI___libc_read (nbytes=1024, buf=0x7fffffffc810, fd=0) at ../sysdeps/unix/sysv/linux/read.c:26
  2    Thread 0x7ffff6afd640 (LWP 296953) "corerun-ust"     syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  3    Thread 0x7ffff62fc640 (LWP 296954) "corerun-ust"     syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  4    Thread 0x7ffff5afb640 (LWP 296955) ".NET SynchManag" 0x00007ffff7b63d7f in __GI___poll (fds=0x7ffff5afa9b8, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
  5    Thread 0x7ffff52fa640 (LWP 296956) ".NET Eventpipe " 0x00007ffff7b63d7f in __GI___poll (fds=0x7fff70008f20, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
  6    Thread 0x7ffff4ad4640 (LWP 296957) ".NET Debug Tran" 0x00007ffff7b5f764 in __libc_open64 (file=0x55555559fa44 "/tmp/clr-debug-pipe-296950-42007406-in", oflag=0) at ../sysdeps/unix/sysv/linux/open64.c:41
  7    Thread 0x7fffe7fff640 (LWP 296958) ".NET DebuggerRC" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x555555644e10) at ./nptl/futex-internal.c:57
  8    Thread 0x7fff77cc5640 (LWP 296959) ".NET Finalizer"  __futex_abstimed_wait_common64 (private=1432319836, cancel=true, abstime=0x7fff77cc40b8, op=137, expected=0, futex_word=0x555555665950) at ./nptl/futex-internal.c:57
  9    Thread 0x7fffe6e02640 (LWP 296960) ".NET SigHandler" __GI___libc_read (nbytes=1, buf=0x7fffe6e01e47, fd=33) at ../sysdeps/unix/sysv/linux/read.c:26

ghost · 2022-09-06T06:55:03Z

Tagging subscribers to this area: @tommcdon
See info in area-owners.md if you want to be subscribed.

Issue Details

It gives names to internal thread so that they can be distinguished easier when debugging, triaging, and/or profiling.

(gdb) info threads
  Id   Target Id                                            Frame
* 1    Thread 0x7ffff7a4a3c0 (LWP 296950) "corerun"         __GI___libc_read (nbytes=1024, buf=0x7fffffffc810, fd=0) at ../sysdeps/unix/sysv/linux/read.c:26
  2    Thread 0x7ffff6afd640 (LWP 296953) "corerun-ust"     syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  3    Thread 0x7ffff62fc640 (LWP 296954) "corerun-ust"     syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  4    Thread 0x7ffff5afb640 (LWP 296955) ".NET SynchManag" 0x00007ffff7b63d7f in __GI___poll (fds=0x7ffff5afa9b8, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
  5    Thread 0x7ffff52fa640 (LWP 296956) ".NET Eventpipe " 0x00007ffff7b63d7f in __GI___poll (fds=0x7fff70008f20, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
  6    Thread 0x7ffff4ad4640 (LWP 296957) ".NET Debug Tran" 0x00007ffff7b5f764 in __libc_open64 (file=0x55555559fa44 "/tmp/clr-debug-pipe-296950-42007406-in", oflag=0) at ../sysdeps/unix/sysv/linux/open64.c:41
  7    Thread 0x7fffe7fff640 (LWP 296958) ".NET DebuggerRC" __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x555555644e10) at ./nptl/futex-internal.c:57
  8    Thread 0x7fff77cc5640 (LWP 296959) ".NET Finalizer"  __futex_abstimed_wait_common64 (private=1432319836, cancel=true, abstime=0x7fff77cc40b8, op=137, expected=0, futex_word=0x555555665950) at ./nptl/futex-internal.c:57
  9    Thread 0x7fffe6e02640 (LWP 296960) ".NET SigHandler" __GI___libc_read (nbytes=1, buf=0x7fffe6e01e47, fd=33) at ../sysdeps/unix/sysv/linux/read.c:26

Author:	HJLeee
Assignees:	-
Labels:	`area-Diagnostics-coreclr`
Milestone:	-

danmoseley · 2022-09-06T07:35:58Z

Note that thread names are limited to 15 chars on Linux and perhaps macOS and FreeBSD, so all names must be distinguishable in the first 15 chars. With ".NET " that leaves 10. The ones in this PR seem fine but we have not done this in some places eg:

runtime/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs

Line 1358 in bcd44bf

internal const string WorkerThreadName = ".NET ThreadPool Worker";

runtime/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.GateThread.cs

Line 244 in bcd44bf

Name = ".NET ThreadPool Gate"

runtime/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WaitThread.cs

Line 190 in 6251d49

Name = ".NET ThreadPool Wait"

Are not distinguishable. Windows from memory has a higher limit, so places in Windows specific code should be fine.

Might be worth follow up cleanup. IMO I'd keep the prefix but use some contracted form like ".NET TPool Wait"

We maybe should Assert in the PAL for > 15

danmoseley · 2022-09-06T07:46:08Z

Mac builds broke


Users/runner/work/1/s/src/native/libs/System.Native/pal_signal.c:532:47: error: too many arguments to function call, expected 1, have 2

HJLeee · 2022-09-06T08:51:56Z

Thank you for the note.
According to the below MAC allows longer thread names. Also, it doesn't allow to change other threads name.

runtime/src/coreclr/pal/src/thread/thread.cpp

Lines 94 to 100 in 2dcd5bf

    
           #ifdef __APPLE__ 
        
           #define MAX_THREAD_NAME_SIZE 63 
        
           #elif defined(__FreeBSD__) 
        
           #define MAX_THREAD_NAME_SIZE MAXCOMLEN 
        
           #else 
        
           #define MAX_THREAD_NAME_SIZE 15 
        
           #endif

runtime/src/coreclr/pal/src/thread/thread.cpp

Lines 1662 to 1668 in 2dcd5bf

    
           #if defined(__APPLE__) 
        
           // on macOS, pthread_setname_np only works for the calling thread. 
        
           if (PlatformGetCurrentThreadId() == pTargetThread->GetThreadId()) 
        
           { 
        
               error = pthread_setname_np(nameBuf); 
        
           } 
        
           #endif

src/coreclr/debug/ee/rcthread.cpp

src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.GateThread.cs

src/coreclr/debug/ee/rcthread.cpp

src/coreclr/vm/eventing/eventpipe/ep-rt-coreclr.h

src/native/libs/System.Native/pal_signal.c

danmoseley · 2022-09-07T04:22:15Z

Test failures seem infrastructural... failed to load CLR, entry point not found, etc

jkotas · 2022-09-07T05:15:47Z

The failures looks like a real problem with the change to me.

HJLeee · 2022-09-07T11:15:29Z

Suspicious mono test is passed with ebca9d4.
#75185 is submitted for double checking and it seems there is no problem with mono tests.

HJLeee · 2022-09-18T21:23:35Z

@danmoseley @jkotas Would you merge this please?

jkotas · 2022-09-19T06:02:24Z

The tests are failing due to changes in this PR

HJLeee · 2022-09-19T06:35:54Z

Build Browser wasm windows Release LibraryTests_AOT
[01:12:42] info: Waiting to flush log messages with a timeout of 120 secs ..
fail: Microsoft.AspNetCore.Server.Kestrel[13]
Connection id "0HMKPRFK1AHDC", Request id "0HMKPRFK1AHDC:00000001": An unhandled exception was thrown by the application.
System.InvalidOperationException: Message processor is not running. Make sure to call RunAsync first
at Microsoft.DotNet.XHarness.CLI.Commands.Wasm.WasmTestMessagesProcessor.InvokeAsync(String message, Boolean isError) in //src/Microsoft.DotNet.XHarness.CLI/Commands/WASM/WasmTestMessagesProcessor.cs:line 94
at Microsoft.DotNet.XHarness.CLI.Commands.Wasm.WasmBrowserTestRunner.RunConsoleMessagesPump(WebSocket socket, CancellationToken token) in //src/Microsoft.DotNet.XHarness.CLI/Commands/WASM/Browser/WasmBrowserTestRunner.cs:line 196
at Microsoft.DotNet.XHarness.CLI.Commands.WebServer.TestWebServerStartup.<>c__DisplayClass3_0.<b__2>d.MoveNext() in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/WebServer.cs:line 163
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
[01:12:42] fail: [out of order message from the browser]: http://127.0.0.1:49193/dotnet.js 4:1872 "[main] websocket closed: [object CloseEvent]" CloseEvent
[01:12:42] fail: Application has finished with exit code TESTS_FAILED but 0 was expected
XHarness exit code: 71 (GENERAL_FAILURE)
Build iOS arm64 Release AllSubsets_Mono
[16:35:12] fail: Application test run crashed
No test log file was produced
Check logs for more information
Libraries Test Run release coreclr windows x86 Release
257 Error(s)
C:\h\w\B6BF09BC\w\A61D091B\e>"C:\h\w\B6BF09BC\p\dotnet.exe" exec --runtimeconfig System.Xml.XmlSerializer.ReflectionOnly.Tests.runtimeconfig.json --depsfile System.Xml.XmlSerializer.ReflectionOnly.Tests.deps.json xunit.console.dll System.Xml.XmlSerializer.ReflectionOnly.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing
Failed to load the dll from [C:\h\w\B6BF09BC\p\shared\Microsoft.NETCore.App\8.0.0\coreclr.dll], HRESULT: 0x8007007F
Libraries Test Run release coreclr windows x64 Debug
257 Error(s)
C:\h\w\A86F0922\w\C3030A5C\e>"C:\h\w\A86F0922\p\dotnet.exe" exec --runtimeconfig Common.Tests.runtimeconfig.json --depsfile Common.Tests.deps.json xunit.console.dll Common.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing
Failed to load the dll from [C:\h\w\A86F0922\p\shared\Microsoft.NETCore.App\8.0.0\coreclr.dll], HRESULT: 0x8007007F
Libraries Test Run checked coreclr windows x86 Debug
257 Error(s)
C:\h\w\9FE50944\w\976A0843\e>"C:\h\w\9FE50944\p\dotnet.exe" exec --runtimeconfig System.Xml.XmlSerializer.ReflectionOnly.Tests.runtimeconfig.json --depsfile System.Xml.XmlSerializer.ReflectionOnly.Tests.deps.json xunit.console.dll System.Xml.XmlSerializer.ReflectionOnly.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing
Failed to load the dll from [C:\h\w\9FE50944\p\shared\Microsoft.NETCore.App\8.0.0\coreclr.dll], HRESULT: 0x8007007F
Libraries Test Run checked coreclr windows x64 Release
257 Error(s)
C:\h\w\9CD30919\w\A683088E\e>"C:\h\w\9CD30919\p\dotnet.exe" exec --runtimeconfig System.Diagnostics.EventLog.Tests.runtimeconfig.json --depsfile System.Diagnostics.EventLog.Tests.deps.json xunit.console.dll System.Diagnostics.EventLog.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing
Failed to load the dll from [C:\h\w\9CD30919\p\shared\Microsoft.NETCore.App\8.0.0\coreclr.dll], HRESULT: 0x8007007F
coreclr Pri0 Runtime Tests Run windows x86 checked
JIT\SIMD\CtorFromArray_ro\CtorFromArray_ro.cmd [FAIL]
Failed to load: 'C:\h\w\B1C6096C\p\coreclr.dll'. Error: 0x0000007f
mono llvmaot Pri0 Runtime Tests Run Linux x64 release
Expected: 100
Actual: 0
END EXECUTION - FAILED
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=/datadisks/disk1/work/B79209F6/p
> /datadisks/disk1/work/B79209F6/w/A5340917/e/JIT/HardwareIntrinsics/X86/Aes/Aes_ro/Aes_ro.sh
Expected: True
Actual: False
Stack Trace:
at JIT_HardwareIntrinsics.X86_Aes_Aes_ro_Aes_ro._X86_Aes_Aes_ro_Aes_ro_sh()
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

Thank you Co-authored-by: Jan Kotas <jkotas@microsoft.com>

Co-authored-by: Jan Kotas <jkotas@microsoft.com>

src/coreclr/debug/shared/dbgtransportsession.cpp

HJLeee · 2022-09-20T02:54:55Z

@jkotas I think it's ready. .NET CI is nice. :)

src/coreclr/debug/shared/dbgtransportsession.cpp

src/coreclr/vm/eventing/eventpipe/ep-rt-coreclr.h

HJLeee · 2022-09-21T00:40:30Z

Now it looks like this on Linux.

(gdb) info threads
  Id   Target Id                                            Frame 
* 1    Thread 0x7ffff7a4c3c0 (LWP 619311) "corerun"         __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737348158400) at ./nptl/pthread_kill.c:44
  2    Thread 0x7ffff6afc640 (LWP 619314) "corerun-ust"     syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  3    Thread 0x7ffff62fb640 (LWP 619315) "corerun-ust"     syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
  4    Thread 0x7ffff5afa640 (LWP 619316) ".NET SynchManag" 0x00007ffff7b65d7f in __GI___poll (fds=0x0, nfds=0, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
  5    Thread 0x7ffff52f9640 (LWP 619317) ".NET EventPipe"  0x00007ffff7b65d7f in __GI___poll (fds=0x7fff70008f40, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
  6    Thread 0x7ffff4ad3640 (LWP 619318) ".NET DebugPipe"  0x00007ffff7b61764 in __libc_open64 (file=0x55555559fa44 "/tmp/clr-debug-pipe-619311-18359507-in", oflag=0) at ../sysdeps/unix/sysv/linux/open64.c:41
  7    Thread 0x7fffe7fff640 (LWP 619319) ".NET Debugger"   __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x555555645670) at ./nptl/futex-internal.c:57
  8    Thread 0x7fff77cc4640 (LWP 619320) ".NET Finalizer"  __futex_abstimed_wait_common64 (private=1432321820, cancel=true, abstime=0x7fff77cc30b8, op=137, expected=0, futex_word=0x555555666360) at ./nptl/futex-internal.c:57
  9    Thread 0x7fffe6e02640 (LWP 619321) ".NET SigHandler" __GI___libc_read (nbytes=1, buf=0x7fffe6e01e47, fd=33) at ../sysdeps/unix/sysv/linux/read.c:26

jkotas · 2022-09-22T03:16:18Z

Thank you!

HJLeee · 2022-09-22T05:50:37Z

Thank you for your kind reviews and advice.

dotnet-issue-labeler bot added the area-Diagnostics-coreclr label Sep 6, 2022

ghost added the community-contribution Indicates that the PR has been added by a community member label Sep 6, 2022

jkotas reviewed Sep 6, 2022

View reviewed changes

src/coreclr/debug/ee/rcthread.cpp Outdated Show resolved Hide resolved

jkotas reviewed Sep 6, 2022

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.GateThread.cs Outdated Show resolved Hide resolved

jkotas reviewed Sep 7, 2022

View reviewed changes

src/coreclr/debug/ee/rcthread.cpp Outdated Show resolved Hide resolved

jkotas reviewed Sep 7, 2022

View reviewed changes

src/coreclr/vm/eventing/eventpipe/ep-rt-coreclr.h Outdated Show resolved Hide resolved

jkotas reviewed Sep 7, 2022

View reviewed changes

src/native/libs/System.Native/pal_signal.c Outdated Show resolved Hide resolved

jkotas approved these changes Sep 7, 2022

View reviewed changes

HJLeee mentioned this pull request Sep 8, 2022

Set name .NET SigHandler #75185

Closed

HJLeee force-pushed the thread_names branch from ebca9d4 to 376457e Compare September 8, 2022 01:08

HJLeee and others added 7 commits September 19, 2022 15:50

Set names to runtime internal threads

52c8b29

Fix MAC build

7b7e47f

Shorten ThreadPool thread names

7f58e1c

Apply Feedback

f4af66c

Update src/coreclr/debug/ee/rcthread.cpp

26a9a8f

Thank you Co-authored-by: Jan Kotas <jkotas@microsoft.com>

Update src/coreclr/vm/eventing/eventpipe/ep-rt-coreclr.h

3a13519

Co-authored-by: Jan Kotas <jkotas@microsoft.com>

Apply Feedback

ea88ce3

Co-authored-by: Jan Kotas <jkotas@microsoft.com>

HJLeee force-pushed the thread_names branch from 931ecd1 to 27bfcf1 Compare September 19, 2022 06:51

Fix not to return SetThreadDescription()

e4f8691

HJLeee force-pushed the thread_names branch from 27bfcf1 to e4f8691 Compare September 19, 2022 08:26

Change thread names on linux only

635addb

jkotas reviewed Sep 19, 2022

View reviewed changes

src/coreclr/debug/shared/dbgtransportsession.cpp Outdated Show resolved Hide resolved

Change SetThreadDescription to SetThreadName

b0e751d

jkotas reviewed Sep 20, 2022

View reviewed changes

src/coreclr/debug/shared/dbgtransportsession.cpp Outdated Show resolved Hide resolved

jkotas reviewed Sep 20, 2022

View reviewed changes

src/coreclr/vm/eventing/eventpipe/ep-rt-coreclr.h Outdated Show resolved Hide resolved

HJLeee requested review from lateralusX and lambdageek as code owners September 21, 2022 00:34

Set thread names inside thread functions

2ee5c1b

HJLeee force-pushed the thread_names branch from 1a315f7 to 2ee5c1b Compare September 21, 2022 00:37

HJLeee added 2 commits September 21, 2022 09:47

Add missing name for MAC

9abb1b3

Move name change info thread function

f4a14fc

HJLeee force-pushed the thread_names branch from 9a1fd19 to b3b156d Compare September 21, 2022 01:49

Fix windows build error

1de0103

HJLeee force-pushed the thread_names branch from b3b156d to 1de0103 Compare September 21, 2022 01:52

jkotas merged commit e013f9d into dotnet:main Sep 22, 2022

HJLeee deleted the thread_names branch September 22, 2022 05:50

ghost locked as resolved and limited conversation to collaborators Oct 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set names to runtime internal threads #75113

Set names to runtime internal threads #75113

HJLeee commented Sep 6, 2022 •

edited

Loading

ghost commented Sep 6, 2022

danmoseley commented Sep 6, 2022 •

edited

Loading

danmoseley commented Sep 6, 2022

HJLeee commented Sep 6, 2022

danmoseley commented Sep 7, 2022

jkotas commented Sep 7, 2022

HJLeee commented Sep 7, 2022

HJLeee commented Sep 18, 2022

jkotas commented Sep 19, 2022

HJLeee commented Sep 19, 2022

HJLeee commented Sep 20, 2022

HJLeee commented Sep 21, 2022

jkotas commented Sep 22, 2022

HJLeee commented Sep 22, 2022

Set names to runtime internal threads #75113

Set names to runtime internal threads #75113

Conversation

HJLeee commented Sep 6, 2022 • edited Loading

ghost commented Sep 6, 2022

danmoseley commented Sep 6, 2022 • edited Loading

danmoseley commented Sep 6, 2022

HJLeee commented Sep 6, 2022

danmoseley commented Sep 7, 2022

jkotas commented Sep 7, 2022

HJLeee commented Sep 7, 2022

HJLeee commented Sep 18, 2022

jkotas commented Sep 19, 2022

HJLeee commented Sep 19, 2022

HJLeee commented Sep 20, 2022

HJLeee commented Sep 21, 2022

jkotas commented Sep 22, 2022

HJLeee commented Sep 22, 2022

HJLeee commented Sep 6, 2022 •

edited

Loading

danmoseley commented Sep 6, 2022 •

edited

Loading