Web Compiler extension consuming large number of threads in the wild (up to 195 threads)

[davkean]

Describe the bug
In the wild we're seeing Web Compiler extension (version 1.14.8.0) consume large amounts of thread pool threads, which is causing Visual Studio responsiveness issues.

In the below session from the dump, thread pool latency at its worst was at 66 seconds, that is measuring the time from when a work item was queued on the thread pool to the actual time it started running. Visual Studio would have been very unpleasant to use during this session.

See https://github.com/microsoft/vs-threading/blob/main/doc/threadpool_starvation.md for more information.

To Reproduce
This is from Visual Studio telemetery, there are no repo steps.

Screenshots
In one dump we saw 195 threads consumed along the following paths. Unfortunately, I have no insight into the WebCompiler stacks as I cannot find the binaries for this version.

Additional context

Stacks:

#1

 	msenv.dll!CThreadAwareServiceProvider::QueryService(__VSTASKRUNCONTEXT taskPriority, const _GUID & guidService, const _GUID & riid, void * * ppv) Line 553	C++
 	msenv.dll!CThreadAwareServiceProvider::QueryService(const _GUID & guidService, const _GUID & riid, void * * ppv) Line 342	C++
 	[Managed to Native Transition]	
 	Microsoft.VisualStudio.Shell.Framework.dll!Microsoft.VisualStudio.Shell.ServiceProvider.QueryService(System.Guid guid, System.Type serviceType, bool setShellErrorInfo, out object service) Line 238	C#
 	Microsoft.VisualStudio.Shell.Framework.dll!Microsoft.VisualStudio.Shell.ServiceProvider.QueryService(System.Type serviceType, bool setShellErrorInfo, out object service) Line 144	C#
	Microsoft.VisualStudio.Shell.Framework.dll!Microsoft.VisualStudio.Shell.ServiceProvider.GetService(System.Type serviceType, bool setShellErrorInfo) Line 110	C#
 	Microsoft.VisualStudio.Shell.15.0.dll!Microsoft.VisualStudio.Shell.Package.GetGlobalService(System.Type serviceType, bool setShellErrorInfo) Line 2984	C#
 	Microsoft.VisualStudio.Shell.UI.Internal.dll!Microsoft.VisualStudio.PlatformUI.GlobalServices.GetGlobalService<System.__Canon, System.__Canon>() Line 29	C#
 	Microsoft.VisualStudio.Shell.UI.Internal.dll!Microsoft.VisualStudio.PlatformUI.Automation.AutomationGuard.AutomationGuard() Line 9	C#
 	Microsoft.VisualStudio.Shell.UI.Internal.dll!Microsoft.VisualStudio.PlatformUI.Automation.AutomationThreadHelper.GetInvocationWrapper() Line 16	C#
 	Microsoft.VisualStudio.Shell.Framework.dll!Microsoft.VisualStudio.Shell.ThreadHelper.Invoke<>(System.Func<> method) Line 401	C#
 	[Frames below may be incorrect and/or missing, no binaries loaded for WebCompilerVsix.dll]	
 	[Frames below may be incorrect and/or missing, no binaries loaded for WebCompilerVsix.dll]	
 	mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 980	C#
 	mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 928	C#
 	mscorlib.dll!System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() Line 1252	C#
 	mscorlib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 820	C#

#2

 	msenv.dll!CThreadAwareServiceProvider::QueryService(__VSTASKRUNCONTEXT taskPriority, const _GUID & guidService, const _GUID & riid, void * * ppv) Line 553	C++
 	msenv.dll!CThreadAwareServiceProvider::QueryService(const _GUID & guidService, const _GUID & riid, void * * ppv) Line 342	C++
 	[Managed to Native Transition]	
 	Microsoft.VisualStudio.Shell.Framework.dll!Microsoft.VisualStudio.Shell.ServiceProvider.QueryService(System.Guid guid, System.Type serviceType, bool setShellErrorInfo, out object service) Line 238	C#
 	Microsoft.VisualStudio.Shell.Framework.dll!Microsoft.VisualStudio.Shell.ServiceProvider.QueryService(System.Type serviceType, bool setShellErrorInfo, out object service) Line 144	C#
 	Microsoft.VisualStudio.Shell.Framework.dll!Microsoft.VisualStudio.Shell.ServiceProvider.GetService(System.Type serviceType, bool setShellErrorInfo) Line 110	C#
 	Microsoft.VisualStudio.Shell.15.0.dll!Microsoft.VisualStudio.Shell.Package.GetGlobalService(System.Type serviceType, bool setShellErrorInfo) Line 2984	C#
 	Microsoft.VisualStudio.Shell.UI.Internal.dll!Microsoft.VisualStudio.PlatformUI.GlobalServices.GetGlobalService<System.__Canon, System.__Canon>() Line 29	C#
 	Microsoft.VisualStudio.Shell.UI.Internal.dll!Microsoft.VisualStudio.PlatformUI.Automation.AutomationGuard.DisposeNativeResources() Line 28	C#
 	Microsoft.VisualStudio.Utilities.dll!Microsoft.VisualStudio.PlatformUI.DisposableObject.Dispose(bool disposing) Line 102	C#
 	Microsoft.VisualStudio.Utilities.dll!Microsoft.VisualStudio.PlatformUI.DisposableObject.Dispose() Line 35	C#
 	Microsoft.VisualStudio.Shell.Framework.dll!Microsoft.VisualStudio.Shell.ThreadHelper.Invoke<>(System.Func<> method) Line 411	C#
 	[Frames below may be incorrect and/or missing, no binaries loaded for WebCompilerVsix.dll]	
 	[Frames below may be incorrect and/or missing, no binaries loaded for WebCompilerVsix.dll]	
 	mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 980	C#
 	mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 928	C#
 	mscorlib.dll!System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() Line 1252	C#
 	mscorlib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 820	C#

#3

 	ntdll.dll!ZwWaitForMultipleObjects() Line 947	Unknown
 	KERNELBASE.dll!WaitForMultipleObjectsEx(unsigned long nCount, void * const * lpHandles, int bWaitAll, unsigned long dwMilliseconds, int bAlertable) Line 1551	C
 	[Managed to Native Transition]	
 	mscorlib.dll!System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle waitableSafeHandle, long millisecondsTimeout, bool hasThreadAffinity, bool exitContext) Line 243	C#
	mscorlib.dll!System.Threading.WaitHandle.WaitOne(int millisecondsTimeout, bool exitContext) Line 194	C#
 	System.dll!System.Diagnostics.Process.WaitForExit(int milliseconds)	Unknown
 	[Frames below may be incorrect and/or missing, no binaries loaded for WebCompiler.exe]	
 	[Frames below may be incorrect and/or missing, no binaries loaded for WebCompiler.exe]	
 	[Frames below may be incorrect and/or missing, no binaries loaded for WebCompiler.exe]	
 	[Frames below may be incorrect and/or missing, no binaries loaded for WebCompiler.exe]	
 	[Frames below may be incorrect and/or missing, no binaries loaded for WebCompilerVsix.dll]	
 	mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 980	C#
 	mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 928	C#
 	mscorlib.dll!System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() Line 1252	C#
 	mscorlib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 820	C#

#4

	ntdll.dll!ZwWaitForMultipleObjects() Line 947	Unknown
 	KERNELBASE.dll!WaitForMultipleObjectsEx(unsigned long nCount, void * const * lpHandles, int bWaitAll, unsigned long dwMilliseconds, int bAlertable) Line 1551	C
 	[Managed to Native Transition]	
 	[Frames below may be incorrect and/or missing, no binaries loaded for WebCompiler.exe]	
 	[Frames below may be incorrect and/or missing, no binaries loaded for WebCompilerVsix.dll]	
 	mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 980	C#
 	mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 928	C#
 	mscorlib.dll!System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() Line 1252	C#
 	mscorlib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 820	C#