Bundle_extraction_is_reused intermittently fails with "text file is busy" #44657

runfoapp · 2020-11-13T20:11:29Z

Runfo Tracking Issue: Bundle_extraction_is_reused intermittently fails with "text file is busy"

Build	Definition	Kind	Run Name

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
0	0	0

Dotnet-GitSync-Bot · 2020-11-13T20:11:31Z

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

ghost · 2020-11-16T01:41:20Z

Tagging subscribers to this area: @agocke, @vitek-karas
See info in area-owners.md if you want to be subscribed.

Issue Details

Description:

Runfo Tracking Issue: Bundle_extraction_is_reused intermittently fails with "text file is busy"

Build	Definition	Kind	Run Name
885716	runtime	PR 44656	Installer-coreclr-Linux_x64-Release
883403	runtime	PR 42972	Installer-coreclr-Linux_x64-Release

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
1	2	2

Author: runfoapp[bot]

Assignees: -

Labels:

area-Single-File, untriaged

Milestone: -

eerhardt · 2020-12-09T21:38:30Z

This failed again here:

https://dev.azure.com/dnceng/public/_build/results?buildId=913629&view=logs&j=488ba465-33f8-5af3-b422-ec6526547df5&t=6c6997ae-4d39-5a90-21e3-089719c051c1&l=525

https://dev.azure.com/dnceng/public/_build/results?buildId=913629&view=ms.vss-test-web.build-test-results-tab&runId=29085272&resultId=100002&paneView=debug

Error message
System.ComponentModel.Win32Exception : Text file busy


Stack trace
   at System.Diagnostics.Process.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec)
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at Microsoft.DotNet.Cli.Build.Framework.Command.Start() in /_/src/installer/tests/TestUtils/Command.cs:line 199
   at Microsoft.DotNet.Cli.Build.Framework.Command.Execute(Boolean fExpectedToFail) in /_/src/installer/tests/TestUtils/Command.cs:line 239
   at Microsoft.DotNet.Cli.Build.Framework.Command.Execute() in /_/src/installer/tests/TestUtils/Command.cs:line 171
   at AppHost.Bundle.Tests.StaticHost.Can_Run_SingleFile_App_With_StatiHost() in /_/src/installer/tests/Microsoft.NET.HostModel.Tests/AppHost.Bundle.Tests/StaticHost.cs:line 44

vitek-karas · 2020-12-09T23:32:36Z

The last thing we do on the executable before we call Process.Start (which fails) is this:

runtime/src/installer/managed/Microsoft.NET.HostModel/AppHost/HostWriter.cs

Lines 175 to 177 in dbe5e2a

    
                       // Memory-mapped write does not updating last write time 
        
                       RetryUtil.RetryOnIOError(() => 
        
                           File.SetLastWriteTimeUtc(appHostPath, DateTime.UtcNow));

The Retry has logic to try 500 times (with a short sleep) and if it still fails - then throw. So unless something is really weird, then the SetLastWriteTimeUtc actually succeeded before it's done.

I don't know Linux file system enough to be able to tell if the SetLastWriteTime even requires the file to be "non-busy" - if it can work on "busy" files, then the one we call which actually modifies the content of the file is this:

runtime/src/installer/managed/Microsoft.NET.HostModel/AppHost/BinaryUtils.cs

Lines 68 to 74 in dbe5e2a

    
           using (var mappedFile = MemoryMappedFile.CreateFromFile(filePath)) 
        
           { 
        
               using (var accessor = mappedFile.CreateViewAccessor()) 
        
               { 
        
                   SearchAndReplace(accessor, searchPattern, patternToReplace, pad0s); 
        
               } 
        
           }

This has usings around the right stuff - so it should close the file once it's done.

eerhardt · 2020-12-11T02:01:13Z

Failed again here:

https://dev.azure.com/dnceng/public/_build/results?buildId=916830&view=logs&j=488ba465-33f8-5af3-b422-ec6526547df5&t=6c6997ae-4d39-5a90-21e3-089719c051c1&l=536

https://dev.azure.com/dnceng/public/_build/results?buildId=916830&view=ms.vss-test-web.build-test-results-tab&runId=29156076&resultId=100011&paneView=debug

agocke · 2020-12-11T07:45:22Z

I think some more info would be helpful. I'm going to try to run strace and get the syscall log to see if I can figure out what's going on under the hood here.

We'll also need to move this entire test into staging unfortunately -- it's too noisy to continue running in all PRs.

There is a race condition somewhere in these tests which is causing dotnet#44657

danmoseley · 2020-12-14T15:54:46Z

Failed again https://dev.azure.com/dnceng/public/_build/results?buildId=920007&view=logs&jobId=a45ba455-ebc4-599b-2570-d14d0585f433

radical · 2020-12-14T22:06:10Z

Failed here - https://dev.azure.com/dnceng/public/_build/results?buildId=920677&view=ms.vss-test-web.build-test-results-tab&runId=29254050&resultId=100009&paneView=debug

* Disable apphost tests on Linux There is a race condition somewhere in these tests which is causing #44657 * Run disabled tests on staging pipeline * Fix yml indentation * Use continue on error on installer tests run * Pass down restore as well * Publish test results in global-build-job * Remove not needed parameter Co-authored-by: Santiago Fernandez Madero <safern@microsoft.com>

directhex · 2021-03-03T17:02:42Z

Closing, fixed by #45976

safern · 2021-03-03T18:32:34Z

I don't know if @agocke once to bring those tests back from the staging pipeline to the main one?

agocke · 2021-03-03T18:42:52Z

Yeah, let's keep it open. I do eventually want to get to the bottom of this.

safern · 2021-04-06T17:40:59Z

Reopening as this is on the runtime-staging pipeline.

VincentBu · 2021-05-24T01:40:02Z

Failed in runtime 20210523.14

Failed test:

Installer-coreclr-Linux_x64-Release

- AppHost.Bundle.Tests.BundleExtractToSpecificPath.Bundle_Extraction_To_Relative_Path_Succeeds(relativePath: \"foo/bar\", bundleOptions: BundleAllContent)

Error message:

System.ComponentModel.Win32Exception : Text file busy


Stack trace
   at System.Diagnostics.Process.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec)
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at Microsoft.DotNet.Cli.Build.Framework.Command.Start() in /_/src/installer/tests/TestUtils/Command.cs:line 199
   at Microsoft.DotNet.Cli.Build.Framework.Command.Execute(Boolean fExpectedToFail) in /_/src/installer/tests/TestUtils/Command.cs:line 239
   at Microsoft.DotNet.Cli.Build.Framework.Command.Execute() in /_/src/installer/tests/TestUtils/Command.cs:line 171
   at AppHost.Bundle.Tests.BundleExtractToSpecificPath.Bundle_Extraction_To_Relative_Path_Succeeds(String relativePath, BundleOptions bundleOptions) in /_/src/installer/tests/Microsoft.NET.HostModel.Tests/AppHost.Bundle.Tests/BundleExtractToSpecificPath.cs:line 87

VincentBu · 2021-06-03T01:47:36Z

Failed again in runtime 20210602.73

Failed test:

Installer-coreclr-Linux_x64-Release

- AppHost.Bundle.Tests.BundleExtractToSpecificPath.Bundle_Extraction_To_Relative_Path_Succeeds(relativePath: \"./foo\", bundleOptions: BundleAllContent)

Error message:

System.ComponentModel.Win32Exception : Text file busy


Stack trace
   at System.Diagnostics.Process.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec)
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at Microsoft.DotNet.Cli.Build.Framework.Command.Start() in /_/src/installer/tests/TestUtils/Command.cs:line 199
   at Microsoft.DotNet.Cli.Build.Framework.Command.Execute(Boolean fExpectedToFail) in /_/src/installer/tests/TestUtils/Command.cs:line 239
   at Microsoft.DotNet.Cli.Build.Framework.Command.Execute() in /_/src/installer/tests/TestUtils/Command.cs:line 171
   at AppHost.Bundle.Tests.BundleExtractToSpecificPath.Bundle_Extraction_To_Relative_Path_Succeeds(String relativePath, BundleOptions bundleOptions) in /_/src/installer/tests/Microsoft.NET.HostModel.Tests/AppHost.Bundle.Tests/BundleExtractToSpecificPath.cs:line 87

danmoseley · 2021-06-03T01:55:33Z

@agocke we seem to be continuing to hit this (I think there is another issue as well)

agocke · 2021-06-03T03:09:02Z

We moved the original stuff to the staging branch, must have missed it on alpine. @VSadov is seeing if he can succeed where I failed -- if we don't have any ideas we can move this one to staging as well.

agocke · 2021-08-30T20:29:07Z

Duplicate of #53587

Dotnet-GitSync-Bot added the untriaged New issue has not been triaged by the area owner label Nov 13, 2020

jkoritzinsky mentioned this issue Nov 13, 2020

Don't build archives or installers for Mono builds. #44656

Merged

am11 mentioned this issue Nov 14, 2020

Can_Run_App_With_StatiHost installer test failure in CI #42101

Closed

jeffschwMSFT added the area-Single-File label Nov 16, 2020

jkotas added the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Dec 6, 2020

jkotas mentioned this issue Dec 6, 2020

Delete redundant DivMod implementation #45652

Merged

runfoapp bot mentioned this issue Dec 7, 2020

Infrastructure - Status/Health #702

Closed

eerhardt mentioned this issue Dec 9, 2020

Address ILLink warning in InteropServices.JavaScript #45727

Merged

eerhardt mentioned this issue Dec 11, 2020

P/Invoke function returning SafeSocketHandle throws MissingMethodException #45911

Merged

agocke added a commit to agocke/runtime that referenced this issue Dec 11, 2020

Disable apphost tests on Linux

3788e12

There is a race condition somewhere in these tests which is causing dotnet#44657

agocke mentioned this issue Dec 11, 2020

Disable apphost tests on Linux #45976

Merged

danmoseley mentioned this issue Dec 14, 2020

Enable File.Copy to EXFAT volumes when not running as root #46027

Merged

radical mentioned this issue Dec 14, 2020

[wasm] Move Wasm app building targets to new WasmApp.targets #45977

Merged

directhex closed this as completed Mar 3, 2021

agocke reopened this Mar 3, 2021

safern removed blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' untriaged New issue has not been triaged by the area owner labels Mar 3, 2021

safern added this to the 6.0.0 milestone Mar 3, 2021

safern assigned agocke Mar 3, 2021

runfoapp bot removed this from the 6.0.0 milestone Apr 6, 2021

runfoapp bot closed this as completed Apr 6, 2021

safern reopened this Apr 6, 2021

agocke mentioned this issue Jun 3, 2021

Disable BundleExtractToSpecificPath tests on Linux due to same problems #53690

Merged

agocke marked this as a duplicate of #53587 Aug 30, 2021

agocke closed this as completed Aug 30, 2021

ghost locked as resolved and limited conversation to collaborators Sep 29, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bundle_extraction_is_reused intermittently fails with "text file is busy" #44657

Bundle_extraction_is_reused intermittently fails with "text file is busy" #44657

runfoapp bot commented Nov 13, 2020 •

edited

Loading

Dotnet-GitSync-Bot commented Nov 13, 2020

ghost commented Nov 16, 2020

eerhardt commented Dec 9, 2020

vitek-karas commented Dec 9, 2020

eerhardt commented Dec 11, 2020

agocke commented Dec 11, 2020

danmoseley commented Dec 14, 2020

radical commented Dec 14, 2020

directhex commented Mar 3, 2021

safern commented Mar 3, 2021

agocke commented Mar 3, 2021

safern commented Apr 6, 2021

VincentBu commented May 24, 2021

VincentBu commented Jun 3, 2021

danmoseley commented Jun 3, 2021

agocke commented Jun 3, 2021

agocke commented Aug 30, 2021

Bundle_extraction_is_reused intermittently fails with "text file is busy" #44657

Bundle_extraction_is_reused intermittently fails with "text file is busy" #44657

Comments

runfoapp bot commented Nov 13, 2020 • edited Loading

Dotnet-GitSync-Bot commented Nov 13, 2020

ghost commented Nov 16, 2020

eerhardt commented Dec 9, 2020

vitek-karas commented Dec 9, 2020

eerhardt commented Dec 11, 2020

agocke commented Dec 11, 2020

danmoseley commented Dec 14, 2020

radical commented Dec 14, 2020

directhex commented Mar 3, 2021

safern commented Mar 3, 2021

agocke commented Mar 3, 2021

safern commented Apr 6, 2021

VincentBu commented May 24, 2021

VincentBu commented Jun 3, 2021

danmoseley commented Jun 3, 2021

agocke commented Jun 3, 2021

agocke commented Aug 30, 2021

runfoapp bot commented Nov 13, 2020 •

edited

Loading