I/O sandbox execution exceptions during compilation

[nikhilkalige]

Description of the bug:

I randomly keep hitting this error during C++ compilation.

Compiling main.cpp failed: I/O exception during sandboxed execution: /home/michaelno/.cache/bazel/_bazel_michaelno/17e16a46687a9496c7794b1f221a19a9/sandbox/linux-sandbox/174/stats.out (No such file or directory)
ERROR: /home/michaelno/o/BUILD.bazel:33:15: Compiling ui.cpp failed: I/O exception during sandboxed execution: /home/michaelno/.cache/bazel/_bazel_michaelno/17e16a46687a9496c7794b1f221a19a9/sandbox/linux-sandbox/170/stats.out (No such file or directory)
... More of them

com.google.devtools.build.lib.skyframe.ActionExecutionFunction$ActionExecutionFunctionException: com.google.devtools.build.lib.actions.AlreadyReportedActionExecution
     Exception: I/O exception during sandboxed execution: /home/michaelno/.cache/bazel/_bazel_michaelno/17e16a46687a9496c7794b1f221a19a9/sandbox/linux-sandbox/170/stats.o
     ut (No such file or directory)
 at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.computeInternal(ActionExecutionFunction.java:357)
 at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:171)
 at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:461)
 at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:414)
 at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1375)
 at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
 at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
 at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
 at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
 at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
 Caused by: com.google.devtools.build.lib.actions.AlreadyReportedActionExecutionException: I/O exception during sandboxed execution: /home/michaelno/.cache/bazel/_baz
     el_michaelno/17e16a46687a9496c7794b1f221a19a9/sandbox/linux-sandbox/170/stats.out (No such file or directory)
 ... 10 more
 Caused by: com.google.devtools.build.lib.actions.UserExecException: I/O exception during sandboxed execution: /home/michaelno/.cache/bazel/_bazel_michaelno/17e16a466
     87a9496c7794b1f221a19a9/sandbox/linux-sandbox/170/stats.out (No such file or directory)
 at com.google.devtools.build.lib.sandbox.AbstractSandboxSpawnRunner.exec(AbstractSandboxSpawnRunner.java:117)
 at com.google.devtools.build.lib.sandbox.SandboxModule$SandboxFallbackSpawnRunner.exec(SandboxModule.java:441)
 at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:159)
 at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:119)
 at com.google.devtools.build.lib.exec.SpawnStrategyResolver.exec(SpawnStrategyResolver.java:45)
 at com.google.devtools.build.lib.rules.cpp.CppCompileAction.execute(CppCompileAction.java:1402)
 at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.executeAction(SkyframeActionExecutor.java:1148)
 at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:1065)
 at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:165)
 at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:94)
 at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:562)
 at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:859)
 at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.computeInternal(ActionExecutionFunction.java:333)
 ... 9 more
 Caused by: java.io.FileNotFoundException: /home/michaelno/.cache/bazel/_bazel_michaelno/17e16a46687a9496c7794b1f221a19a9/sandbox/linux-sandbox/170/stats.out (No such
      file or directory)
 at java.base/java.io.FileInputStream.open0(Native Method)
 at java.base/java.io.FileInputStream.open(FileInputStream.java:216)
 at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
 at com.google.devtools.build.lib.unix.UnixFileSystem.createFileInputStream(UnixFileSystem.java:497)
 at com.google.devtools.build.lib.vfs.AbstractFileSystem.createMaybeProfiledInputStream(AbstractFileSystem.java:90)
 at com.google.devtools.build.lib.vfs.AbstractFileSystem.getInputStream(AbstractFileSystem.java:59)
 at com.google.devtools.build.lib.vfs.Path.getInputStream(Path.java:765)
 at com.google.devtools.build.lib.shell.ExecutionStatistics.getResourceUsage(ExecutionStatistics.java:35)
 at com.google.devtools.build.lib.sandbox.AbstractSandboxSpawnRunner.run(AbstractSandboxSpawnRunner.java:310)
 at com.google.devtools.build.lib.sandbox.AbstractSandboxSpawnRunner.runSpawn(AbstractSandboxSpawnRunner.java:148)
 at com.google.devtools.build.lib.sandbox.AbstractSandboxSpawnRunner.exec(AbstractSandboxSpawnRunner.java:111)
 at com.google.devtools.build.lib.sandbox.AbstractSandboxSpawnRunner.exec(AbstractSandboxSpawnRunner.java:111)
 ... 21 more

Which category does this issue belong to?

Local Execution

Which operating system are you running Bazel on?

Linux

What is the output of bazel info release?

7.0.0

[fmeum]

Could you try again with 7.0.1?

[nikhilkalige]

I haven't seen this again in the past week or so.. So, will close this until I can reproduce/see it. Thank you..

[opikalo]

Still happens with 7.1.0

[fmeum]

@opikalo Do you happen to have a reproducer you could share? Could you also test with 7.1.1rc1?