Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SteamVR crashes after 10-20 minutes running in Sandboxie-Plus x64 #1243

Closed
GT500org opened this issue Oct 3, 2021 · 28 comments
Closed

SteamVR crashes after 10-20 minutes running in Sandboxie-Plus x64 #1243

GT500org opened this issue Oct 3, 2021 · 28 comments
Labels
crash dump Dump file attached for a detailed analysis fixed in next build Fixed in the next Sandboxie version High priority To be done as soon as possible Log available Trace log attached for a detailed analysis Workaround Temporary or alternative solution

Comments

@GT500org
Copy link

GT500org commented Oct 3, 2021

When running in the sandbox, SteamVR will work normally for about 10-20 minutes. There are two exceptions to this:

  1. SteamVR will have no access to apps/games owned on Steam if the Steam client was not run in the sandbox as well.
  2. vrdashboard.exe will repeatedly crash for the first couple of minutes, until SteamVR stops trying to re-launch it.

During this 10-20 minutes you can play VR games that will work without the Steam client normally. Eventually other SteamVR processes will begin to crash, leaving you with a sudden black screen in the VR HMD. This happens even if you never launch games, never turn on the controllers, and are just standing in place in SteamVR Home.

Background - If you download a free VR game from a website such as itch.io you don't necessarily know if it's trustworthy. It's not possible for SteamVR to connect to a VR HMD from within a virtual machine, and it isn't possible for a VR game running in a sandbox to connect to SteamVR, so up until now the only way I could play downloaded VR games with some degree of safety was to run SteamVR in the sandbox as well. A few months ago SteamVR began crashing when running in a sandbox, and I would run it like this too infrequently to know if it was an update to SteamVR or to Sandboxie that was responsible for it breaking.

Steps to reproduce the behavior:

  1. Make sure HMD is powered on and connected to PC.
  2. Open the following EXE in Sandboxie:
    C:\Program Files (x86)\Steam\steamapps\common\SteamVR\bin\win64\vrstartup.exe
  3. SteamVR process will begin to launch in the sandbox.
  4. vrdashboard.exe will begin crashing almost right after launching.
  5. Other SteamVR processes will continue running, and VR HMD will display 360 background until SteamVR Home launches.
  6. Once SteamVR Home launches and displays in VR HMD, wait 10-20 minutes while holding the VR HMD and moving it slightly to keep it from going into standby mode.
  7. After about 10-20 minutes SteamVR processes will start to crash.

Note: It should be possible to simulate a VR HMD with a smartphone using software such as VRidge or Trinus VR, so it may be possible to test this without actually needing an HTC Vive or an Oculus Rift (they connect over the network, so testing in a VM with DirectX 11 support should work):
https://uploadvr.com/budget-vr-101-get-pc-vr-streaming-phone/

Expected behavior
I expect SteamVR to continue running in the sandbox without issues.

System details and installed software (please provide the following information):

  • Windows 10 Pro 20H2
  • Sandboxie Plus 0.9.7
  • I'm not certain exactly when the issue started happening (at leas 4 months ago). I don't know what version of Sandboxie and SteamVR was installed at the time it started.
  • Currently Malwarebytes 4.4.7, previously Emsisoft Anti-Malware (multiple 2021 versions).
@isaak654
Copy link
Collaborator

isaak654 commented Oct 6, 2021

I'm not certain exactly when the issue started happening (at leas 4 months ago). I don't know what version of Sandboxie and SteamVR was installed at the time it started.

Could you test the Plus versions released in June (four in total) and verify if a specific Sandboxie version introduced the issue?
https://github.com/sandboxie-plus/Sandboxie/tags?after=0.8.9
(a backup is recommended)

I would also suggest to provide a log. Assuming you're on the main Sandboxie Plus window and you've closed any running program in the sandbox:

  • Go to Options menu -> Edit ini file -> confirm yes
  • Add IpcTrace=* below [DefaultBox] (or your sandbox name)
  • Save the ini file and close it

In order to take a log with Plus, you need to enable 'Trace Log' tab by opening 'Options' menu -> 'Trace Logging'.
Once enabled, you can reproduce your issue.
After reproducing it, just right click -> 'Copy Panel' to copy the full log:
trace_log
You can post the output here or in an external link to share.

@isaak654 isaak654 added the more info needed More information is needed to move forward label Oct 6, 2021
@GT500org
Copy link
Author

GT500org commented Oct 8, 2021

The contents of the panel appear to be more than can fit in the clipboard. Is the log saved to disk, or can it be saved to disk?

@GT500org
Copy link
Author

GT500org commented Oct 8, 2021

Upon further checking, the issue isn't the length of the log, but rather certain Asian characters in the log. If I reverse the order of the log before copying the panel, it pastes more of it in Notepad++, however it still stops pasting (or copying?) after getting to one of these Asian characters:

Screenshot

I'll need a workaround for this issue if you want to see the entire log. Until then, here's all I was able to copy (barely anything, as the timestamps run from 16:37:25 to 16:52:48):
https://www.gt500.org/debug/sandboxie/2021-10-08_SandBoxie-Plus_Partial_Trace_Log_SteamVR_Crashing.txt

@isaak654
Copy link
Collaborator

isaak654 commented Oct 9, 2021

it still stops pasting (or copying?) after getting to one of these Asian characters

This log bug should be fixed on 0.9.7d.

After seeing your partial trace log, I think you may try to apply these lines and verify if they help with your issue:

OpenIpcPath=*\BaseNamedObjects\vrdashboard.exe
OpenIpcPath=*\BaseNamedObjects\VR_*
OpenIpcPath=*\BaseNamedObjects\VRBootstrapper
OpenIpcPath=*\BaseNamedObjects\SteamVR_*
OpenPipePath=\Device\NamedPipe\SteamVR_Namespace
OpenPipePath=\Device\NamedPipe\VR_CompositorPipe_*
OpenPipePath=\Device\NamedPipe\VR_ServerPipe_*

If you need to take another log, you could use temporarily for debugging purposes other trace options described here and also applying OpenIpcPath=*, OpenPipePath=*, OpenWinClass=* individually or combined to exclude which OpenXxxPath setting is not needed to allow access.

@GT500org
Copy link
Author

GT500org commented Oct 9, 2021

I have 0.9.7d installed, and that's the version I'm experiencing the log copying bug with:

screenshot

I'll try adding those lines to the config file as soon as I find time.

@GT500org
Copy link
Author

GT500org commented Oct 9, 2021

Still crashed after about 15 minutes. Did you want a trace log for that as well?

@isaak654
Copy link
Collaborator

Still crashed after about 15 minutes. Did you want a trace log for that as well?

Apparently the Trace log feature is struggling because you're running it for more than 10 minutes.

I passed this issue to the maintainer, in the meantime you may want to take a log for a smaller number of minutes and/or replacing IpcTrace=* with IpcTrace=ad.

I would also suggest to the maintainer a new option to use a hotkey to stop/restart a trace log quickly in order to avoid a compulsive use of the Options -> Trace Logging menu entry.

@isaak654 isaak654 added Must-fix Log available Trace log attached for a detailed analysis labels Oct 10, 2021
@GT500org
Copy link
Author

I've gone back to Sandboxie-Plus 0.7.0 testing this, and I can't find a version that it runs in that it doesn't crash after about 15 minutes. It's possible it was a change to SteamVR that broke things.

@isaak654
Copy link
Collaborator

isaak654 commented Oct 20, 2021

I've gone back to Sandboxie-Plus 0.7.0 testing this, and I can't find a version that it runs in that it doesn't crash after about 15 minutes. It's possible it was a change to SteamVR that broke things.

Yesterday I installed 0.9.8b and there is a new button on the Trace tab to save a log on file... I think it's more reliable than the "Copy Panel" feature, although the "Unknown" lines (like yours here) can still be displayed.

In my case, these are the "Unknown" lines I logged with the new "save as log" feature while accessing to Windows Explorer:

22:08:03.869 Sconosciuto 3 7636 Unknown: 215 (D) (34844) ? (IA) 001F0001 \Sandbox\Test\test\Session_1\Sessions\1\BaseNamedObjects\C::Users:Test:AppData:Local:Microsoft:Window

22:08:12.116 Sconosciuto 3 7636 Unknown: 41 (D) (321) ? (IA) 001F0001 \Sandbox\Test\test\Session_1\Sessions\1\BaseNamedObjects\C::Users:Test:AppData:Local:Microsoft:Window

It looks like it can't handle very long paths and the Trace Log cuts them... with "Copy Panel" is even worse, because it doesn't allow you to copy almost anything when any "Unknown" entry is displayed on the Trace tab.

EDIT: What happens if you apply TraceBufferPages=2560 in the [GlobalSettings] section of the Sandboxie.ini file?

@GT500org
Copy link
Author

Here's (hopefully) a full trace log:
https://www.gt500.org/debug/sandboxie/2021-10-26_Sandboxie-Plus_0.9.8c_Trace_Log_SteamVR_Crashing.log

Let me know if you need anything else.

@isaak654
Copy link
Collaborator

isaak654 commented Oct 30, 2021

This is the only SteamVR error I was able to reproduce in a sandbox (without having no game or controller attached).
Test performed on 0.9.8c / 0.9.8d and Windows 10 21H1, with Steam and SteamVR installed outside of the sandbox:

  1. Run the following exe in a default sandbox:
    C:\Program Files (x86)\Steam\steamapps\common\SteamVR\bin\win64\vrstartup.exe
  2. Opening the SteamVR Media Player results in the warning "HMD not found (108)" that will close automatically the window if you wait there for 5 minutes. This doesn't happen if you run SteamVR Media Player outside of the sandbox.

I didn't have any luck to find out the cause after analyzing the previous logs, so I'm going to attach my log below in case someone wants to try:
https://gist.githubusercontent.com/isaak654/0d03a6dc086b4de6356d7663f814c564/raw/c8b872e2f2da9fe6824bf984b3f677449bf4ba94/SteamVR.txt

@GT500org
Copy link
Author

For debugging you may need to simulate an HMD (VR device) with one of the programs detailed in the following article:
https://uploadvr.com/budget-vr-101-get-pc-vr-streaming-phone/

Note that you may need to install an app on your phone for them to work, however I don't think they require VR or XR support on the phone so you should be able to do it without any VR equipment whatsoever.

@isaak654 isaak654 removed the more info needed More information is needed to move forward label Dec 10, 2021
@isaak654
Copy link
Collaborator

isaak654 commented Feb 16, 2022

I have some news about this issue:

  • I can't reproduce it at all when I run SteamVR in the "App Compartment" sandbox type on Plus 1.0.11.
  • I can still reproduce the crash on the standard sandbox type, but I just used the new EnableMiniDump=y command to generate a crash dump on C:\Sandbox\%USER%\MySandbox that should help the developer.

Crash dump: steamvr_media_player.exe.1899435299.zip

@DavidXanatos

@isaak654 isaak654 added the crash dump Dump file attached for a detailed analysis label Feb 16, 2022
@isaak654 isaak654 added the Workaround Temporary or alternative solution label Feb 21, 2022
@DavidXanatos
Copy link
Member

Its strange the crash happens inside a hooking routine, wil investigate this furtner

@DavidXanatos
Copy link
Member

it is strange on my system this hook works correctly, not sure why it runs into a crash for you, may be there is already some other hook present that interfears,
if you configure the box for compartment mode, this hook will not be installed in the first place, so that might be worth a try.

@isaak654
Copy link
Collaborator

Indeed I already explained that I can't reproduce it in the compartment mode (that would be the workaround).
Even if it's replicable just on the standard box, a crash is still an issue.

In order to reproduce it, you need to follow the required steps I posted here.

My GlobalSettings:

[GlobalSettings]
Template=WindowsRasMan
Template=7zipShellEx
Template=RTSS
Template=OfficeLicensing
ClosePrintSpooler=y
NotifyDirectDiskAccess=n
SeparateUserFolders=y
KeyRootPath=\REGISTRY\USER\Sandbox_%USER%_%SANDBOX%
IpcRootPath=\Sandbox\%USER%\%SANDBOX%\Session_%SESSION%
EditAdminOnly=n
ForceDisableAdminOnly=n
ForgetPassword=n
TraceBufferPages=2560
TemplateReject=WindowsLive
NetworkEnableWFP=n
EnableObjectFiltering=y
EnableWin32kHooks=n
StartRunAlertDenied=n
NotifyStartRunAccessDenied=y
ForceDisableSeconds=60

@DavidXanatos
Copy link
Member

do i need to wait 20 minutes?
on my system C:\Program Files (x86)\Steam\steamapps\common\SteamVR\bin\win64\vrstartup.exe starts just fine and says its waiting for me to connect a headset

@isaak654
Copy link
Collaborator

isaak654 commented Feb 21, 2022

Nope, the problem I reproduced here appears after opening the SteamVR Media Player from the hamburger menu.
After opening it, just wait 5 minutes and see:
immagine

@GT500org
Copy link
Author

do i need to wait 20 minutes? on my system C:\Program Files (x86)\Steam\steamapps\common\SteamVR\bin\win64\vrstartup.exe starts just fine and says its waiting for me to connect a headset

Of course it starts fine. The bug report is about a crash that happens after a period of time, not an issue with the software starting in the sandbox.

The bug report is also not about the "App Compartment" mode. That's not a sandbox. A workaround would be finding a way to run VR games in the sandbox while allowing them to communicate with SteamVR when it's running outside of the sandbox, not running SteamVR and whatever game you want to try in a non-sandboxed mode.

Also note that non-Patreon users do not appear to be able to use the "App Compartment" feature, or at least some of its capabilities appear to be locked behind a paywall. This makes it a non-viable workaround even if it could have worked as a workaround in the first place, as not everyone would have access to it.

BTW: I've upgraded to an HTC Vive Pro 2, which requires software from HTC to function (VIVE Console). While this software and its associated service do run in a sandbox, SteamVR does not appear to be able to output to the device's screens (possibly due to a vrcompositor.exe crash that happens shortly after SteamVR launches in the sandbox).

@DavidXanatos
Copy link
Member

Ok, so the media player closes with the error "HMD not found (108)", that needs some open something presumably...
but how do I get to experiencing the crash which created the crash dump posted above?

"App Compartment" mode, is as secure as comodo sandbox, so its still a sandbox, also steam is not one of your regular suspects of aggressively trying to escape isolation.
IMHO "App Compartment" is the right choice for gaming, and since you mentioned paywall, its cheeper then most games.

@isaak654
Copy link
Collaborator

Ok, so the media player closes with the error "HMD not found (108)", that needs some open something presumably...
but how do I get to experiencing the crash which created the crash dump posted above?

The crash dump was generated with EnableMiniDump=y already added in the same sandbox, maybe you could try to expand the output with MiniDumpFlags=Extended (if needed).

@DavidXanatos
Copy link
Member

DavidXanatos commented Feb 21, 2022

I was able to reproduce the issue in my debugger and it seams the problem is quite not so trivial,
as it looked initially, the software is loading and unloading a certain hooked dll all the time, sandboxie however is not designed to properly handle the unload, it leaves some memory behind each time and at some point we run out of memory in a pre-alocated table, making it larger is not an option that would just delay the inevitable.
This is why my earlier testing seamed fine, the hook works as its intended, well at least for the first few 100 times.
as a temporary workaround you can use DllSkipHook=wtsapi32.dll this disables the hooking of the problematic dll but may cause other issues, the hooks are in place for a reason.

A proper fix for this issue will require a significant redesign of the internal hooking mechanism.

I presume other crashes which occur only after a longer runtime may have the same cause.

It is a significant architectural oversight not to have properly cleaned up the hooks once a dll is unloaded.

@DavidXanatos DavidXanatos added the ToDo To be done label Feb 21, 2022
@GT500org
Copy link
Author

Do you see any possibility of a game running in Sanboxie's sandbox being able to communicate with SteamVR when it is not running via Sandboxie? While that obviously wouldn't fix the hooking issue, it would work around it for the SteamVR crash in the short term.

There's an example of a free downloadable VR game at the following link that can be used for testing if needed (to my knowledge the game is safe, but the reality is that can't be guaranteed when downloading games from sources like this, which is why I want to be able to run the games in a sandbox, but right now that also means running SteamVR in the sandbox with the game as the sandbox blocks the communication between the game and SteamVR):
https://further-beyond.itch.io/wolf3dvr

@DavidXanatos
Copy link
Member

I have not investigated yet how the communication works,
if it does not to heavenly relay on COM, (which I dont think as steam is also available on linux afaik, so thay probably use more generic communication mechnisms) it should be possible

@isaak654 isaak654 mentioned this issue Feb 23, 2022
@DavidXanatos DavidXanatos added the High priority To be done as soon as possible label Apr 6, 2022
@isaak654 isaak654 removed ToDo To be done Must-fix labels Apr 11, 2022
@DavidXanatos DavidXanatos added fixed in next build Fixed in the next Sandboxie version and removed fixed in next build Fixed in the next Sandboxie version labels May 7, 2022
@DavidXanatos
Copy link
Member

I have a fix in an internal dev build it will be included in one of the builds to come, it changes the hooking management a lot so may not be 1.0.xx but in the 1.1.xx line we will see

@DavidXanatos
Copy link
Member

build 1.2.0 that is to be releases withing the next week will contain the fix for this issue, thx for your patience

@GT500org
Copy link
Author

Awesome. Thanks.

@DavidXanatos
Copy link
Member

build is released

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
crash dump Dump file attached for a detailed analysis fixed in next build Fixed in the next Sandboxie version High priority To be done as soon as possible Log available Trace log attached for a detailed analysis Workaround Temporary or alternative solution
Projects
None yet
Development

No branches or pull requests

3 participants