quant_pre_process on Windows will raise PermissionError when trying to remove temporary directory

[guotuofeng]

Describe the issue

In ORT 1.16.0, the quant_pre_process will raise the following exception when Olive is running in Windows.

The quant_pre_process works fine for ORT 1.15.1.

Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.8.16\x64\lib\shutil.py", line 616, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\CLOUDT~1\\AppData\\Local\\Temp\\pre.quant.om44m0mg\\bert.embeddings.LayerNorm.bias'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.8.16\x64\lib\tempfile.py", line 802, in onerror
    _os.unlink(path)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\CLOUDT~1\\AppData\\Local\\Temp\\pre.quant.om44m0mg\\bert.embeddings.LayerNorm.bias'
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\a\_work\1\s\***\passes\onnx\quantization.py", line 442, in _quant_preprocess
    quant_pre_process(
  File "C:\hostedtoolcache\windows\Python\3.8.16\x64\lib\site-packages\onnxruntime\quantization\shape_inference.py", line 133, in quant_pre_process
    model = onnx.load(inferred_model_path)
  File "C:\hostedtoolcache\windows\Python\3.8.16\x64\lib\tempfile.py", line 827, in __exit__
    self.cleanup()
  File "C:\hostedtoolcache\windows\Python\3.8.16\x64\lib\tempfile.py", line 831, in cleanup
    self._rmtree(self.name)
  File "C:\hostedtoolcache\windows\Python\3.8.16\x64\lib\tempfile.py", line 813, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\hostedtoolcache\windows\Python\3.8.16\x64\lib\shutil.py", line 740, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\hostedtoolcache\windows\Python\3.8.16\x64\lib\shutil.py", line 618, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\hostedtoolcache\windows\Python\3.8.16\x64\lib\tempfile.py", line 805, in onerror
    cls._rmtree(path)
  File "C:\hostedtoolcache\windows\Python\3.8.16\x64\lib\tempfile.py", line 813, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\hostedtoolcache\windows\Python\3.8.16\x64\lib\shutil.py", line 740, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\hostedtoolcache\windows\Python\3.8.16\x64\lib\shutil.py", line 599, in _rmtree_unsafe
    onerror(os.scandir, path, sys.exc_info())
  File "C:\hostedtoolcache\windows\Python\3.8.16\x64\lib\shutil.py", line 596, in _rmtree_unsafe
    with os.scandir(path) as scandir_it:
NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\CLOUDT~1\\AppData\\Local\\Temp\\pre.quant.om44m0mg\\bert.embeddings.LayerNorm.bias'

To reproduce

https://aiinfra.visualstudio.com/PublicPackages/_build/results?buildId=357498&view=logs&jobId=aaa819ba-d3b3-5f6a-9167-927525a92668&j=aaa819ba-d3b3-5f6a-9167-927525a92668&t=11258177-43ed-56c7-2af4-75f6baa69786

Urgency

The issue cannot be reproed locally. But it will be reproed 100% in CI pipelines.

Platform

Windows

OS Version

10

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.16

ONNX Runtime API

Python

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response

[guotuofeng]

I am currently use microsoft/Olive@86c461a to workaround the issue

[trajepl]

https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/quantization/shape_inference.py#L79-L102
@xadupre the failure may be: the created onnxruntime inference session does not be destroyed in time.

https://github.com/microsoft/Olive/blob/f04ed7123d9683a384e966979eb14fe72902ba6b/test/unit_test/passes/onnx/test_model_optimizer.py#L43-L45

The above fix worked for me.

[trajepl]

Somehow when the context of temp directory is end, the ort session do not be cleaned which is occupying the external data file.
I have several quick tries:

1. save model with external data file.
2. create a session and key it alive
   • I cannot remove the external data file manually in windows as it is opened.
3. del sess:
   • I can remove the external data file successfully.

[yufenglee]

@trajepl, could you please create a PR to fix the issue?

[trajepl]

@trajepl, could you please create a PR to fix the issue?

Sure. Created one into main branch. #17697. Please help take a look~ :)

Oh, besides, I have no idea about how to write a sufficient test case for this issue. Just test it in Olive side. microsoft/Olive#609