Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to acquire fcntl.LOCK_SH lock when cache directory is mounted on NFS #515

Closed
tonywang16 opened this issue Sep 19, 2023 · 0 comments · Fixed by #517
Closed

Unable to acquire fcntl.LOCK_SH lock when cache directory is mounted on NFS #515

tonywang16 opened this issue Sep 19, 2023 · 0 comments · Fixed by #517

Comments

@tonywang16
Copy link
Contributor

tonywang16 commented Sep 19, 2023 •
edited
Loading

Error showing failed to acquire shared file lock during downloading model weight file from hub when cache directory is on a NFS mount.

Traceback (most recent call last):
  File "/root/petals/src/petals/server/from_pretrained.py", line 139, in _load_state_dict_from_file
    with allow_cache_reads(cache_dir):
  File "/usr/lib/python3.9/contextlib.py", line 117, in __enter__
    return next(self.gen)
  File "/root/petals/src/petals/utils/disk_cache.py", line 26, in _blocks_lock
    fcntl.flock(lock_fd.fileno(), mode)
OSError: [Errno 9] Bad file descriptor`

Reproduce:
Make sure open the file on a NFS mounted directory:

import os
import fclnt
f = open("/nfs-mounted-dir/abc","wb")
fcntl.flock(f.fileno(), fcntl.LOCK_SH)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 9] Bad file descriptor

Suggested resolution:
Open file in "wb+" mode

import os
import fclnt
f = open("/nfs-mounted-dir/abc","wb+")
fcntl.flock(f.fileno(), fcntl.LOCK_SH)
tonywang16 pushed a commit to tonywang16/petals that referenced this issue Sep 19, 2023
bigscience-workshop#515 fix file lock failed to acquire on NFS mounte…
172f593 
…d directory
@tonywang16 tonywang16 mentioned this issue Sep 19, 2023
Closed
tonywang16 added a commit to tonywang16/petals that referenced this issue Sep 19, 2023
@tonywang16
Fix bigscience-workshop#515 unable to acquire shared lock on NFS moun…
a14c0f3 
…ted directory
@tonywang16 tonywang16 mentioned this issue Sep 19, 2023
Merged
tonywang16 added a commit to tonywang16/petals that referenced this issue Sep 19, 2023
@tonywang16
bigscience-workshop#515 update file open mode to avoid acquire file l…
d57917b 
…ock fail on NFS shared mount
borzunov pushed a commit that referenced this issue Sep 20, 2023
@tonywang16
Fix file locks in NFS-mounted directories (#517)
a2484b3 
Fix #515.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
1 participant
@tonywang16