You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When using a pytorch Dataloader with many workers, access to lance dataset will hang forever. This is due to the fact that torch.utils.data.Dataloader use "fork" method for multiprocessing by default. The need of using "spawn" method with lance is mentionned in lancedb FAQ : https://lancedb.github.io/lancedb/faq/#does-lancedb-support-concurrent-operations but nowhere in lance doc, example, guide or repo. There is this issue : #2204 which was send to me by someone of lance team on discord. So as it was aked to me, i'm creating this issue to let you know that this information should maybe be written in docs or guides.
Maybe here : https://github.com/lancedb/lance-deeplearning-recipes as deep-learning usually use multi-gpu.
The text was updated successfully, but these errors were encountered:
When using a pytorch Dataloader with many workers, access to lance dataset will hang forever. This is due to the fact that torch.utils.data.Dataloader use "fork" method for multiprocessing by default. The need of using "spawn" method with lance is mentionned in lancedb FAQ : https://lancedb.github.io/lancedb/faq/#does-lancedb-support-concurrent-operations but nowhere in lance doc, example, guide or repo. There is this issue : #2204 which was send to me by someone of lance team on discord. So as it was aked to me, i'm creating this issue to let you know that this information should maybe be written in docs or guides.
Maybe here : https://github.com/lancedb/lance-deeplearning-recipes as deep-learning usually use multi-gpu.
The text was updated successfully, but these errors were encountered: