Fix /learn in 2.14.0

[michaelchia]

Fixed /learn not working properly for directories in 2.14.0. For directories, it only kept the files from the final iteration of os.walk(path) instead of accumulating the filepaths across iterations.

[srdas]

@michaelchia Thanks for the PR, the key missing line is 80 below, and thanks for adjusting the rest of the code accordingly.

I fixed the failing test, and @dlqqq @3coins -- please take a look to confirm, the fix looks good, I have tested it as well.

[3coins]

@michaelchia
Thanks for submitting the fix. This is a critical piece of logic that should have some unit tests to make sure there are no regressions. Let us know if you need help with this.

[michaelchia]

I did look into the tests but didn't find any similar tests for other handlers or their supporting functions. I wasn't sure what the convention was for this project. It is probably best that the core team create the tests for this.

[3coins]

@michaelchia
Looks good, here are the results of my testing. @srdas will take care of the unit tests in a separate PR.

(jupyter-ai-lab4) ➜  playground git:(fix-learn) ✗ tree misc
misc
├── Chemical Equations and Stoichiometry.ipynb
├── Python Decorators: A Comprehensive Guide.ipynb
├── README.md
├── Testing, Training, and Deploying a Model with Amazon SageMaker.ipynb
├── Understanding Python Decorators.ipynb
├── Untitled.ipynb
├── Using the GitHub API: A Jupyter Notebook.ipynb
├── config.example.py
├── config.py
├── magics-demo.ipynb
├── magics.ipynb
├── neptune_utils.ipynb
├── notebooks
│   ├── An Overview of Traitlets.ipynb
│   ├── Introduction to Differential Equations.ipynb
│   └── Matplotlib Usage and Customization.ipynb
├── state-of-union
│   └── state_of_the_union.txt
├── test
│   ├── PDF generation test.ipynb
│   └── empty.ipynb
├── test.ipynb
└── test_generate_components.ipynb

Before

>>> from jupyter_ai.document_loaders.directory import split
>>> split('README.md', False, None)
Processing file: README.md
>>> split('misc', False, None)
dir is misc
subdirs before filtering: ['test', 'state-of-union', '.ipynb_checkpoints', 'notebooks']
subdirs after filtering: ['test', 'state-of-union', 'notebooks']
dir is misc/test
subdirs before filtering: ['.ipynb_checkpoints']
subdirs after filtering: []
dir is misc/state-of-union
subdirs before filtering: ['.ipynb_checkpoints']
subdirs after filtering: []
dir is misc/notebooks
subdirs before filtering: ['.ipynb_checkpoints']
subdirs after filtering: []
Processing file: misc/notebooks/Introduction to Differential Equations.ipynb
Processing file: misc/notebooks/An Overview of Traitlets.ipynb
Processing file: misc/notebooks/Matplotlib Usage and Customization.ipynb

After

>>> from jupyter_ai.document_loaders.directory import split
>>> split('README.md', False, None)
Processing file: README.md

>>> split('misc', False, None)
dir is misc
subdirs before filtering: ['test', 'state-of-union', '.ipynb_checkpoints', 'notebooks']
subdirs after filtering: ['test', 'state-of-union', 'notebooks']
dir is misc/test
subdirs before filtering: ['.ipynb_checkpoints']
subdirs after filtering: []
dir is misc/state-of-union
subdirs before filtering: ['.ipynb_checkpoints']
subdirs after filtering: []
dir is misc/notebooks
subdirs before filtering: ['.ipynb_checkpoints']
subdirs after filtering: []
Processing file: misc/test_generate_components.ipynb
Processing file: misc/magics-demo.ipynb
Processing file: misc/config.py
Processing file: misc/Untitled.ipynb
Processing file: misc/Testing, Training, and Deploying a Model with Amazon SageMaker.ipynb
Processing file: misc/magics.ipynb
Processing file: misc/README.md
Processing file: misc/config.example.py
Processing file: misc/Python Decorators: A Comprehensive Guide.ipynb
Processing file: misc/Chemical Equations and Stoichiometry.ipynb
Processing file: misc/neptune_utils.ipynb
Processing file: misc/Understanding Python Decorators.ipynb
Processing file: misc/Using the GitHub API: A Jupyter Notebook.ipynb
Processing file: misc/test.ipynb
Processing file: misc/test/PDF generation test.ipynb
Processing file: misc/test/empty.ipynb
Processing file: misc/state-of-union/state_of_the_union.txt
Processing file: misc/notebooks/Introduction to Differential Equations.ipynb
Processing file: misc/notebooks/An Overview of Traitlets.ipynb
Processing file: misc/notebooks/Matplotlib Usage and Customization.ipynb
Processing file: misc/test_generate_components.ipynb
Processing file: misc/magics-demo.ipynb
Processing file: misc/config.py
Processing file: misc/Untitled.ipynb
Processing file: misc/Testing, Training, and Deploying a Model with Amazon SageMaker.ipynb
Processing file: misc/magics.ipynb
Processing file: misc/README.md
Processing file: misc/config.example.py
Processing file: misc/Python Decorators: A Comprehensive Guide.ipynb
Processing file: misc/Chemical Equations and Stoichiometry.ipynb
Processing file: misc/neptune_utils.ipynb
Processing file: misc/Understanding Python Decorators.ipynb
Processing file: misc/Using the GitHub API: A Jupyter Notebook.ipynb
Processing file: misc/test.ipynb
Processing file: misc/test/PDF generation test.ipynb
Processing file: misc/test/empty.ipynb
Processing file: misc/state-of-union/state_of_the_union.txt
Processing file: misc/notebooks/Introduction to Differential Equations.ipynb
Processing file: misc/notebooks/An Overview of Traitlets.ipynb
Processing file: misc/notebooks/Matplotlib Usage and Customization.ipynb

[dlqqq]

@meeseeksdev please backport to 1.x