/learn not working properly for directories in 2.14.0

[michaelchia]

Description

/learn does not properly index all the files in the directory.

jupyter-ai/packages/jupyter-ai/jupyter_ai/document_loaders/directory.py

Lines 64 to 92 in 3e6b61d

	def split(path, all_files: bool, splitter):
	chunks = []
	# Check if the path points to a single file
	if os.path.isfile(path):
	dir = os.path.dirname(path)
	filenames = [os.path.basename(path)]
	else:
	for dir, subdirs, filenames in os.walk(path):
	# Filter out hidden filenames, hidden directories, and excluded directories,
	# unless "all files" are requested
	if not all_files:
	subdirs[:] = [
	d for d in subdirs if not (d[0] == "." or d in EXCLUDE_DIRS)
	]
	filenames = [f for f in filenames if not f[0] == "."]
	for filename in filenames:
	filepath = Path(os.path.join(dir, filename))
	# Lower case everything to make sure file extension comparisons are not case sensitive
	if filepath.suffix.lower() not in {j.lower() for j in SUPPORTED_EXTS}:
	continue
	document = dask.delayed(path_to_doc)(filepath)
	chunk = dask.delayed(split_document)(document, splitter)
	chunks.append(chunk)
	flattened_chunks = dask.delayed(flatten)(*chunks)
	return flattened_chunks

split only keeps the files in the last iteration of os.walk(path).

I've made a PR that fixes this. #747

[dlqqq]

Closed by #747. Thank you @michaelchia !