Allows specifying chunk size and overlap with /learn

[3coins]

Summary

Current implementation of /learn has fixed values for chunk size and overlap which is impractical for all document types and models. Adding options for specifying these attributes will allow power users to quickly experiment and select the options that work best for their document types and model selected. This PR adds these new options to the /learn command and stores these values in the metadata, so that relearn works when embedding models are switched.

Usage

# default chunk size of 2000, and chunk overlap of 100
/learn <directory>

# chunk size of 500, and chunk overlap of 50
/learn -c 500 -o 50 <directory>

# chunk size of 1000, and chunk overlap of 200
/learn --chunk-size 1000 --chunk-overlap 200 <directory>

[JasonWeill]

We need to document chunk size and overlap, including the default, in both the user docs and the user interface. As a user, why would I need to modify these values?

[3coins]

@JasonWeill
These defaults won't work for every document and model, I was using this as a user to experiment with which size and overlap value works best and would have to update the code in order to do that without these options.

[JasonWeill]

Can you please add something about these new options to the docs for the /learn command? I can help write or edit this info. There's no help option for the chat UI's commands yet (#91), so we need to keep our written docs up to date.