Integrate semantic search with OpenAI embedings API

[umaar]

Is the search a traditional exact text match? If so, having semantic searching via embeddings/completion would be great!

[NotJoeMartinez]

The current implementation uses the SQLite LIKE keyword, meaning an exact match isn't necessary, but it is still somewhat limited. I plan to implement true full-text search with sqlite-utils, which should enhance query speed and introduce more flexibility for features like relevancy ranking. #24

Several users brought up the idea of semantic search on the HN thread, and it's an area I'm interested in exploring. I don't have much experience with vector search databases, but my rough plan is the following:

• Using the OpenAI Python library to authenticate users with their API key.
• Adding a command to generate embeddings from one of their models.
• Somehow creating a searchable vector database with the embeddings data, possibly using the openai-to-sqlite library.

As I mentioned, I'm not very familiar with this area, so any suggestions are highly appreciated. Thank you!

[umaar]

Sounds excellent, thank you for this!

[NotJoeMartinez]

@umaar Added experimental integration with semantic search via OpenAI embeddings on #48

• https://github.com/NotJoeMartinez/yt-fts#Semantic-Search-via-OpenAI-embeddings-API