Hybrid search #2969
Merged
Hybrid search #2969
+303
−23
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Adam Treat <treat.adam@gmail.com>
…s in exact same database on quality dataset. Signed-off-by: Adam Treat <treat.adam@gmail.com>
Signed-off-by: Adam Treat <treat.adam@gmail.com>
Signed-off-by: Adam Treat <treat.adam@gmail.com>
cebtenzzre
reviewed
Sep 24, 2024
Signed-off-by: Adam Treat <treat.adam@gmail.com>
Signed-off-by: Adam Treat <treat.adam@gmail.com>
Signed-off-by: Adam Treat <treat.adam@gmail.com>
Signed-off-by: Adam Treat <treat.adam@gmail.com>
This reverts commit eb39a36. Signed-off-by: Adam Treat <treat.adam@gmail.com>
…d results in" This reverts commit 552b471. Signed-off-by: Adam Treat <treat.adam@gmail.com>
Signed-off-by: Adam Treat <treat.adam@gmail.com>
Signed-off-by: Adam Treat <treat.adam@gmail.com>
Signed-off-by: Adam Treat <treat.adam@gmail.com>
manyoso
force-pushed
the
hybrid_search
branch
from
4c23020
to
3f04b01
Compare
Signed-off-by: Adam Treat <treat.adam@gmail.com>
Signed-off-by: Adam Treat <treat.adam@gmail.com>
Signed-off-by: Adam Treat <treat.adam@gmail.com>
Signed-off-by: Adam Treat <treat.adam@gmail.com>
cebtenzzre
approved these changes
Sep 26, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The first three commits are not strictly necessary for hybrid search. The first one is important though as we should maintain the same order of results that the embedding search returns. Even this has an impact on beir test results.
The second and third commits are about addressing a problem in our current chunking strategy where the maximum chunk size is not strictly enforced. These two changes enforce a strict maximum chunk size while not changing anything else about our chunking strategy.
The fourth commit is the actual hybrid search. It introduces the fts virtual table and implements reciprocal rank fusion to combine bm25 keyword search with the embedding search.
RRF paper: https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf
Beir dataset paper: https://arxiv.org/pdf/2306.07471
The following changes improve our performance across four beir datasets. Future changes will be forthcoming integrating the test harness used to assess and make these changes. For now, here are screenshots showing some of the results:
I also tested k=3 with 512 chunk size which matches our localdocs defaults and the numbers again showed improvements for hybrid search.
The one dataset that doesn't show clear improvement at 512 chunk size is fiqa, but it does show improvement with document sized chunks. Still researching how to improve performance on this one.
Also: I'm considering adding a configuration option to turn on/off hybrid search, but I think this is good to go in now.