Pinecone Hybrid Search #659
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
|
I had an oversight with using the |
|
@nfcampos @hwchase17 I think this is ready to be reviewed whenever the chance. I am using |
|
Note to me: use tiktoken instead of bert |
| const embeddings = new OpenAIEmbeddings(); | ||
| const pineconeIndex = client.Index(process.env.PINECONE_INDEX!); | ||
|
|
||
| const tokenizer = new BertTokenizer(undefined, true, 512); |
There was a problem hiding this comment.
Consider using js-tiktoken instead, which should already be a required dependency.
| this.topK = args.topK; | ||
| this.alpha = args.alpha; |
There was a problem hiding this comment.
nit: the default values are being overriden, should that be the case? If not, consider making alpha: number and topK: number
| this.topK = args.topK; | |
| this.alpha = args.alpha; | |
| this.topK = args.topK ?? 4; | |
| this.alpha = args.alpha ?? 0.5; |
| @@ -1,5 +1,9 @@ | |||
| export { RemoteRetriever } from "./remote/base.js"; | |||
| export { ChatGPTPluginRetriever } from "./remote/chatgpt-plugin.js"; | |||
| export { | |||
| PineconeHybridSearchRetriever, | |||
There was a problem hiding this comment.
Instead of adding a file into a barrel file, create a new entrypoint instead.
|
|
||
| pineconeIndex: VectorOperationsApi; | ||
|
|
||
| tokenizer: BertTokenizer; |
There was a problem hiding this comment.
I think it would make more sense for the sparse encoder to be more configurable, not assuming BERT tokenization.
| alpha: number | ||
| ): [number[], SparseValues] { | ||
| if (alpha < 0 || alpha > 1) { | ||
| throw new Error("Alpha must be between 0 and 1"); |
There was a problem hiding this comment.
nit: consider validating this.alpha in the constructor.
nit: do we need to pass this.alpha as a parameter? Why not access it directly?
There was a problem hiding this comment.
nit: consider validating this.alpha in the constructor.
nit: do we need to pass this.alpha as a parameter? Why not access it directly?
| } from "@pinecone-database/pinecone/dist/pinecone-generated-ts-fetch"; | ||
| import { BertTokenizer } from "bert-tokenizer"; | ||
|
|
||
| import { PineconeLibArgs, PineconeMetadata } from "vectorstores/pinecone.js"; |
There was a problem hiding this comment.
Avoid absolute imports, as this causes the example not to build
I tried to emulate the langchain python implementation of Pinecone Hybrid Search. I also used the Supabase Hybrid Search implementation as a reference. Thanks @hwchase17 @ericciarla
I went with
BertWordPieceTokenizerfromtokenizersas the tokenizer. I also had to update the@pinecone-database/pineconeto^0.0.12as I could not addsparseVectorsotherwise.Not 100% sure if everything is in working order yet, but I will work on testing it in my local project when I get the chance. A good starting point nonetheless!