Not working propperly under CPU/RAM usage - Confirmed connection

[mironkraft]

Hello there Sebax

Maybe we are wrong about the usage of this app, but, afaik we seen, should run under CPU/RAM, is that right?

I was yesterday setting up and making some test, all kind of failed, some screenshots:

Next image, shows that he read the PDF I uploaded:

Are we doing it wrong by using it under CPU/RAM? It is supposed to work only with GPU?

Thanks, kind regards!

[mironkraft]

Late tries goes like this:

Also, other error that I get:

UniqueConstraintError: Collection 6d029a2e-76e8-40ad-9ad0-34b35a807bf5 already exists

File "C:\Users\ASUS\Desktop\IA\oobabooga_windows\installer_files\env\lib\site-packages\streamlit\runtime\scriptrunner\script_runner.py", line 552, in _run_script exec(code, module.__dict__) File "C:\Users\ASUS\Desktop\IA\Juridia-main\st_interface_en.py", line 188, in <module> main() File "C:\Users\ASUS\Desktop\IA\Juridia-main\st_interface_en.py", line 70, in main st.session_state.db = VectorStore.get(name=name, chunk_size=chunk_size, chunk_overlap=chunk_overlap, model_name=repo_id, device=device) File "C:\Users\ASUS\Desktop\IA\Juridia-main\src\main.py", line 142, in get vectorstore = VectorStore.create(persist_directory=index_dir, splitted_docs=splitted_docs, embeddings=embeddings, name=name) File "C:\Users\ASUS\Desktop\IA\Juridia-main\src\main.py", line 79, in create return Chroma.from_documents(documents=splitted_docs, embedding=embeddings, persist_directory=persist_directory, collection_name='name') File "C:\Users\ASUS\Desktop\IA\oobabooga_windows\installer_files\env\lib\site-packages\langchain\vectorstores\chroma.py", line 578, in from_documents return cls.from_texts( File "C:\Users\ASUS\Desktop\IA\oobabooga_windows\installer_files\env\lib\site-packages\langchain\vectorstores\chroma.py", line 533, in from_texts chroma_collection = cls( File "C:\Users\ASUS\Desktop\IA\oobabooga_windows\installer_files\env\lib\site-packages\langchain\vectorstores\chroma.py", line 115, in __init__ self._collection = self._client.get_or_create_collection( File "C:\Users\ASUS\Desktop\IA\oobabooga_windows\installer_files\env\lib\site-packages\chromadb\api\segment.py", line 153, in get_or_create_collection return self.create_collection( File "C:\Users\ASUS\Desktop\IA\oobabooga_windows\installer_files\env\lib\site-packages\chromadb\api\segment.py", line 134, in create_collection self._sysdb.create_collection(coll) File "C:\Users\ASUS\Desktop\IA\oobabooga_windows\installer_files\env\lib\site-packages\chromadb\db\mixins\sysdb.py", line 96, in create_collection raise UniqueConstraintError(

[mironkraft]

Second Test is kinda working but not ok, it has processed and answered, but cannot ask more:

[mironkraft]

as you can see, it works on the CMD, but it gets blocked to interact with on the interface:

[sebaxzero]

1. for UniqueConstraintError: Collection 6d029a2e-76e8-40ad-9ad0-34b35a807bf5 already exists.
   theres is an issue with how langchain saves multiples vectorstore, im trying to solve it but for now delete "sessions" folder where you cloned the repo, this will delete the process document and you will need to process again.

2. for no response from the llm backend (oobabooga error 404), this is an error from the textgen itself, maybe you didnt enable the api or used a wrong url, if you are using cpu-only i might suggest to use llama.cpp directly instead of oobabooga

[mbaggnx]

2. llama.cpp

hello there, we are still fighting to get this work jejeje, when you mean "use llama.cpp" you mean the option inside of oobabooga right?
If yes, It does not loads correctly:

[sebaxzero]

2. llama.cpp

hello there, we are still fighting to get this work jejeje, when you mean "use llama.cpp" you mean the option inside of oobabooga right? If yes, It does not loads correctly:

no, i meant to install it as a python package in the project, without using oobabooga.
you can use oobabooga with llama.cpp but it never work for me so i cant help you much with that.

you can create a python virtual enviroment and use it from there as shown in #installation

to install:

1. open cmd, 2. clone repo, 3. navigate, 4. create venv, 5. activate venv, 6. install requirements, 7. install llama.cpp.

to use:

1. activate venv, 2. streamlit run command.

llama.cpp is a library to run ggml models in cpu-ram, oobabooga allows you to use it but only with compatible ggml models, the error you are getting might be a wrong model format, the model you are trying to load i think is a sentence transformer model, not an llm, for example you can use: TheBloke/vicuna-13b-v1.3.0-GGML.

[mironkraft]

2. llama.cpp

hello there, we are still fighting to get this work jejeje, when you mean "use llama.cpp" you mean the option inside of oobabooga right? If yes, It does not loads correctly:

no, i meant to install it as a python package in the project, without using oobabooga. you can use oobabooga with llama.cpp but it never work for me so i cant help you much with that.

you can create a python virtual enviroment and use it from there as shown in #installation

to install:

1. open cmd, 2. clone repo, 3. navigate, 4. create venv, 5. activate venv, 6. install requirements, 7. install llama.cpp.

to use:

1. activate venv, 2. streamlit run command.

llama.cpp is a library to run ggml models in cpu-ram, oobabooga allows you to use it but only with compatible ggml models, the error you are getting might be a wrong model format, the model you are trying to load i think is a sentence transformer model, not an llm, for example you can use: TheBloke/vicuna-13b-v1.3.0-GGML.

Hi, i think we are starting to understand better the repo, you can even work without oobabooga right? becuase you can load also a llama.cpp

We have already conected oobabooga to the juridia and it seems working (done in low profile PC)

I guess this means that it is conected and working right? We have to test it into a better PC

for now I loaded like this:

We will keep you informed in any case, thanks a lot for your time, appreacite it :D