You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm trying to assess a model that does not fit into a single GPU, and therefore needs to be loaded with device_map="auto", and torch_dtype=torch.float16. As far as I understand, evaluate.evaluator.compute(), does not support passing arguments to AutoModelForCausalLM.from_pretrained(), nor does it support an instantiated model, so I had to resort to using a pipe.
$ python3.11
Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import evaluate, torch, accelerate
[2023-08-07 17:06:13,359] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
>>> from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
>>> from datasets import load_dataset
>>> model_id = "TheBloke/Llama-2-7b-chat-fp16"
>>> my_pipe = pipeline(model=model_id, device_map="auto", torch_dtype=torch.float16, max_length=256)
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:37<00:00, 18.81s/it]
>>> my_data = load_dataset("Abirate/english_quotes", split='train').select([1])
>>> my_metric=evaluate.load("perplexity")
>>> my_evaluator = evaluate.evaluator(task="text-generation")
>>> result = my_evaluator.compute(model_or_pipeline=my_pipe,
... data=my_data, input_column = "quote",
... metric=my_metric,
... tokenizer=AutoTokenizer.from_pretrained(model_id))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/hans/.local/lib/python3.11/site-packages/evaluate/evaluator/base.py", line 261, in compute
metric_results = self.compute_metric(
^^^^^^^^^^^^^^^^^^^^
File "/home/hans/.local/lib/python3.11/site-packages/evaluate/evaluator/base.py", line 467, in compute_metric
result = metric.compute(**metric_inputs, **self.METRIC_KWARGS)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/hans/.local/lib/python3.11/site-packages/evaluate/module.py", line 433, in compute
self._finalize()
File "/home/hans/.local/lib/python3.11/site-packages/evaluate/module.py", line 385, in _finalize
file_paths, filelocks = self._get_all_cache_files()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/hans/.local/lib/python3.11/site-packages/evaluate/module.py", line 302, in _get_all_cache_files
raise ValueError(
ValueError: Evaluation module cache file doesn't exist. Please make sure that you call `add` or `add_batch` at least once before calling `compute`.
How can one compute perplexity with a custom pipe?
The text was updated successfully, but these errors were encountered:
As a user, I found a walkaround: using customized metrics, carefully. It works for me and thus I am here to share with you.
Step 1: at some /path/to/somewhere, create a folder my_perplexity, under which further create a Python file my_perplexity.py. Copy all the content from the officially defined perplexity.py (e.g., here) and paste it there.
Step 2: Comment out the following two lines in the definition of _compute() in my_perplexity.py:
This way, one receives a customized model and tokenizer piggybacked by the argument model_id. By the way, the argument model_id is the only argument you can override with no side-effect in this context.
Step 3: In the original evaluation code, we can now use a customized model and tokenizer in this way:
perplexity=evaluate.load("/path/to/somewhere/my_perplexity", module_type="metric")
results=metric.compute(
model_id=(model, tokenizer),
predictions=predictions# a list of input text
)
Still, I hope the developer can directly support this. It is of no intrinsic difficulty.
I'm trying to assess a model that does not fit into a single GPU, and therefore needs to be loaded with
device_map="auto"
, andtorch_dtype=torch.float16
. As far as I understand,evaluate.evaluator.compute()
, does not support passing arguments toAutoModelForCausalLM.from_pretrained()
, nor does it support an instantiated model, so I had to resort to using apipe
.How can one compute perplexity with a custom pipe?
The text was updated successfully, but these errors were encountered: