Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Llama-v2 to Llama.cpp notebook #7913

Merged
merged 1 commit into from
Jul 18, 2023
Merged

Add Llama-v2 to Llama.cpp notebook #7913

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Add Llama-v2
  • Loading branch information
@rlancemartin
rlancemartin committed Jul 18, 2023
commit 8a1ff2e30236602a8414a4ed50e37b9619b86d2d
103 changes: 87 additions & 16 deletions docs/extras/modules/model_io/models/llms/integrations/llamacpp.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand All @@ -14,7 +13,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand All @@ -40,7 +38,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand All @@ -61,7 +58,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand All @@ -78,7 +74,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand All @@ -99,7 +94,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand All @@ -116,15 +110,13 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Usage"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand All @@ -135,7 +127,7 @@
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": 4,
"metadata": {
"tags": []
},
Expand All @@ -148,7 +140,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand All @@ -157,7 +148,7 @@
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": 5,
"metadata": {
"tags": []
},
Expand All @@ -172,7 +163,7 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": 6,
"metadata": {
"tags": []
},
Expand All @@ -184,13 +175,96 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### CPU"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`Llama-v2`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Make sure the model path is correct for your system!\n",
"llm = LlamaCpp(\n",
" model_path=\"/Users/rlm/Desktop/Code/llama/llama-2-7b-ggml/llama-2-7b-chat.ggmlv3.q4_0.bin\",\n",
" input={\"temperature\": 0.75, \"max_length\": 2000, \"top_p\": 1},\n",
" callback_manager=callback_manager,\n",
" verbose=True,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Stephen Colbert:\n",
"Yo, John, I heard you've been talkin' smack about me on your show.\n",
"Let me tell you somethin', pal, I'm the king of late-night TV\n",
"My satire is sharp as a razor, it cuts deeper than a knife\n",
"While you're just a british bloke tryin' to be funny with your accent and your wit.\n",
"John Oliver:\n",
"Oh Stephen, don't be ridiculous, you may have the ratings but I got the real talk.\n",
"My show is the one that people actually watch and listen to, not just for the laughs but for the facts.\n",
"While you're busy talkin' trash, I'm out here bringing the truth to light.\n",
"Stephen Colbert:\n",
"Truth? Ha! You think your show is about truth? Please, it's all just a joke to you.\n",
"You're just a fancy-pants british guy tryin' to be funny with your news and your jokes.\n",
"While I'm the one who's really makin' a difference, with my sat"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n",
"llama_print_timings: load time = 358.60 ms\n",
"llama_print_timings: sample time = 172.55 ms / 256 runs ( 0.67 ms per token, 1483.59 tokens per second)\n",
"llama_print_timings: prompt eval time = 613.36 ms / 16 tokens ( 38.33 ms per token, 26.09 tokens per second)\n",
"llama_print_timings: eval time = 10151.17 ms / 255 runs ( 39.81 ms per token, 25.12 tokens per second)\n",
"llama_print_timings: total time = 11332.41 ms\n"
]
},
{
"data": {
"text/plain": [
"\"\\nStephen Colbert:\\nYo, John, I heard you've been talkin' smack about me on your show.\\nLet me tell you somethin', pal, I'm the king of late-night TV\\nMy satire is sharp as a razor, it cuts deeper than a knife\\nWhile you're just a british bloke tryin' to be funny with your accent and your wit.\\nJohn Oliver:\\nOh Stephen, don't be ridiculous, you may have the ratings but I got the real talk.\\nMy show is the one that people actually watch and listen to, not just for the laughs but for the facts.\\nWhile you're busy talkin' trash, I'm out here bringing the truth to light.\\nStephen Colbert:\\nTruth? Ha! You think your show is about truth? Please, it's all just a joke to you.\\nYou're just a fancy-pants british guy tryin' to be funny with your news and your jokes.\\nWhile I'm the one who's really makin' a difference, with my sat\""
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"prompt = \"\"\"\n",
"Question: A rap battle between Stephen Colbert and John Oliver\n",
"\"\"\"\n",
"llm(prompt)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`Llama-v1`"
]
},
{
"cell_type": "code",
"execution_count": 18,
Expand Down Expand Up @@ -260,7 +334,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand Down Expand Up @@ -366,7 +439,6 @@
"source": []
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand Down Expand Up @@ -405,7 +477,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand Down