Add TPU service and task example #1416
Conversation
|
What about a README file? Do you have an idea of what could be written there? |
@peterschmidt85 Do you mean this README file? |
|
@Bihan I mean typically every example we have has a README.md like https://github.com/dstackai/dstack/blob/master/examples/fine-tuning/axolotl/README.md (a detailed one)or https://github.com/dstackai/dstack/blob/master/examples/fine-tuning/qlora/README.md (a short one - didn't have time to write a good one). |
@peterschmidt85 I had added a README.md here for the TPU/TGI example. I will add more details in this README file. |
Oh, I missed it when reviewed it. If you want to add more details what can we add that could be very useful? |
@peterschmidt85 We can add about the |
|
Sounds good it me
Best regards,
Andrey
…On Thu 18. Jul 2024 at 11:01, Bihan Rana ***@***.***> wrote:
@Bihan <https://github.com/Bihan> I mean typically every example we have
has a README.md like
https://github.com/dstackai/dstack/blob/master/examples/fine-tuning/axolotl/README.md
(a detailed one)or
https://github.com/dstackai/dstack/blob/master/examples/fine-tuning/qlora/README.md
(a short one - didn't have time to write a good one).
@peterschmidt85 <https://github.com/peterschmidt85> I had added a
README.md here
<https://github.com/Bihan/dstack/blob/tpu_examples/examples/deployment/tpu/tgi/README.md>
for the TPU/TGI example. I will add more details in this README file.
Oh, I missed it when reviewed it. If you want to add more details what can
we add that could be very useful?
@peterschmidt85 <https://github.com/peterschmidt85> We can add about the image
huggingface/optimum-tpu:latest. Also point to optimum-tpu repo
<http://huggingface/optimum-tpu:latest> for more details. Mention the
models (Gemma (2b, 7b), Llama2 (7b) and Llama3 (8b)) supported by it.
Finally, we can mention that all TPU Cores are utilized in Inference.
—
Reply to this email directly, view it on GitHub
<#1416 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AM5DXRUXCJSH4EIDJ5OVKCDZM576PAVCNFSM6AAAAABLARWBASVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZVHE4TIOJYHE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
… example temporarily; minor edits; WIP
… example temporarily; minor edits; WIP
Change optimum-tpu fork from bihan to dstack repo
| [Optimum TPU :material-arrow-top-right-thin:{ .external }](https://github.com/huggingface/optimum-tpu){:target="_blank"}. | ||
| Llama 3.1 8B using | ||
| [Optimum TPU :material-arrow-top-right-thin:{ .external }](https://github.com/huggingface/optimum-tpu){:target="_blank"} | ||
| and [vLLM :material-arrow-top-right-thin:{ .external }](https://github.com/huggingface/optimum-tpu){:target="_blank"}. |
There was a problem hiding this comment.
Noticed vLLM link is wrong. I will update it.
|
Closing PR Due to Squashing Complexity and for Repo Cleanliness This PR is being closed to address issues with squashing commits and maintaining a clean commit history. New PR is opened with the appropriate changes consolidated into a single, clean commit. |
No description provided.