0.18.11rc1

AMD

With the latest update, you can now specify an AMD GPU under resources. Below is an example.

type: service
name: amd-service-tgi

image: ghcr.io/huggingface/text-generation-inference:sha-a379d55-rocm
env:
  - HUGGING_FACE_HUB_TOKEN
  - MODEL_ID=meta-llama/Meta-Llama-3.1-70B-Instruct
  - TRUST_REMOTE_CODE=true
  - ROCM_USE_FLASH_ATTN_V2_TRITON=true
commands:
  - text-generation-launcher --port 8000
port: 8000

resources:
  gpu: MI300X
  disk: 150GB

spot_policy: auto

model:
  type: chat
  name: meta-llama/Meta-Llama-3.1-70B-Instruct
  format: openai

Note

AMD accelerators are currently supported only with the runpod backend. Support for on-prem fleets and more backends
is coming soon.

Other

• [Docs] Document projects #1547 by @peterschmidt85 in #1548
• [UI] Ensure users can create projects #191 by @olgenn in #1554
• [UI] Use a toggle button switching themes #190 by @olgenn in #1556
• [Bugfix] Force root in Kubernetes runs by @jvstme in #1555
• [Bugfix] Avoid TGI error logit_bias: invalid type by @jvstme in #1557
• Support the vendor property under gpu @un-def in #1558
• [Internal] Improve gateway auth issues troubleshooting by @jvstme in #1569
• [Feature] Implement "encryption at rest" by @r4victor in #1561
• [Feature] Implement project manager role by @r4victor in #1572
• [Feature] Implement user activation/deactivation by @r4victor in #1575
• [Bugfix] Support users without projects @olgenn in #1578
• [UI] Fix the Logs component appearance for the dark theme by @olgenn in #1579
• [UI] Minor restyle of the side navigation by @olgenn in #1580
• [Internal] Replace pkg_resources with importlib.resources by @r4victor in #1582
• [UI] Support manager project role @olgenn in #1566
• [Bugfix] Provision AWS instances in all eligible availability zones by @r4victor in #1585
• [Feature] Implement configurable default permissions by @r4victor in #1591
• [Internal] Reintroduce tpu- prefix; add tpu vendor alias by @un-def in #1587
• [Docs] Mention AMD GPUs, describe gpu.vendor property by @un-def in #1570
• [Bugfix] Fix global admin restricted by manager role by @r4victor in #1592
• [Bugfix] Fixed defect with incorrect setting project role in the UI by @olgenn in #1593
• [Internal] Order project members by @r4victor in #1594
• [Feature] Introduce default permissions #1559 by @olgenn in #1567
• [Bugfix] Abort provisioning fleet when parsing ssh key fails(#1442) by @swsvc in #1589
• [Feature] Add LogStorage interface, CloudWatch Logs impl by @un-def in #1597
• [Docs] Document AMD support on RunPod by @peterschmidt85 in #1598

New contributors

• @swsvc made their first contribution in #1589

Full changelog: 0.18.10...0.18.11rc1