0.12.0

Server configuration

Previously, the only way to configure clouds for a project was through the UI. Additionally, you had to specify not only the credentials but also set up a storage bucket for each cloud to store metadata.

Now, you can configure clouds for a project via ~/.dstack/server/config.yml. Example:

projects:
- name: main
  backends:
  - type: aws
    creds:
      type: access_key
      access_key: AIZKISCVKUKO5AAKLAEH
      secret_key: QSbmpqJIUBn1V5U3pyM9S6lwwiu8/fOJ2dgfwFdW

Enhanced Python API

The earlier introduced Python API is now greatly refined.

Creating a dstack client is as easy as this:

from dstack.api import Client, ClientError

try:
    client = Client.from_config()
except ClientError:
    print("Can't connect to the server")

Now, you can submit a task or a service:

from dstack.api import Task, Resources, GPU

task = Task(
    image="ghcr.io/huggingface/text-generation-inference:latest",
    env={"MODEL_ID": "TheBloke/Llama-2-13B-chat-GPTQ"},
    commands=[
        "text-generation-launcher --trust-remote-code --quantize gptq",
    ],
    ports=["80"],
)

run = client.runs.submit(
    run_name="my-awesome-run",
    configuration=task,
    resources=Resources(gpu=GPU(memory="24GB")),
)

The dstack.api.Run instance provides methods for various operations including attaching to the run,
forwarding ports to localhost, retrieving status, stopping, and accessing logs. For more details, refer to
the example and reference.

Other changes

• Because we've prioritized CLI and API UX over the UI, the UI is no longer bundled.
  Please inform us if you experience any significant inconvenience related to this.
• Gateways should now be configured using the dstack gateway command, and their usage requires you to specify a domain.
  Learn more about how to set up a gateway.
• The dstack start command is now dstack server.
• The Python API classes were moved from the dstack package to dstack.api.

Migration

Unfortunately, when upgrading to 0.12.0, there is no automatic migration for data.
This means you'll need to delete ~/.dstack and configure dstack from scratch.

1. pip install "dstack[all]==0.12.0"
2. Delete ~/.dstack
3. Configure clouds via ~/.dstack/server/config.yml (see the new guide)
4. Run dstack server

0.11.3

Full Changelog: 0.11.2...0.11.3

0.11.1

Default gateway

Previously, to run a service, you had to create a gateway using the dstack gateway create command and pass its address via the gateway property in the service configuration file.

Now, you don't need to use the gateway property anymore, as you can create a gateway via the UI and mark it as default.

Gateway domain

Once the gateway is created (and assigned an external IP), you can set up an A record with your DNS provider to map *.<your domain name> to the gateway's IP and specify this wildcard domain in the gateway's settings.

If a wildcard domain is configured, dstack automatically enables HTTPS and runs services at https://<run name>.<your domain name>.

Retry policy

In other news, the update fixes a few bugs with the --retry-limit argument in dstack run. Now, it works again, allowing you to schedule tasks even if there is no required capacity at the moment.

Last but not least, we've updated the entire documentation and examples.

0.11

Multiple backends per project

Now, dstack run leverages price data from multiple configured cloud providers and regions to automatically suggest the most cost-effective options.

To implement this change, we have modified the way projects are configured. You can now configure multiple clouds and regions within a single project.

Custom domains and HTTPS

Also, it is now possible to deploy services using HTTPS. All you need to do is configure a wildcard domain (e.g., *.mydomain.com), point it to the gateway IP address, and then pass the subdomain you want to use (e.g., myservice.mydomain.com) to the gateway property in YAML (instead of the gateway IP address).

Other changes

.dstack/profiles.yml

• The project property is no longer supported.
• You can now use max_price to set the maximum price per hour in dollars.

dstack run

• Using the dstack run command, you are now able to utilize options such as --gpu, --memory, --env, --max-price, and several other arguments to override the profile settings.

Lastly, the local backend is no longer supported. Now, you can run everything using only a cloud backend.

The documentation is updated to reflect the changes in the release.

Migration

The dstack version 0.11 update brings significant changes that break backward compatibility. If you used prior dstack versions, after updating to dstack==0.11, you'll need to log in to the UI and reconfigure clouds. We apologize for any inconvenience and aim to ensure future updates maintain backward compatibility.

0.10.8

This minor update addresses a crucial issue that was causing services to malfunction. If you're utilizing services, be sure to update.

Changelog: 0.10.7...0.10.8

0.10.7

Services

Until now, dstack has supported dev-environment and task as configuration types. With the 0.10.7 update, we introduce
service, a dedicated configuration type for serving.

Usage example:

type: service

gateway: ${{ secrets.GATEWAY_ADDRESS }}

image: ghcr.io/huggingface/text-generation-inference:0.9.3

port: 8000

commands: 
  - text-generation-launcher --hostname 0.0.0.0 --port 8000 --trust-remote-code

The gateway property represents the address of a special cloud instance that wraps the running service with a public endpoint.

Gateways

Before you can run a service, you have to configure a gateway.

First, you have to create a gateway in a project of your choice using the dstack gateway create command:

dstack gateway create

Once the gateway is up, the command will print its address. Go ahead and create a secret with this address.

dstack secrets add GATEWAY_ADDRESS <gateway address>

That's it! Now you can run your service using the dstack run command, which deploys the service and forwards the traffic to the gateway, thereby providing you with a public endpoint.

This initial support for services is the first step towards providing multi-cloud and cost-effective inference. In the near future, we plan to make it more functional and easier to use.

What's changed

• Support restarting local and GCP runs by @r4victor in #590
• Add API for listing all runs by @r4victor in #591
• The dstack init should handle invalid Git credentials by @peterschmidt85 in #594
• Support custom run names by @r4victor in #595
• dstack doesn't work if the repo contains a submodule with SSH URL by @peterschmidt85 in #598
• Always use cuda images for instances with GPU by @r4victor in #602
• Introduce gateways for services publication by @Egor-S in #596
• Small ports refactoring by @Egor-S in #603
• #588 Created All run list page. by @olgenn in #607
• Improve gateway security, show verbose errors by @Egor-S in #608
• Do not require sshd in task configuration for custom docker images by @Egor-S in #609
• #588 all runs list by @olgenn in #611
• [Bug]: Doesn't run a dev environment if code is not configured in PATH by @peterschmidt85 in #613
• Allow to create gateways in AWS and Azure by @Egor-S in #614
• [Bug]: dstack init doesn't work if .git/config or ~/.gitconfig doesn't have the user section by @peterschmidt85 in #617
• [Bug]: dstack config doesn't work if ~/.dstack doesn't exist by @peterschmidt85 in #618
• Migrate to Gen2 images for Azure, add A100 support by @Egor-S in #619
• Add time.Sleep() in /logsws handler by @r4victor in #623
• Services & gateway docs by @Egor-S in #620
• Replace localhost with gateway hostname in service logs by @Egor-S in #624

Changelog: 0.10.6...0.10.7

0.10.6

Port mapping

Any task that is running on dstack can expose ports. Here's an example:

type: task

ports:
  - 7860

commands:
  - pip install -r requirements.txt
  - gradio app.py

When you run it with dstack run, by default, dstack forwards the traffic from the specified port to the same port on your local machine.

With this update, you now have the option to override the local machine's port for traffic forwarding.

dstack run . -f serve.dstack.yml --port 3000:7860

This command forwards the traffic to port 3000 on your local machine.

If you specify a port on your local machine already taken by another process, dstack will notify you before provisioning cloud resources.

Max duration

Previously, when running a dev environment or task with dstack and forgetting about it, it would continue indefinitely. Now, you can use the max_duration property in .dstack/profiles.yml to set a maximum time for workloads.

Example:

profiles:
  - name: gcp-t4
    project: gcp
    resources:
      memory: 24GB
      gpu:
        name: T4
    max_duration: 2h

With this profile, dstack will automatically stop the workload after 2 hours.

If you don't specify max_duration, dstack defaults to 6h for dev environments and 72h for tasks.

To disable max duration, you can set it to off.

Imagine the amount of money your team can save with this minor configuration.

More supported GPUs

With the CUDA version updated to 11.8, dstack now supports additional GPU types, including NVIDIA T4 and NVIDIA L4. These GPUs are highly efficient for LLM development, offering excellent performance at low costs!

If you are using a custom Docker image, you can now utilize a CUDA version up to 12.2.

Last but not least, the K80 GPU is no longer supported.

Examples

Make sure to check the new page with examples.

The documentation is updated to reflect the changes in the release.

What's changed

• Elaborate error message on unmatched requirements for local backend by @r4victor in #563
• Handle no permissions for listing aws buckets by @r4victor in #566
• Support max_duration by @r4victor in #571
• Fix aws not using default region without default creds by @r4victor in #572
• Add --ports to dstack run by @Egor-S in #573
• Add T4 support for GCP by @Egor-S in #575
• Close #537 Added field extra_regions for aws backend by @olgenn in #577
• Reserve ports before creating instance by @Egor-S in #578
• Upgrade instance images by @Egor-S in #581
• Update CUDA version to 11.8 by @peterschmidt85 in #584
• Use identity ports mapping by default by @Egor-S in #587

Changelog: 0.10.5...0.10.6

0.10.5

Lambda

The Lambda Cloud integration has significantly improved with this release. We've added the possibility to create Lambda Cloud projects via the user interface.

All you need to do is provide your Lambda Cloud API key and specify an S3 bucket and AWS credentials for storing state and artifacts.
Check the docs for more details.

Custom Docker images

By default, dstack uses its own base Docker images to run dev environments and tasks. These base images come pre-configured with Python, Conda, and essential CUDA drivers. However, there may be times when you need additional dependencies that you don't want to install every time you run your dev environment or task.

To address this, dstack now allows specifying custom Docker images. Here's an example:

type: task

image: ghcr.io/huggingface/text-generation-inference:0.9

env:
  - MODEL_ID=tiiuae/falcon-7b

ports:
 - 3000

commands: 
  - text-generation-launcher --hostname 0.0.0.0 --port 3000 --trust-remote-code

Note
Dev environments require the Docker image to have openssh-server pre-installed.

What's Changed

• Refactor providers, add custom docker images support by @Egor-S in #544
• Use nvidia docker runtime for build if available by @Egor-S in #552
• Allow env variables interpolation by @Egor-S in #553
• Display docker pull progress by @Egor-S in #554
• Handle backend creds becoming invalid by @r4victor in #556
• Add PATH to .profile by @Egor-S in #557
• Close #529 Supported lambda backend type in UI by @olgenn in #558
• #529 Fixed labels by @olgenn in #560
• 529 fixes after review by @olgenn in #562

Changelog: 0.10.4...0.10.5

0.10.4

What's changed

• [Bug]: Provisioning on Lambda Cloud doesn't work #542 by @peterschmidt85 in #543
• [Bug]: Cannot create an AWS project #545 by @peterschmidt85 in #546

Changelog: 0.10.3...0.10.4

dstack 0.10.3: A preview of Lambda Cloud support

With the 0.10.3 update, dstack now allows provisioning infrastructure in Lambda Cloud while storing state and artifacts in an S3 bucket.

See the Reference for detailed instructions on how to configure a project that uses Lambda Cloud.

Note, there are a few limitations in the preview:

1. Since Lambda Cloud does not have its own object storage, dstack requires you to specify an S3 bucket, along with AWS credentials for storing state and artifacts.
2. At the moment, there is no possibility to create a Lambda project via the UI. Currently, you can only create a Lambda project through an
   API request.

In other news, we have pre-configured the base Docker image with the required Conda channel, enabling you to install additional CUDA tools like nvcc using conda install cuda. Note that you only need it for building a custom CUDA kernel; otherwise, the essential CUDA drivers are already pre-installed and not necessary.

The documentation and examples are updated to reflect the changes.

Give it a try and share feedback

Go ahead, and install the update, give it a spin, and share your feedback in our Slack community.

What's Changed

• Updated fields in run details page by @olgenn in #515
• Update gcp.md by @axitkhurana in #517
• Warn user about long git diff by @Egor-S in #518
• Close #523 Implemented building run status by @olgenn in #525
• Implement Lambda Labs backend by @r4victor in #528
• Restore cache functionality by @Egor-S in #527
• Runner storages refactoring by @Egor-S in #532
• Refactor backends by @r4victor in #533
• Add multi-region compute support for AWS by @r4victor in #536
• Check if the build exists before starting an instance by @Egor-S in #534
• Pre-configure the nvidia/label/cuda-11.4.3 channel for conda in the CUDA image by @peterschmidt85 in #541

New Contributors

• @axitkhurana made their first contribution in #517

Changelog: 0.10.2...0.10.3