Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix cli connection issue with TPU #1705

Merged
merged 1 commit into from
Sep 18, 2024

Conversation

Bihan
Copy link
Collaborator

@Bihan Bihan commented Sep 18, 2024

CLI was failing to connect during TPU provisioning with following error:

dstack apply -f examples/.dstack.yml
 Configuration          examples/.dstack.yml                          
 Project                main                                          
 User                   admin                                         
 Pool                   default-pool                                  
 Min resources          2..xCPU, 8GB.., 1xv5litepod-1, 100GB.. (disk) 
 Max price              -                                             
 Max duration           6h                                            
 Spot policy            auto                                          
 Retry policy           no                                            
 Creation policy        reuse-or-create                               
 Termination policy     destroy-after-idle                            
 Termination idle time  5m                                            

 #  BACKEND  REGION       INSTANCE     RESOURCES                      SPOT  PRICE   
 1  gcp      us-central1  v5litepod-1  1xv5litepod-1, 100.0GB (disk)  yes   $0.6    
 2  gcp      us-east5     v5litepod-1  1xv5litepod-1, 100.0GB (disk)  yes   $0.6    
 3  gcp      us-south1    v5litepod-1  1xv5litepod-1, 100.0GB (disk)  yes   $0.6    
    ...                                                                             
 Shown 3 of 12 offers, $1.56 max

Submit the run vscode? [y/n]: y
vscode provisioning completed (running)
Traceback (most recent call last):
  File "/Users/bihan/Projects/dstack-bihan/venv/bin/dstack", line 33, in <module>
    sys.exit(load_entry_point('dstack', 'console_scripts', 'dstack')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bihan/Projects/dstack-bihan/dstack/src/dstack/_internal/cli/main.py", line 75, in main
    args.func(args)
  File "/Users/bihan/Projects/dstack-bihan/dstack/src/dstack/_internal/cli/commands/apply.py", line 70, in _command
    configurator.apply_configuration(
  File "/Users/bihan/Projects/dstack-bihan/dstack/src/dstack/_internal/cli/services/configurators/run.py", line 147, in apply_configuration
    if run.attach(bind_address=bind_address):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bihan/Projects/dstack-bihan/dstack/src/dstack/api/_public/runs.py", line 293, in attach
    self._ssh_attach.attach()
  File "/Users/bihan/Projects/dstack-bihan/dstack/src/dstack/_internal/core/services/ssh/attach.py", line 144, in attach
    raise SSHError("Can't connect to the remote host")
dstack._internal.core.errors.SSHError: Can't connect to the remote host

Discussed Solutions:

  1. Increasing the ConnectTimeout value in SSH_DEFAULT_OPTIONS eg: 10 instead of 3.

  2. Disable SSH options by passing options={}.

Implemented Solution
Option 2 is recommended and implemented because it resembles the settings before Commit

Copy link
Collaborator

@jvstme jvstme left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@jvstme jvstme merged commit 277584d into dstackai:master Sep 18, 2024
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants