mosh does not work with ssh gateway

[axonasif]

Bug description

Being able to use mosh would be very helpful to overcome latency issues with SSH. I might be able to use it via local-companion but that's too much effort to put each time. Tailscale should also work (nope), but again, that will make the latency even worse.

Steps to reproduce

• Open a workspace
• Run sudo install-packages mosh so that we have mosh server on our workspace
• From gitpod dashboard, copy the ssh address for workspace
• Connect with mosh command from your PC
• See the following screen:

Workspace affected

No response

Expected behavior

No response

Example repository

No response

Anything else?

No response

[loujaybee]

Thanks @axonasif ! Not sure why this doesn't work, but it can be looked into, I dropped the bug tag for now, as not sure it was something we said we'd previously supported (though it would be nice to). Any details, you or others could find when debugging about why it's not connecting would be useful! 🙏

[iQQBot]

I'm afraid there's no way around this. mosh expects the ssh server to be on the public network, or at least the firewall to be able to directly penetrate the specified UDP port, but that's not possible in gitpod

[mrzarquon]

you can get around the UDP limitation using tailscale, but since we don't support tailscale ssh on the workspace, you end using a gitpod ssh gateway session to start the mosh-server but then connect the mosh-client via the tailscale IP / port / session token

[abitrolly]

@iQQBot why allowing UDP traffic Kubernets LoadBalancer policy won't work? Not sure what Gitpod is using, but I guess it is here https://github.com/gitpod-io/gitpod/blob/main/install/installer/pkg/common/networkpolicies.go

[iQQBot]

@iQQBot why allowing UDP traffic Kubernets LoadBalancer policy won't work? Not sure what Gitpod is using, but I guess it is here https://github.com/gitpod-io/gitpod/blob/main/install/installer/pkg/common/networkpolicies.go

Unfortunately, this doesn't work because mossh expects the server to have a fixed port for authentication

[abitrolly]

@iQQBot mosh allows to specify the port, so I guess it can be fixed. So what it should do?

[abitrolly]

@iQQBot wdyt?

[iQQBot]

mosh allows to specify the port, so I guess it can be fixed. So what it should do?

Unfortunately we can't do this, there is not only one workspace in a node, so the public IP and workspace are not one-to-one, and there is no way to solve the mapping problem by opening the port

Unless we use UDP hole-punching, but this requires modifying the mosh source code

[axonasif]

@iQQBot we appreciate you sharing all the technical/cost related issues that this may have. 🙏

But as an user who's looking to default to SSH for Gitpod usage, this is a big blocker. Having this ability would make SSH more sane to use with Gitpod. ssh-gateway is such an amazing capability Gitpod got that unlocks many possibilities, now only if mosh worked with it, then nothing like it.

Or maybe in the future if we have a client-side gp CLI with features of local-companion but with UDP port forwarding support. That'd work too.

[abitrolly]

@iQQBot I still don't understand. How does SSH attach to the specified workspace then, if there is no IP?

[iQQBot]

@iQQBot I still don't understand. How does SSH attach to the specified workspace then, if there is no IP?

We have a component called ssh gateway, which you can simply think of as nginx, that resolves usernames and credentials and routes them to the correct workspace, so there is no public IP for each workspace

[axonasif]

I assume #459 is the main issue?

[abitrolly]

We have a component called ssh gateway, which you can simply think of as nginx, that resolves usernames and credentials and routes them to the correct workspace, so there is no public IP for each workspace

Found this diagram here.

If I have multiple workspaces open, how SSH Gateway knows which one to forward to? IP is the same, user name is the same, SSH key is the same.

[andreafalzetti]

@abitrolly each workspace has a different workspace ID, https://<WORKSPACE_ID>.<CLUSTER>.gitpod.io/ the SSH connection cmd is ssh '<WORKSPACE_ID>@<WORKSPACE_ID>.ssh.<CLUSTER>.gitpod.io'

[abitrolly]

@andreafalzetti mosh uses user@ too at least for initial connection.

Usage: /usr/bin/mosh [options] [--] [user@]host [command...]
...

So to make traffic pass LoadBalancer should allow UDP packets to travel to SSH Gateway, and then the gateway needs to figure out from which user the packet comes from. I am not sure how it is done for SSH. SSH Gateway then should be running SSH server to terminate secure connection, but then traffic to workspace will be unencrypted. How does that work?

[iQQBot]

SSH gateway no to terminate secure connection, it will start a new ssh connect with workspace using different private key

for mosh, it's not, the username only use for ssh, this is in order to start mosh-server in server, after mosh-server is start, it will disconnect ssh connection, and direct using mosh-server, here, they negotiate another set of aes keys and then communicate directly with that set of keys; simply put, mosh uses ssh for

1. start mosh-server
2. get the IP address of the server
3. negotiate the communication key

[abitrolly]

I mean SSH Gateway is kind of https://en.wikipedia.org/wiki/TLS_termination_proxy It terminates SSH connection - meaning it doesn't forward it, and initiates new SSH connection instead.

So for mosh the SSH Gateway can not terminate mosh connection, because that would mean mosh-server would run on the gateway and will forward events from gateway's terminal. So there should be some kind of proxy for mosh, which will forward the UDP traffic depending on communication key the client is using.

Can SSH gateway already read the communication key from mosh negotiation step?

[abitrolly]

There is a related issue in mosh mobile-shell/mosh#970