Remote worker registry and cloud service discovery

[hairyhum]

Background

In order to send messages between workers they should know routes to each other. These routes may be known in advance or retrieved from messages. Addresses in these routes may be static or dynamic.

In simple setup workers are accessible via static routes which are known in advance, like in this transport example

In more complex cases some remote workers may be created dynamically without a static route leading there. In that case the calling worker needs to find out this remote route before sending a message, like in forwarding example

Also during development and integration, even the static routes to remote workers may not be known in advance. For example if you're developing an application which uses some cloud service worker, you would need to know a route to this worker.
Usually cloud service workers have static addresses, but depending on the deployment could be complex.

Worker registry

In order to simplify discovery, we can introduce a "registry" worker with well known address which can store routes to workers and provide them to other workers by request.

This worker could also have ability to add routes to the storage (register workers).

Registry worker would also contain some sort of identifiers for registered workers to have some meaningful distinction between them.
Registry might also contain some additional information about remote workers, some of which can be used to access them (e.g. public keys)

In general the registry worker API include operations like:

• register(id, info) : ok
• list() : [{id, info}]
• get(id) : info

Where info contains route and potentially some additional metadata

Worker accessibility routes relativity

Important to note that routes registered in the registry are only accessible from the registry worker context, in order to access them from "local" workers (which get "remote" worker routes from the registry), they should construct a "full route" to the "remote" worker.

For example if registry is accessible via [TCP#registry_node, 0#"registry"] and remote route registered is [TCP#remote, 0#"service"], then the route to "service" worker that local worker can use should be: [TCP#registry_node, TCP#remote, 0#service], GIVEN THAT tcp address TCP#remote is accessible from tcp handler of TCP#registry_node.

Relying on TCP handlers being able to access registered routes is not recommended but it's currently how things work.

Alternatively explicit channels can be used. A local worker would set up a channel which would have access to registry worker and registered workers:

In that case registry is accessible via [channel, 0#"registry"] and "full route" would be [channel, TCP#remote, 0#"service"]

We can call the first fraction of the route ([TCP#registry_node] or [channel]) access_route or bridge_route and the second fraction ([0#registry] or [TCP#remote, 0#"service"]) target_route

Cloud services discovery

On Ockam Cloud Nodes services are workers (or groups of workers) which perform certain functions and accessible to remote workers. Those workers can serve as integration with other cloud services (for example Apache Kafka)

Each service has an access point worker which implement some message API. To get the route to this access point from the perspective of the cloud node we can use a registry worker which will register services when they're started and be able to retrieve their routes on demand.
We can call this registry worker "Service discovery worker".

For simplicity since services currently don't provide cryptographic identity we can use a simple "name id".

Routes relativity concert applies to service discovery same as for any remote workers registry.

Service discovery implementation and API

Discovery service API is request-response and identifies requests by routes (meaning that there is no request IDs)

Request to the service is sending and empty binary.
NOTE: not an empty string, but data<0> in BARE terms

Response contains a list of routes encoded as following BARE type:

type Response []ServiceInfo
type ServiceInfo {
  name: string
  route: Route
}
type Route []Address
type Address {
 type: uint,
 value: data
}

Name ID means that there cannot be multiple services with the same name

Service info might be extended in the future.

Future extension of this API can have different request formats corresponding to different response formats.

Discovery VS remote aliases

Service discovery serves two purposes:

• listing available services
• getting routes to them

Second issue can be addressed also by remote static aliases.
For example we can create a static alias worker [0#"stream"] which will forward all messages to the stream service access point. This allows client applications to communicate with this service by [TCP#cloud, 0#"stream"] route without additional discovery step.

This simplifies the client application code since it doesn't need to contain discovery step, but complicates route negotiation because alias worker would not be in the return route and routes in the further communication will not include the alias.
This may be fine for some communication, but workers need to be ready to handle that

[thomcc]

We have to solve this anyway, but my only real concern here is that being able to get a list of available routes seems... very useful to an attacker (e.g. a hostile intermediary that's impersonating a node), especially given that they'd be able to forge whatever routes they need (and thus convince legitimate nodes to send messages they want along return routes and such).

[thomcc]

IOW some of our current issues are mitigated by the fact that it is difficult for an attacker to determine the addresses/routes to certain workers, which this makes much easier.

That said, we need to solve this regardless.

[hairyhum]

The idea here is to have workers which are supposed to be easily discovered. This doesn't make arbitrary workers to be easily discoverable though.

I left the access control out in this proposal, but I think it makes sense to address it:

• access_route would have some access control (e.g. authorization) implemented in there. This could be a secure channel, tls connection or something else. Access route would allow workers to access certain remote workers or not.
• Registry/discovery service could also have some sort of access control, for example only listing certain routes to authorized requesters.

I don't have a definitive answer which one is better, but to me the access route approach seems to be reasonably composable to start with and then move to more or different access control.
I think @SanjoDeundiak already started laying some ground work regarding access control within a node which I hope to use.

[thomcc]

Thanks for the clarification, that helps. I agree that access route sounds like the better place to start (it also doesn't imply folks should rely on workers being non-discoverable, which they shouldn't, as the onward/return routes are not encrypted and so an attacker will at least know that much).

[spacekookie]

This is a pretty exciting proposal in terms of what it'll allow us to do in the future. It feels like a weird cross-section between authentication and DNS.

For example if registry is accessible via [TCP#registry_node, 0#"registry"] and remote route registered is [TCP#remote, 0#"service"], then the route to "service" worker that local worker can use should be: [TCP#registry_node, TCP#remote, 0#service], GIVEN THAT tcp address TCP#remote is accessible from tcp handler of TCP#registry_node.

Relying on TCP handlers being able to access registered routes is not recommended but it's currently how things work.

Something that will have to factor into this is the way we implement access control mechanisms. I agree that relying on the implementation detail that any TCP handler would be accessible to the registered routes in not great. I think one way to approach this is to build a handshake mechanism with whatever system handles the access control to certain workers.

So for example, if [TCP#remote, 0#service] should only be accessible after some kind of authentication mechanism then the service registry should be aware of this and instead of handing out a route to "service" instead initiate a handshake with the initiator to build/ resolve the final route.

---

One thing I'd like to clarify is the relationship between the worker registry and the service discovery worker. From my understanding of this proposal:

• Worker registry provides a lookup table to routes via well-known addresses
• Service discovery worker is started for a particular service which then registers the set of service addresses with the worker registry.

Is that understanding accurate?

[hairyhum]

Regarding access control and session handshake:
I'm more in favour of doing the "data approach" as providing route and some metadata rather than doing some handshake on discovery. Because discovery might be done from a different context from using the discovered data: this could be a different time, different code or even different device.
It's totally reasonable to imagine an API which would say something like "make a channel all the way to service X" in the "remote aliases" approach, but it doesn't solve the discovery problem, you have to know that "service X" exists.

Service discovery is a special case of worker registry, just for the cloud services. Services are accessed as workers.
Registry is more of a concept, while discovery service is an implementation of this concept.

We might consider calling all registry workers "discovery workers"

[hairyhum]

Remote nodes access

As mentioned in the "Worker accessibility routes relativity" section above, worker routes are relative to the discovery service and all workers would be accessed through the access route and through the discovery service node.

This also means that there would be some access channels from the discovery node to the registered worker node.

This access channels would be reused for all "clients" using the registered worker and might create a bottleneck depending on how they're configured. This is just something to keep in mind when designing connections between the nodes.

[hairyhum]

Alternatively there should be some sort of "reference route" sections which would instruct clients to use some other access channels to access remote nodes.
These route sections would serve as a reference to an actual route known to the client node.