Service dependencies

[achingbrain]

We need a way for services to express their dependencies (e.g. KAD-DHT needs Identify to function properly).

I tried doing this through types but it didn't really work, plus it doesn't help anyone writing vanilla js.

@maschad tried doing this through config validation but that has it's own problems - mandated service names, libp2p needing to know about which service depends on what others, etc.

How about each service could have a string identifier, and each service could return a list of other services it requires, then on startup libp2p could verify that everything is present and correct?

// in @libp2p/interface
export const serviceName = Symbol.for('@libp2p/service-name')
export const serviceDependencies = Symbol.for('@libp2p/service-dependencies')

// in @foobar/my-service
import { serviceName, serviceDependencies } from '@libp2p/interface' 

/**
 *  A service class, requires Identify to function
 */
class MyService {
  get [serviceName] (): string {
    return '@foobar/my-service'
  }

  get [serviceDependencies] (): string[] {
    return [
      '@libp2p/identify'
    ]
  }
}

It would be tempting to export the service name, but that would mean (for example) KAD-DHT would need to depend (npm-style) on the Identify implementation which currently lives in libp2p (for now anyway) so we'd probably start off with duplicated strings.

We could even have some sort of versioning, read out of the package.json - my-service@1.0.0, this might help with working out which module is compatible with which version of libp2p, something that's a continual pain point for people.

[wemeetagain]

Some questions / thoughts:

Do service implementations really depend on other service implementations?
Or are there certain capabilities / interfaces / protocols / behaviors that are independent of specific implementations that are depended on?

Is this a problem that actually needs to be solved? It seems like identify is a special case, and this can be solved by better documentation in services that require specific configurations.

[achingbrain]

Do service implementations really depend on other service implementations?

Ideally in a modular network stack the protocols would be modular too but there are interdependencies, yes. For example anything that registers a topology needs Identify, the WebRTC transport needs some other transport to do the SDP handshake over (effectively the Circuit Relay transport), etc.

this can be solved by better documentation

We're assuming people will actually read the documentation 😩 - an error message on startup will help them "fail their way to success" or whatever the pithy term for this is, and also means we won't need to endlessly link people to the docs.

[SgtPooki]

TLDR for below thoughts:

There are other patterns out there that are related to the problem we're trying to solve, but it would be a much more significant haul to implement & maintain them. Let's start with the simple approach and see how it goes. If we want to move to something more complex later, we should ensure the work here doesn't make that harder.

---

Quote & Response

1.

How about each service could have a string identifier, and each service could return a list of other services it requires, then on startup libp2p could verify that everything is present and correct?

I think this is a fairly straightforward and simple approach that would work for the stated problem. It reminds me a lot of older AngularJS Dependency Injection (DI) and AMD (requirejs) type approach to modules.

libp2p logic for this would just be something like

import { serviceName, serviceDependencies } from '@libp2p/interface' 

/**
 * Ensure that every service's `serviceDependencies` in the array of services is present in the array of services.
 */
function checkLibp2pServiceDepAlignment(services) {
    const serviceNamesSet = new Set(...services.map(service => service[serviceName]))
    services.forEach(service => {
        if (!service[serviceDependencies].every(dependency => serviceNames.has(dependency))) {
            throw new Error(`Service ${service.serviceName} could not find all of its dependencies in the provided services array.`)
        }
    })
    // TODO: catch circular deps
    // TODO: catch duplicate services?
    // TODO: print out a nice error message including ALL errors if ANY (missing/circ/duplicate)
}

Can we force the serviceName to be the same as the npm package name? This could make it harder to forget to set serviceName on services but would require some decorator/parent-class/similar. If not, we should start requiring any Object.values(services) to have these keys set.

2.

We could even have some sort of versioning, read out of the package.json - my-service@1.0.0, this might help with working out which module is compatible with which version of libp2p, something that's a continual pain point for people.

I like this idea but I feel like versions in the service key will ultimately result in two "version not compatible" error messages: compiletime + runtime, and be very troublesome. If we could guarantee the serviceDependencies missing errors cover at least the full Set of npm version mismatch errors (forever), then I think that's an easy YES from me; but I don't think it's that simple. I vote we should continue to use npm versioning for this.

---

additional brain-dump:

• This doesn't seem like a full DI problem, more like a "Dependency Awareness" or "Service Registry" + half a "Service Discovery" problem, but DI may help. DI in JS/TS is not common so we would want to keep it light for js-libp2p to remain approachable for devs. Would DI make using/extending/maintaining js-libp2p easier/harder?
  • DI codifies dependencies (half of the awareness problem) but is a big problem to solve (and DI patterns aren't idiomatic in JS)
    • DI would make solving the "Error: X default service" problem easier on maintainers.
      • devs WILL run into this when modifying only some services and not knowing what services will pop up.
  • Has anyone working on js-libp2p used a DI library they would recommend?
    • a quick search resulted in me discovering an older one (new to me) from MicroSoft: https://github.com/microsoft/tsyringe (how big? how reliable, etc?)
    • https://github.com/mgechev/injection-js seems decent and i've looked into it before, but 5.2k seems like a lot for this issue.
    • Learnings with DI patterns should be easy to find by looking at AngularJS history. I can't seem to find a good docs link for old patterns, but https://github.com/mgechev/injection-js is related. Their new patterns can be glimpsed in https://angular.io/guide/creating-injectable-service
  • Do we want to write our own DI functionality inside of libp2p? DI is straightforward, but if you look at Angular's DI path, or any other discussion about IOC/DI, DI not something that should live inside libp2p as application/library logic IMHO
• Do we need a Service Locator (SL)? This wasn't brought up by anyone else, but it's related and I want to just callout that we should not do this.
  • https://blog.ploeh.dk/2010/02/03/ServiceLocatorisanAnti-Pattern/ is a really good example that covers our problem somewhat and advises against the Service Locator pattern.
• Service Registry Pattern?

[SgtPooki]

A question popped up while I was re-reading @wemeetagain's question & @achingbrain's answer. Particularly this one:

Or are there certain capabilities / interfaces / protocols / behaviors that are independent of specific implementations that are depended on?

WebRTC transport needs some other transport to do the SDP handshake over (effectively the Circuit Relay transport), etc.

Should we make this more broadly scoped than just to services? I imagine there are (or will be) M parts of libp2p that could depend N-Z types of other parts.

  get [libp2pComponent.id] (): string {
    return `${libp2pComponent.type.SERVICE}_@foobar/my-service`
	// or `${libp2pComponent.type.TRANSPORT}_@wemeetagain/http4`
	// or `${libp2pComponent.type.OTHER}_@achingbrain/newcoolthing`
  }

The main concern I have after bringing this up is the naming of serviceName and serviceDependencies locking us into the idea that this is only solving service dependency validation, instead of a broader any-libp2p-part cross-dependency validation standard.

[achingbrain]

Perhaps something like this?

import { capabilities, capabilityRequirements } from '@libp2p/interface'

/**
 *  A service class, requires Identify to function
 */
class MyService {
  // provides pubsub and peer discovery
  get [capabilities] (): string[] {
    return [
      '@libp2p/pubsub',
      '@libp2p/peer-discovery'
    ]
  }

  // needs identify
  get [capabilityRequirements] (): string[] {
    return [
      '@libp2p/identify'
    ]
  }
}

[marcus-pousette]

I am in need for this feature, but rather just being a type help (like the identify + kad or identity + circuit relay example). I want to access another service functionality from a service. What I have been thinking about is that having services in a map fundamentally makes it hard to build this cross dependencies, for the same reason it is hard to force a field with a type in a class to be present if another field with a specific type is present. IMO (service) maps are good for storing data that have not dependencies on each other, and one can/should be able to put/remove any value by its key without trouble.

My dream scenario (I think) would be something much more simple

const libp2p: Libp2p = await createLibp2p({ /*  transports, connnection encryption, peerId  */ }) 
const withIdentify: Libp2p & { services: { identify: IdentifyService } }  = identifyService(libp2p) 
const withKad: Libp2p & {services: {  identify: IdentifyService, kad: DHT } }  = kadDHT(withIdentify) 

// another service that depends on kad dht
const somethingThatDependsOnKad: Libp2p & { services: { identify: IdentifyService, kad: DHT, anotherService: AnotherService} }  = anotherService(withKad) 

await libp2p.start()

or just a builder pattern (this might be even better)

const libp2p: Libp2p & { services: {  identify: IdentifyService, kad: DHT}  }
    = await createLibp2p()
   .withTransport(tcp)
   .withService(identifyService) 
   .withService(kadDHT)
   .build().start()

Now inside kadDHT constructor I would be able to get access to the IdentifyService because I know it has been constructed before kadDHT() is invoked.

This would allow dependencies to both be validated with types but also during runtime as a service can check whether another service is present in its constructor.

[achingbrain]

Out of interest why do you need access to the identify service? If it's to run identify on a connection manually it's a delicate one because by the protocol spec prevents it being run in parallel - it runs by default on connection opening so if you're not careful you can introduce hard-to-debug failures.

[marcus-pousette]

I don't need access to the identify service. I have made two custom services, and would want one to get access of functionality that exists in the other. (More specifically the service I want to get access to has some routing information I want to use in the other service)

[achingbrain]

You can do this already, if your services are Startable and you defer access until the start method:

class Service1 {
  constructor (components) {
    this.components = components
  }

  start () {
    console.info('components.service2', this.components.service2)
  }

  stop () {}
}

class Service2 {
  constructor (components) {
    this.components = components
  }

  start () {
    console.info('components.service1', this.components.service1)    
  }

  stop () { }
}

const libp2p = await createLibp2p({
  //... other config
  services: {
    service1: (components) => new Service1(components),
    service2: (components) => new Service2(components)
  }
})

[marcus-pousette]

Oh did not know that. Thanks, thats really helpful!

[maschad]

I created an issue to track @achingbrain 's suggestion #2263