Proposal: Flow services

[rfratto]

The most recent copy of this proposal can be found on Google docs. The below may be out of date, but is left in for posterity.

Background

In the context of Flow, a "service" is a low-level primitive which exposes additional behavior on top of the Flow controller. Today, there are three services that exist in pkg/flow:

• The telemetry service, allowing components to create metrics, logs, and traces for component observability.
• The HTTP service, allowing components to expose an HTTP endpoint on a shared server for things such as Prometheus exporters.
• The clustering service, allowing components to distribute work amongst themselves.
  • The clustering service depends on the HTTP service to function properly.

All of the services today are started prior to the Flow controller.

One of the remaining items for Flow feature parity is adding support for configuring TLS settings for the HTTP service at runtime. This would be added as a new config block into the Flow configuration file. This leads to a minimum set of functional requirements:

• For security reasons, plaintext HTTP traffic must only be accepted when a root config is loaded that does not configure TLS.
• If the TLS settings reference a component export (such as remote.vault), those components must be evaluated first to be able to evaluate the TLS settings properly.
• TLS settings must not have a cyclic dependency: it is not valid for the TLS settings to depend on a component which depends on the HTTP service to function.
  • For example, prometheus.exporter.unix depends on the HTTP service to create its targets.

These functional requirements extend to the clustering service, which must only start after the HTTP service is ready.

To allow services to be configured at runtime, those services (any any service which depends on them) must now exist in the DAG to ensure proper evaluation order; today, only the telemetry service exists in the DAG.

Additionally, there has been ongoing work to prepare Grafana Agent for a 1.0 release. This included some preliminary offline discussions about moving Flow and River to a separate repository so they may have their own versions separate from Grafana Agent.

This proposal aims to solve both problems at once:

1. Allow TLS settings to be configurable at runtime.
2. Decouple the agent-specific HTTP and clustering services from pkg/flow in preparation for Flow and River to be moved to a dedicated Git repository.

Goals

• Allow TLS settings to be configurable at runtime.
• Ensure that plaintext HTTP traffic is only accepted when TLS is not configured.
• Prevent cyclic dependencies between services and components.
• Allow services to depend on other services.
• Decouple the HTTP and clustering service from pkg/flow.
• Keep definition of a service to a minimum for initial implementation.

Non-goals

• Decouple telemetry APIs (metrics, logs, traces) from Flow.
  • Telemetry APIs are fundamental to all components, so it makes sense for telemetry to be the one thing always provided to Flow components, even after Flow is moved to a separate repository.
• Ensure that existing usages of the service are unchanged.
  • Usages of the clustering service will have to change following the decoupling.

Design

A new service package will be introduced which defines the API that services will have at their disposal:

package service // import "github.com/grafana/agent/service"

Package service defines a pluggable service for the Flow system.

Services are low-level constructs which run for the lifetime of the 
Flow controller, and are given higher levels of access to the system 
compared to components, such as the ability to request components
which depend on a service.

TYPES

type Definition struct {
	// Name uniquely defines a service.
	Name string

	// ConfigType is an optional config type to configure a 
	// service at runtime.
	//
	// If provided, ConfigType must be a struct type with River
	// tags for decoding as a config block.
	//
	// If ConfigType is non-nil, the Name of the service is used
	// as the block name in River to configure the service.
	ConfigType interface{}

	// DependsOn defines a set of dependencies for a
	// specific service by name.
	DependsOn []string
}
    Definition describes an individual Flow service. Services have 
    unique names and optional ConfigTypes where they can be configured 
    within a root Flow module.

type Host interface {
	// GetComponent gets a running component by ID.
	GetComponent(id component.ID, opts component.InfoOptions) (*component.Info, error)

	// ListComponent lists all running components.
	ListComponent(opts component.InfoOptions) []*component.Info

	// GetServiceConsumers gets the list of components and services 
	// which depend on a service by name. The returned values will be 
	// an instance of [component.Component] or [Service].
	GetServiceConsumers(serviceName string) []any
}
    The Host is responsible for running a given service.

type Service interface {
	// Definition returns the Definition of the Service. Definition 
	// must always return the same value across all calls.
	Definition() Definition

	// Run starts a Service. Run must block until the provided context
	// is canceled. Returning an error should be treated as a fatal
	// error for the Service.
	Run(ctx context.Context) error

	// Update updates a Service at runtime. Update is never called
	// if [Definition.ConfigType] is nil. newConfig will be the same
	// type as ConfigType; if ConfigType is a pointer to a type,
	// newConfig will be a pointer to the same type.
	//
	// Update will be called once before Run, and may be called
	// while Run is active. 
	Update(newConfig interface{}) error

	// Data returns the Data associated with a Service. Data must
	// always return the same value across multiple calls, as callers
	// are expected to be able to cache the result.
	//
	// Data may be invoked before Run.  
	Data() interface{}
}
    Service is an individual service to run.

The component package will be updated to be aware of services:

package component // import "github.com/grafana/agent/component"

type Registration struct {
	... 

	// NeedsServices holds the set of service names which this 
	// component depends on to run. 
	NeedsServices []string  
}

type Options struct {
	...

	// GetServiceData retrieves data for a service by key. 
	//
	// GetServiceData will return an error if the service does not 
	// exist or was not listed as a dependency with the registration 
	// of the component.
	//
	// The result of GetServiceData may be cached as the value will 
	// not change at runtime.  
	GetServiceData func(name string) (interface{}, error)   
}

The flow package will also be updated to be aware of services:

package flow // import "github.com/grafana/agent/pkg/flow"

type Options struct {
	...

	// List of Services to run with the Flow controller.
	// 
	// Services will start when the Flow controller runs. 
	// Services will be configured when LoadFile is invoked. 
	Services []service.Service
}
	Options holds static options for a Flow controller.

Notes

• The service.Host interface will be implemented by the Flow controller.
• service.Host.GetServiceConsumers is required for the HTTP interface to install HTTP handlers for the clustering service, and for an optimization, look up components which depend on a service.
  • This must unfortunately return any to avoid circular dependencies. The component, flow, and service packages should be merged in the future to avoid this, but this change is out of scope for this proposal.
• service.Service.Data can return anything which might be useful to components, such as information about the service in a struct (such as the HTTP listen address), or an interface so components can invoke methods (such as looking up the clustering owner for at token).
• There is no way for a service to request a reload for a component; the clustering service will have to change its interface so that re-sharding data happens outside of a call to component.Component.Update.

Details

The Flow controller will create DAG nodes for each service assigned to it. Edges from service nodes will be based on:

• Other services that each service depends on by their service.Definition.
• Services that components depend on by their component.Registration.

Having services in the DAG will ensure that there are no cyclic dependencies.

Services will only run once the Flow controller is run; Flow.LoadFile must be called before Flow.Run to make sure that the HTTP service does not run before TLS is configured.

Refer to the design above for other aspects of how the services API behaves.

Thanks

Thanks to @erikbaranowski for his help on this proposal.

[rfratto]

This proposal has been reviewed by all maintainers with consensus for implementing it.

Given there are no objections on the issue or on the doc, I'm marking this proposal as accepted.