FireFly is a multiparty system for enterprise data flows, powered by blockchain. It solves all of the layers of complexity that sit between the low level blockchain and high level business processes and user interfaces. FireFly enables developers to build blockchain apps for enterprise radically faster by allowing them to focus on business logic instead of infrastructure.
FireFly focusses on:
- Providing a great developer API and experience, with a CLI and UI as first class project components
- Pluggability for implementations of multi-party system infrastructure (blockchains, off-chain data exchange, identity, compute etc.)
- Making proven multi-party system patterns easy for new projects to adopt
- Providing developer friendly access to custom transactions+events in the underlying blockchain platforms
- Giving visibility and control on the private data exchange that occurs between businesses in a multi-party system
- Simplifying the journey of building multi-party business processes, by empowering non-blockchain developers to build great APIs+UX
You will see enterprise focussed code in FireFly solving hard "plumbing" problems like on-chain/off-chain event sequencing and aggregation, and enough smart contract code to make the patterns possible. You will then find patterns of integration with the individual communities that are already building the deep blockchain & multi-party compute tech, like Hyperledger Fabric, Hyperledger Besu, Quorum, Corda, IPFS, Hyperledger Avalon, OpenZeppelin, NodeRED etc.
Watch this space for patterns on integrating Tokens into the model (fungible token value exchange, and NFTs), which is a big current focus of evolution in the gen2 FireFly architecture (building on the work done in gen1, also in this repo). The tokens working group is being lead by Jim Zhang
https://labs.hyperledger.org/firefly/
FireFly has a plugin based architecture design, with a microservice runtime footprint. As such there are a number of repos, and the list will grow as the community evolves.
But not to worry, one of those repos is a CLI designed to get you running with all the components you need in minutes!
- CLI / Developer experience - https://github.com/hyperledger-labs/firefly-cli
- UI Explorer - https://github.com/hyperledger-labs/firefly-ui
- Sample applications - https://github.com/hyperledger-labs/firefly-samples
- Core (this repo) - https://github.com/hyperledger-labs/firefly
- HTTP Data Exchange - https://github.com/hyperledger-labs/firefly-dataexchange-https
- Ethereum (Hyperledger Besu / Quorum) connector: https://github.com/hyperledger-labs/firefly-ethconnect
- Corda connector: https://github.com/hyperledger-labs/firefly-cordaconnect - contributed from Kaleido generation 1 - porting to generation 2
- Hyperledger Fabric connector - in design phase, including collaboration with https://github.com/hyperledger-labs/fabric-smart-client
Note only the projects that are primarily built to support FireFly are listed here, not all of the ecosystem of projects that integrate underneath the plugins. See below for more information on the landscape of plugins and components.
Use the FireFly CLI for fast bootstrap: https://github.com/hyperledger-labs/firefly-cli
There are two core codebases currently active in this repo:
Directories:
- internal: The core Golang implementation code
- pkg: Interfaces intended for external project use
- cmd: The command line entry point
- solidity_firefly: Ethereum/Solidity smart contract code
This latest generation is re-engineered from the ground up to improve developer experience, runtime performance, and extensibility.
This means a simplified REST/WebSocket programming model for app development, and a wider range of infrastructure options for deployment.
It also means a focus on an architecture and code structure for a vibrant open source community.
A few highlights:
- Golang codebase
- Strong coding standards, including unit test coverage, translation support, logging and more
- Fast starting, low memory footprint, multi-threaded runtime
- OpenAPI 3.0 API specification (Swagger)
- Generated from the API router code, to avoid divergence with the implementation
- Active/active HA architecture for the core runtime
- Deferring to the core database for state high availability
- Exploiting leader election where required
- Fully pluggable architecture
- Everything from Database through to Blockchain, and Compute
- Golang plugin infrastructure to decouple the core code from the implementation
- Remote Agent model to decouple code languages, and HA designs
- Updated API resource model
Asset
,Data
,Message
,Event
,Topic
,Transaction
- Added flexibility, with simplified the developer experience:
- Versioning of data definitions
- Introducing a first class
Context
construct link related events into a single sequence - Allow many pieces of data to be attached to a single message, and be automatically re-assembled on arrival
- Clearer separation of concerns between the FireFly DB and the Application DB
- Better search, filter and query support
Directories:
- kat: The core TypeScript runtime
- solidity_kat: Ethereum/Solidity smart contract code
- cordapp_kat: The Corda smart contract (CorDapp)
This was the original implementation of the multi-party systems API by Kaleido, and is already deployed in a number production projects.
The codebase distilled years of learning, into a set of patterns for performing blockchain orchestrated data exchange.
It depends on the following Kaleido services:
- Blockchain nodes
- Ethereum with the Kaleido Kaleido REST API Gateway
- Corda with the Kaleido built-in API for streaming KAT transactions
- Kaleido Event Streams
- Kaleido App2App Messaging
- Kaleido Document Exchange
ββββββββββββ βββββββββββββββββ
β cmd ββββ€ firefly [Ff]β - CLI entry point
ββββββββββββ β β - Creates parent context
β β - Signal handling
βββββββ¬ββββββββββ
β
ββββββββββββ βββββββ΄ββββββββββ - HTTP listener (Gorilla mux)
β internal ββββ€ api [As]β * TLS (SSL), CORS configuration etc.
ββββββββββββ β server β * WS upgrade on same port
β β - REST route definitions
βββββββ¬ββββββββββ * Simple routing logic only, all processing deferred to orchestrator
β
βββββββ΄ββββββββββ - REST route definition framework
β openapi [Oa]β * Standardizes Body, Path, Query, Filter semantics
β spec | - OpenAPI 3.0 (Swagger) generation
βββββββ¬ββββββββββ * Including Swagger. UI
β
βββββββ΄ββββββββββ - WebSocket server
β [Ws]β * Developer friendly JSON based protocol business app development
β websockets β * Reliable sequenced delivery
βββββββ¬ββββββββββ * _Event interface [Ei] supports lower level integration with other compute frameworks/transports_
β
βββββββ΄ββββββββββ - Core data types
β fftypes [Ft]β * Used for API and Serialization
β β * APIs can mask fields on input via router definition
βββββββ¬ββββββββββ
β
βββββββ΄ββββββββββ - Core runtime server. Initializes and owns instances of:
β [Or]β * Components: Implement features
βββββββββ¬ββββ€ orchestrator β * Plugins: Pluggable infrastructure services
β β β β - Exposes actions to router
β β βββββββββββββββββ * Processing starts here for all API calls
β β
β Components: Components do the heavy lifting within the engine
β β
β β βββββββββββββββββ - Maintains a view of the entire network
β βββββ€ network [Nm]β * Integrates with network permissioning [NP] plugin
β β β map β * Integrates with broadcast plugin
β β βββββββββββββββββ * Handles hierarchy of member identity, node identity and signing identity
β β
β β βββββββββββββββββ - Broadcast of data to all parties in the network
β βββββ€ broadcast [Bm]β * Implements dispatcher for batch component
β β β manager | * Integrates with public storage interface [Ps] plugin
β β βββββββββββββββββ * Integrates with blockchain interface [Bi] plugin
β β
β β βββββββββββββββββ - Send private data to individual parties in the network
β βββββ€ private [Pm]β * Implements dispatcher for batch component
β β β messaging | * Integrates with the data exchange [Dx] plugin
β β ββββββββ¬βββββββββ * Messages can be pinned and sequenced via the blockchain, or just sent
β β β
β β ββββββββ΄βββββββββ - Groups of parties, with isolated data and/or blockchains
β β β group [Gm]β * Integrates with data exchange [Dx] plugin
β β β manager β * Integrates with blockchain interface [Bi] plugin
β β βββββββββββββββββ
β β
β β βββββββββββββββββ - Private data management and validation
β βββββ€ data [Dm]β * Implements dispatcher for batch component
β β β manager β * Integrates with data exchange [Dx] plugin
β β ββββββββ¬βββββββββ * Integrates with blockchain interface [Bi] plugin
β β β
β β ββββββββ΄βββββββββ - JSON data shema management and validation (architecture extensible to XML and more)
β β β json [Jv]β * JSON Schema validation logic for outbound and inbound messages
β β β validator β * Schema propagatation
β β ββββββββ¬βββββββββ * Integrates with broadcast plugin
β β β
β β ββββββββ΄βββββββββ - Binary data addressable via ID or Hash
β β β blobstore [Bs]β * Integrates with data exchange [Dx] plugin
β β β β * Hashes data, and maintains mapping to payload references in blob storage
β β βββββββββββββββββ * Integrates with blockchain interface [Bi] plugin
β β
β β βββββββββββββββββ - Private data management and validation
β βββββ€ event [Em]β * Implements dispatcher for batch component
β β β manager β * Integrates with data exchange [Dx] plugin
β β ββββββββ¬βββββββββ * Integrates with blockchain interface [Bi] plugin
β β β
β β ββββββββ΄βββββββββ - Handles incoming external data
β β β [Ag]β * Integrates with data exchange [Dx] plugin
β β β aggregator β * Integrates with public storage interface [Ps] plugin
β β β β * Integrates with blockchain interface [Bi] plugin
β β β β - Ensures valid events are dispatched only once all data is available
β β ββββββββ¬βββββββββ * Context aware, to prevent block-the-world scenarios
β β β
β β ββββββββ΄βββββββββ - Subscription manager
β β β [Sm]β * Creation and management of subscriptions
β β β subscription β * Creation and management of subscriptions
β β β manager β * Message to Event matching logic
β β ββββββββ¬βββββββββ
β β β
β β ββββββββ΄βββββββββ - Manages delivery of events to connected applications
β β β event [Ed]β * Integrates with data exchange [Dx] plugin
β β β dispatcher β * Integrates with blockchain interface [Bi] plugin
β β βββββββββββββββββ
β β
β β βββββββββββββββββ - Token operations
β βββββ€ asset [Am]β * NFT coupling with contexts
β β β manager β * Transfer coupling with data describing payment reason
β β β β - ...
β β βββββββββββββββββ
β β
β β βββββββββββββββββ
β βββββ€ sync / [Sa] β - Sync/Async Bridge
β β β async bridge β * Provides synchronous request/reply APIs
β β β β * Translates to underlying event-driven API
β β βββββββββββββββββ
β β
β β βββββββββββββββββ - Aggregates messages and data, with rolled up hashes for pinning
β βββββ€ batch [Ba]β * Pluggable dispatchers
β β β manager β - Database decoupled from main-line API processing
β β β β * See architecture diagrams for more info on active/active sequencing
β β ββββββββ¬βββββββββ - Manages creation of batch processor instances
β β β
β β ββββββββ΄βββββββββ - Short lived agent spun up to assemble batches on demand
β β β batch [Bp]β * Coupled to an author+type of messages
β β β processor β - Builds batches of 100s messages for efficient pinning
β β β β * Aggregates messages and data, with rolled up hashes for pinning
β β βββββββββββββββββ - Shuts down automatically after a configurable inactivity period
β ... more TBD
β
Plugins: Each plugin comprises a Go shim, plus a remote agent microservice runtime (if required)
β
β βββββββββββββββββ - Blockchain Interface
βββββββββββββ€ [Bi]β * Transaction submission - including signing key management
β β blockchain β * Event listening
β β interface β * Standardized operations, and custom on-chain coupling
β βββββββ¬ββββββββββ
β β
β βββββββββββββββββββββββ¬ββββββββββββββββββββ
β βββββββ΄ββββββββββ βββββββββ΄ββββββββ βββββββββ΄βββββββββ
β β ethereum β β corda β β fabric β
β βββββββββββββββββ βββββββββββββββββ ββββββββββββββββββ
β
β βββββββββββββββββ - P2P Content Addresssed Filesystem
βββββββββββββ€ public [Pi]β * Payload upload / download
β β storage β * Payload reference management
β β interface β
β βββββββ¬ββββββββββ
β β
β ββββββββββ ... extensible to any shared storage sytem, accessible to all members
β βββββββ΄ββββββββββ
β β ipfs β
β βββββββββββββββββ
β
β βββββββββββββββββ - Private Data Exchange
βββββββββββββ€ data [Dx]β * Blob storage
β β exchange β * Private secure messaging
β βββββββ¬ββββββββββ * Secure file transfer
β β
β βββββββββββββββββββββββ¬ββββββββββ ... extensible to any private data exchange tech
β βββββββ΄ββββββββββ βββββββββ΄ββββββββ
β β httpdirect β β kaleido β
β βββββββββββββββββ βββββββββββββββββ
β
β βββββββββββββββββ - Identity resolution and mapping
βββββββββββββ€ identity [Ii]β * Resolves opaque identifiers used throughout FireFly
β β interface β * Maps to and from blockchain signing identities
β βββββββ¬ββββββββββ * Map API/user identities from authentication, to network/organizational identities
β β
β ββββββββββββββ ... extensible to DIDs etc.
β βββββββ΄ββββββββββ
β β onchain β
β βββββββββββββββββ
β
β βββββββββββββββββ - API Authentication and Authorization Interface
βββββββββββββ€ api auth [Aa]β * Authenticates security credentials (OpenID Connect id token JWTs etc.)
β β β * Extracts API/user identity (for identity interface to map)
β βββββββ¬ββββββββββ * Enforcement point for fine grained API access control
β β
β βββββββββββββββββββββββ¬ββββββββββ ... extensible other single sign-on technologies
β βββββββ΄ββββββββββ βββββββββ΄ββββββββ
β β apikey β β jwt β
β βββββββββββββββββ βββββββββββββββββ
β
β βββββββββββββββββ - Database Interactions
βββββββββββββ€ database [Di]β * Create, Read, Update, Delete (CRUD) actions
β β interace β * Filtering and update definition interace
β βββββββ¬ββββββββββ * Migrations and Indexes
β β
β ββββββββββ ... extensible to NoSQL (CouchDB / MongoDB etc.)
β βββββββ΄ββββββββββ
β β sqlcommon β
β βββββββ¬ββββββββββ
β βββββββββββββββββββββββββ¬βββββββββ ... extensible other SQL databases
β βββββββ΄ββββββββββ βββββββββ΄βββββββββ
β β postgres β β sqlite3 β
β βββββββββββββββββ ββββββββββββββββββ
β
β βββββββββββββββββ - Connects the core event engine to external frameworks and applications
βββββββββββββ€ event [Ei]β * Supports long-lived (durable) and ephemeral event subscriptions
β β interface β * Batching, filtering, all handled in core prior to transport
β βββββββ¬ββββββββββ * Interface supports connect-in (websocket) and connect-out (broker runtime style) plugins
β β
β ββββββββββ ... extensible to integrate off-chain compute framework (Hyperledger Avalon, TEE, ZKP, MPC etc.)
β β ... extensible to additional event delivery brokers/subsystems (Webhooks, Kafka, AMQP etc.)
β βββββββ΄ββββββββββ
β β websockets β
β βββββββββββββββββ
β ... more TBD
Additional utility framworks
βββββββββββββββββ - REST API client
β rest [Re]β * Provides convenience and logging
β client β * Standardizes auth, config and retry logic
βββββββββββββββββ * Built on Resty
βββββββββββββββββ - WebSocket client
β wsclient [Wc]β * Provides convenience and logging
β β * Standardizes auth, config and reconnect logic
βββββββββββββββββ * Built on Gorilla WebSockets
βββββββββββββββββ - Translation framework
β i18n [In]β * Every translations must be added to `en_translations.json` - with an `FF10101` key
β β * Errors are wrapped, providing extra features from the `errors` package (stack etc.)
βββββββββββββββββ * Description translations also supported, such as OpenAPI description
βββββββββββββββββ - Logging framework
β log [Lo]β * Logging framework (logrus) integrated with context based tagging
β β * Context is used throughout the code to pass API invocation context, and logging context
βββββββββββββββββ * Example: Every API call has an ID that can be traced, as well as a timeout
βββββββββββββββββ - Configuration
β config [Co]β * File and Environment Variable based logging framework (viper)
β β * Primary config keys all defined centrally
βββββββββββββββββ * Plugins integrate by returning their config structure for unmarshaling (JSON tags)
REST collections provide filter, skip
, limit
and sort
support.
- The field in the message is used as the query parameter
- When multiple query parameters are supplied these are combined with AND
- When the same query parameter is supplied multiple times, these are combined with OR
GET
/api/v1/messages?confirmed=>0&type=broadcast&topic=t1&topic=t2&context=@someprefix&sort=sequence&descending&skip=100&limit=50
This states:
- Filter on
confirmed
greater than 0 - Filter on
type
exactly equal tobroadcast
- Filter on
topic
exactly equal tot1
ort2
- Filter on
context
containing the case-sensitive stringsomeprefix
- Sort on
sequence
indescending
order - Paginate with
limit
of50
andskip
of100
(e.g. get page 3, with 50/page)
Table of filter operations, which must be the first character of the query string (after the =
in the above URL path example)
Operator | Description |
---|---|
(none) | Equal |
! |
Not equal |
< |
Less than |
<= |
Less than or equal |
> |
Greater than |
>= |
Greater than or equal |
@ |
Containing - case sensitive |
!@ |
Not containing - case sensitive |
^ |
Containing - case insensitive |
!^ |
Not containing - case insensitive |