Skip to content

Latest commit

 

History

History
727 lines (573 loc) · 27.9 KB

appendix_protocol_messages.asciidoc

File metadata and controls

727 lines (573 loc) · 27.9 KB

Appendix A: Wire Protocol Messages

This appendix lists all the currently defined message types used in the Lightning P2P protocol. Additionally, we show the structure of each message, grouping the messages into logical groupings based on the protocol flows.

Note

Lightning Protocol messages are extensible and their structure may change during network-wide upgrades. For the authoritative information, consult the latest version of the BOLTs found in the Github - Lightning-RFC repository.

Message Types

Currently defined message types are:

Table 1. Message Types
Type Integer Message Name Category

16

init

Connection Establishment

17

error

Error Communication

18

ping

Connection Liveness

19

pong

Connection Liveness

32

open_channel

Channel Funding

33

accept_channel

Channel Funding

34

funding_created

Channel Funding

35

funding_signed

Channel Funding

36

funding_locked

Channel Funding + Channel Operation

38

shutdown

Channel Closing

39

closing_signed

Channel Closing

128

update_add_htlc

Channel Operation

130

update_fulfill_hltc

Channel Operation

131

update_fail_htlc

Channel Operation

132

commit_sig

Channel Operation

133

revoke_and_ack

Channel Operation

134

update_fee

Channel Operation

135

update_fail_malformed_htlc

Channel Operation

136

channel_reestablish

Channel Operation

256

channel_announcement

Channel Announcement

257

node_announcement

Channel Announcement

258

channel_update

Channel Announcement

259

announce_signatures

Channel Announcement

261

query_short_chan_ids

Channel Graph Syncing

262

reply_short_chan_ids_end

Channel Graph Syncing

263

query_channel_range

Channel Graph Syncing

264

reply_channel_range

Channel Graph Syncing

265

gossip_timestamp_range

Channel Graph Syncing

In [message_types], the Category field allows us to quickly categorize a message based on its functionality within the protocol itself. At a high level, we place a message into one of 8 (non exhaustive) buckets including:

  • Connection Establishment: Sent when a peer to peer connection is first established. Also used in order to negotiate the set of feature supported by a new connection.

  • Error Communication: Used by peer to communicate the occurrence of protocol level errors to each other.

  • Connection Liveness: Used by peers to check that a given transport connection is still live.

  • Channel Funding: Used by peers to create a new payment channel. This process is also known as the channel funding process.

  • Channel Operation: The act of updating a given channel off-chain. This includes sending and receiving payments, as well as forwarding payments within the network.

  • Channel Announcement: The process of announcing a new public channel to the wider network so it can be used for routing purposes.

  • Channel Graph Syncing: The process of downloading & verifying the channel graph.

Notice how messages that belong to the same category typically share an adjacent message type as well. This is done on purpose in order to group semantically similar messages together within the specification itself.

Message Structure

We now detail each message category in order to define the precise structure and semantics of all defined messages within the LN protocol.

Connection Establishment Messages

Messages in this category are the very first message sent between peers once they establish a transport connection. At the time of writing of this chapter, there exists only a single messages within this category, the init message. The init message is sent by both sides of the connection once it has been first established. No other messages are to be sent before the init message has been sent by both parties.

The structure of the init message is defined as follows:

The init message
  • type: 16

  • fields:

    • uint16: global_features_len

    • global_features_len*byte: global_features

    • uint16: features_len

    • features_len*byte: features

    • tlv_stream_tlvs

Structurally, the init message is composed of two variable size bytes slices that each store a set of feature bits. As we see in [feature_bits], feature bits are a primitive used within the protocol in order to advertise the set of protocol features a node either understands (optional features), or demands (required features).

Note that modern node implementations will only use the features field, with items residing within the global_features vector for primarily historical purposes (backwards compatibility).

What follows after the core message is a series of T.L.V, or Type Length Value records which can be used to extend the message in a forwards+backwards compatible manner in the future. We’ll cover what TLV records are and how they’re used later in the chapter.

An init message is then examined by a peer in order to determine if the connection is well defined based on the set of optional and required feature bits advertised by both sides.

An optional feature means that a peer knows about a feature, but they don’t consider it critical to the operation of a new connection. An example of one would be something like the ability to understand the semantics of a newly added field to an existing message.

On the other hand, required feature indicate that if the other peer doesn’t know about the feature, then the connection isn’t well defined. An example of such a feature would be a theoretical new channel type within the protocol: if your peer doesn’t know of this feature, they you don’t want to keep the connection as they’re unable to open your new preferred channel type.

Error Communication Messages

Messages in this category are used to send connection level errors between two peers. Another type of error exists in the protocol: an HTLC forwarding level error. Connection level errors may signal things like feature bit incompatibility, or the intent to force close (unilaterally broadcast the latest signed commitment)

The sole message in this category is the error message:

The error message
  • type: 17

  • fields:

    • channel_id : chan_id

    • uint16 : data_len

    • data_len*byte : data

An error message can be sent within the scope of a particular channel by setting the channel_id, to the channel_id of the channel under going this new error state. Alternatively, if the error applies to the connection in general, then the channel_id field should be set to all zeroes. This all zero channel_id is also known as the connection level identifier for an error.

Depending on the nature of the error, sending an error message to a peer you have a channel with may indicate that the channel cannot continue without manual intervention, so the only option at that point is to force close the channel by broadcasting the latest commitment state of the channel.

Connection Liveness

Messages in this section are used to probe to determine if a connection is still live or not. As the LN protocol somewhat abstracts over the underlying transport being used to transmit the messages, a set of protocol level ping and pong messages are defined.

The ping message
  • type: 18

  • fields:

    • uint16 : num_pong_bytes

    • uint16 : ping_body_len

    • ping_body_len*bytes : ping_body

Next it’s companion, the pong message.

The pong message
  • type: 19

  • fields:

    • uint16 : pong_body_len

    • ping_body_len*bytes : pong_body

A ping message can be sent by either party at any time.

The ping message includes a num_pong_bytes field that is used to instruct the receiving node with respect to how large the payload it sends in its pong message is. The ping message also includes a ping_body opaque set of bytes which can be safely ignored. It only serves to allow a sender to pad out ping messages they send, which can be useful in attempting to thwart certain de-anonymization techniques based on packet sizes on the wire.

A pong message should be sent in response to a received ping message. The receiver should read a set of num_pong_bytes random bytes to send back as the pong_body field. Clever use of these fields/messages may allow a privacy concious routing node to attempt to thwart certain classes of network de-anonymization attempts, as they can create a "fake" transcript that resembles other messages based on the packet sizes set across. Remember that by default the LN uses an encrypted transport, so a passive network monitor cannot read the plaintext bytes, thus only has timing and packet sizes to go off of.

Channel Funding

As we go on, we enter into the territory of the core messages that govern the functionality and semantics of the Lightning Protocol. In this section, we explore the messages sent during the process of creating a new channel. We’ll only describe the fields used as we leave a in in-depth analysis of the funding process to [payment_channels].

Messages that are sent during the channel funding flow belong to the following set of 5 messages: open_channel, accept_channel, funding_created, funding_signed, funding_locked.

The detailed protocol flow using these messages is described in [payment_channels].

The open_channel message
  • type: 32

  • fields:

    • chain_hash : chain_hash

    • 32*byte : temp_chan_id

    • uint64 : funding_satoshis

    • uint64 : push_msat

    • uint64 : dust_limit_satoshis

    • uint64 : max_htlc_value_in_flight_msat

    • uint64 : channel_reserve_satoshis

    • uint64 : htlc_minimum_msat

    • uint32 : feerate_per_kw

    • uint16 : to_self_delay

    • uint16 : max_accepted_htlcs

    • pubkey : funding_pubkey

    • pubkey : revocation_basepoint

    • pubkey : payment_basepoint

    • pubkey : delayed_payment_basepoint

    • pubkey : htlc_basepoint

    • pubkey : first_per_commitment_point

    • byte : channel_flags

    • tlv_stream : tlvs

This is the first message sent when a node wishes to execute a new funding flow with another node. This message contains all the necessary information required for both peers to constructs both the funding transaction as well as the commitment transaction.

At the time of writing of this chapter, a single TLV record is defined within the set of optional TLV records that may be appended to the end of a defined message:

  • type: 0

  • data: upfront_shutdown_script

The upfront_shutdown_script is a variable sized byte slice that MUST be a valid public key script as accepted by the Bitcoin networks' consensus algorithm. By providing such an address, the sending party is able to effectively create a "closed loop" for their channel, as neither side will sign off an cooperative closure transaction that pays to any other address. In practice, this address is usually one derived from a cold storage wallet.

The channel_flags field is a bitfield of which at the time of writing, only the first bit has any sort of significance. If this bit is set, then this denotes that this channel is to be advertised to the public network as a route bal channel. Otherwise, the channel is considered to be unadvertised, also commonly referred to as a "private" channel.

The accept_channel message is the response to the open_channel message:

The accept_channel message
  • type: 33

  • fields:

    • 32*byte : temp_chan_id

    • uint64 : dust_limit_satoshis

    • uint64 : max_htlc_value_in_flight_msat

    • uint64 : channel_reserve_satoshis

    • uint64 : htlc_minimum_msat

    • uint32 : minimum_depth

    • uint16 : to_self_delay

    • uint16 : max_accepted_htlcs

    • pubkey : funding_pubkey

    • pubkey : revocation_basepoint

    • pubkey : payment_basepoint

    • pubkey : delayed_payment_basepoint

    • pubkey : htlc_basepoint

    • pubkey : first_per_commitment_point

    • tlv_stream : tlvs

The accept_channel message is the second message sent during the funding flow process. It serves to acknowledge an intent to open a channel with a new remote peer. The message mostly echoes the set of parameters that the responder wishes to apply to their version of the commitment transaction. In [payment_channels], when we go into the funding process in details, we do a deep dive to explore the implications of the various parameters that can be set when opening a new channel.

In response, the initiator will send the funding_created message.

The funding_created message
  • type: 34

  • fields:

    • 32*byte : temp_chan_id

    • 32*byte : funding_txid

    • uint16 : funding_output_index

    • sig : commit_sig

Once the initiator of a channel receives the accept_channel message from the responder, they they have all the materials they need in order to construct the commitment transaction, as well as the funding transaction. As channels by default are single funder (only one side commits funds), only the initiator needs to construct the funding transaction. As a result, in order to allow the responder to sign a version of a commitment transaction for the initiator, the initiator, only needs to send the funding outpoint of the channel.

To conclude the responder sends the funding_signed message.

The funding_signed message
  • type: 34

  • fields:

    • channel_id : channel_id

    • sig : signature

To conclude after the responder receivers the funding_created message, they now own a valid signature of the commitment transaction by the initiator. With this signature they’re able to exit the channel at any time by signing their half of the multi-sig funding output, and broadcasting the transaction. This is referred to as a force close. In order to give the initiator the ability to do so was well, before the channel can be used, the responder then signs the initiator’s commitment transaction as well.

Once this message has been received by the initiator, it’s safe for them to broadcast the funding transaction, as they’re now able to exit the channel agreement unilaterally.

Once the funding transaction has received enough confirmations, the funding_locked is sent.

The funding_locked message
  • type: 36

  • fields:

    • channel_id : channel_id

    • pubkey : next_per_commitment_point

Once the funding transaction obtains a minimum_depth number of confirmations, then the funding_locked message is to be sent by both sides. Only after this message has been received, and sent can the channel being to be used.

Channel Closing

Channel closing is a multi-step process. One node initiates by sending the shutdown message. The two channel partners then exchange a series of channel_closing messages to negotiate mutually acceptable fees for the closing transaction. The channel funder sends the first closing_signed message and the other side can accept by sending a closing_signed message with the same fee values.

The shutdown message
  • type: 38

  • fields:

    • channel_id : channel_id

    • u16 : len

    • len*byte : scriptpubkey

The closing_signed message
  • type: 39

  • fields:

    • channel_id : channel_id

    • u64 : fee_satoshis

    • signature : signature

Channel Operation

In this section, we briefly describe the set of messages used to allow nodes to operate a channel. By operation, we mean being able to send receive, and forward payments for a given channel.

In order to send, receive or forward a payment over a channel, an HTLC must first be added to both commitment transactions that comprise of a channel link.

The update_add_htlc message allows either side to add a new HTLC to the opposite commitment transaction.

The update_add_htlc message
  • type: 128

  • fields:

    • channel_id : channel_id

    • uint64 : id

    • uint64 : amount_msat

    • sha256 : payment_hash

    • uint32 : cltv_expiry

    • 1366*byte : onion_routing_packet

Sending this message allows one party to initiate either sending a new payment, or forwarding an existing payment that arrived via in incoming channel. The message specifies the amount (amount_msat) along with the payment hash that unlocks the payment itself. The set of forwarding instructions of the next hop are onion encrypted within the onion_routing_packet field. In [onion_routing], on multi-hop HTLC forwarding, we detail the onion routing protocol used in the Lighting Network in detail.

Note that each HTLC sent uses an auto incrementing ID which is used by any message which modifies an HTLC (settle or cancel) to reference the HTLC in a unique manner scoped to the channel.

The update_fulfill_hltc allow redemption (receipt) of an active HTLC.

The update_fulfill_hltc message
  • type: 130

  • fields:

    • channel_id : channel_id

    • uint64 : id

    • 32*byte : payment_preimage

This message is sent by the HTLC receiver to the proposer in order to redeem an active HTLC. The message references the id of the HTLC in question, and also provides the pre-image (which unlocks the HLTC) as well.

The update_fail_htlc is sent to remove an HTLC from a commitment transaction.

The update_fail_htlc message
  • type: 131

  • fields:

    • channel_id : channel_id

    • uint64 : id

    • uint16 : len

    • len*byte : reason

The update_fail_htlc is the opposite of the update_fulfill_hltc message as it allows the receiver of an HTLC to remove the very same HTLC. This message is typically sent when an HTLC cannot be properly routed upstream, and needs to be sent back to the sender in order to unravel the HTLC chain. As we explore in Chapter XX, the message contains an encrypted failure reason (reason) which may allow the sender to either adjust their payment route, or terminate if the failure itself is a terminal one.

The commitment_signed message is used to stamp the creation of a new commitment transaction

The commitment_signed message
  • type: 132

  • fields:

    • channel_id : channel_id

    • sig : signature

    • uint16 : num_htlcs

    • num_htlcs*sig : htlc_signature

In addition to sending a signature for the next commitment transaction, the sender of this message also needs to send a signature for each HTLC that’s present on the commitment transaction. This is due to the existence of the

The revoke_and_ack is sent to revoke a dated commitment:

The revoke_and_ack message
  • type: 133

  • fields:

    • channel_id : channel_id

    • 32*byte : per_commitment_secret

    • pubkey : next_per_commitment_point

As the Lightning Network uses a replace-by-revoke commitment transaction, after receiving a new commitment transaction via the commit_sig message, a party must revoke their past commitment before they’re able to receive another one. While revoking a commitment transaction, the revoker then also provides the next commitment point that’s required to allow the other party to send them a new commitment state.

The update_fee is sent to update the fee on the current commitment transactions.

The update_fee message
  • type: 134

  • fields:

    • channel_id : channel_id

    • uint32 : feerate_per_kw

This message can only be sent by the initiator of the channel they’re the ones that will pay for the commitment fee of the channel as along as it’s open.

The update_fail_malformed_htlc is sent to remove a corrupted HTLC:

The update_fail_malformed_htlc message
  • type: 135

  • fields:

    • channel_id : channel_id

    • uint64 : id

    • sha256 : sha256_of_onion

    • uint16 : failure_code

This message is similar to the update_fail_htlc but it’s rarely used in practice. As mentioned above, each HTLC carries an onion encrypted routing packet that also covers the integrity of portions of the HTLC itself. If a party receives an onion packet that has somehow been corrupted along the way, then it won’t be able to decrypt the packet. As a result it also can’t properly forward the HTLC, therefore it’ll send this message to signify that the HTLC has been corrupted somewhere along the route back to the sender.

Channel Announcement

Messages in this category are used to announce components of the Channel Graph authenticated data structure to the wider network. The Channel Graph has a series of unique properties due to the condition that all data added to the channel graph MUST also be anchored in the base Bitcoin blockchain. As a result, in order to add a new entry to the channel graph, an agent must be an on chain transaction fee. This serves as a natural spam de tenace for the Lightning Network.

The channel_announcement is used to announce a new channel to the wider network.

The channel_announcement message
  • type: 256

  • fields:

    • sig : node_signature_1

    • sig : node_signature_2

    • sig : bitcoin_signature_1

    • sig : bitcoin_signature_2

    • uint16 : len

    • len*byte : features

    • chain_hash : chain_hash

    • short_channel_id : short_channel_id

    • pubkey : node_id_1

    • pubkey : node_id_2

    • pubkey : bitcoin_key_1

    • pubkey : bitcoin_key_2

The series of signatures and public keys in the message serves to create a proof that the channel actually exists within the base Bitcoin blockchain. As we detail in [scid], each channel is uniquely identified by a locator that encodes it’s location within the blockchain. This locator is called this short_channel_id and can fit into a 64-bit integer.

The node_announcement allows a node to announce/update it’s vertex within the greater Channel Graph.

The node_announcemen` message
  • type: 257

  • fields:

    • sig : signature

    • uint64 : flen

    • flen*byte : features

    • uint32 : timestamp

    • pubkey : node_id

    • 3*byte : rgb_color

    • 32*byte : alias

    • uint16 : addrlen

    • addrlen*byte : addresses

Note that if a node doesn’t have any advertised channel within the Channel Graph, then this message is ignored in order to ensure that adding an item to the Channel Graph bares an on-chain cost. In this case, the on-chain cost will the cost of creating the channel which this node is connected to.

In addition to advertising its feature set, this message also allows a node to announce/update the set of network addresses that it can be reached at.

The channel_update message is sent to update the properties and policies of an active channel edge within the Channel graph.

The channel_update message
  • type: 258

  • fields:

    • signature : signature

    • chain_hash : chain_hash

    • short_channel_id : short_channel_id

    • uint32 : timestamp

    • byte : message_flags

    • byte : channel_flags

    • uint16 : cltv_expiry_delta

    • uint64 : htlc_minimum_msat

    • uint32 : fee_base_msat

    • uint32 : fee_proportional_millionths

    • uint16 : htlc_maximum_msat

In addition to being able to enable/disable a channel this message allows a node to update it’s routing fees as well as other fields that shape the type of payment that is permitted to flow through this channel.

The announce_signatures message is exchange by channel peers in order to assemble the set of signatures required to produce a channel_announcement message.

The announce_signatures message
  • type: 259

  • fields:

    • channel_id : channel_id

    • short_channel_id : short_channel_id

    • sig : node_signature

    • sig : bitcoin_signature

After the funding_locked message has been sent, if both sides wish to advertise their channel to the network, then they’ll each send the announce_signatures message which allows both sides to emplace the 4 signatures required to generate a announce_signatures message.

Channel Graph Syncing

The query_short_chan_ids allows a peer to obtain the channel information related to a series of short channel IDs.

The query_short_chan_ids message
  • type: 261

  • fields:

    • chain_hash : chain_hash

    • u16 : len

    • len*byte : encoded_short_ids

    • query_short_channel_ids_tlvs : tlvs

As we learn in [gossip], these channel IDs may be a series of channels that were new to the sender, or were out of date which allows the sender to obtain the latest set of information for a set of channels.

The reply_short_chan_ids_end message is sent after a peer finishes responding to a prior query_short_chan_ids message.

The reply_short_chan_ids_end message
  • type: 262

  • fields:

    • chain_hash : chain_hash

    • byte : full_information

This message signals to the receiving party that if they wish to send another query message, they can now do so.

The query_channel_range message allows a node to query for the set of channel opened within a block range.

The query_channel_range message
  • type: 263

  • fields:

    • chain_hash : chain_hash

    • u32 : first_blocknum

    • u32 : number_of_blocks

    • query_channel_range_tlvs : tlvs

As channels are represented using a short channel ID that encodes the location of a channel in the chain, a node on the network can use a block height as a sort of cursor to seek through the chain in order to discover a set of newly opened channels.

The reply_channel_range message is the response to query_channel_range and includes the set of short channel IDs for known channels within that range.

The reply_channel_range message
  • type: 264

  • fields:

    • chain_hash : chain_hash

    • u32 : first_blocknum

    • u32 : number_of_blocks

    • byte : sync_complete

    • u16 : len

    • len*byte : encoded_short_ids

    • reply_channel_range_tlvs : tlvs

As a response to query_channel_range, this message sends back the set of channels that were opened within that range. This process can be repeated with the requester advancing their cursor further down the chain in order to continue syncing the Channel Graph.

The gossip_timestamp_range message allows a peer to start receiving new incoming gossip messages on the network.

The gossip_timestamp_range message
  • type: 265

  • fields:

    • chain_hash : chain_hash

    • u32 : first_timestamp

    • u32 : timestamp_range

Once a peer has synced the channel graph, they can send this message if they wish to receive real-time updates on changes in the Channel Graph. They can also set the first_timestamp and timestamp_range fields if they wish to receive a backlog of updates they may have missed while they were down.