F3 passive testing

[jennijuju]

coming soon

[jennijuju]

Note that the f3 implementation team will not deploy anything that initiates f3 in mainnet environment until the vast majority of the network has stabilized from the network upgrade. The current plan is to bootstrap the initial f3 instance among a small amount of the nodes on Aug 7th/8th (>24 hours after the upgrade), we will monitor the network and adjust the deployment plan accordingly, and keep you updated on what to expect in this discussion!

[jennijuju]

🚀F3 Passive Testing Kick off

Aug 7th, 2024

We're excited to announce the soft launch of Fast Finality for Filecoin (F3) as part of FIP-0086! 🏎️If you've been following our progress, you might already know that this has been in the works for some time. Fast Finality aims to dramatically reduce the finality time from 900 epochs to just around 2, increasing the speed by 450X and ensuring that "finality" truly means finality. This means transactions will be completed reliably and quickly.

With the mandatory release of Lotus v1.28.1 and above, we're now able to conduct passive testing on the mainnet. This testing is a critical step toward fully implementing Fast Finality in the upcoming network upgrade (nv24) later this year. After the full rollout, we expect significant UX improvements for both token holders and dApp users.

Additionally, F3 will enable the creation of trustless clients and bridges that can send messages over the network and verify finality without needing to run a full node, making it both efficient and cost-effective.

To learn more about Fast Finality and its implications for bridging to other networks, check out this talk. We're thrilled to take this step forward and look forward to your feedback as we move closer to full implementation.

F3 Passive Testing

F3 has undergone extensive testing in both code simulation and on the Butterfly testnet. However, to ensure it's ready for mainnet production, we need to perform large-scale testing that mirrors the number of nodes on the mainnet. This will help verify the performance of F3, GPBFT, and Lotus integration. Our goal is to execute this testing as passively as possible, minimizing disruptions.

The F3 engineering team plans to start with a small group of F3 nodes to confirm everything works as expected. We will gradually increase the number of F3 participants, aiming to reach a scale close to mainnet production.

Why Passive Testing?

The transition to F3 marks a significant advancement for Filecoin and its ecosystem, offering faster transaction speeds and enabling trustless bridges. However, this evolution introduces complexities in testing, verification, and ensuring confidence in the system's implementation across the network.

Passive testing is essential for several reasons:

1. Building Robust Confidence: Before a widespread rollout, it's crucial to achieve strong confidence in F3's deployment. While existing tests in simulated environments provide a framework for assessing consensus correctness, they can't fully capture the complexities of interacting software layers. Real-world testing is needed to ensure that all components work together seamlessly.
2. Addressing Software Layer Complexities: Simulations often fall short in testing critical components such as the libp2p transport protocol, GossipSub’s optimistic message propagation mechanism, Lotus EC integration, and the dynamics of a large number of network participants and GPBFT instances. Passive testing allows us to evaluate these elements in a realistic environment.
3. Non-Disruptive Testing Environment: The nv23 upgrade provides an opportunity to test the F3 protocol in a live, but non-disruptive manner. Passive testing integrates F3 as an optional alternative consensus mechanism, enabled by a feature flag, without replacing the existing Filecoin consensus. This approach mitigates risks associated with modifying a fundamental component of the Filecoin protocol.
4. Identifying Integration Gaps: This phase is crucial for identifying potential integration gaps among the various system components affected by F3. It allows us to assess network adoption and response in a real-world setting.
5. Evaluating Consensus Protocol Functionality: Most importantly, passive testing provides a definitive examination of the consensus protocol’s functionality under real-world conditions. This helps ensure that F3 is fully ready for production use.

Passive Testing allows the F3 engineering team to test a full-scale rollout on a real network without affecting consensus. This approach ensures that the testing does not disrupt existing network operations.

Testing Plan Summary

A more thorough testing plan is laid out in this issue, in a nutshell:

• A dedicated server will deploy an F3 Manifest connected to the Lotus node.
• The F3 implementation team can configure and deploy the Manifest to bootstrap, pause, and restart the F3 participation selectively.
• Once F3 is deployed and live, selected nodes will begin signing and broadcasting messages to finalize tipsets on confirmation that everything functions as expected.
• The Manifest can assign “fake power” to selected Storage Providers (SPs), allowing us to verify the protocol as nodes (power) join and leave the network, and simulate scenarios that are otherwise (near) impossible to achieve in simulations.

It's important to note that none of the above affects the Expected Consensus (EC) in nv23. F3 runs in the background and finalizes tipsets independently from EC for testing purposes, hence the term “passive.”

Observable Metrics for Monitoring

The F3 team has implemented several metrics to monitor node and consensus performance:

• Lotus Node Metrics: Goroutines, heap object allocation rate, garbage collection performance, CPU usage.
• GPBFT Metrics: Head divergence, rounds to finalize an instance, broadcast and rebroadcast rate, errors by type, consensus phase transitions.
• Validation Performance: Message validation time, proposal fetch time, duplicate validation time.
• Pubsub Status: Message and RPC rate.
• Certificate Exchange: Polling intervals and peers, latencies.

We will share updates and results with the community at least bi-weekly, if not more frequently. You can also monitor F3 metrics via the Prometheus debug metrics endpoint at http://<lotus-node-address>:1234/debug/metrics, looking for metric names prefixed with f3_.

sample dashboard

Initial F3 Deployment

The F3 team is excited to announce the initial deployment of Fast Finality (F3) starting on August 7th. We will begin with 50 randomly selected participants, including some of our dedicated alpha testers from the community 💙.

Once the deployment begins, you will start seeing F3 logs similar to the following:

{"level":"info","ts":"2024-08-07T02:17:00.149Z","logger":"f3","caller":"go-f3@v0.0.8-0.20240802170449-e0aa03140cb9/host.go:618","msg":"reached a decision","instance":38671,"ecHeadEpoch":37328}

This log indicates that F3 has successfully finalized a tipset.

Adjusting Log Levels

If you wish to adjust the log level for F3 in Lotus or Lotus Miner, you can do so by running the following commands:

$ lotus log set-level --system f3
$ lotus-miner log set-level --system f3

Monitoring and Reporting

The F3 engineering team will be closely monitoring all participating nodes. If you notice any irregular behavior on your nodes, please report it in the #fil-fast-finality channel on Filecoin Slack. Ideally, include a log dump from your daemon node to help us diagnose the issue. If any problems arise, we will promptly redeploy a manifest to pause all F3 activities, ensuring your node returns to normal without any manual intervention.

Mainnet Verifications

Below is a list of verifications that will be performed on the mainnet during the initial F3 deployment:

• Successful Bootstrap: Ensure F3 is successfully bootstrapped.
• Consensus Formation: Instances are formed, reaching consensus, and producing a finality certificate.
• GPBFT Performance: Verify that GPBFT performance is as expected: e.g. rate of message broadcast, justifications weighted by power.
• Pubsub Performance: Check that Pubsub performance is as expected: e.g. RPC byte transfer rate, GossipSub duplicate messages, validation time.
• Lotus Daemon Performance: Ensure Lotus daemon performance is as expected: e.g. CPU usage, memory consumption, unaffected pre-existing functionality.
• Network Joining: Verify that a new daemon node can join the network and catch up with F3 with expected timeframe.
• Power Drop Tolerance: Ensure that if there is a <33% power drop from F3, the F3 chain continues to progress.
• Power Drop Halting: Confirm that if there is a >66% power drop, the F3 chain progress eventually falls back on existing EC.
• Power Rejoining: Verify that the F3 chain resumes once power re-joins.

We look forward to this exciting phase and appreciate your support and feedback as we work towards fully implementing F3 across the network.

🚢 Launch Time!

Fast Finality has been one of the most anticipated features for Filecoin participants since the network's inception. We are thrilled to collaborate with YOU to finally bring this feature to the Filecoin mainnet by the end of the year!

We want to extend our gratitude to all community members participating in this testing phase. Special thanks go to our early testers for their invaluable support with Slack handles: TippyFlits, Reiers, stuberman, beck, and @marco-storswift.

Stay tuned for updates in the #fil-fast-finality channel over the coming weeks. Please join the channel to keep up with announcements and potential support requests. We may need your node logs and profiles to assist our implementation team with debugging.

Happy finalizing fast! 🏎️💨🏁

[jennijuju]

Testing update - Aug 14th

In the past week..

1. We’ve successfully tested F3 networks with up to 120 nodes.
2. We’ve significantly reduced the CPU usage of F3 through code optimisations and aggressive caching.
3. We’ve improved the behavior/speed of F3 when it falls behind.
4. We’ve introduced a ton of (local) metrics for monitoring the state of the network. As usual, these are available on your local node (only to you) via prometheus.

🐛 Bandwidth Usage Spike

Lotus node operators have reported irregular traffic spike starting Wednesday morning and we have confirmed it is caused by F3 implementation. The excessive F3 bandwidth was caused by a routing loop in pubsub. The “manifest” server (used to facilitate testing) broadcast a small message every 20 seconds, which shouldn’t have introduced an excessive load. Unfortunately, pubsub’s routing loop prevention mechanism appears to have been ineffective in this case so each message cycled around the network over and over.

We’ve fixed this issue by:

1. Adding additional filtering in the manifest pubsub topic to ensure that, even if we end up with routing loops, we’ll have at most one message rebroadcast every 2 minutes (negligible traffic).
2. Identifying pubsub messages by content, instead of by sequence number and sender (aligning this logic with the other Filecoin pubsub topics). This ensures that even if a message gets stuck in the network for a while, we’ll only have one copy instead of one copy for each time it was sent.

Note that even tho the bandwidth usage on nodes was increased, we believe that does not impact/cause issue towards node synchronization. However, we would also like to avoid any unexpected node performance degradation as much as possible. Therefore, we will release a lotus patch in NA working hours (The fix is already available here #12390, we will merge it after more testing).

Next round..

Next week, we aim to scale our testing to 500-1000 nodes. We will gradually increase the number of nodes participating in F3 and continue monitoring their performance throughout the process. One piece of good news is that we haven't encountered any issues so far that impact node synchronization or block production, and we will remain conservative with our testing to avoid any potential problems. If you notice anything irregular, please don't hesitate to reach out to us in #fil-fast-finality!
Once again, we want to thank the node operators who have been supporting the testing, helping us identify potential consensus protocol bugs and areas for performance improvement, and ensuring a smooth rollout of the fast finality consensus change to the mainnet 💙!

[rjan90]

F3 (Fast Finality) passive testing update - 2024-09-05

🗂 F3 Readiness Review and Timeline Adjustment

With the proposed nv24 upgrade date rapidly approaching, we conducted a thorough review of the tasks necessary to complete our "Hardening and Mainnet Deployment Readiness" milestone. Our assessment revealed a time deficit of approximately three weeks. In light of this finding, we've taken two important steps:

• We've thoroughly documented and prioritized tasks, focusing only on what's essential for safely deploying fast finality to mainnet.
• We've communicated our progress and requested revised dates for the nv24 network upgrade.

This time extension is important for completing our "Hardening and Mainnet Deployment Readiness" milestone, ensuring we address all critical items in our backlog. The additional time will help us maintain quality and minimize risks as we approach this significant network upgrade.

⏮️ Since the last update: Progress over the past weeks

Let's dive into the specifics of our recent hardening, fixes and testing we have done in the past weeks:

Hardening and Fixes:

• ✅ QUALITY Rebroadcast Issue Resolved: [The QUALITY rebroadcast issue](Fast Finality in Filecoin (FIP-0086) FIPs#809 (comment)), which was causing some inefficiencies in instance progress documented [here](Fast Finality in Filecoin (FIP-0086) FIPs#809 (comment)), has been [fixed](Handle QUALITY censorship go-f3#591). This fix is now merged, and ready for live testing.
• 🛠️ Fixed the "Next Best Converge Value" Issue: We successfully addressed the last remaining recommendation from our external audit regarding the "next best converge value selection". This is the [item No. 4 in the FIP discussion](Fast Finality in Filecoin (FIP-0086) FIPs#809 (comment)) that details the fix to lack of progress during CONVERGE phase.
• ⚡ Fixed Power Overflow Issue: A bug related to power overflow was identified and fixed. This fix is important for maintaining the stability and accuracy of power calculations within the network.
• 📡 Improved Propagation Strategy for DECIDE Messages: We've developed a better strategy for propagating DECIDE messages, which should help nodes that are behind by approximately two epochs to catch up more quickly. This new approach reduces the reliance on certificate exchange protocol, which ultimately aids faster finality.

Testing Efforts:

• 🧪 Expanded Test Coverage: We've broadened our test suite to include more scenarios observed during live testing. This expansion will help us catch potential issues earlier in the development process, leading to a more robust and reliable network.

⏭️ Upcoming Week's Focus:

Hardening and Fixes:

• 📝 Write-Ahead Logging (WAL): As part of our efforts to reduce and detect self-equivocations (a situation where a node contradicts itself), we are implementing WAL. This feature will help us better track and manage state changes, enhancing overall network reliability and security for cases where a set of miners may be using a single identity.
• 🔄 Signature Generation Optimizations: We are working on two optimizations to reduce the resource consumption associated with signature generation. These changes will also contribute to the Drand BLS library, benefiting the broader community.
• 🔍 Inferring Unseen Messages from Justifications: We are developing methods to infer unseen messages from existing justifications. This technique aims to reduce the impact of equivocations, while at the same time further reducing reliance on message delivery for correctness.
• ✂️ Reducing Network Spam: To optimize network performance, we're implementing a strategy to drop messages from previous rounds. This change will help reduce network spam and lower the workload on the PubSub system, allowing us to defer some PubSub-related work and save several weeks of effort.

Testing Efforts:

• 🏗️ Infrastructure Improvements for Easier Testing: We're making several infrastructure upgrades to simplify the testing process. These improvements will make it easier for the team to test changes on butterflynet and calibnet.
• 🏁 Goals before next update:
  • Conduct thorough Butterflynet testing of all recent hardening and fixes to ensure their effectiveness and stability.
  • Continue improving test coverage, focusing on areas that require additional attention based on Butterflynet testing results.

Stay tuned for more updates in the #fil-fast-finality channel as we continue to harden F3 and move closer to the nv24 rollout! 💪

[rjan90]

F3 (Fast Finality) passive testing update - 2024-09-15

Hey all! 👋 

Another week has passed, so here is another weekly update on the F3 passive testing and hardening efforts in preparation for the mainnet launch.

As mentioned in last week's update, the team requested a time adjustment for the nv24 timeline which would allow us to complete all tasks in our "Hardening and Mainnet Deployment Readiness" milestone, and complete critical items in our backlog. This timeline adjustment has now been accepted, and the the nv24 timeline and F3 rollout looks like this now:

• October 23rd: Calibration network upgrade
• November 20th: Mainnet network upgrade
  • December 4th: F3-activation

⏮️ Since the last update: Progress over the past week

Hardening and Fixes:

• ✅ Infrastructure Improvements for Easier Testing: We have completed our infrastructure upgrades to simplify the testing process. These improvements now make it easier for the team to test the changes faster and with more confidence.
• ➡️✅ Signature Generation Optimizations: The work signature generation optimization is awaiting a final approval. Based on our benchmarks we achieved up to 88x speedup for public key aggregation, which will substantially reduce resource consumption associated with signature generation. (Link to PR)
• ➡️✅ Reducing Network Spam: The implementation of a strategy to drop messages from previous rounds is currently in progress. This optimization will reduce pubsub traffic between instances, saving bandwidth and improving network efficiency.
• 🚧 📝 Write-Ahead Logging (WAL): A complete implementation of WAL has been opened for preliminary review. Integration of the write-ahead logging is currently in progress. This feature will help us better track and manage state changes, enhancing overall network reliability and security for cases where a set of miners may be using a single identity. (Link to PR
• 🚧 Make F3 consensus final: We have have made quite some progress on making the F3 consensus final; implementing checkpoints for F3 decisions, expanding the EC backend API, and improving invalid base chain handling in Lotus. (Link to PR #1 and #2

Testing Efforts:

• ✅ Test Coverage Improvements: We've added a lot more test coverage over the week, and have marked the test coverage improvements ticket completed.
• 🏁 Butterflynet Testing: We are currently running the Butterfly network with the latest F3-changes to ensure the effectiveness and stability of all the recent hardening and fixes, and so far it looks like the recent fixes and improvements are have made the network safer. We will continue to let the network spin over the next couple of days, while we tweak and run more edge-case testing.

⏭️ Upcoming Week's Focus:

Hardening and Fixes:

• 🛬 Land Outstanding in-flight PRs:  Merge and integrate all pending pull requests and features that have been developed but not yet incorporated into the main codebase. This includes the Signature Generation Optimizations, Reducing Network Spam, Write-Ahead Logging, and Make F3 Consensus Final.
• 🛼 Mainnet Rollout Strategy: Finalize the F3 bootstrap strategy for mainnet deployment, building on discussions from [issue #596. Our goal is to decide on an approach that balances implementation simplicity, network security, and user intervention.
• 🛠️ Address Testing Issues: We anticipate that the expanded testing will potentially surface new issues. We have set some time off to rapidly address and resolve these issues.

Testing Efforts:

• ⬆️🦋 Expand Butterflynet Testing: Scale up amount of participants in the Butterfly significantly, aiming for 800-1000 nodes. This expanded testing will build on our runs with the latest F3 changes, allowing us to validate the effectiveness and stability of recent hardening and fixes at a larger scale

Next week the F3 team will be co-locating. This focused sprint aims to complete the remaining tasks in Milestone 2: "Hardening and Mainnet Deployment Readiness". By working side-by-side, we expect to accelerate our progress, and ensure we're fully prepared for the upcoming mainnet deployment 🚀.

Additionally, we are working on preparing an operators’ guide to F3 to help get the community ready for the F3 launch. This guide will cover key topics such as setup, configuration, and best practices. We aim to have it published by the end of next week.

Stay tuned for more updates in the #fil-fast-finality channel!