cordinator(ticdc): Fix Puller Resolved TS Lag Calculation and Deprecate current_ts Field in Stats (#11624)

[ti-chi-bot]

This is an automated cherry-pick of #11624

What problem does this PR solve?

Issue Number: close #11561

What is changed and how it works?

Summary:

This PR addresses a critical issue in the Puller Resolved TS Lag metric, where the lag was incorrectly displayed as -55 years. The root cause of the problem was the improper initialization of the CurrentTs field in the Stats structure within table.pb.go. This led to a situation where CurrentTs remained zero while ResolvedTs was correctly populated, resulting in a negative lag calculation that erroneously reflected a time difference of 55 years (from 1970 to the current year).

Detailed Explanation:

• Issue Background:

  • The Puller Resolved TS Lag was calculated as the difference between CurrentTs and ResolvedTs from the Stats structure.
  • However, the NewReplicationSet function did not correctly initialize CurrentTs, leading to incorrect lag values when metrics were collected.

• Design Intent of current_ts:

  • The current_ts field was initially designed to record the time at the processor side, which would then be used by the coordinator for lag calculation. This design was meant to avoid including the transmission delay and processing time on the coordinator side, as these could introduce inaccuracies in the lag calculation.
  • However, this potential inaccuracy is minimal since the communication between the processor and coordinator happens within the TiCDC cluster, where network delays are typically in the millisecond range. Given that the lag metric is reported in seconds, any millisecond-level discrepancy is considered acceptable.
  • Deprecating the current_ts field may introduce a slight inaccuracy in the lag calculation, as the "current time" will now be fetched directly by the coordinator rather than from the processor. This could result in a minor error due to transmission delays and processing time. However, this inaccuracy is negligible for practical purposes, as the delay is typically within milliseconds and does not significantly impact the metric, which is measured in seconds.
  • Additionally, since this field was part of the protobuf (pb) structure, it added extra overhead to every network transmission by carrying an additional value. It also introduced more complexity into the code, requiring maintenance of this field, which was part of the issue that led to the incorrect metric calculations in this case.

• Root Cause:

  • The use of CurrentTs from the Stats structure was flawed because this timestamp was not reliably up-to-date.
  • The metric calculation requires the "current time," which should always be fetched directly from PD to ensure accuracy, rather than relying on a potentially outdated CurrentTs stored in the Stats structure.

• Solution Implemented:

  • This PR updates the metric calculation to fetch the latest "current time" directly from PD whenever calculating the lag.
  • All references to the CurrentTs field within the codebase, particularly in metric calculations, have been updated to reflect this change.

• Deprecation of current_ts Field:

  • The current_ts field in the Stats structure, defined in cdc/processor/tablepb/table.proto, has been marked as deprecated.
  • The Stats structure is used for data transmission between the Coordinator and Processor, where current_ts was unnecessarily included in the state for each Table Span.
  • This PR removes all dependencies on current_ts within the project, ensuring that the Coordinator and Processor no longer rely on this outdated field.

Impact:

This PR ensures accurate lag calculation in the Puller Resolved TS Lag metric and removes the outdated current_ts field, improving both performance and maintainability by reducing unnecessary transmission overhead and field maintenance. While this change may introduce a slight inaccuracy in the lag calculation due to the use of coordinator-side timestamps, this impact is minimal and well within acceptable thresholds.

Check List

Tests

• Unit test
• Integration test

Questions

Will it cause performance regression or break compatibility?

No. The deprecation of the current_ts field in the Stats structure is backward-compatible because the field is not actively used elsewhere in the project beyond the affected metrics. The existing field remains in the protocol buffer definition but is marked as deprecated, so it won’t break compatibility with older versions that still use the field.

Do you need to update user documentation, design documentation or monitoring documentation?

No.

Release note

Fix the issue that the puller resolved ts lag metric shows incorrect values. #11561

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign 5kbpers for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

This cherry pick PR is for a release branch and has not yet been approved by triage owners.
Adding the do-not-merge/cherry-pick-not-approved label.

To merge this cherry pick:

1. It must be approved by the approvers firstly.
2. AFTER it has been approved by approvers, please wait for the cherry-pick merging approval from triage owners.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ti-chi-bot]

@ti-chi-bot: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cdc-integration-mysql-test	3f07a1d	link	true	/test cdc-integration-mysql-test

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.