Persisting the lease using --experimental-enable-lease-checkpoint is not working as expected

[ishan16696]

What happened?

It has been observed that lease time-to-live are getting renewed on 3rd leadership change in a 3 node etcd cluster.
I haven't tried with 5 node etcd cluster.

What did you expect to happen?

If TTL>5mins, then after 5mins of checkpointing lease TTLs should get reset to last checkpoint rather than getting renewed after any no. of leadership change.

How can we reproduce it (as minimally and precisely as possible)?

1. Start a 3 node etcd cluster with flag --experimental-enable-lease-checkpoint=true
2. Create a lease with TTL 600sec(say).
3. Then wait for 300sec as leader will checkpoints lease time-to-live after every 300seconds.
4. Then after 300sec, trigger a leadership change(restart a pod or restart a process by killing it).
5. Check lease TTL ----> this time TTL will correctly reset to last checkpoint i.e 300sec (600s-300s=300s)
6. Again, trigger a leadership change and check the lease TTL ----> this time TTL will also correctly reset to last checkpoint i.e 300sec.
7. Again, trigger a leadership change and check the lease TTL ----> this time TTL is not correctly reset to last checkpoint i.e 300sec but it get renewed to 600sec

Anything else we need to know?

~ > etcdctl lease grant 600                                                                                                                                                                               
lease 75cc8c7b79747704 granted with TTL(600s)

~ > etcdctl endpoint status --endpoints=http://127.0.0.1:2379 --cluster -w table                                                                                                                          

+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|        ENDPOINT        |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://127.0.0.1:23379 | 292104365751cc93 |   3.5.9 |   98 kB |      true |      false |         2 |         10 |                 10 |        |
| http://127.0.0.1:22379 | 91bc3c398fb3c146 |   3.5.9 |   98 kB |     false |      false |         2 |         10 |                 10 |        |
|  http://127.0.0.1:2379 | bf9071f4639c75cc |   3.5.9 |   98 kB |     false |      false |         2 |         10 |                 10 |        |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

~ > etcdctl put fun fun2 --lease 75cc8c7b79747704                                                                                                                                                         
OK

~ > etcdctl lease timetolive 75cc8c7b79747704                                                                                                                                                             
lease 75cc8c7b79747704 granted with TTL(600s), remaining(556s)

~ > etcdctl lease timetolive 75cc8c7b79747704                                                                                                                                                             
lease 75cc8c7b79747704 granted with TTL(600s), remaining(394s)

~ > etcdctl lease timetolive 75cc8c7b79747704                                                                                                                                                             
lease 75cc8c7b79747704 granted with TTL(600s), remaining(326s)

~ > etcdctl lease timetolive 75cc8c7b79747704                                                                                                                                                             
lease 75cc8c7b79747704 granted with TTL(600s), remaining(269s)

// now, first lease checkpoint is done, we can trigger a leadership change

~ > etcdctl endpoint status --endpoints=http://127.0.0.1:2379 --cluster -w table                                                                                                                          
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|        ENDPOINT        |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://127.0.0.1:23379 | 292104365751cc93 |   3.5.9 |   98 kB |      true |      false |         2 |         12 |                 12 |        |
| http://127.0.0.1:22379 | 91bc3c398fb3c146 |   3.5.9 |   98 kB |     false |      false |         2 |         12 |                 12 |        |
|  http://127.0.0.1:2379 | bf9071f4639c75cc |   3.5.9 |   98 kB |     false |      false |         2 |         12 |                 12 |        |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

// leader has been changed

~ > etcdctl endpoint status --endpoints=http://127.0.0.1:2379 --cluster -w table                                                                                                                          
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|        ENDPOINT        |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://127.0.0.1:23379 | 292104365751cc93 |   3.5.9 |   98 kB |     false |      false |         3 |         14 |                 14 |        |
| http://127.0.0.1:22379 | 91bc3c398fb3c146 |   3.5.9 |   98 kB |      true |      false |         3 |         14 |                 14 |        |
|  http://127.0.0.1:2379 | bf9071f4639c75cc |   3.5.9 |   98 kB |     false |      false |         3 |         14 |                 14 |        |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

// lease get reset to last checkpoint i.e 300s
~ > etcdctl lease timetolive 75cc8c7b79747704                                                                                                                                                             
lease 75cc8c7b79747704 granted with TTL(600s), remaining(290s)

// again trigger a leadership change
~ > etcdctl endpoint status --endpoints=http://127.0.0.1:2379 --cluster -w table                                                                                                                          
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|        ENDPOINT        |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://127.0.0.1:23379 | 292104365751cc93 |   3.5.9 |   98 kB |     false |      false |         4 |         16 |                 16 |        |
| http://127.0.0.1:22379 | 91bc3c398fb3c146 |   3.5.9 |   98 kB |     false |      false |         4 |         16 |                 16 |        |
|  http://127.0.0.1:2379 | bf9071f4639c75cc |   3.5.9 |   98 kB |      true |      false |         4 |         16 |                 16 |        |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

// lease get reset to last checkpoint i.e 300s
~ > etcdctl lease timetolive 75cc8c7b79747704                                                                                                                                                             
lease 75cc8c7b79747704 granted with TTL(600s), remaining(286s)

// again trigger a leadership change
~ > etcdctl endpoint status --endpoints=http://127.0.0.1:2379 --cluster -w table         
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|        ENDPOINT        |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://127.0.0.1:23379 | 292104365751cc93 |   3.5.9 |   98 kB |      true |      false |         5 |         18 |                 18 |        |
| http://127.0.0.1:22379 | 91bc3c398fb3c146 |   3.5.9 |   98 kB |     false |      false |         5 |         18 |                 18 |        |
|  http://127.0.0.1:2379 | bf9071f4639c75cc |   3.5.9 |   98 kB |     false |      false |         5 |         18 |                 18 |        |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

// lease doesn't get reset to last checkpoint i.e 300s but completely renew to 600s
~ > etcdctl lease timetolive 75cc8c7b79747704                                                                                                                                                             
lease 75cc8c7b79747704 granted with TTL(600s), remaining(590s)

Etcd version (please run commands below)

$ etcd --version
etcd Version: 3.5.9
Git SHA: bdbbde998
Go Version: go1.20.4
Go OS/Arch: darwin/arm64

$ etcdctl version
etcdctl version: 3.5.9
API version: 3.5

Etcd configuration

etcd configuration

etcd --name etcd0 --initial-advertise-peer-urls http://127.0.0.1:2380 \
  --listen-peer-urls http://127.0.0.1:2380 \
  --listen-client-urls http://127.0.0.1:2379 \
  --advertise-client-urls http://127.0.0.1:2379 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-cluster etcd0=http://127.0.0.1:2380,etcd1=http://127.0.0.1:22380,etcd2=http://127.0.0.1:23380 \
  --initial-cluster-state new --data-dir infra0.etcd --experimental-enable-lease-checkpoint=true

etcd --name etcd1 --initial-advertise-peer-urls http://127.0.0.1:22380 \
  --listen-peer-urls http://127.0.0.1:22380 \
  --listen-client-urls http://127.0.0.1:22379 \
  --advertise-client-urls http://127.0.0.1:22379 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-cluster etcd0=http://127.0.0.1:2380,etcd1=http://127.0.0.1:22380,etcd2=http://127.0.0.1:23380 \
  --initial-cluster-state new --data-dir infra1.etcd --experimental-enable-lease-checkpoint=true

etcd --name etcd2 --initial-advertise-peer-urls http://127.0.0.1:23380 \
  --listen-peer-urls http://127.0.0.1:23380 \
  --listen-client-urls http://127.0.0.1:23379 \
  --advertise-client-urls http://127.0.0.1:23379 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-cluster etcd0=http://127.0.0.1:2380,etcd1=http://127.0.0.1:22380,etcd2=http://127.0.0.1:23380 \
  --initial-cluster-state new --data-dir infra2.etcd --experimental-enable-lease-checkpoint=true

Etcd debug information

Relevant log output

No response

[ahrtr]

Thanks @ishan16696

Could you double check whether you can reproduce this issue in the main branch?

Is anyone interested in having a deep dive on this issue?

[serathius]

This is a known issue, experimental-enable-lease-checkpoint will prevent ttl to be reset on a single leader change, however it's not persisted. So if all members are restarted before the next checkpoint it will cause ttl to be reset.

The fix is the --experimental-enable-lease-checkpoint-persist flag. #13508

[serathius]

@ishan16696 Can you try again with --experimental-enable-lease-checkpoint-persist flag?

[ishan16696]

The fix is the --experimental-enable-lease-checkpoint-persist flag. #13508

yes, I saw this PR and this flag --experimental-enable-lease-checkpoint-persist but I was unable to find it in a required version of etcd which I mentioned in the issue.
In, CHANGELOG.md file says this flag is available from etcd v3.4.20 onwards but I couldn't find it in etcd version 3.4.26
And similarly CHANGELOG.md file says this flag is available from etcd v3.5.2 onwards but I couldn't find it in etcd version 3.5.9

~ > etcd --version                                                                                                                                                     
etcd Version: 3.5.9
Git SHA: bdbbde998
Go Version: go1.20.4
Go OS/Arch: darwin/arm64
~ > etcd --help | grep lease                                                                                                                                            
  --experimental-enable-lease-checkpoint 'false'
    ExperimentalEnableLeaseCheckpoint enables primary lessor to persist lease remainingTTL to prevent indefinite auto-renewal of long lived leases.

Am I missing something ?

[ishan16696]

I tried similar experiment with etcd version 3.5.9 by enabling both of the flag as suggested by @serathius and it's working fine.
But why we have 2 flags performing the similar functionality 🤔 ? Isn't that's confusing ?

[ishan16696]

Hi @serathius ,
As I mentioned in this comment that flag: --experimental-enable-lease-checkpoint-persist is missing in etcd --help in etcd version 3.4.x and 3.5.x.
I have opened the PR to add this flag on respective etcd verison: #17189 and #17190

[serathius]

We should backport #16034 to fix missing flags in help on older releases.

[serathius]

But why we have 2 flags performing the similar functionality 🤔 ? Isn't that's confusing ?

Sorry for that, the --experimental-enable-lease-checkpoint-persist is invasive enough that we needed to separate it from the original flag. Hopefully with v3.6 all those options will be the default.