slowLogTailer: rotated slow logs not cleaned

[aylei]

Bug Report

What version of TiDB Operator are you using?

v1.0.6

What's the status of the TiDB cluster pods?

NAME                            READY   STATUS    RESTARTS   AGE
db-discovery-65d54bbb55-zp272   1/1     Running   0          10h
db-monitor-669c96fb55-x2h75     3/3     Running   0          11h
db-pd-0                         1/1     Running   0          11h
db-pd-1                         1/1     Running   0          11h
db-pd-2                         1/1     Running   0          11h
db-tidb-0                       2/2     Running   0          4h28m
db-tidb-1                       2/2     Running   0          4h28m
db-tikv-0                       1/1     Running   0          11h
db-tikv-1                       1/1     Running   0          11h
db-tikv-2                       1/1     Running   0          11h

What did you do?

Load data to tidb cluster.

What did you expect to see?
Load data succeed

What did you see instead?
Load data failed because TiDB Pods were evicted to reclaim ephemeral storage:

And it turns out the slow log consumed a lot of ephemeral storage, whereas the tailer does not rotate the slow log:

/var/log/tidb # du -sh ./*
70.8M	./slowlog
299.9M	./slowlog-2020-02-26T02-10-16.452
299.9M	./slowlog-2020-02-26T02-11-33.170
299.9M	./slowlog-2020-02-26T02-12-27.762
299.9M	./slowlog-2020-02-26T02-13-44.140
299.9M	./slowlog-2020-02-26T02-16-08.322
299.9M	./slowlog-2020-02-26T02-18-27.989
300.0M	./slowlog-2020-02-26T02-22-49.095
300.0M	./slowlog-2020-02-26T02-26-00.591
299.9M	./slowlog-2020-02-26T02-29-03.809
299.9M	./slowlog-2020-02-26T02-31-16.241
299.9M	./slowlog-2020-02-26T02-32-11.446
300.0M	./slowlog-2020-02-26T02-33-12.131
300.0M	./slowlog-2020-02-26T02-34-16.872
299.9M	./slowlog-2020-02-26T02-35-14.501
299.9M	./slowlog-2020-02-26T02-38-52.075
300.0M	./slowlog-2020-02-26T02-43-13.551
299.9M	./slowlog-2020-02-26T02-49-34.426
299.9M	./slowlog-2020-02-26T02-53-30.816
299.9M	./slowlog-2020-02-26T02-54-38.124
299.9M	./slowlog-2020-02-26T02-55-52.087
300.0M	./slowlog-2020-02-26T03-02-16.277
299.9M	./slowlog-2020-02-26T03-07-40.269

[Yisaer]

@aylei @cofyc I found that the busybox we used in the tidb sidecar didn't have logrotate command. What's your opinion to logrotate slowlog file?

[cofyc]

the log file is already rotated by tidb periodically. is it possible to configure tidb to discard or limit the total size of rotated log files? otherwise, we need to delete them ourselves. we can use rm to remove them, the problem is we need to update the command of slow log tailer container which will trigger a rolling upgrade.

[aylei]

Yes, I misphrase the problem. I've not found any configurations about cleaning rotated logs

[Yisaer]

It seems we could use max-size and max-backup configuration in tidb-server. This would limit the total storage usage of the log.
ref: https://pingcap.com/docs-cn/stable/reference/configuration/tidb-server/configuration-file/#max-size
https://pingcap.com/docs-cn/stable/reference/configuration/tidb-server/configuration-file/#max-backups

[cofyc]

that's great! it's good practice to collect the slow logs to other places, e.g. elasticsearch. maybe we can limit the max-backup to a reasonable value in the cloud by default. this is the same as if we're going to clean the logs for the user.

[Yisaer]

Yes, I think maybe we could set a reasonable default value for max-size and max-backup for tidb considering the ephemeral storage.