Long running stability

[yahoNanJing]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

To make Ballista able for long running without manually intervention, there are still something left, especially for cleaning up:

• Log
  • Rolling Add a config for tracing log rolling policy for both scheduler and executor #486, Add a config for tracing log rolling policy for both scheduler and executor #487
  • Retention [feat] Enable enable delete log periodically. #280, [arrow-ballista] enable delete log periodically (#280) #285
• Shuffle data
  • Add some validation for the remove_job_data grpc service #467, Add some validation for remove_job_data in the executor server #468
  • Periodically cleanup Need clean up intermediate data in Ballista #9, Cleanup job/stage status from TaskManager and clean up shuffle data after a period after JobFinished #185, Clean up job data on both Scheduler and Executor #188
  • Make the delayed time interval for cleanup job data in both scheduler and executor configurable #469, Add SchedulerConfig for the scheduler configurations, like event_loop_buffer_size, finished_job_data_clean_up_interval_seconds, finished_job_state_clean_up_interval_seconds #472
  • Add grpc service for the scheduler to make it able to be triggered by client explicitly #458, Add grpc service of cleaning up job shuffle data for the scheduler to make it able to be triggered by client explicitly #485
  • Clean up legacy job shuffle data #459, Enable periodic cleanup of work_dir directories in ballista executor datafusion#1783, Remove executor config executor_cleanup_enable and make the configuation name for executor cleanup more intuitive #489
• Job metadata
  • Periodically cleanup Cleanup job/stage status from TaskManager and clean up shuffle data after a period after JobFinished #185, Clean up job data on both Scheduler and Executor #188
  • Backup historical ones to the remote shared storage

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.