-
Notifications
You must be signed in to change notification settings - Fork 3
Data Movement Operations 3x Daily Checklist
-
Look at Edinburgh FTS monitoring dashboard. Check for FTS3 failures
-
Check for Ingest and Declaration Daemon failures
If failures are seen in these plots then follow instructions in Checking Ingest Daemon works
If problems are seen in the Declaration Daemon follow instructions here Checking Declaration Daemon Works. In particular investigate if any files are quarantined.
-
Check quota dashboard to see if anything is close to quota. Edinburgh Quota Dashboard
-
Do following Metacat query:
metacat query --summary count 'files from dune:all where core.run_type=hd-protodune and core.file_type=detector and core.data_tier=raw and core.data_stream=physics and created_timestamp > 2024-06-19'
Would be nice to incorporate this query into the dashboard.
- Check messages for completed JustIN keep-up workflows
If there are any follow
- Check status of all rucio FNAL_DCACHE rules in flight that were created today, make sure none STUCK or SUSPENDED
rucio list-rules --account=dunepro | grep FNAL_DCACHE | grep -v OK | grep 'today's date'
-
Check the main FTS3 page. CERN FTS3 Main Status Page
-
Check the main Justin AWT Page, look for storage element failures. Justin AWT Testing Page
-
Check main Fermilab dCache page Fermilab dCache pools Check to make sure in the protoDUNE pools that we don't have a lot of precious files building up.