Skip to content
This repository has been archived by the owner on Nov 3, 2023. It is now read-only.

Record task start and end times for Acute-Evals #4208

Merged
merged 21 commits into from
Jan 6, 2022

Conversation

EricMichaelSmith
Copy link
Contributor

@EricMichaelSmith EricMichaelSmith commented Nov 24, 2021

Patch description
Record HIT start and end times for Acute-Evals so that Mephisto's DataBrowser can access them. This will allow the Acute-Eval analysis script to record the task completion times properly. NOTE: the actual recording of the start and end times is in Mephisto, as part of facebookresearch/Mephisto#636 : this PR bumps the crowdsourcing CI check to use that update.

Testing steps
Existing tests have been updated to handle this change: the tests will discount the 'time_taken' and 'task_start' columns, which are now (correctly) variable from one run to the next

Testing that HIT times are now saved properly:

python parlai/crowdsourcing/tasks/acute_eval/run.py
mkdir -p ${DUMMY_SAVE_FOLDER}
python parlai/crowdsourcing/tasks/acute_eval/analysis.py -ids ${PRINTED_RUN_ID} --root-dir ${DUMMY_SAVE_FOLDER}

@EricMichaelSmith EricMichaelSmith changed the base branch from main to EricMichaelSmith-patch-1 December 22, 2021 21:32
Base automatically changed from EricMichaelSmith-patch-1 to main December 22, 2021 21:41
@EricMichaelSmith EricMichaelSmith changed the title [WIP] Record task start and end times for Acute-Evals Record task start and end times for Acute-Evals Dec 22, 2021
@EricMichaelSmith EricMichaelSmith marked this pull request as ready for review December 22, 2021 22:08
Copy link
Contributor

@JackUrb JackUrb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Glad to have start and end times for these! These get_task_start and get_task_end functions are likely general enough to live in the underlying StaticAgentState though.

@EricMichaelSmith
Copy link
Contributor Author

EricMichaelSmith commented Jan 5, 2022

LGTM! Glad to have start and end times for these! These get_task_start and get_task_end functions are likely general enough to live in the underlying StaticAgentState though.

Good point @JackUrb - just opened up a PR for this at facebookresearch/Mephisto#636

@EricMichaelSmith EricMichaelSmith merged commit d23a623 into main Jan 6, 2022
@EricMichaelSmith EricMichaelSmith deleted the acute-task-times branch January 6, 2022 22:28
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants