Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix.all xtriggers on an itask are the same #5791

Merged

Conversation

wxtim
Copy link
Member

@wxtim wxtim commented Oct 25, 2023

Closes #5783

The task proxy object was caching the value of clock trigger time. Only the first xtrigger was checked!

Check List

  • I have read CONTRIBUTING.md and added my name as a Code Contributor.
  • Contains logically grouped changes (else tidy your branch by rebase).
  • Does not contain off-topic changes (use other PRs for other changes).
  • Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
  • Tests are included (or explain why tests are not needed).
  • CHANGES.md entry included if this is a change that can affect users
  • Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
  • If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

@wxtim wxtim self-assigned this Oct 25, 2023
@wxtim wxtim added this to the cylc-8.2.3 milestone Oct 25, 2023
@wxtim wxtim force-pushed the fix.all_xtriggers_on_an_itask_are_the_same branch from 239893e to e751dcb Compare October 25, 2023 15:00
@wxtim wxtim added the bug Something is wrong :( label Oct 25, 2023
Copy link
Member

@hjoliver hjoliver left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we had (wrongly) assumed a task would only ever have one clock trigger, so cache to avoid doing the datetime computation every time the trigger is checked.

With multiple clock triggers, it might still be worth caching all the values?

@wxtim
Copy link
Member Author

wxtim commented Oct 26, 2023

multiple clock triggers, it might still be worth caching all the values?

I have done this, although I'm not sure how much time the memoization saves.

@wxtim wxtim requested a review from hjoliver October 26, 2023 09:07
@oliver-sanders oliver-sanders removed the request for review from hjoliver October 26, 2023 09:08
Copy link
Member

@oliver-sanders oliver-sanders left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

cylc/flow/task_proxy.py Outdated Show resolved Hide resolved
tests/integration/test_xtrigger_mgr.py Outdated Show resolved Hide resolved
@wxtim wxtim force-pushed the fix.all_xtriggers_on_an_itask_are_the_same branch from 65fed81 to a99576c Compare October 26, 2023 10:03
@wxtim wxtim force-pushed the fix.all_xtriggers_on_an_itask_are_the_same branch from a99576c to fbc4c89 Compare October 26, 2023 12:38
@oliver-sanders oliver-sanders removed the request for review from hjoliver October 26, 2023 13:31
cylc/flow/task_proxy.py Outdated Show resolved Hide resolved
if offset_str is None:
trigger_time = self.point
# None cannot be used as a dict key:
offset_str = offset_str if offset_str else 'P0Y'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could avoid this line by just storing None in the clock_trigger_times dict?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, although I've never tried using None as a dictionary key before, and was somewhat surprised by it as a thing.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a bit more complex in practice - because if offset_str is in self.clock_trigger_times it makes the following logic invalid and you have to write a separate loop caching the conversion of P0Y into seconds since epoch. I ended up with this diff:

-        # None cannot be used as a dict key:
-        offset_str = offset_str if offset_str else 'P0Y'
-        if offset_str not in self.clock_trigger_times:
-            if offset_str == 'P0Y':
-                trigger_time = point
-            else:
-                trigger_time = point + ISO8601Interval(offset_str)
-
+        trigger_time = None
+        if (
+            offset_str is None
+            and self.clock_trigger_times[offset_str] == 'P0Y'
+        ):
+            trigger_time = point
+        elif offset_str not in self.clock_trigger_times:
+            trigger_time = point + ISO8601Interval(offset_str)
+        if trigger_time:

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

None is hashable so can be used as a dict key

Copy link
Member Author

@wxtim wxtim Nov 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

None is hashable so can be used as a dict key

Yes. TIL - still doesn't deal with the other problem.
Removed the plain wrong comment about using None as a dictionary key.

Copy link
Member

@MetRonnie MetRonnie Nov 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a bit more complex in practice - because if offset_str is in self.clock_trigger_times it makes the following logic invalid and you have to write a separate loop caching the conversion of P0Y into seconds since epoch. I ended up with this diff:

I'm not sure what you mean by this. What I am suggesting is:

diff --git a/cylc/flow/task_proxy.py b/cylc/flow/task_proxy.py
index ac1ec5c1c..1fcd91b71 100644
--- a/cylc/flow/task_proxy.py
+++ b/cylc/flow/task_proxy.py
@@ -248,5 +248,5 @@ class TaskProxy:
         self.non_unique_events = Counter()  # type: ignore # TODO: figure out
 
-        self.clock_trigger_times: Dict[str, int] = {}
+        self.clock_trigger_times: Dict[Optional[str], int] = {}
         self.expire_time: Optional[float] = None
         self.late_time: Optional[float] = None
@@ -370,15 +370,13 @@ class TaskProxy:
 
         """
-        # None cannot be used as a dict key:
-        offset_str = offset_str if offset_str else 'P0Y'
         if offset_str not in self.clock_trigger_times:
-            if offset_str == 'P0Y':
+            if not offset_str:
                 trigger_time = point
             else:
                 trigger_time = point + ISO8601Interval(offset_str)
 
-            offset = int(
-                point_parse(str(trigger_time)).seconds_since_unix_epoch)
-            self.clock_trigger_times[offset_str] = offset
+            self.clock_trigger_times[offset_str] = int(
+                point_parse(str(trigger_time)).seconds_since_unix_epoch
+            )
         return self.clock_trigger_times[offset_str]

Copy link
Member Author

@wxtim wxtim Nov 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Too late - Oliver's merged it! - And yes, I had misunderstood.

cylc/flow/task_proxy.py Outdated Show resolved Hide resolved
Copy link
Member

@hjoliver hjoliver left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Leaving umerged while @MetRonnie 's comments are open.

Co-authored-by: Ronnie Dutta <61982285+MetRonnie@users.noreply.github.com>
cylc/flow/task_proxy.py Outdated Show resolved Hide resolved
@oliver-sanders oliver-sanders merged commit b23a486 into cylc:8.2.x Nov 2, 2023
18 of 22 checks passed
@wxtim wxtim deleted the fix.all_xtriggers_on_an_itask_are_the_same branch November 2, 2023 11:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something is wrong :(
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Task that depends on multiple wall_clock xtriggers will submit when first is satisfied
4 participants