Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize ID/tokens code #5321

Merged
merged 1 commit into from
Jan 24, 2023
Merged

Optimize ID/tokens code #5321

merged 1 commit into from
Jan 24, 2023

Conversation

MetRonnie
Copy link
Member

@MetRonnie MetRonnie commented Jan 23, 2023

Profiling revealed Tokens computed properties and detokenise() were being called many times and so was acting as a bit of a bottleneck (within the bottleneck of data store manager's increment_graph_window() - #5315).

  • Remove a couple of unnecessary nested comprehensions
  • Precompute certain sets

Partially addresses #5315 & #5183

Context

Click to see `flow.cylc`
#!Jinja2

{% set TASKS = TASKS | default(100) %}
{% set CYCLES = CYCLES | default(1) %}
{% set SLEEP = SLEEP | default(1) %}

[meta]
    description = """
        Example scaling workflow with {{ (TASKS * 2) + 2 }} tasks and {{ (TASKS ** 2) + (TASKS * 2) }}
        dependencies per cycle which runs for {{ CYCLES }} cycles.
    """

[task parameters]
    x = 1..{{ TASKS }}
    y = 1..{{ TASKS }}

[scheduling]
    cycling mode = integer
    initial cycle point = 1
    final cycle point = {{ CYCLES }}
    [[graph]]
        P1 = """
            d[-P1] => a => b<x> => c<y> => d
        """

[runtime]
    [[b<x>, c<y>]]
        script = sleep {{ SLEEP }}
    [[a, d]]
$ cylc play efficiency/ --profile -s 'TASKS=16'

c1

Expanded to edge_id:

c2

After

c3

Check list

  • I have read CONTRIBUTING.md and added my name as a Code Contributor.
  • Contains logically grouped changes (else tidy your branch by rebase).
  • Does not contain off-topic changes (use other PRs for other changes).
  • No dependency changes
  • No new tests needed
  • No changelog entry - presumably will be part of a bunch of efficiency improvements for 8.1.1
  • No docs
  • If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

@MetRonnie MetRonnie added small efficiency For notable efficiency improvements labels Jan 23, 2023
@MetRonnie MetRonnie added this to the 8.1.1 milestone Jan 23, 2023
@MetRonnie MetRonnie self-assigned this Jan 23, 2023
Copy link
Member

@oliver-sanders oliver-sanders left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, 16% passive speedup for the given example!

cylc/flow/id.py Outdated Show resolved Hide resolved
Profiling revealed `Tokens.relative_id()` was being called many times and was acting as a bit of a bottleneck.

- Remove a couple of unecessary nested comprehensions
- Precompute certain sets
Copy link
Member

@hjoliver hjoliver left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, nice speedup 👍

@hjoliver hjoliver merged commit 5d7a225 into cylc:8.1.x Jan 24, 2023
@MetRonnie MetRonnie deleted the tokens branch January 24, 2023 09:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
efficiency For notable efficiency improvements small
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants