Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(python): ~30% faster iter_rows(named=True) and to_dicts(), if pyarrow available #6493

Merged
merged 1 commit into from
Jan 27, 2023

Conversation

alexander-beedie
Copy link
Collaborator

@alexander-beedie alexander-beedie commented Jan 27, 2023

This is on top of the previous 10% speedup for to_dicts, but only if pyarrow is available. Speedup varies with dict composition but is always significant (I'm seeing 25-45% gains with a freshly-compiled release build).

Also:

  • Repointed to_dicts internally to use iter_rows(named=True).
  • Added some Struct coverage to the iter_rows unit test.

The degree of speedup makes me think I should look at generating the iter-values down in Rust with pyo3 after all; python loops are notoriously slow compared to lower-level languages, and it seems there is some low-hanging fruit here...

@github-actions github-actions bot added enhancement New feature or an improvement of an existing feature python Related to Python Polars labels Jan 27, 2023
@ritchie46
Copy link
Member

The degree of speedup makes me think I should look at generating the iter-values down in Rust with pyo3 after all; python loops are notoriously slow compared to lower-level languages, and it seems there is some low-hanging fruit here...

Yeap, we can construct the whole python objects on the rust side with pyo3. We already implemented traits for the conversion so this should be fairly simple.

@ritchie46 ritchie46 merged commit 3271e14 into pola-rs:master Jan 27, 2023
@alexander-beedie alexander-beedie deleted the iterrows-speedup branch January 30, 2023 17:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or an improvement of an existing feature python Related to Python Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants