Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The results of selecting columns with regular expressions in the context of groupby are wrong on Polars 0.18.7. #9947

Closed
2 tasks done
Teamon9161 opened this issue Jul 18, 2023 · 1 comment · Fixed by #9952
Labels
bug Something isn't working python Related to Python Polars

Comments

@Teamon9161
Copy link

Checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import polars as pl

df = pl.DataFrame({'g':[3], 'abc': [1],'abcd': [3]})

print(df.groupby('g').agg(pl.col('^ab.*$')))

Issue description

The result of the above code is correct in Polars 0.18.6 and below, but the result is incorrect in version 0.18.7.
Correct output:

shape: (1, 3)
┌─────┬───────────┬───────────┐
│ gabcabcd      │
│ ---------       │
│ i64list[i64] ┆ list[i64] │
╞═════╪═══════════╪═══════════╡
│ 3   ┆ [1]       ┆ [3]       │
└─────┴───────────┴───────────┘

Wrong output

shape: (1, 1)
┌─────┐
│ g   │
│ --- │
│ i64 │
╞═════╡
│ 3   │
└─────┘

When I run df.select(pl.col('^ab.*$')) on Polars 0.18.7, the result is also correct.

shape: (1, 2)
┌─────┬──────┐
│ abcabcd │
│ ------  │
│ i64i64  │
╞═════╪══════╡
│ 13    │
└─────┴──────┘

Expected behavior

See above

Installed versions

polars 0.18.6

--------Version info---------
Polars:              0.18.6
Index type:          UInt32
Platform:            Windows-10-10.0.22621-SP0
Python:              3.10.11 | packaged by Anaconda, Inc. | (main, Apr 20 2023, 18:56:50) [MSC v.1916 64 bit (AMD64)]

----Optional dependencies----
adbc_driver_sqlite:  <not installed>
connectorx:          0.3.1
deltalake:           0.9.0
fsspec:              2023.5.0
matplotlib:          3.7.1
numpy:               1.24.3
pandas:              1.5.3
pyarrow:             12.0.0
pydantic:            <not installed>
sqlalchemy:          2.0.15
xlsx2csv:            0.8.1
xlsxwriter:          3.1.2

polars 0.18.7

--------Version info---------
Polars:              0.18.7
Index type:          UInt32
Platform:            Windows-10-10.0.22621-SP0
Python:              3.10.11 | packaged by Anaconda, Inc. | (main, Apr 20 2023, 18:56:50) [MSC v.1916 64 bit (AMD64)]

----Optional dependencies----
adbc_driver_sqlite:  <not installed>
connectorx:          0.3.1
deltalake:           0.9.0
fsspec:              2023.5.0
matplotlib:          3.7.1
numpy:               1.24.3
pandas:              1.5.3
pyarrow:             12.0.0
pydantic:            <not installed>
sqlalchemy:          2.0.15
xlsx2csv:            0.8.1
xlsxwriter:          3.1.2
@Teamon9161 Teamon9161 added bug Something isn't working python Related to Python Polars labels Jul 18, 2023
@cmdlineluser
Copy link
Contributor

cmdlineluser commented Jul 18, 2023

Looks like it was introduced when fixing .exclude() #9827

It appears that the block on L148 is never entered.

[/Users/user/git/polars/polars/polars-lazy/polars-plan/src/logical_plan/projection.rs:150] exclude = {
    "g",
}
[/Users/user/git/polars/polars/polars-lazy/polars-plan/src/logical_plan/projection.rs:153] e = col("^ab.*$")

// iterate until we find the Exclude node
// we remove that node from the expression
for e in expr.into_iter() {
if let Expr::Exclude(e, _) = e {

So it looks like groupby keys are added to exclude meaning it's possible for expr to not contain any exclude nodes?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working python Related to Python Polars
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants