Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(any|all)_horizontal evaluate predicates that expand to zero expressions as true #17515

Closed
2 tasks done
cmdlineluser opened this issue Jul 9, 2024 · 2 comments
Closed
2 tasks done
Labels
bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars

Comments

@cmdlineluser
Copy link
Contributor

cmdlineluser commented Jul 9, 2024

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import polars as pl

df = pl.DataFrame({"foo": [1, 2]})

df.filter(pl.all_horizontal(pl.col(pl.String)))

# shape: (2, 1)
# ┌─────┐
# │ foo │
# │ --- │
# │ i64 │
# ╞═════╡
# │ 1   │
# │ 2   │
# └─────┘

Log output

No response

Issue description

I'm not sure if this is a "bug", but it was a bit surprising.

Polars reports a ComputeError for this example as pl.col(pl.String) matches 0 columns:

df.filter(pl.col(pl.String) == 'hi')
ComputeError: The predicate expanded to zero expressions. 
This may for example be caused by a regex not matching column names 
or a column dtype match not hitting any dtypes in the DataFrame

But introducing any/all horizontal ends up evaluating to true so all rows are returned:

df.filter(pl.any_horizontal(pl.col(pl.String) == 'hi'))
# shape: (2, 1)
# ┌─────┐
# │ foo │
# │ --- │
# │ i64 │
# ╞═════╡
# │ 1   │
# │ 2   │
# └─────┘

df.filter(pl.all_horizontal(pl.col(pl.String) == 'hi'))
# shape: (2, 1)
# ┌─────┐
# │ foo │
# │ --- │
# │ i64 │
# ╞═════╡
# │ 1   │
# │ 2   │
# └─────┘

Expected behavior

Should this raise the same ComputeError?

Installed versions

--------Version info---------
Polars:               1.1.0
Index type:           UInt32
Platform:             macOS-13.6.1-arm64-arm-64bit
Python:               3.12.2 (main, Feb  6 2024, 20:19:44) [Clang 15.0.0 (clang-1500.1.0.2.5)]

----Optional dependencies----
adbc_driver_manager:  <not installed>
cloudpickle:          <not installed>
connectorx:           <not installed>
deltalake:            <not installed>
fastexcel:            <not installed>
fsspec:               <not installed>
gevent:               <not installed>
great_tables:         <not installed>
hvplot:               <not installed>
matplotlib:           <not installed>
nest_asyncio:         <not installed>
numpy:                1.26.4
openpyxl:             <not installed>
pandas:               2.2.1
pyarrow:              15.0.2
pydantic:             <not installed>
pyiceberg:            <not installed>
sqlalchemy:           <not installed>
torch:                <not installed>
xlsx2csv:             <not installed>
xlsxwriter:           <not installed>
@cmdlineluser cmdlineluser added bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars labels Jul 9, 2024
@mcrumiller
Copy link
Contributor

mcrumiller commented Jul 9, 2024

I would think any_horizontal evaluates to False and all_horizontal evaluates to True, in the same way that Python evaluates these on empty lists:

>>> any([])
False
>>> all([])
True

Rust works the same way:

  • any: "An empty iterator returns false."
  • all: "An empty iterator returns true."

@cmdlineluser
Copy link
Contributor Author

Fixed by #18545

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars
Projects
None yet
Development

No branches or pull requests

2 participants