Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ComputeError / panic on projection pushdown using .count() #11841

Closed
2 tasks done
nameexhaustion opened this issue Oct 19, 2023 · 0 comments · Fixed by #11843
Closed
2 tasks done

ComputeError / panic on projection pushdown using .count() #11841

nameexhaustion opened this issue Oct 19, 2023 · 0 comments · Fixed by #11843
Labels
bug Something isn't working python Related to Python Polars

Comments

@nameexhaustion
Copy link
Collaborator

nameexhaustion commented Oct 19, 2023

Checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import polars as pl

(
    pl.LazyFrame(dict(x=1))
    .select(records=pl.count())
    .select(pl.lit(1).alias("x"), pl.all())
    .collect()
)

Log output

polars.exceptions.ComputeError: column 'records' not available in schema Schema:
name: x, data type: Int64

Issue description

Originally came across this when running a long lazy query that led to a panic pyo3_runtime.PanicException: called Result::unwrap()on anErr value: ColumnNotFound(ErrString("records")). When reducing to the provided MRE, the error instead becomes a ComputeError.

This is the panic backtrace of the original query, but it's likely the issue happens much earlier from where the panic happens. The records column is created via .select(records=pl.count()) (in a similar fashion to the one in the MRE):

Panic backtrace
thread '<unnamed>' panicked at /home/runner/work/polars/polars/crates/polars-plan/src/utils.rs:384:68:
called `Result::unwrap()` on an `Err` value: ColumnNotFound(ErrString("records"))
stack backtrace:
   0: rust_begin_unwind
             at /rustc/475c71da0710fd1d40c046f9cee04b733b5b2b51/library/std/src/panicking.rs:597:5
   1: core::panicking::panic_fmt
             at /rustc/475c71da0710fd1d40c046f9cee04b733b5b2b51/library/core/src/panicking.rs:72:14
   2: core::result::unwrap_failed
             at /rustc/475c71da0710fd1d40c046f9cee04b733b5b2b51/library/core/src/result.rs:1653:5
   3: polars_plan::utils::aexprs_to_schema
   4: polars_plan::logical_plan::builder_alp::ALogicalPlanBuilder::project
   5: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::push_down
   6: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::pushdown_and_assign
   7: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::push_down
   8: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::pushdown_and_assign
   9: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::push_down
  10: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::pushdown_and_assign
  11: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::push_down
  12: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::pushdown_and_assign
  13: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::push_down
  14: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::pushdown_and_assign
  15: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::push_down
  16: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::pushdown_and_assign
  17: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::push_down
  18: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::pushdown_and_assign
  19: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::push_down
  20: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::pushdown_and_assign
  21: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::push_down
  22: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::pushdown_and_assign
  23: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::push_down
  24: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::pushdown_and_assign
  25: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::push_down
  26: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::pushdown_and_assign
  27: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::push_down
  28: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::pushdown_and_assign
  29: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::push_down
  30: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::pushdown_and_assign
  31: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::push_down
  32: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::pushdown_and_assign
  33: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::push_down
  34: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::pushdown_and_assign
  35: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::push_down
  36: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::pushdown_and_assign
  37: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::push_down
  38: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::pushdown_and_assign
  39: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::push_down
  40: core::iter::adapters::map::map_try_fold::{{closure}}
  41: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::no_pushdown_restart_opt
  42: polars_plan::logical_plan::optimizer::projection_pushdown::functions::melt::process_melt
  43: polars_plan::logical_plan::optimizer::projection_pushdown::ProjectionPushDown::push_down
  44: polars_plan::logical_plan::optimizer::optimize
  45: polars_lazy::frame::LazyFrame::optimize_with_scratch
  46: polars_lazy::frame::LazyFrame::prepare_collect
  47: polars_lazy::frame::LazyFrame::collect
  48: polars::lazyframe::_::<impl polars::lazyframe::PyLazyFrame>::__pymethod_collect__
  49: pyo3::impl_::trampoline::trampoline
  50: _PyEval_EvalFrameDefault
             at /tmp/python-build.20231011043018.472516/Python-3.11.6/Python/ceval.c:5284:29
  51: _PyEval_EvalFrame
             at /tmp/python-build.20231011043018.472516/Python-3.11.6/./Include/internal/pycore_ceval.h:73:16
  52: _PyEval_Vector
             at /tmp/python-build.20231011043018.472516/Python-3.11.6/Python/ceval.c:6425:24
  53: _PyVectorcall_Call
             at /tmp/python-build.20231011043018.472516/Python-3.11.6/Objects/call.c:245:16
  54: _PyObject_Call
             at /tmp/python-build.20231011043018.472516/Python-3.11.6/Objects/call.c:328:16
  55: do_call_core
             at /tmp/python-build.20231011043018.472516/Python-3.11.6/Python/ceval.c:7343:12
  56: _PyEval_EvalFrameDefault
             at /tmp/python-build.20231011043018.472516/Python-3.11.6/Python/ceval.c:5367:22
  57: _PyEval_EvalFrame
             at /tmp/python-build.20231011043018.472516/Python-3.11.6/./Include/internal/pycore_ceval.h:73:16
  58: _PyEval_Vector
             at /tmp/python-build.20231011043018.472516/Python-3.11.6/Python/ceval.c:6425:24
  59: PyEval_EvalCode
             at /tmp/python-build.20231011043018.472516/Python-3.11.6/Python/ceval.c:1140:21
  60: run_eval_code_obj
             at /tmp/python-build.20231011043018.472516/Python-3.11.6/Python/pythonrun.c:1710:9
  61: run_mod
             at /tmp/python-build.20231011043018.472516/Python-3.11.6/Python/pythonrun.c:1731:19
  62: pyrun_file
             at /tmp/python-build.20231011043018.472516/Python-3.11.6/Python/pythonrun.c:1626:15
  63: _PyRun_SimpleFileObject
             at /tmp/python-build.20231011043018.472516/Python-3.11.6/Python/pythonrun.c:440:13
  64: _PyRun_AnyFileObject
             at /tmp/python-build.20231011043018.472516/Python-3.11.6/Python/pythonrun.c:79:15
  65: pymain_run_file_obj
             at /tmp/python-build.20231011043018.472516/Python-3.11.6/Modules/main.c:360:15
  66: pymain_run_file
             at /tmp/python-build.20231011043018.472516/Python-3.11.6/Modules/main.c:379:15
  67: pymain_run_python
             at /tmp/python-build.20231011043018.472516/Python-3.11.6/Modules/main.c:601:21
  68: Py_RunMain
             at /tmp/python-build.20231011043018.472516/Python-3.11.6/Modules/main.c:680:5
  69: pymain_main
             at /tmp/python-build.20231011043018.472516/Python-3.11.6/Modules/main.c:710:12
  70: Py_BytesMain
             at /tmp/python-build.20231011043018.472516/Python-3.11.6/Modules/main.c:734:12
  71: __libc_start_main
  72: _start
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
...
pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: ColumnNotFound(ErrString("records"))

Expected behavior

The MRE should provide the following output instead of erroring (it works if projection_pushdown is set to False):

shape: (1, 2)
┌─────┬─────────┐
│ xrecords │
│ ------     │
│ i32u32     │
╞═════╪═════════╡
│ 11       │
└─────┴─────────┘

Installed versions

--------Version info---------
Polars:              0.19.9
Index type:          UInt32
Platform:            Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.31
Python:              3.11.6 (main, Oct  4 2023, 13:44:03) [GCC 9.4.0]

----Optional dependencies----
adbc_driver_sqlite:  <not installed>
cloudpickle:         <not installed>
connectorx:          <not installed>
deltalake:           <not installed>
fsspec:              2023.9.2
gevent:              <not installed>
matplotlib:          3.8.0
numpy:               1.26.0
openpyxl:            <not installed>
pandas:              2.1.1
pyarrow:             13.0.0
pydantic:            2.4.2
pyiceberg:           <not installed>
pyxlsb:              <not installed>
sqlalchemy:          <not installed>
xlsx2csv:            <not installed>
xlsxwriter:          3.1.6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working python Related to Python Polars
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant