Python Polars 0.20.27

Warning

This release was yanked. Please use the 0.20.28 release instead.

⚠️ Deprecations

• Change parameter chunked to allow_chunks in parametric testing strategies (#16264)

🚀 Performance improvements

• Use branchless uleb128 decoding for parquet (#16352)
• Reduce error bubbling in parquet hybrid_rle (#16348)
• use is_sorted in ewm_mean_by, deprecate check_sorted (#16335)
• Optimize is_sorted for numeric data (#16333)
• do not use pyo3-built (#16309)
• Faster bitpacking for Parquet writer (#16278)
• Improve Series.to_numpy performance for chunked Series that would otherwise be zero-copy (#16301)
• Further optimise initial polars import (#16308)
• Avoid importing ctypes.util in CPU check script if possible (#16307)
• Don't rechunk when converting DataFrame to numpy/ndarray (#16288)
• use zeroed vec in ewm_mean_by for sorted fastpath (#16265)

✨ Enhancements

• expose BooleanFunction in expr IR (#16355)
• Allow read_excel to handle bytes/BytesIO directly when using the "calamine" (fastexcel) engine (#16344)
• Raise when joining on the same keys twice (#16329)
• Don't require data to be sorted by by column in rolling_*_by operations (#16249)
• Support List types in Series.to_numpy (#16315)
• Add to_jax methods to support Jax Array export from DataFrame and Series (#16294)
• Enable generating data with time zones in parametric testing (#16298)
• Add struct.field expansion (regex, wildcard, columns) (#16320)
• Add new alpha, alphanumeric and digit selectors (#16310)
• Faster bitpacking for Parquet writer (#16278)
• Add require_all parameter to the by_name column selector (#15028)
• Start updating BytecodeParser for Python 3.13 (#16304)
• Add struct.with_fields (#16305)
• Handle implicit SQL string → temporal conversion in the BETWEEN clause (#16279)
• Expose string expression nodes to python (#16221)
• Add new index/range based selector cs.by_index, allow multiple indices for nth (#16217)
• Show warning if expressions are very deep (#16233)
• Fix some issues in parametric testing with nested dtypes (#16211)

🐞 Bug fixes

• pick a consistent order for the sort options in PyIR (#16350)
• Infer CSV schema as supertype of all files (#16349)
• Fix issue in parametric testing where excluded_dtypes list would grow indefinitely (#16340)
• Address overflow combining u64 hashes in Debug builds (#16323)
• Don't exclude explicitly named columns in group-by context' expr expansion (#16318)
• Improve map_elements typing (#16257)
• Harden Series.reshape against invalid parameters (#16281)
• Fix list.sum dtype for boolean (#16290)
• Don't stackoverflow on all/any horizontal (#16287)
• Fix Series.to_numpy for Array types with nulls and nested Arrays (#16230)
• `rolling_*_by was throwing incorrect error when dataframe was sorted by contained multiple chunks (#16247)
• Don't allow passing missing data to generalized ufuncs (#16198)
• Address overly-permissive expand_selectors function, minor fixes (#16250)
• Add missing support for parsing instantiated Object dtypes Object() (#16260)
• Reading CSV with low_memory gave no data (#16231)
• Add missing read_database overload (#16229)
• Fix a rounding error in parametric test datetimes generation (#16228)
• Fix some issues in parametric testing with nested dtypes (#16211)

📖 Documentation

• Add missing word in join docstring (#16299)
• document that month_start/month_end preserve the current time (#16293)
• Add example for separator parameter in pivot (#15957)

📦 Build system

• Fix allocator features (#16365)
• Update Rust nightly toolchain version (#16222)

🛠️ Other improvements

• Move DataFrame.to_numpy implementation to Rust side (#16354)
• Organize PyO3 NumPy code into interop::numpy module (#16346)
• simplify interpolate code, add test for rolling_*_by with nulls (#16334)
• Very minor refactor of DataFrame.to_numpy code (#16325)
• InterchangeDataFrame.version should be a ClassVar (not a property) (#16312)
• Add polars-expr README (#16316)
• Raise import timing test threshold (#16302)
• Use cls (not self) in classmethods (#16303)
• Use Scalar instead of Series some aggregations (#16277)
• Do not hardcode bash path in Makefile (#16263)
• Add IR::Reduce (not yet implemented) (#16216)
• move all describe, describe_tree and dot-viz code to IR instead of DslPlan (#16237)

Thank you to all our contributors for making this release possible!
@MarcoGorelli, @NickCondron, @ShivMunagala, @alexander-beedie, @brandon-b-miller, @coastalwhite, @datenzauberai, @itamarst, @jsarbach, @max-muoto, @nameexhaustion, @orlp, @r-brink, @ritchie46, @stinodego, @thalassemia, @twoertwein and @wence-

Python Polars 0.20.26

⚠️ Deprecations

• Deprecate allow_infinities and null_probability args to parametric test strategies (#16183)

🚀 Performance improvements

• Avoid needless copy when converting chunked Series to NumPy (#16178)
• use zeroable_vec in ewm_mean_by (#16166)
• Improve cost of chunk_idx compute (#16154)
• Don't rechunk by default in concat (#16128)
• Ensure rechunk is parallel (#16127)

✨ Enhancements

• Clarify to_torch "features" and "label" parameter behaviour when return type is not "dataset" (#16218)
• Native CSV file list reading (#16180)
• Register memory mapped files and raise when written to (#16208)
• Implement support for Struct types in parametric tests (#16197)
• Enable Null datatype and null values by default in parametric testing (#16192)
• Support Enum types in parametric testing (#16188)
• Raise when encountering invalid supertype in functions during conversion (#16182)
• Add SQL support for GROUP BY ALL syntax and fix several issues with aliased group keys (#16179)
• Overhaul parametric test implementations and update Hypothesis to latest version (#16062)
• Avoid an extra copy when converting Boolean Series to writable NumPy array (#16164)
• Allow implicit string → temporal conversion in SQL comparisons (#15958)
• Add run-length encoding to Parquet writer (#16125)
• Support passing instantiated adbc/alchemy connection objects to write_database (#16099)
• Add top-level nth(n) method, to go with existing first and last (#16112)

🐞 Bug fixes

• Empty unique (#16214)
• Fix empty drop nulls (#16213)
• Fix get expression group-by state (#16189)
• Fix rolling empty group OOB (#16186)
• offset=-0i was being treated differently to offset=0i in rolling (#16184)
• Fix panic on empty frame joins (#16181)
• Fix streaming glob slice (#16174)
• Fix CSV skip_rows_after_header for streaming (#16176)
• Flush parquet at end of batches tick (#16073)
• Check CSE name aliases for collisions. (#16149)
• Don't override CSV reader encoding with lossy UTF-8 (#16151)
• Respect dtype and strict in pl.Series's constructor for pyarrow arrays, numpy arrays, and pyarrow-backed pandas (#15962)
• Ensure hex and bitstring literals work inside SQL IN clauses (#16101)

📖 Documentation

• Add examples for multiple Series functions (#16172)
• Add deprecated messages to cumfold and cumreduce (#16173)
• StringNameSpace.replace_all docstring (#16169)
• Explain how Polars floor division differs from Python (#16054)
• Clarify arrow usage (#16152)
• Add example to polars.Series.flags (#16123)

🛠️ Other improvements

• Switch over some of the custom Python date/time conversions to native PyO3 conversions (#16203)
• move describe to IR instead of DSL (#16191)
• More PyO3 0.21 Bound<> APIs, and finally disable gil-refs backwards compat feature on pyo3 crate (#16143)
• Move hypothesis tests into unit test module (#16185)
• Remove unused code (#16175)
• Update plugin example to PyO3 0.21 (#16157)
• Return correct temporal type from Rust in to_numpy (#14353)
• Create NumPy view for Datetime/Duration Series on the Rust side (#16148)
• Don't override CSV reader encoding with lossy UTF-8 (#16151)
• Continue converting to PyO3 0.21 Bound<> APIs (#16081)

Thank you to all our contributors for making this release possible!
@MarcoGorelli, @YichiZhang0613, @alexander-beedie, @bertiewooster, @coastalwhite, @dangotbanned, @itamarst, @janpipek, @jrycw, @luke396, @nameexhaustion, @pydanny, @ritchie46, @stinodego, @thalassemia and @tharunsuresh-code

Python Polars 0.20.25

🐞 Bug fixes

• Revert "Add RLE to RLE_DICTIONARY encoder"
• Improve error handling of ParameterCollisionError in read_excel (#16100)

Thank you to all our contributors for making this release possible!
@nameexhaustion, @ritchie46 and @wsyxbcl

Python Polars 0.20.24

Warning

This release was yanked. Please use the 0.20.25 release instead.

🏆 Highlights

• Support pytorch Tensor and Dataset export with new to_torch DataFrame/Series method (#15931)

🚀 Performance improvements

• Don't traverse deep datasets that we repr as union in CSE (#16096)
• Ensure better chunk sizes (#16071)

✨ Enhancements

• split out rolling_*(..., by='foo') into rolling_*_by('foo', ...) (#16059)
• add date pattern dd.mm.YYYY (#16045)
• split Expr.top_k and Expr.top_k_by into separate functions (#16041)
• Support non-coalescing joins in default engine (#16036)
• Support pytorch Tensor and Dataset export with new to_torch DataFrame/Series method (#15931)
• Minor DB type inference updates (#16030)
• Move diagonal & horizontal concat schema resolving to IR phase (#16034)
• raise more informative error messages in rolling_* aggregations instead of panicking (#15979)
• Convert concat during IR conversion (#16016)
• Improve dynamic supertypes (#16009)
• Additional uint datatype support for the SQL interface (#15993)
• Add post-optimization callback (#15972)
• Support Decimal read from IPC (#15965)
• Expose plan and expression nodes through NodeTraverser to Python (#15776)
• Add typed collection from par iterators (#15961)
• Add by argument for Expr.top_k and Expr.bottom_k (#15468)

🐞 Bug fixes

• Respect user passed 'reader_schema' in 'scan_csv' (#16080)
• Lazy csv + projection; respect null values arg (#16077)
• Materialize dtypes when converting to arrow (#16074)
• Fix casting decimal to decimal for high precision (#16049)
• Fix Series constructor failure for Array types for large integers (#16050)
• Fix printing max scale decimals (#16048)
• Decimal supertype for dyn int (#16046)
• Correctly handle large timedelta objects in Series constructor (#16043)
• Do not close connection just because we're not returning Arrow data in batches (#16031)
• properly handle nulls in DictionaryArray::iter_typed (#16013)
• Fix CSE case where upper plan has no projection (#16011)
• Crash/incorrect group_by/n_unique on categoricals created by (q)cut (#16006)
• converting from numpy datetime64 and overriding dtype with a different resolution was returning incorrect results (#15994)
• Ternary supertype dynamics (#15995)
• Fix PartialEq for DataType::Unknown (#15992)
• Finish adding typed_lit to help schema determination in SQL "extract" func (#15955)
• Fix dtype parameter in pandas_to_pyseries function (#15948)
• do not panic when comparing against categorical with incompatible dtype (#15857)
• Join validation for multiple keys (#15947)
• Add missing "truncate_ragged_lines" parameter to read_csv_batched (#15944)

📖 Documentation

• Ensure consistent docstring warning in fill_nan methods (pointing out that nan isn't null) (#16061)
• add filter docstring examples to date and datetime (#15996)
• Fix docstring mistake for polars.concat_str (#15937)
• Update reference to apply (#15982)
• Remove unwanted linebreaks from docstrings (#16002)
• correct default in rolling_* function examples (#16000)
• Improve user-guide doc of UDF (#15923)
• update the link to R API docs (#15973)

🛠️ Other improvements

• Bump sccache action (#16088)
• Fix failures in test coverage workflow (#16083)
• Update benchmarks/coverage jobs with "requirements-ci" (#16072)
• Add TypeGuard to is_polars_dtype util (#16065)
• Clean up hypothesis decimal strategy (#16056)
• split Expr.top_k and Expr.top_k_by into separate functions (#16041)
• Use UnionArgs for DSL side (#16017)
• Add some comments (#16008)
• Improve hypothesis strategy for decimals (#16001)
• Set up TPC-H benchmark tests (#15908)
• Even more Pyo3 0.21 Bound<> APIs (#15914)
• Fix failing test (#15936)

Thank you to all our contributors for making this release possible!
@CanglongCl, @JulianCologne, @KDruzhkin, @MarcoGorelli, @alexander-beedie, @avimallu, @bertiewooster, @c-peters, @dependabot, @dependabot[bot], @eitsupi, @haocheng6, @itamarst, @luke396, @marenwestermann, @nameexhaustion, @orlp, @ritchie46, @stinodego, @thalassemia, @wence- and @wsyxbcl

Python Polars 0.20.23

🏆 Highlights

• Add a low-friction sql method for DataFrame and LazyFrame (#15783)

🚀 Performance improvements

• Don't rechunk in parallel collection (#15907)
• Improve non-trivial list aggregations (#15888)
• Ensure we hit specialized gather for binary/strings (#15886)
• Limit the cache size for to_datetime (#15826)
• skip initial null items and don't recompute slope in interpolate (#15819)

✨ Enhancements

• don't require pyarrow for converting pandas to Polars if all columns have simple numpy-backed datatypes (#15933)
• Add option to disable globbing in csv (#15930)
• Add option to disable globbing in parquet (#15928)
• Expressify dt.round (#15861)
• Improve error messages in context stack (#15881)
• Add dynamic literals to ensure schema correctness (#15832)
• add timestamp time travel in delta scan/read (#15813)

🐞 Bug fixes

• Set default limit for String column display to 30 and fix edge cases (#15934)
• Change recognition of numba ufunc (#15916)
• series.search_sorted could support more types of input (#15940)
• Remove ffspec from parquet reader (#15927)
• avoid WRITE+EXEC for CPUID check (#15912)
• fix inconsistent decimal formatting (#15457)
• Preserve NULLs for is_not_nan (#15889)
• double projection check should only take the upstream projections into account (#15901)
• Ensure we don't create invalid frames when combining unit lit + … (#15903)
• Clear cached rename schema (#15902)
• Fix OOB in struct lit/agg aggregation (#15891)
• Refine interaction of "schema_overrides" with read_excel when using "calamine" engine (#15827)
• Don't modify user-supplied storage_options dict (take a shallow-copy) (#15859)
• create (q)cut labels in fixed order (#15843)
• Tag shrink_dtype as non-streaming (#15828)

📖 Documentation

• improve graphviz install documentation/error message (#15791)
• Extend docstring examples for asof_join (#15810)

📦 Build system

• Don't import jemalloc (#15942)
• Use default allocator for lts-cpu (#15941)
• replace all macos-latest referrals with macos-13 (#15926)
• pin mimalloc and macos-13 (#15925)
• use jemalloc in lts-cpu (#15913)
• Update Cargo.lock (#15865)
• Bump ruff version and improve make clean on the Python side (#15858)
• Exclude rust-toolchain.toml from wheels (#15840)

🛠️ Other improvements

• Replace copy/paste import handling with import_optional utility function (#15906)
• Reorganize from_iter and dispatch to collect_ca when possible (#15904)
• More PyO3 0.21 bound APIs (#15872)
• Improve type-coercion (#15879)
• Move type coercion to IR conversion phase (#15868)
• Fix Python test coverage upload (#15853)
• More upgrades to PyO3 0.21 Bound<> APIs (#15790)
• Use uv for installing Python dependencies in CI (#15848)
• Update benchmark tests (#15825)

Thank you to all our contributors for making this release possible!
@JulianCologne, @MarcoGorelli, @NedJWestern, @NexVeridian, @alexander-beedie, @deanm0000, @dependabot, @dependabot[bot], @ion-elgreco, @itamarst, @jr200, @nameexhaustion, @orlp, @reswqa, @ritchie46 and @stinodego

Python Polars 0.20.22

🚀 Performance improvements

• Improved type-inference for read_excel and read_ods, use calamine engine for read_ods (#15808)
• Fix quadratic in binview growable same source (#15734)
• use two binary searches for equality mask when data is sorted (#15702)
• improve filter parallelism (#15686)

✨ Enhancements

• Minor type-inference update for read_database (#15809)
• Improved type-inference for read_excel and read_ods, use calamine engine for read_ods (#15808)
• dt.truncate supports broadcasting lhs (#15768)
• Expressify str.json_path_match (#15764)
• raise if storage_options is passed to read_csv but fsspec isnt available (#15778)
• Support decimal float parsing in CSV (#15774)
• Add context trace to LazyFrame conversion errors (#15761)
• Improve error message when passing invalid input to lit (#15718)
• Remove outdated join validation checks (#15701)

🐞 Bug fixes

• drop-nulls edge case; remove drop-nulls special case (#15815)
• ewm_mean_by was skipping initial nulls when it was already sorted by "by" column (#15812)
• Consult cgroups to determine free memory (#15798)
• raise if index count like 2i is used when performing rolling, group_by_dynamic, upsample, or other temporal operatios (#15751)
• Don't deduplicate sort that has slice pushdown (#15784)
• Allow passing files opened by fsspec in read_parquet (#15770)
• Fix incorrect is_between pushdown to scan_pyarrow_dataset (#15769)
• Handle null index correctly for list take (#15737)
• Preserve lexical ordering on concat (#15753)
• Remove incorrect unsafe pointer cast for int -> enum (#15740)
• pass series name to apply for cut/qcut (#15715)
• count of null column shouldn't panic in agg context (#15710)
• manual cache (#15711)
• Ensure we don't hold onto Mutex when grabbing join tuples (#15704)
• allow null dtypes in UDFs if they match the schema (#15699)
• Respect join_null argument for semi/anti joins (#15696)
• Ensure we don't hold RwLock when spawning group parallelism in w… (#15697)
• Ensure empty with_columns is a no-op (#15694)
• Include predicate in cache state union (#15693)
• Add the missing feature flag for ewm_mean_by (#15687)
• 8/16-bits int could also apply in place for log expr (#15680)
• prepare_expression_for_context shouldn't panic if exceptions raised from optimizer (#15681)

📖 Documentation

• Add docstring examples for datetimes (#13161) (#15804)
• Fix a typo in categorical section of the user guide (#15777)
• Fix a docstring mistake for DataType.is_float (#15773)
• Remove incorrect "1i (1 index count)" from some docs methods (#15750)
• Add example for Config.set_tbl_width_chars (#15566)
• Align docstring phrasing in Series/Expr.dt.truncate/round (#15698)
• Various deprecation docstring improvements (#15648)

🛠️ Other improvements

• Always expand horizontal_any/all (#15816)
• Rename decimal_float to decimal_comma (#15817)
• Split coverage calculation (#15780)
• Update readme (#15787)
• Start at using new Bound<> API from PyO3 (#15752)
• Make json_path_match expr non-anonymous (#15682)

Thank you to all our contributors for making this release possible!
@MarcoGorelli, @NedJWestern, @Robinsane, @TobiasDummschat, @alexander-beedie, @c-peters, @dependabot, @dependabot[bot], @gasmith, @henryharbeck, @itamarst, @kszlim, @mbuhidar, @nameexhaustion, @orlp, @reswqa, @ritchie46, @stinodego and @wsyxbcl

Rust Polars 0.39.2

🚀 Performance improvements

• use two binary searches for equality mask when data is sorted (#15702)
• improve filter parallelism (#15686)

✨ Enhancements

• Remove outdated join validation checks (#15701)

🐞 Bug fixes

• manual cache (#15711)
• Ensure we don't hold onto Mutex when grabbing join tuples (#15704)
• allow null dtypes in UDFs if they match the schema (#15699)
• Respect join_null argument for semi/anti joins (#15696)
• Ensure we don't hold RwLock when spawning group parallelism in w… (#15697)
• Ensure empty with_columns is a no-op (#15694)
• Include predicate in cache state union (#15693)
• Add the missing feature flag for ewm_mean_by (#15687)
• 8/16-bits int could also apply in place for log expr (#15680)
• prepare_expression_for_context shouldn't panic if exceptions raised from optimizer (#15681)

🛠️ Other improvements

• Make json_path_match expr non-anonymous (#15682)

Thank you to all our contributors for making this release possible!
@henryharbeck, @kszlim, @orlp, @reswqa and @ritchie46

Python Polars 0.20.22-rc.1

🚀 Performance improvements

• improve filter parallelism (#15686)

🐞 Bug fixes

• Add the missing feature flag for ewm_mean_by (#15687)
• 8/16-bits int could also apply in place for log expr (#15680)
• prepare_expression_for_context shouldn't panic if exceptions raised from optimizer (#15681)

📖 Documentation

• Various deprecation docstring improvements (#15648)

🛠️ Other improvements

• Make json_path_match expr non-anonymous (#15682)

Thank you to all our contributors for making this release possible!
@henryharbeck, @reswqa and @ritchie46

Rust Polars 0.39.1

🚀 Performance improvements

• Fix regression that led to using only a single thread (#15667)

✨ Enhancements

• add ewm_mean_by (#15638)

🐞 Bug fixes

• Ensure profile of simple-projection only take own runtime (#15671)
• Panic if invalid array in object (#15664)
• Ensure 'CachedSchema' doesn't get synced between plans (#15661)
• group_by multiple null columns produce phantom row (#15659)
• rolling_* aggs were behaving as if they return scalars in group-by (#15657)
• Correct the unsoundness slice range of arr.min/max (#15654)
• list.mean fast path shouldn't produce NaN (#15652)
• Fix Display implementation of Duration (#15647)

📖 Documentation

• Fix typo in legacy install instructions (#15662)
• Include prelude import in the example (#15633)

🛠️ Other improvements

• Remove the remaining usage of deprecated numpy crate APIs (#15668)
• make Duration.is_constant_duration less strict for non-timezone-aware case (#15639)
• Fix some typos in comments (#15665)
• remove unnecessary unsafe in list mean/sum (#15660)

Thank you to all our contributors for making this release possible!
@MarcoGorelli, @Priyansh4444, @StevenMia, @itamarst, @mcrumiller, @orlp, @reswqa, @ritchie46 and @stinodego

Python Polars 0.20.21

🚀 Performance improvements

• Fix regression that led to using only a single thread (#15667)

✨ Enhancements

• add ewm_mean_by (#15638)

🐞 Bug fixes

• Ensure profile of simple-projection only take own runtime (#15671)
• Panic if invalid array in object (#15664)
• Ensure 'CachedSchema' doesn't get synced between plans (#15661)
• group_by multiple null columns produce phantom row (#15659)
• rolling_* aggs were behaving as if they return scalars in group-by (#15657)
• Correct the unsoundness slice range of arr.min/max (#15654)
• list.mean fast path shouldn't produce NaN (#15652)

📖 Documentation

• Add missing deprecation warning to DataFrame.replace (#15612)
• Fix typo in legacy install instructions (#15662)

🛠️ Other improvements

• make Duration.is_constant_duration less strict for non-timezone-aware case (#15639)
• Fix some typos in comments (#15665)
• remove unnecessary unsafe in list mean/sum (#15660)
• fixup failing test due to offset deprecation in upsample (#15636)

Thank you to all our contributors for making this release possible!
@MarcoGorelli, @Priyansh4444, @StevenMia, @eitsupi, @itamarst, @mcrumiller, @orlp, @reswqa, @ritchie46 and @stinodego