Releases: pola-rs/polars
Python Polars 0.16.2
🚀 Performance improvements
- improve dynamic groupby performance on sorted keys (#6599)
✨ Enhancements
- implement fill_null for list data (#6635)
- expression functions should be nullable (#6629)
- Implement unary plus operation on exprs and series (#6517)
- add streamable udfs (#6614)
- is_first for struct dtype (#6595)
- Added from_str_radix method to StringNameSpace that allows to parse strings from any base to i32 (#6570)
- Implement DataFrame Interchange Protocol through
pyarrow
(#6581) - improve predicate pushdown (#6579)
- raise error on invalid binary cmp (#6564)
🐞 Bug fixes
- make string_repr private (#6636)
- treat literal values consistently in
select
context, improve related typing (#6628) - Fix _repr_html_ double-height rows (#5645) (#6534)
- fix(rust, python) cast to and from fixed offsets (#6602)
- raise error on string numeric arithmetic (#6601)
- don't convert "ns"-precision temporal types via
pyarrow
(#6592) - partially assert sortedness in groupby dynamic (#6593)
- fix(rust, python); raise oob if negative index given to take (#6590)
- fix predicate pushdown key check (#6577)
- fix schema of apply with many inputs on empty df (#6571)
- let lhs determine struct order in supertype (#6572)
- ensure consistent handling of 1D numpy arrays with respect to other sequences (#6569)
- fix(rust, python) validate utc, fmt, and tz-aware in strptime (#6550)
- add strptime to filter boundary (#6560)
🛠️ Other improvements
- make string_repr private (#6636)
- add example of using
is_between
with string bounds, and extend test coverage for the same (#6627) - provide additional examples for
diff
methods (#6630) - Consistent handling of env vars (#6626)
- make
structify
behaviour experimental, while also extending it to aliased expressions (#6615) - Disallow clippy borrow deref ref (#6605)
- Update
ruff
version and some settings (#6588) - Add release flow info to contributing guide (#6480)
- Use
assert_series_equal
instead ofs.series_equal(...)
(#6582) - cleanup last vestiges of experimental kwargs setting (#6568)
- Use
assert_frame_equal
instead ofassert df.frame_equal(...)
(#6553) - Update to PyO3 to 0.18.0 (#6531)
Thank you to all our contributors for making this release possible!
@2-5, @MarcoGorelli, @abalkin, @alexander-beedie, @cojmeister, @dependabot, @dependabot[bot], @jjerphan, @plaflamme, @ritchie46 and @stinodego
Python Polars 0.16.1
🏆 Highlights
- Formalize list aggregation difference between groupbys, selection and window functions (#6487)
- automagically upconvert
with_columns
kwarg expressions with multiple output names to struct; extend**named_kwargs
support toselect
(#6497)
⚠️ Breaking changes
- error on string <-> date cmp (#6498)
- Formalize list aggregation difference between groupbys, selection and window functions (#6487)
- show where error messages originated (#6482)
- Remove deprecated paths from
Series.__getitem__
(#6048) - change behaviour of named rows (#6302)
- Remove deprecated
read/write_json
arguments (#5990) - make
schema
,schema_overrides
, andorient
consistent on all user-facing interfaces (#6387) - Groupby iteration now returns tuples of (name, data) (#6350)
- Remove
Groupby.pivot
(#6016) - Remove deprecated argument aliases (#5993)
- Change
Series.shuffle
default behaviour (#5991) - Change
Expr.is_between
default behaviour (#5985) - Restrict certain function parameters to be keyword-only (#6464)
✨ Enhancements
- let cast_time_zone accept None (#6539)
- automagically upconvert
with_columns
kwarg expressions with multiple output names to struct; extend**named_kwargs
support toselect
(#6497) - add some missing type annotation in
series
dispatch methods (#6523) - better errors in get_ptr and a probability on a boolean… (#6522)
- add utc parameter to strptime (#6496)
- add meta 'has_multiple_outputs', 'is_regex_projec… (#6500)
- error on string <-> date cmp (#6498)
- ~30% faster
iter_rows(named=True)
andto_dicts()
, if pyarrow available (#6493) - show where error messages originated (#6482)
- Remove deprecated paths from
Series.__getitem__
(#6048) - change behaviour of named rows (#6302)
- Remove deprecated
read/write_json
arguments (#5990) - Groupby iteration now returns tuples of (name, data) (#6350)
- Remove
Groupby.pivot
(#6016) - Remove deprecated argument aliases (#5993)
- Change
Series.shuffle
default behaviour (#5991) - Change
Expr.is_between
default behaviour (#5985) - Restrict certain function parameters to be keyword-only (#6464)
🐞 Bug fixes
- implement ser/de for BinaryChunked (#6543)
- on frame-init from generator, initial
chunk_size
cannot be smaller thaninfer_schema_length
(#6541) - raise if tz_localize called on UTC-aware (#6526)
- make concat_list group aware (#6527)
- error on invalid expanding expression (#6521)
- create from dicts directly as struct categorical (#6520)
- fix oob in arr.get by expressions (#6519)
- fix cse schema (#6518)
- panic when max_len -1 is reached (#6494)
- Formalize list aggregation difference between groupbys, selection and window functions (#6487)
- fix(rust, python) validate tz in with_time_zone (#6417)
🛠️ Other improvements
- Remove
verify_series_and_expr_api
util (#6524) - Disable some tests for Windows (#6532)
- Remove unnecessary brackets in doc examples (#6332)
- Enable some tests for Windows (#6511)
- Fix test issue with tmp directory (#6508)
- Fix some deprecation warnings (#6495)
- added all missing examples for temporal expressions (#6488)
- Utilize pytest-xdist for faster unittests (#6483)
- test(python) I/O test improvements (#6475)
- make
schema
,schema_overrides
, andorient
consistent on all user-facing interfaces (#6387) - improved error message from Expr on incorrect usage in boolean context (#6473)
Thank you to all our contributors for making this release possible!
@MarcoGorelli, @alexander-beedie, @gab23r, @papparapa, @ritchie46, @romanovacca, @stinodego and @zundertj
Python Polars 0.15.18
✨ Enhancements
- More precise pipe type annotation (#6457)
🐞 Bug fixes
🛠️ Other improvements
- Specify deltalake minimum version (#6363)
- deprecate
iterrows
in favour ofiter_rows
, add new@redirect
class decorator (#6461) - Improve IO test structure (#6453)
Thank you to all our contributors for making this release possible!
@alexander-beedie, @josh, @ritchie46 and @stinodego
Python Polars 0.15.17
✨ Enhancements
- allow expr in str.contains (#6443)
- Deprecate
with_column
(#6128) - expose efficient iterator over
DataFrame
slices (#6414) - add float formatting option (#6432)
- 10% speedup for
to_dicts
method (#6415) - add datetime/duration dtype selector groups covering the different timeunits (#6425)
- allow internal api to get pointer to values buffer (#6385)
- infer ISO8601 datetimes (#6357)
- minor improvement to auto-detection of ambiguous data orientation (#6376)
- allow expressions as arguments in
str.ends_with
(#6361) - Make groupby rolling/dynamic iterable (#6372)
- accept expr in
str.starts_with
(#6355) - Move
explode
to namespaces (#6351) - Rename
Series.struct.to_frame
to.struct.unnest
(#6352) - auto-detect %+ as tz-aware (#6434)
🐞 Bug fixes
- fix projection pushdown on double semi join (#6440)
- ensure column-exclusion works with the new
dtype
groups, and improve some related typing (#6442) - ensure
from_dicts
andDataFrame
init from list of dicts behave consistently, update/improve related docstrings (#6431) - cumulative_eval ensure output dtype is respected (#6435)
- allow from pandas null structs (#6430)
- fixed interaction of
schema_overrides
with frame-init from list of dicts (#6424) - only use float simd on specific alignment (#6427)
- no early escape when window is equal to len in rolling_float (#6408)
is_between
typing with time in start and end (#6393)- dont incorrectly infer Zulu time (#6378)
- raise error on invalid sort_by argument (#6382)
- take offset into account with str.explode (#6384)
- Return empty batch for pl.read_csv_batched().next_… (#6381)
- ensure pyarrow.compute module is loaded (#6353)
- implement ser/de for StructChunked (#6359)
- series of empty structs (#6347)
🛠️ Other improvements
- add explicit note about use of
Config
as a context manager (#6439) - ensure
from_dicts
andDataFrame
init from list of dicts behave consistently, update/improve related docstrings (#6431) - Fix docstring of series.interpolate (#6399)
- Remove duplicate test (#6390)
- deprecate
columns
param forDataFrame
init; transitioning toschema
(#6366) - Add docs and tests to
Expr.flatten
(#6370) - Example of filtering partitioned delta tables (#6365)
- Uppercase project URL refs (#6362)
Thank you to all our contributors for making this release possible!
@ChayimFriedman2, @MarcoGorelli, @alexander-beedie, @c-peters, @flowlight0, @gab23r, @gam-phon, @ghuls, @jgmartin, @josh, @ritchie46, @romanovacca, @stinodego, @universalmind303 and @zundertj
Python Polars 0.15.16
🚀 Performance improvements
- Improve rechunk check (#6268)
- reuse allocated scratches in ipc writer (#6287)
- use dedicated writer thread for sink_parquet (#6285)
✨ Enhancements
- add strict parameter to decoding expressions (#6342)
- allow unordered struct creating from anyvalues (#6321)
- allow pass_name in aggregation apply (#6318)
- parse abbrev month name (#6314)
- Add warning for new behaviour of named rows (#6300)
- add
dt.combine
for combining date and time components (#6121) - improvements to dtype-based column selection (#6295)
- add sink_ipc (#6286)
- additional
schema_overrides
param for more ergonomicDataFrame
init (#6230)
🐞 Bug fixes
- don't cast nulls before trying normal cast (#6339)
- properly dispatch categorical string comparison (#6336)
- expand all nested wildcards in functions (#6334)
- fix groupby rolling by_key if groups are empty (#6333)
- Fix some type hints and bugs for groupby (#6329)
- Reject
None
input forhead
/tail
(#6326) - parse abbrev month name (#6314)
- default to pyarrow for writing parquet (#6313)
- disallow alias in inline join expressions (#6312)
- block proj-pd and pred-pd on swapping rename (#6303)
- convert nested dictionary with i64 keys (#6299)
- fix(python) Print instantiated dtypes in glimpse (#6298)
- infer y-m-d datetime even if single element (#6297)
- fix panic dynamic_groupby on empty dataframe (#6294)
- implement missing DataFrame
__floordiv__
op (#6280) - Allow low and high in date_range to be str (#6275)
- allow integer-compatible row indexes that are not strictly typed as
int
(#6266) - Parse negative dates with polars parser (#6256)
🛠️ Other improvements
- run cse optimization only if joins and caches… (#6337)
- Fix wrong description for variable_name argument in melt (#6331)
- Fix random groupby test failure (#6327)
- fixup test names, adjust test_struct (#6317)
- simplify
_from_pandas
constructor (#6310) - Ignore hash doctests (#6304)
- Fix docstring formatting for truncate (#6291)
- Move package metadata to
pyproject.toml
(#6271) - Move
io
tests to the same folder (#6277) - Enable Dependabot (#5036)
Thank you to all our contributors for making this release possible!
@MarcoGorelli, @alexander-beedie, @c-peters, @dependabot, @dependabot[bot], @ghuls, @n8henrie, @ritchie46, @stinodego and @universalmind303
Python Polars 0.15.15
✨ Enhancements
- ensure ooc sort works ooc with all-constant values (#6235)
- The 1 billion row sort (#6156)
- optionally treat missing UTF8 values as the empty string at CSV parse-time (#6203)
- check file target is not an existing directory (#6187)
- support -ve indexing for DataFrame
head
andtail
methods (#6173) - Implement
DataFrame.unique(keep="none")
(#6169) - support use of explicit
Struct
dtypes on DataFrame/Series init (#6145)
🐞 Bug fixes
- Add list inner dtype when printing Series (#6233)
- strptime now respects pl.Datetime's time_unit (#6231)
- fix when then otherwise with arity and aggregation… (#6224)
- collect now uses the storage_options given to scan_parquet (#6223)
- set_sorted keep schema (#6222)
- pass name to value counts in aggregation (#6221)
- don't set fast_explode on list of structs (#6220)
- address a frame init/construction error, and expose
infer_schema_length
to frame init (#6210) - explode of empty nullable list (#6190)
- fix oob arr.take (#6189)
- Make
with_columns
inwith_columns_kwargs
mode compatible with more data types (#6126) - Update docstring
with_columns
to reflect a new dataframe is being returned (#6122) - fix empty streaming joins (#6149)
- fix streaming joins where the join order has been … (#6143)
- write tz-aware datetimes to csv (#6135)
- add null behavior for oob indices (#6133)
🛠️ Other improvements
- Create
DataFrame
from schema (#6225) - don't set aggregated flag on null propagated aggregation. (#6191)
- undo cargo.toml change (#6219)
- Improve drop_nulls docstrings (#6127)
- Clarify docstrings for
closed
argument (#6198) - minor docs and typing updates (plus additional test coverage for related areas) (#6182)
- explain n_field_strategy (#6158)
Thank you to all our contributors for making this release possible!
@MarceColl, @MarcoGorelli, @alexander-beedie, @gab23r, @ghuls, @jvanbuel, @n8henrie, @rben01, @ritchie46, @ropoctl, @sorhawell, @stinodego, @winding-lines and @zundertj
Python Polars 0.15.14
🚀 Performance improvements
- first check rev-map on categorical equality check (#6085)
✨ Enhancements
- add
arr.take
expression (#6116) - allow
extend_constant
to work with date literals (#6114) - allow nested categorical cast (#6113)
- add a
rounded_corners
modifier topl.Config.set_tbl_formatting
(#6108) - huge speedup of scalar-to-array expansion on frame init from dict (#6111)
- extend existing fast range->Series init to lists of ranges in a Series (#6099)
- additional (opt-in) options for assert_frame_equal (#6096)
- add search_sorted for arrays and utf8 dtype (#6083)
🐞 Bug fixes
- ensure multi-line type hints are parenthesised (#6100)
- fix invalid dtype in chunked array after struct cast (#6093)
- don't run cse cache_states if no projections found (#6087)
- Support all datatypes in glimpse and align with head/tail (#6091)
- Update
read_csv
error message (#6082) - propogate nulls in binary arithmetic/aggregation (#6076)
🛠️ Other improvements
- Fix docstring with_context (#6118)
- Use Dataframe.item internally and in tests (#6109)
- Assert deprecation warning on check_column_names (#6110)
- enable
unused import
autofix via ruff (#6102)
Thank you to all our contributors for making this release possible!
@alexander-beedie, @gitkwr, @huitseeker, @ritchie46, @stinodego and @zundertj
Python Polars 0.15.13
✨ Enhancements
- Improve iterating over
GroupBy
(#6051) - much faster lazy type-checks (#6064)
- support array-expansion of scalars on frame init from dict (#6034)
- improve error message when writing nested data to… (#6040)
🐞 Bug fixes
- bound complex type from 3.8 to 3.11 (#6071)
- deal with unnest schema expansion in projection pd (#6063)
- correct output dtype for cummin/cumsum/cummax (#6062)
- block streaming on literal series/range (#6058)
- improve handling of dict-type "columns" param on frame-init (#6045)
- Fix typing for
DataFrame.select
(#6047) - ndjson struct inference (#6049)
- fix stringcache. latest refactor introduced a hashing error (#6056)
- allow mixed field order and availability in apply that r… (#6041)
- deal with empty structs (#6039)
- fix aggregation that filters out all data (#6036)
- fix diff overflow (#6033)
- keep column names in is_null/is_not_null (#6032)
- keep name when sorting categorical in lexial order (#6029)
- tweaked property/accessor behaviour (#6021)
- properly set null anyvalue if categorical is neste… (#6025)
- Fix
from_epoch
function signature (#6024) - Validate
estimated_size
parameter (#6018)
🛠️ Other improvements
- suggest forward fill in cumsum/cummax (#6061)
- Fix SIM105 issues. (#6042)
- Remove trailing spaces in glimpse output (#6037)
- Remove unnecessary noqa's (#6035)
- Fix flake8-pytest-style errors in tests. (#6031)
- update
read_sql
androw
docstrings (#6028) - Enable the
isort
-style import autofix viaruff
(#6020) - Update py-polars/Cargo.lock (#6013)
- Refactor pivot tests (#6012)
- Use ruff instead of isort, flake8 and pyupgrade (#5916)
- Properly deprecate
groupby.pivot
(#6000)
Thank you to all our contributors for making this release possible!
@alexander-beedie, @ghuls, @ritchie46, @stinodego and @universalmind303
Python Polars 0.15.11
🚀 Performance improvements
- ensure set_at_idx is O(1) (#5977)
✨ Enhancements
- allow eq,ne,lt etc (#5995)
- Improve
Expr.is_between
API (#5981) - large speedup for
df.iterrows
(~200-400%) (#5979) - updated default table format from "UTF8_FULL" to "UTF8_FULL_CONDENSED" (#5967)
- Access rows as namedtuples (#5966)
- Improve
assert_frame_equal
messages (#5962)
🐞 Bug fixes
- make weekday tz-aware (#5989)
- fix categorical in struct anyvalue issue (#5987)
- fix invalid boolean simplification (#5976)
- allow empty sort on any dtype (#5975)
- properly deal with categoricals in streaming queries (#5974)
Thank you to all our contributors for making this release possible!
@alexander-beedie, @ritchie46 and @stinodego
Python Polars 0.15.9
🚀 Performance improvements
- improve reducing window function performance ~33% (#5878)
✨ Enhancements
str.strip
with multiple chars (#5929)- add iterrrows (#5945)
- read decimal as f64 (#5938)
- improve query plan scan formatting (#5937)
- allow all null cast (#5933)
- allow objects in struct types (#5925)
- handle Series init from python sequence of numpy arrays (#5918)
- merge sorted dataframes (#5817)
- impl hex and base64 for binary (#5892)
- Add datatype hierarchy (#5901)
- Add .item() on DataFrame and Series (#5893)
- make get_any_value fallible (#5877)
- Add string representation for data types (#5861)
- directly push all operator result into sink, prev… (#5856)
🐞 Bug fixes
- don't panic on ignored context (#5958)
- don't allow named expression in arr.eval (#5957)
- error on invalid dtype (#5956)
- fix panic in join expressions (#5954)
- block ordered predicates before explode (#5951)
- adhere to schema in arr.eval of empty list (#5947)
- fix from_dict schema_inference=0 (#5948)
- fix arrow nested null conversion (#5946)
- allow None in arr.slice length (#5934)
- fix time to duration cast (#5932)
- error on addition with datetime/time (#5931)
- don't create categoricals in streaming (#5926)
- object filter should keep single chunk (#5913)
- csv, read escaped "" as missing (#5912)
- fix pivot of signed integers (#5909)
- don't allow duplicate columns in read_csv arg (#5908)
- fix latest oob in streaming convertion (#5902)
- adapt k to len in topk (#5888)
- fix lazy swapping rename (#5884)
- fix window function with nullable values; regression due… (#5874)
- improve equality consistency between types (#5873)
- evaluate whole branch expression to determine if r… (#5864)
- fix top_k on empty (#5865)
- fix slice in streaming (#5854)
- Fix type hint for IO
*_options
arguments (#5852)
🛠️ Other improvements
- Fix docs for sink_parquet (#5952)
- Fix misspelling in LazyFrame docstring (#5917)
- add bin, series.is_sorted and merge_sorted (#5914)
Thank you to all our contributors for making this release possible!
@AnatolyBuga, @alexander-beedie, @cannero, @chitralverma, @dannyvankooten, @johngunerli, @ozgrakkurt, @ritchie46, @stinodego, @winding-lines and @zundertj