[BEAM-12563] swaplevel general function for dataframe and series #15944

AlikRodriguez · 2021-11-11T16:57:05Z

Implementation of swaplevel

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

`ValidatesRunner` compliance status (on master branch)

Lang	ULR	Twister2
Go	---	---
Java
Python	---	---
XLang		---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

codecov · 2021-11-11T17:20:25Z

Codecov Report

Merging #15944 (12fcda5) into master (6a573e4) will increase coverage by 1.27%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master   #15944      +/-   ##
==========================================
+ Coverage   83.51%   84.78%   +1.27%     
==========================================
  Files         445      453       +8     
  Lines       61390    71025    +9635     
==========================================
+ Hits        51267    60217    +8950     
- Misses      10123    10808     +685

Flag	Coverage Δ
python	`83.62% <100.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdks/python/apache_beam/dataframe/frames.py	`95.89% <100.00%> (+0.97%)`	⬆️
sdks/python/apache_beam/internal/pickler.py	`77.27% <0.00%> (-9.59%)`	⬇️
...on/apache_beam/examples/dataframe/flight_delays.py	`25.00% <0.00%> (-2.91%)`	⬇️
...runners/interactive/options/interactive_options.py	`83.33% <0.00%> (-2.39%)`	⬇️
...hon/apache_beam/runners/direct/test_stream_impl.py	`94.02% <0.00%> (-2.24%)`	⬇️
sdks/python/apache_beam/io/gcp/bigtableio.py	`77.11% <0.00%> (-2.11%)`	⬇️
sdks/python/apache_beam/io/jdbc.py	`82.92% <0.00%> (-2.08%)`	⬇️
...eam/io/gcp/datastore/v1new/rampup_throttling_fn.py	`93.75% <0.00%> (-1.38%)`	⬇️
...s/python/apache_beam/examples/wordcount_minimal.py	`92.00% <0.00%> (-0.86%)`	⬇️
...ks/python/apache_beam/runners/interactive/utils.py	`94.69% <0.00%> (-0.77%)`	⬇️
... and 89 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6a573e4...12fcda5. Read the comment docs.

AlikRodriguez · 2021-12-03T19:10:41Z

@TheNeuralBit Hi, I'm having troubles with the docs in this issue, the error refers to the pandas documentation, with the message NameError: name 'df' is not defined, is something I'm not considering here?

TheNeuralBit · 2021-12-03T21:01:07Z

The PythonDocs issue is really strange. I can take a look at what's going on there. Is this ready to go otherwise?

TheNeuralBit · 2021-12-03T22:45:27Z

The PythonDocs issue is because the docstring for swaplevel is a little mangled and it breaks some of our automation:

Swap levels i and j in a :class:`MultiIndex`.

Default is to swap the two innermost levels of the index.

Parameters
----------
i, j : int or str
    Levels of the indices to be swapped. Can pass level name as string.
axis : {0 or 'index', 1 or 'columns'}, default 0
            The axis to swap levels on. 0 or 'index' for row-wise, 1 or
            'columns' for column-wise.

Returns
-------
DataFrame
    DataFrame with levels swapped in MultiIndex.

Examples
        --------
        >>> df = pd.DataFrame(
        ...     {"Grade": ["A", "B", "A", "C"]},
        ...     index=[
        ...         ["Final exam", "Final exam", "Coursework", "Coursework"],
        ...         ["History", "Geography", "History", "Geography"],
        ...         ["January", "February", "March", "April"],
        ...     ],
        ... )
        >>> df
...

I proposed a change upstream (pandas-dev/pandas#44740), if that gets merged and released in 1.3.5 this issue should go away.

If that takes a while or doesn't happen, we'll have to come up with some kind of workaround. I'll have to think on that. You should feel free to ignore this issue for now.

AlikRodriguez · 2021-12-06T16:30:53Z

@TheNeuralBit great! Aside of that, do you think is anything else I should change?

TheNeuralBit

LGTM, this is ready to merge aside from the docs issue. I think the fix there may just be to wait until pandas 1.4.0 is out, pandas maintainers are uninterested in backporting the fix to 1.3.5

TheNeuralBit · 2021-12-08T17:30:40Z

sdks/python/apache_beam/dataframe/frames.py

+            'swaplevel',
+            lambda df: df.swaplevel(**kwargs), [self._expr],
+            requires_partition_by=partitionings.Arbitrary(),
+            preserves_partition_by=partitionings.Arbitrary()))


I wouldn't think this should preserve Index partitioning, since it modifies the index. But our test framework does verify this.

It turns out this expression does preserve index partitioning, since we generate hashes by summing hashes on the individual index levels, which is independent of the order of the index levels.

TheNeuralBit · 2021-12-14T18:04:07Z

Just to be clear, no further action is needed from you @AlikRodriguez - I will take care of making sure this gets merged when the docs issue is resolved.

github-actions · 2022-02-13T12:43:42Z

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.

TheNeuralBit · 2022-02-14T17:03:24Z

retest this please

TheNeuralBit · 2022-02-15T23:45:01Z

retest this please

TheNeuralBit · 2022-02-15T23:45:37Z

Run PythonDocs PreCommit

aaltay · 2022-02-25T00:53:37Z

@TheNeuralBit - What is the next step on this PR?

TheNeuralBit · 2022-02-25T16:49:16Z

Run PythonLint PreCommit

TheNeuralBit · 2022-02-25T16:49:28Z

Run Python PreCommit

TheNeuralBit · 2022-02-25T16:49:50Z

@TheNeuralBit - What is the next step on this PR?

Thanks for the ping - this just needs a green CI run.

TheNeuralBit · 2022-02-25T18:27:56Z

Run PythonLint PreCommit

TheNeuralBit · 2022-02-25T19:46:11Z

Run PythonLint PreCommit

TheNeuralBit · 2022-02-25T20:13:51Z

Run PythonLint PreCommit

TheNeuralBit · 2022-02-25T21:06:22Z

Run PythonLint PreCommit

TheNeuralBit · 2022-02-25T21:21:32Z

Run PythonLint PreCommit

swaplevel general function for dataframe and series

357e796

pandas doctest

9820d0b

AlikRodriguez changed the title ~~WIP [BEAM-12563]swaplevel general function for dataframe and series~~ [BEAM-12563]swaplevel general function for dataframe and series Dec 2, 2021

AlikRodriguez changed the title ~~[BEAM-12563]swaplevel general function for dataframe and series~~ [BEAM-12563] swaplevel general function for dataframe and series Dec 2, 2021

AlikRodriguez and others added 2 commits December 2, 2021 16:41

doctest

3546ea8

Update frames.py

2244ee0

lint error

12fcda5

TheNeuralBit self-requested a review December 8, 2021 16:48

TheNeuralBit approved these changes Dec 8, 2021

View reviewed changes

github-actions bot added the stale label Feb 13, 2022

github-actions bot removed the stale label Feb 15, 2022

TheNeuralBit merged commit ba2aef5 into apache:master Feb 25, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-12563] swaplevel general function for dataframe and series #15944

[BEAM-12563] swaplevel general function for dataframe and series #15944

AlikRodriguez commented Nov 11, 2021

codecov bot commented Nov 11, 2021 •

edited

Loading

AlikRodriguez commented Dec 3, 2021

TheNeuralBit commented Dec 3, 2021

TheNeuralBit commented Dec 3, 2021 •

edited

Loading

AlikRodriguez commented Dec 6, 2021

TheNeuralBit left a comment

TheNeuralBit Dec 8, 2021

TheNeuralBit commented Dec 14, 2021

github-actions bot commented Feb 13, 2022

TheNeuralBit commented Feb 14, 2022

TheNeuralBit commented Feb 15, 2022

TheNeuralBit commented Feb 15, 2022

aaltay commented Feb 25, 2022

TheNeuralBit commented Feb 25, 2022

TheNeuralBit commented Feb 25, 2022

TheNeuralBit commented Feb 25, 2022

TheNeuralBit commented Feb 25, 2022

TheNeuralBit commented Feb 25, 2022

TheNeuralBit commented Feb 25, 2022

TheNeuralBit commented Feb 25, 2022

TheNeuralBit commented Feb 25, 2022

[BEAM-12563] swaplevel general function for dataframe and series #15944

[BEAM-12563] swaplevel general function for dataframe and series #15944

Conversation

AlikRodriguez commented Nov 11, 2021

ValidatesRunner compliance status (on master branch)

Examples testing status on various runners

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

codecov bot commented Nov 11, 2021 • edited Loading

Codecov Report

AlikRodriguez commented Dec 3, 2021

TheNeuralBit commented Dec 3, 2021

TheNeuralBit commented Dec 3, 2021 • edited Loading

AlikRodriguez commented Dec 6, 2021

TheNeuralBit left a comment

Choose a reason for hiding this comment

TheNeuralBit Dec 8, 2021

Choose a reason for hiding this comment

TheNeuralBit commented Dec 14, 2021

github-actions bot commented Feb 13, 2022

TheNeuralBit commented Feb 14, 2022

TheNeuralBit commented Feb 15, 2022

TheNeuralBit commented Feb 15, 2022

aaltay commented Feb 25, 2022

TheNeuralBit commented Feb 25, 2022

TheNeuralBit commented Feb 25, 2022

TheNeuralBit commented Feb 25, 2022

TheNeuralBit commented Feb 25, 2022

TheNeuralBit commented Feb 25, 2022

TheNeuralBit commented Feb 25, 2022

TheNeuralBit commented Feb 25, 2022

TheNeuralBit commented Feb 25, 2022

`ValidatesRunner` compliance status (on master branch)

codecov bot commented Nov 11, 2021 •

edited

Loading

TheNeuralBit commented Dec 3, 2021 •

edited

Loading