Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Interoperability updates for py3.8, py3.9 and py3.10 #6157

Closed
wants to merge 63 commits into from

Conversation

nzlosh
Copy link
Contributor

@nzlosh nzlosh commented Feb 28, 2024

This branch started as an exploration for upgrading mongoengine/pymongo but has evolved into aligning module dependencies and fixing code related to updating module versions to support py3.8, py3.9 and py3.10.

@pull-request-size pull-request-size bot added the size/L PR that changes 100-499 lines. Requires some effort to review. label Feb 28, 2024
@nzlosh nzlosh marked this pull request as draft February 28, 2024 13:37
@pull-request-size pull-request-size bot added size/XS PR that changes 0-9 lines. Quick fix/merge. and removed size/L PR that changes 100-499 lines. Requires some effort to review. labels Feb 28, 2024
@pull-request-size pull-request-size bot added size/S PR that changes 10-29 lines. Very easy to review. size/M PR that changes 30-99 lines. Good size to review. and removed size/XS PR that changes 0-9 lines. Quick fix/merge. size/S PR that changes 10-29 lines. Very easy to review. labels Feb 28, 2024
@nzlosh
Copy link
Contributor Author

nzlosh commented Mar 12, 2024

Currently attempting to get tests passing with pymongo 3.13 to display deprecation warns for python3.8 to python3.10.
nose is blocking python3.10 (Will attempt to run tests with pytest)
oslo.config is blocking python3.10 (Upgraded to latest, but behavioural changes break string output. Further investigation required.)

@pull-request-size pull-request-size bot added size/XL PR that changes 500-999 lines. Consider splitting work into several ones that easier to review. and removed size/M PR that changes 30-99 lines. Good size to review. labels Mar 12, 2024
@cognifloyd
Copy link
Member

I cherry picked your unittest2 changes to #6187.

@cognifloyd
Copy link
Member

cognifloyd commented Apr 19, 2024

Cherry-picked PRs:

Next PR(s) to extract:

  • Fix bash syntax error in test_pause_resume_with_error
  • ❓ Removed test for application_headers when no compression enabled. and doc comment fix
  • stub test files for pytest
    • 2998a0d (only contrib/runners/{local,remote}_runner/tests/unit/*)
    • 02b7beb (only contrib/ changes)

Probably keep in the py3.10 specific PR:

  • change interpreter constraints to include py3.10 and regen lockfiles
  • py3.10 message differences

pants.toml Outdated
Comment on lines 99 to 100
# pants plugins - 202404 disabled because pants doesn't support >py3.9 and conflicts with other targets using >py3.9
# "/pants-plugins",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What issue(s) led to commenting this?

I believe this is what we need to change to add py3.10 to the interpreter constraints:
753974d

(That commit is in the pants-ic-py3.10 branch which includes the update-oslo.config branch)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On ubuntu 22.04 there is no python3.9 interpreter and no matter if I tried py3.10 or py3.11 or bump pants to 2.18 it refused to run because it claimed to need py3.9. I couldn't figure out how to find a version combination that appeased the constraints so I commented to be able to move on the st2 code base rather than battle with pants plugins. I eventually installed python3.9 from source on u2204 to be able to run pants 2.17.

pants.toml Outdated
@@ -111,38 +111,36 @@ enable_resolves = true
default_resolve = "st2"
interpreter_constraints = [
# python_distributions needs a single constraint (vs one line per python version).
"CPython>=3.8,<3.10",
"CPython==3.9.*",
Copy link
Member

@cognifloyd cognifloyd May 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the interpreter constraints for our code, so your change tells pants that st2 only supports py 3.9.

Suggested change
"CPython==3.9.*",
"CPython>=3.8,<3.11",

Copy link
Contributor Author

@nzlosh nzlosh May 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That probably explains why I couldn't get pants to run without py3.9

@@ -2,7 +2,7 @@ __defaults__(
all=dict(
resolve="pants-plugins",
skip_pylint=True,
interpreter_constraints=["CPython==3.9.*"],
interpreter_constraints=["CPython<=3.11,>=3.8"],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code only runs in pants, so it should match pants constraints.

Suggested change
interpreter_constraints=["CPython<=3.11,>=3.8"],
interpreter_constraints=["CPython==3.9.*"],

BTW Pants will jump to python 3.11 in a future version. Once we update up that version, we can adjust this constraint.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a hard time with pants, so take all those settings with that in mind. :insert dog typing on keyboard meme here:

@pull-request-size pull-request-size bot added size/L PR that changes 100-499 lines. Requires some effort to review. and removed size/XXL PR that changes 1000+ lines. You should absolutely split your PR into several. labels May 16, 2024
Note that this only adjusts interpreter constraints.
python 3.10 is not officially supported yet.

Lockfile diff: lockfiles/st2.lock [st2]

==                    Upgraded dependencies                     ==

  orjson                         3.10.1       -->   3.10.2

Lockfile diff: lockfiles/bandit.lock [bandit]

==                    Upgraded dependencies                     ==

  setuptools                     69.2.0       -->   69.5.1

Lockfile diff: lockfiles/flake8.lock [flake8]

==                    Upgraded dependencies                     ==
  setuptools                     69.2.0       -->   69.5.1

Lockfile diff: lockfiles/black.lock [black]

==                    Upgraded dependencies                     ==

  platformdirs                   4.2.0        -->   4.2.1
  typing-extensions              4.10.0       -->   4.11.0

Lockfile diff: lockfiles/pytest.lock [pytest]

==                    Upgraded dependencies                     ==

  coverage                       7.4.4        -->   7.5.0
  execnet                        2.0.2        -->   2.1.1
  pluggy                         1.4.0        -->   1.5.0

Lockfile diff: lockfiles/twine.lock [twine]

==                    Upgraded dependencies                     ==

  idna                           3.6          -->   3.7
  jaraco-context                 4.3.0        -->   5.3.0
  jaraco-functools               4.0.0        -->   4.0.1
  keyring                        25.1.0       -->   25.2.0

==                      Added dependencies                      ==

  backports-tarfile              1.1.1

Lockfile diff: lockfiles/pants-plugins.lock [pants-plugins]

==                    Upgraded dependencies                     ==

  pluggy                         1.4.0        -->   1.5.0
@pull-request-size pull-request-size bot added size/XXL PR that changes 1000+ lines. You should absolutely split your PR into several. and removed size/L PR that changes 100-499 lines. Requires some effort to review. labels May 16, 2024
Lockfile diff: lockfiles/st2.lock [st2]

==                    Upgraded dependencies                     ==

  bcrypt                         4.1.2        -->   4.1.3
  chardet                        3.0.4        -->   5.2.0
  cryptography                   42.0.5       -->   42.0.7
  dnspython                      1.16.0       -->   2.6.1
  flask                          2.1.3        -->   3.0.3
  jinja2                         3.1.3        -->   3.1.4
  markupsafe                     2.0.1        -->   2.1.5
  networkx                       2.8.8        -->   3.1
  orjson                         3.10.2       -->   3.10.3
  platformdirs                   4.2.1        -->   4.2.2
  pygments                       2.17.2       -->   2.18.0
  tenacity                       8.2.3        -->   8.3.0
  ujson                          5.9.0        -->   5.10.0
  virtualenv                     20.26.1      -->   20.26.2
  werkzeug                       2.1.2        -->   3.0.3
  zipp                           3.18.1       -->   3.18.2

==                      Added dependencies                      ==

  blinker                        1.8.2

==                     Removed dependencies                     ==

  colorama                       0.4.6
  nose                           1.3.7
  nose-parallel                  0.4.0
  nose-timer                     1.0.1
  rednose                        1.3.0
  termstyle                      0.1.11
Apparently, argparse has been in the standard library since Python 3.2.
We want the std lib version, not the old version from pypi.
So, drop the argparse dep.
@cognifloyd
Copy link
Member

cognifloyd commented May 17, 2024

@nzlosh I resolved the merge conflicts, fixed the interpreter constraints, and updated the lockfiles. Hopefully this provides a good basis for you to keep keep working on py 3.10 support. I'm not planning to do much else with requirements, so I hope my other pants work won't create [too many] merge conflicts for you.

@nzlosh
Copy link
Contributor Author

nzlosh commented May 17, 2024

@cognifloyd You're a legend!! Thanks for sorting that mess out. I will create smaller PRs against master as I progress when I pickup the 3.10 compatibility work. 17 out of 29 checks passing is encouraging.

@nzlosh
Copy link
Contributor Author

nzlosh commented Oct 3, 2024

Closing this PR as most changes have been cherry picked and merged to master.

@nzlosh nzlosh closed this Oct 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size/XXL PR that changes 1000+ lines. You should absolutely split your PR into several.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants