Rewrite repo-build and package-build in Python

[matteodelabre]

(See #211.)

The new build scripts are backward-compatible with existing recipes. Apart from some minor improvements on generated maintainer scripts, there are no changes in the final .ipk files.

• Rewrite to Python 3.8
  • Create toltec Python support library (in scripts/toltec)
  • Remove superseded opkg Python library (in scripts/opkg)
  • Add Python linting and formatting checks
  • Replace scripts/repo-build (Bash script) with scripts/repo_build.py
  • Replace scripts/package-build (Bash script) with scripts/package_build.py
  • Update Makefile and workflows to work with the new scripts
• Package changes
  • Update koreader to not use the $recipedir variable (not documented, not supported by the new scripts, and should not be used anymore)
  • Update zoneinfo-utils to fix use of an undefined variable
• Documentation changes
  • Add info on how required Python version and modules
  • Document custom fields and functions support
• New features
  • Improve support for custom fields and functions in recipes
  • Parameterize the https://toltec-dev.org value in workflows, using a REMOTE_HTTP secret.
  • Rename the REMOTE secret to REMOTE_SSH for clarity
• Additional checks
  • Check that all custom declarations start with _ to guard against typos on optional standard declarations
  • Check that timestamp fields contain a valid ISO-8601 value
  • Check that pkgver fields contain a valid version number
  • Run all Bash scripts with set -euo pipefail flags enabled

[raisjn]

how are you testing this? did you generate full repo and compare with the old repo?

[matteodelabre]

how are you testing this? did you generate full repo and compare with the old repo?

I generated the repo, installed the packages and tested them. I haven’t automatically checked that the data.tar.gz sub-archives match with the existing ones yet.

[LinusCDE]

I have not checked out all the code yet. It seems really impressive so far!

The comments only only some minor things I found. I need to dig into the new code deeper and do some more

The data.tar.gz currently contains absolute paths only. I'll make a fix for that.

[matteodelabre]

Thanks for your review @LinusCDE.

The data.tar.gz currently contains absolute paths only. I'll make a fix for that.

It worked before and then I switched to using absolute paths, which triggers a special behavior in tarfile. Should be fixed now!

[LinusCDE]

Cool. Just first fixed the paths, then redid the adding of files manually, since it seemed to me they were empty. I was stupid (looked at a symlink) and the work was for nothing. xD

Ideally I would like to replace str.removeprefix with os.path.relpath in _clean_info(...).

[matteodelabre]

Ideally I would like to replace str.removeprefix with os.path.relpath in _clean_info(...).

That’s also what I did in my fix. :-)

[raisjn]

overall, your python looks good! i don't have much to say about the code itself (i've read through it a few times), just comments+questions on usage and how things fit together

[matteodelabre]

Thanks for your review @raisjn!

[raisjn]

i'm running this on python 3.8.5 on my machine locally, misc notes:

• from collections.abc import Iterable causes an error about ABCMeta, using from typing import Iterable works
• keeping the package dir during build causes an error because there are mkdir lines that throw errors (src_dir). consider using makedirs() with exist_ok=True for most of the creates.
• AttributeError: 'str' object has no attribute 'removeprefix' - https://www.python.org/dev/peps/pep-0616/ - looks like it is python 3.9 feature added a year ago?
• the output is missing the tree (see details below) in rmkit package nvm, this is because i deleted the lines that used removeprefix

[matteodelabre]

i'm running this on python 3.8.5 on my machine locally, misc notes:

• from collections.abc import Iterable causes an error about ABCMeta, using from typing import Iterable works
• AttributeError: 'str' object has no attribute 'removeprefix' - https://www.python.org/dev/peps/pep-0616/ - looks like it is python 3.9 feature added a year ago?

I think we’re seeing differences between 3.8 and 3.9 (I used the latter for testing). Should be fairly easy to make the code 3.8-compatible.

[matteodelabre]

Thanks for the report. Should be fixed now.

[raisjn]

shipit

[matteodelabre]

Thanks for the review! I’ll do a final test of all packages to make sure everything works fine tomorrow, and then I’ll merge this.

[matteodelabre]

I went through all the packages and checked for differences between the versions generated with the new scripts and the versions existing in stable. I found some bugs that I addressed in the second commit of this branch (which is the diff since you last reviewed @raisjn).

There remains no difference in the data.tar.gz archives that contain the installed files. There are some differences in the generated maintainer scripts in control.tar.gz since the generation was slightly improved with the rewrite. I have validated on my rM1 that those differences do not cause problems. I don’t believe that those differences could have an impact on rM2.

This is now awaiting a final review before merging.

[raisjn]

looks good to me, whats the roll-out plan?

i tested by building the whole repository (repo-local), then pushed packages and tested installation and removal of various packages: rm2fb, oxide, remux, fingerterm, draft, etc

[matteodelabre]

Could you clarify what you mean by roll-out plan in this context @raisjn?

[matteodelabre]

Summary of our Discord discussion: @raisjn was “wondering if it makes sense to build a repo using the new build scripts while keeping the old repo, then have them run side by side and switch over down the line (so it gets live testing of packages). but after thinking about it, its much work”. @matteodelabre: “If we merge it into testing, we'll have a bit of time to look for remaining issues before it hits stable. Those remaining bugs should be fairly minor and affect relatively few users.”