Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update doc_url calls for new website #20583

Merged
merged 6 commits into from
Feb 21, 2024
Merged

Update doc_url calls for new website #20583

merged 6 commits into from
Feb 21, 2024

Conversation

huonw
Copy link
Contributor

@huonw huonw commented Feb 20, 2024

This PR fixes #20427 by updating our use of doc_url to reflect the new website structure. It does this semi-automatically, spread across separate commits:

  1. Update the doc_url function to reflect the new website structure, including support for unversioned URLs like https://www.pantsbuild.org/community/getting-help
  2. Apply automated rewrites (see below for details)
  3. Do one or two manual fix-ups

The automatic rewrites were done by:

  1. finding the args passed to doc_url: rg --only-matching "doc_url\([^)]*\)" . | cut -f2 -d\( | cut -f1 -d\) | tr \' \" | sort -u
  2. put them into CSV file, then use https://github.com/pantsbuild/pantsbuild.org/blob/92d5ce8d3442890c3fbeb0fade620b762ae2aa7e/old_site_redirects.js to find the new URL for each one
  3. transform the CSV into a long series of s@...@...@ substitutions for sed
  4. apply them with git ls-files '*.py' '*.rs' | xargs sed -i '' "..."
  5. do a basic check that the appropriate files that call doc_url were updated: identify non-updated files via comm -13 <(git diff --name-only be42bc0def^...be42bc0def) <( rg --files-with-matches doc_url | sort). The files without updates are expected, e.g.

This is marked for cherry-picking to 2.20.x. There are redirects for the URLs for 2.20, but it seems a bit annoying to rely on them, if we don't have to.

old,new
"getting-help","community/getting-help"
"advanced-target-selection","docs/using-pants/advanced-target-selection"
"anonymous-telemetry","docs/using-pants/anonymous-telemetry"
"awslambda-python","docs/python/integrations/aws-lambda"
"enabling-backends","docs/using-pants/key-concepts/backends"
"google-cloud-function-python","docs/python/integrations/google-cloud-functions"
"installation","docs/getting-started/installing-pants"
"macros","docs/writing-plugins/macros"
"options","docs/using-pants/key-concepts/options"
"options#addremove-semantics","docs/using-pants/key-concepts/options#addremove-semantics"
"options#config-file-entries","docs/using-pants/key-concepts/options#config-file-entries"
"options#config-file-interpolation","docs/using-pants/key-concepts/options#config-file-interpolation"
"options#pantsrc-file","docs/using-pants/key-concepts/options#pantsrc-file"
"options#reading-individual-option-values-from-files","docs/using-pants/key-concepts/options#reading-individual-option-values-from-files"
"pex","docs/python/overview/pex"
"plugin-upgrade-guide","docs/writing-plugins/common-plugin-tasks/plugin-upgrade-guide"
"plugins-overview","docs/writing-plugins/overview"
"plugins-setup-py","docs/writing-plugins/common-plugin-tasks/custom-python-artifact-kwargs"
"protobuf-go","docs/go/integrations/protobuf"
"protobuf-python","docs/python/integrations/protobuf-and-grpc"
"python-check-goal","docs/python/goals/check"
"python-distributions","docs/python/overview/building-distributions"
"python-interpreter-compatibility","docs/python/overview/interpreter-compatibility"
"python-lockfiles#lockfiles-for-tools","docs/python/overview/lockfiles#lockfiles-for-tools"
"python-test-goal","docs/python/goals/test"
"python-test-goal#pytest-version-and-plugins","docs/python/goals/test#pytest-version-and-plugins"
"python-third-party-dependencies","docs/python/overview/third-party-dependencies"
"python-third-party-dependencies#local-requirements","docs/python/overview/third-party-dependencies#local-requirements"
"python-third-party-dependencies#multiple-lockfiles","docs/python/overview/lockfiles#multiple-lockfiles"
"python-third-party-dependencies#user-lockfiles","docs/python/overview/third-party-dependencies#user-lockfiles"
"reference-deploy_jar#coderesolvecode","reference/targets/deploy_jar#resolve"
"reference-global#remote_oauth_bearer_token","reference/global-options#remote_oauth_bearer_token"
"remote-caching-execution","docs/using-pants/remote-caching-and-execution"
"source-roots","docs/using-pants/key-concepts/source-roots"
"tagging-docker-images","docs/docker/tagging-docker-images"
"target-api-concepts","docs/writing-plugins/the-target-api/concepts"
"targets","docs/using-pants/key-concepts/targets-and-build-files"
"targets#field-default-values","docs/using-pants/key-concepts/targets-and-build-files#field-default-values"
"targets#dependencies-and-dependency-inference","docs/using-pants/key-concepts/targets-and-build-files#dependencies-and-dependency-inference"
"thrift-python","docs/python/integrations/thrift"
"troubleshooting","docs/using-pants/troubleshooting-common-issues"
"troubleshooting#import-errors-and-missing-dependencies","docs/using-pants/troubleshooting-common-issues#import-errors-and-missing-dependencies"
"upgrade-tips","docs/releases/upgrade-tips"
import pandas as pd
print("\n".join(
  f"""s@\\(doc_url.*[\\"']\\){r.old}\\([\\"']\\)@\\1{r.new}\\2@g;"""
  for _, r in pd.read_csv("replacements.csv").iterrows()
))
git ls-files '*.py' '*.rs' | xargs sed -i '' "
s@\(doc_url.*[\"']\)getting-help\([\"']\)@\1community/getting-help\2@g;
s@\(doc_url.*[\"']\)advanced-target-selection\([\"']\)@\1docs/using-pants/advanced-target-selection\2@g;
s@\(doc_url.*[\"']\)anonymous-telemetry\([\"']\)@\1docs/using-pants/anonymous-telemetry\2@g;
s@\(doc_url.*[\"']\)awslambda-python\([\"']\)@\1docs/python/integrations/aws-lambda\2@g;
s@\(doc_url.*[\"']\)enabling-backends\([\"']\)@\1docs/using-pants/key-concepts/backends\2@g;
s@\(doc_url.*[\"']\)google-cloud-function-python\([\"']\)@\1docs/python/integrations/google-cloud-functions\2@g;
s@\(doc_url.*[\"']\)installation\([\"']\)@\1docs/getting-started/installing-pants\2@g;
s@\(doc_url.*[\"']\)macros\([\"']\)@\1docs/writing-plugins/macros\2@g;
s@\(doc_url.*[\"']\)options\([\"']\)@\1docs/using-pants/key-concepts/options\2@g;
s@\(doc_url.*[\"']\)options#addremove-semantics\([\"']\)@\1docs/using-pants/key-concepts/options#addremove-semantics\2@g;
s@\(doc_url.*[\"']\)options#config-file-entries\([\"']\)@\1docs/using-pants/key-concepts/options#config-file-entries\2@g;
s@\(doc_url.*[\"']\)options#config-file-interpolation\([\"']\)@\1docs/using-pants/key-concepts/options#config-file-interpolation\2@g;
s@\(doc_url.*[\"']\)options#pantsrc-file\([\"']\)@\1docs/using-pants/key-concepts/options#pantsrc-file\2@g;
s@\(doc_url.*[\"']\)options#reading-individual-option-values-from-files\([\"']\)@\1docs/using-pants/key-concepts/options#reading-individual-option-values-from-files\2@g;
s@\(doc_url.*[\"']\)pex\([\"']\)@\1docs/python/overview/pex\2@g;
s@\(doc_url.*[\"']\)plugin-upgrade-guide\([\"']\)@\1docs/writing-plugins/common-plugin-tasks/plugin-upgrade-guide\2@g;
s@\(doc_url.*[\"']\)plugins-overview\([\"']\)@\1docs/writing-plugins/overview\2@g;
s@\(doc_url.*[\"']\)plugins-setup-py\([\"']\)@\1docs/writing-plugins/common-plugin-tasks/custom-python-artifact-kwargs\2@g;
s@\(doc_url.*[\"']\)protobuf-go\([\"']\)@\1docs/go/integrations/protobuf\2@g;
s@\(doc_url.*[\"']\)protobuf-python\([\"']\)@\1docs/python/integrations/protobuf-and-grpc\2@g;
s@\(doc_url.*[\"']\)python-check-goal\([\"']\)@\1docs/python/goals/check\2@g;
s@\(doc_url.*[\"']\)python-distributions\([\"']\)@\1docs/python/overview/building-distributions\2@g;
s@\(doc_url.*[\"']\)python-interpreter-compatibility\([\"']\)@\1docs/python/overview/interpreter-compatibility\2@g;
s@\(doc_url.*[\"']\)python-lockfiles#lockfiles-for-tools\([\"']\)@\1docs/python/overview/lockfiles#lockfiles-for-tools\2@g;
s@\(doc_url.*[\"']\)python-test-goal\([\"']\)@\1docs/python/goals/test\2@g;
s@\(doc_url.*[\"']\)python-test-goal#pytest-version-and-plugins\([\"']\)@\1docs/python/goals/test#pytest-version-and-plugins\2@g;
s@\(doc_url.*[\"']\)python-third-party-dependencies\([\"']\)@\1docs/python/overview/third-party-dependencies\2@g;
s@\(doc_url.*[\"']\)python-third-party-dependencies#local-requirements\([\"']\)@\1docs/python/overview/third-party-dependencies#local-requirements\2@g;
s@\(doc_url.*[\"']\)python-third-party-dependencies#multiple-lockfiles\([\"']\)@\1docs/python/overview/lockfiles#multiple-lockfiles\2@g;
s@\(doc_url.*[\"']\)python-third-party-dependencies#user-lockfiles\([\"']\)@\1docs/python/overview/third-party-dependencies#user-lockfiles\2@g;
s@\(doc_url.*[\"']\)reference-deploy_jar#coderesolvecode\([\"']\)@\1reference/targets/deploy_jar#resolve\2@g;
s@\(doc_url.*[\"']\)reference-global#remote_oauth_bearer_token\([\"']\)@\1reference/global-options#remote_oauth_bearer_token\2@g;
s@\(doc_url.*[\"']\)remote-caching-execution\([\"']\)@\1docs/using-pants/remote-caching-and-execution\2@g;
s@\(doc_url.*[\"']\)source-roots\([\"']\)@\1docs/using-pants/key-concepts/source-roots\2@g;
s@\(doc_url.*[\"']\)tagging-docker-images\([\"']\)@\1docs/docker/tagging-docker-images\2@g;
s@\(doc_url.*[\"']\)target-api-concepts\([\"']\)@\1docs/writing-plugins/the-target-api/concepts\2@g;
s@\(doc_url.*[\"']\)targets\([\"']\)@\1docs/using-pants/key-concepts/targets-and-build-files\2@g;
s@\(doc_url.*[\"']\)targets#field-default-values\([\"']\)@\1docs/using-pants/key-concepts/targets-and-build-files#field-default-values\2@g;
s@\(doc_url.*[\"']\)targets#dependencies-and-dependency-inference\([\"']\)@\1docs/using-pants/key-concepts/targets-and-build-files#dependencies-and-dependency-inference\2@g;
s@\(doc_url.*[\"']\)thrift-python\([\"']\)@\1docs/python/integrations/thrift\2@g;
s@\(doc_url.*[\"']\)troubleshooting\([\"']\)@\1docs/using-pants/troubleshooting-common-issues\2@g;
s@\(doc_url.*[\"']\)troubleshooting#import-errors-and-missing-dependencies\([\"']\)@\1docs/using-pants/troubleshooting-common-issues#import-errors-and-missing-dependencies\2@g;
s@\(doc_url.*[\"']\)upgrade-tips\([\"']\)@\1docs/releases/upgrade-tips\2@g;
"

```shell
git ls-files '*.py' '*.rs' | xargs sed -i '' "
s@\(doc_url.*[\"']\)getting-help\([\"']\)@\1community/getting-help\2@g;
s@\(doc_url.*[\"']\)advanced-target-selection\([\"']\)@\1docs/using-pants/advanced-target-selection\2@g;
s@\(doc_url.*[\"']\)anonymous-telemetry\([\"']\)@\1docs/using-pants/anonymous-telemetry\2@g;
s@\(doc_url.*[\"']\)awslambda-python\([\"']\)@\1docs/python/integrations/aws-lambda\2@g;
s@\(doc_url.*[\"']\)enabling-backends\([\"']\)@\1docs/using-pants/key-concepts/backends\2@g;
s@\(doc_url.*[\"']\)google-cloud-function-python\([\"']\)@\1docs/python/integrations/google-cloud-functions\2@g;
s@\(doc_url.*[\"']\)installation\([\"']\)@\1docs/getting-started/installing-pants\2@g;
s@\(doc_url.*[\"']\)macros\([\"']\)@\1docs/writing-plugins/macros\2@g;
s@\(doc_url.*[\"']\)options\([\"']\)@\1docs/using-pants/key-concepts/options\2@g;
s@\(doc_url.*[\"']\)options#addremove-semantics\([\"']\)@\1docs/using-pants/key-concepts/options#addremove-semantics\2@g;
s@\(doc_url.*[\"']\)options#config-file-entries\([\"']\)@\1docs/using-pants/key-concepts/options#config-file-entries\2@g;
s@\(doc_url.*[\"']\)options#config-file-interpolation\([\"']\)@\1docs/using-pants/key-concepts/options#config-file-interpolation\2@g;
s@\(doc_url.*[\"']\)options#pantsrc-file\([\"']\)@\1docs/using-pants/key-concepts/options#pantsrc-file\2@g;
s@\(doc_url.*[\"']\)options#reading-individual-option-values-from-files\([\"']\)@\1docs/using-pants/key-concepts/options#reading-individual-option-values-from-files\2@g;
s@\(doc_url.*[\"']\)pex\([\"']\)@\1docs/python/overview/pex\2@g;
s@\(doc_url.*[\"']\)plugin-upgrade-guide\([\"']\)@\1docs/writing-plugins/common-plugin-tasks/plugin-upgrade-guide\2@g;
s@\(doc_url.*[\"']\)plugins-overview\([\"']\)@\1docs/writing-plugins/overview\2@g;
s@\(doc_url.*[\"']\)plugins-setup-py\([\"']\)@\1docs/writing-plugins/common-plugin-tasks/custom-python-artifact-kwargs\2@g;
s@\(doc_url.*[\"']\)protobuf-go\([\"']\)@\1docs/go/integrations/protobuf\2@g;
s@\(doc_url.*[\"']\)protobuf-python\([\"']\)@\1docs/python/integrations/protobuf-and-grpc\2@g;
s@\(doc_url.*[\"']\)python-check-goal\([\"']\)@\1docs/python/goals/check\2@g;
s@\(doc_url.*[\"']\)python-distributions\([\"']\)@\1docs/python/overview/building-distributions\2@g;
s@\(doc_url.*[\"']\)python-interpreter-compatibility\([\"']\)@\1docs/python/overview/interpreter-compatibility\2@g;
s@\(doc_url.*[\"']\)python-lockfiles#lockfiles-for-tools\([\"']\)@\1docs/python/overview/lockfiles#lockfiles-for-tools\2@g;
s@\(doc_url.*[\"']\)python-test-goal\([\"']\)@\1docs/python/goals/test\2@g;
s@\(doc_url.*[\"']\)python-test-goal#pytest-version-and-plugins\([\"']\)@\1docs/python/goals/test#pytest-version-and-plugins\2@g;
s@\(doc_url.*[\"']\)python-third-party-dependencies\([\"']\)@\1docs/python/overview/third-party-dependencies\2@g;
s@\(doc_url.*[\"']\)python-third-party-dependencies#local-requirements\([\"']\)@\1docs/python/overview/third-party-dependencies#local-requirements\2@g;
s@\(doc_url.*[\"']\)python-third-party-dependencies#multiple-lockfiles\([\"']\)@\1docs/python/overview/lockfiles#multiple-lockfiles\2@g;
s@\(doc_url.*[\"']\)python-third-party-dependencies#user-lockfiles\([\"']\)@\1docs/python/overview/third-party-dependencies#user-lockfiles\2@g;
s@\(doc_url.*[\"']\)reference-deploy_jar#coderesolvecode\([\"']\)@\1reference/targets/deploy_jar#resolve\2@g;
s@\(doc_url.*[\"']\)reference-global#remote_oauth_bearer_token\([\"']\)@\1reference/global-options#remote_oauth_bearer_token\2@g;
s@\(doc_url.*[\"']\)remote-caching-execution\([\"']\)@\1docs/using-pants/remote-caching-and-execution\2@g;
s@\(doc_url.*[\"']\)source-roots\([\"']\)@\1docs/using-pants/key-concepts/source-roots\2@g;
s@\(doc_url.*[\"']\)tagging-docker-images\([\"']\)@\1docs/docker/tagging-docker-images\2@g;
s@\(doc_url.*[\"']\)target-api-concepts\([\"']\)@\1docs/writing-plugins/the-target-api/concepts\2@g;
s@\(doc_url.*[\"']\)targets\([\"']\)@\1docs/using-pants/key-concepts/targets-and-build-files\2@g;
s@\(doc_url.*[\"']\)targets#field-default-values\([\"']\)@\1docs/using-pants/key-concepts/targets-and-build-files#field-default-values\2@g;
s@\(doc_url.*[\"']\)targets#dependencies-and-dependency-inference\([\"']\)@\1docs/using-pants/key-concepts/targets-and-build-files#dependencies-and-dependency-inference\2@g;
s@\(doc_url.*[\"']\)thrift-python\([\"']\)@\1docs/python/integrations/thrift\2@g;
s@\(doc_url.*[\"']\)troubleshooting\([\"']\)@\1docs/using-pants/troubleshooting-common-issues\2@g;
s@\(doc_url.*[\"']\)troubleshooting#import-errors-and-missing-dependencies\([\"']\)@\1docs/using-pants/troubleshooting-common-issues#import-errors-and-missing-dependencies\2@g;
s@\(doc_url.*[\"']\)upgrade-tips\([\"']\)@\1docs/releases/upgrade-tips\2@g;
"
```
Copy link
Member

@thejcannon thejcannon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome!

src/python/pants/util/docutil.py Outdated Show resolved Hide resolved
Copy link
Member

@thejcannon thejcannon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pending maybe typo

src/python/pants/util/docutil.py Outdated Show resolved Hide resolved
@huonw huonw merged commit ef03520 into main Feb 21, 2024
24 checks passed
@huonw huonw deleted the huonw/20427-doc-url branch February 21, 2024 00:22
WorkerPants pushed a commit that referenced this pull request Feb 21, 2024
This PR fixes #20427 by updating our use of `doc_url` to reflect the new
website structure. It does this semi-automatically, spread across
separate commits:

1. Update the `doc_url` function to reflect the new website structure,
including support for unversioned URLs like
https://www.pantsbuild.org/community/getting-help
2. Apply automated rewrites (see below for details)
3. Do one or two manual fix-ups

The automatic rewrites were done by:

1. finding the args passed to `doc_url`: `rg --only-matching
"doc_url\([^)]*\)" . | cut -f2 -d\( | cut -f1 -d\) | tr \' \" | sort -u`
2. put them into CSV file, then use
https://github.com/pantsbuild/pantsbuild.org/blob/92d5ce8d3442890c3fbeb0fade620b762ae2aa7e/old_site_redirects.js
to find the new URL for each one
3. transform the CSV into a long series of `s@...@...@` substitutions
for `sed`
4. apply them with `git ls-files '*.py' '*.rs' | xargs sed -i '' "..."`
5. do a basic check that the appropriate files that call `doc_url` were
updated: identify non-updated files via `comm -13 <(git diff --name-only
be42bc0^...be42bc0) <( rg --files-with-matches doc_url | sort)`.
The files without updates are expected, e.g.
- `src/python/pants/util/docutil.py` that defines `doc_url` itself (and
the test file etc.)
- `src/python/pants/backend/codegen/avro/target_types.py` that didn't
have an immediately obvious replacement, with more examples in #20584

This is marked for cherry-picking to 2.20.x. There _are_ redirects for
the URLs for 2.20, but it seems a bit annoying to rely on them, if we
don't have to.

<details>

```csv
old,new
"getting-help","community/getting-help"
"advanced-target-selection","docs/using-pants/advanced-target-selection"
"anonymous-telemetry","docs/using-pants/anonymous-telemetry"
"awslambda-python","docs/python/integrations/aws-lambda"
"enabling-backends","docs/using-pants/key-concepts/backends"
"google-cloud-function-python","docs/python/integrations/google-cloud-functions"
"installation","docs/getting-started/installing-pants"
"macros","docs/writing-plugins/macros"
"options","docs/using-pants/key-concepts/options"
"options#addremove-semantics","docs/using-pants/key-concepts/options#addremove-semantics"
"options#config-file-entries","docs/using-pants/key-concepts/options#config-file-entries"
"options#config-file-interpolation","docs/using-pants/key-concepts/options#config-file-interpolation"
"options#pantsrc-file","docs/using-pants/key-concepts/options#pantsrc-file"
"options#reading-individual-option-values-from-files","docs/using-pants/key-concepts/options#reading-individual-option-values-from-files"
"pex","docs/python/overview/pex"
"plugin-upgrade-guide","docs/writing-plugins/common-plugin-tasks/plugin-upgrade-guide"
"plugins-overview","docs/writing-plugins/overview"
"plugins-setup-py","docs/writing-plugins/common-plugin-tasks/custom-python-artifact-kwargs"
"protobuf-go","docs/go/integrations/protobuf"
"protobuf-python","docs/python/integrations/protobuf-and-grpc"
"python-check-goal","docs/python/goals/check"
"python-distributions","docs/python/overview/building-distributions"
"python-interpreter-compatibility","docs/python/overview/interpreter-compatibility"
"python-lockfiles#lockfiles-for-tools","docs/python/overview/lockfiles#lockfiles-for-tools"
"python-test-goal","docs/python/goals/test"
"python-test-goal#pytest-version-and-plugins","docs/python/goals/test#pytest-version-and-plugins"
"python-third-party-dependencies","docs/python/overview/third-party-dependencies"
"python-third-party-dependencies#local-requirements","docs/python/overview/third-party-dependencies#local-requirements"
"python-third-party-dependencies#multiple-lockfiles","docs/python/overview/lockfiles#multiple-lockfiles"
"python-third-party-dependencies#user-lockfiles","docs/python/overview/third-party-dependencies#user-lockfiles"
"reference-deploy_jar#coderesolvecode","reference/targets/deploy_jar#resolve"
"reference-global#remote_oauth_bearer_token","reference/global-options#remote_oauth_bearer_token"
"remote-caching-execution","docs/using-pants/remote-caching-and-execution"
"source-roots","docs/using-pants/key-concepts/source-roots"
"tagging-docker-images","docs/docker/tagging-docker-images"
"target-api-concepts","docs/writing-plugins/the-target-api/concepts"
"targets","docs/using-pants/key-concepts/targets-and-build-files"
"targets#field-default-values","docs/using-pants/key-concepts/targets-and-build-files#field-default-values"
"targets#dependencies-and-dependency-inference","docs/using-pants/key-concepts/targets-and-build-files#dependencies-and-dependency-inference"
"thrift-python","docs/python/integrations/thrift"
"troubleshooting","docs/using-pants/troubleshooting-common-issues"
"troubleshooting#import-errors-and-missing-dependencies","docs/using-pants/troubleshooting-common-issues#import-errors-and-missing-dependencies"
"upgrade-tips","docs/releases/upgrade-tips"
```

```python
import pandas as pd
print("\n".join(
  f"""s@\\(doc_url.*[\\"']\\){r.old}\\([\\"']\\)@\\1{r.new}\\2@g;"""
  for _, r in pd.read_csv("replacements.csv").iterrows()
))
```

```shell
git ls-files '*.py' '*.rs' | xargs sed -i '' "
s@\(doc_url.*[\"']\)getting-help\([\"']\)@\1community/getting-help\2@g;
s@\(doc_url.*[\"']\)advanced-target-selection\([\"']\)@\1docs/using-pants/advanced-target-selection\2@g;
s@\(doc_url.*[\"']\)anonymous-telemetry\([\"']\)@\1docs/using-pants/anonymous-telemetry\2@g;
s@\(doc_url.*[\"']\)awslambda-python\([\"']\)@\1docs/python/integrations/aws-lambda\2@g;
s@\(doc_url.*[\"']\)enabling-backends\([\"']\)@\1docs/using-pants/key-concepts/backends\2@g;
s@\(doc_url.*[\"']\)google-cloud-function-python\([\"']\)@\1docs/python/integrations/google-cloud-functions\2@g;
s@\(doc_url.*[\"']\)installation\([\"']\)@\1docs/getting-started/installing-pants\2@g;
s@\(doc_url.*[\"']\)macros\([\"']\)@\1docs/writing-plugins/macros\2@g;
s@\(doc_url.*[\"']\)options\([\"']\)@\1docs/using-pants/key-concepts/options\2@g;
s@\(doc_url.*[\"']\)options#addremove-semantics\([\"']\)@\1docs/using-pants/key-concepts/options#addremove-semantics\2@g;
s@\(doc_url.*[\"']\)options#config-file-entries\([\"']\)@\1docs/using-pants/key-concepts/options#config-file-entries\2@g;
s@\(doc_url.*[\"']\)options#config-file-interpolation\([\"']\)@\1docs/using-pants/key-concepts/options#config-file-interpolation\2@g;
s@\(doc_url.*[\"']\)options#pantsrc-file\([\"']\)@\1docs/using-pants/key-concepts/options#pantsrc-file\2@g;
s@\(doc_url.*[\"']\)options#reading-individual-option-values-from-files\([\"']\)@\1docs/using-pants/key-concepts/options#reading-individual-option-values-from-files\2@g;
s@\(doc_url.*[\"']\)pex\([\"']\)@\1docs/python/overview/pex\2@g;
s@\(doc_url.*[\"']\)plugin-upgrade-guide\([\"']\)@\1docs/writing-plugins/common-plugin-tasks/plugin-upgrade-guide\2@g;
s@\(doc_url.*[\"']\)plugins-overview\([\"']\)@\1docs/writing-plugins/overview\2@g;
s@\(doc_url.*[\"']\)plugins-setup-py\([\"']\)@\1docs/writing-plugins/common-plugin-tasks/custom-python-artifact-kwargs\2@g;
s@\(doc_url.*[\"']\)protobuf-go\([\"']\)@\1docs/go/integrations/protobuf\2@g;
s@\(doc_url.*[\"']\)protobuf-python\([\"']\)@\1docs/python/integrations/protobuf-and-grpc\2@g;
s@\(doc_url.*[\"']\)python-check-goal\([\"']\)@\1docs/python/goals/check\2@g;
s@\(doc_url.*[\"']\)python-distributions\([\"']\)@\1docs/python/overview/building-distributions\2@g;
s@\(doc_url.*[\"']\)python-interpreter-compatibility\([\"']\)@\1docs/python/overview/interpreter-compatibility\2@g;
s@\(doc_url.*[\"']\)python-lockfiles#lockfiles-for-tools\([\"']\)@\1docs/python/overview/lockfiles#lockfiles-for-tools\2@g;
s@\(doc_url.*[\"']\)python-test-goal\([\"']\)@\1docs/python/goals/test\2@g;
s@\(doc_url.*[\"']\)python-test-goal#pytest-version-and-plugins\([\"']\)@\1docs/python/goals/test#pytest-version-and-plugins\2@g;
s@\(doc_url.*[\"']\)python-third-party-dependencies\([\"']\)@\1docs/python/overview/third-party-dependencies\2@g;
s@\(doc_url.*[\"']\)python-third-party-dependencies#local-requirements\([\"']\)@\1docs/python/overview/third-party-dependencies#local-requirements\2@g;
s@\(doc_url.*[\"']\)python-third-party-dependencies#multiple-lockfiles\([\"']\)@\1docs/python/overview/lockfiles#multiple-lockfiles\2@g;
s@\(doc_url.*[\"']\)python-third-party-dependencies#user-lockfiles\([\"']\)@\1docs/python/overview/third-party-dependencies#user-lockfiles\2@g;
s@\(doc_url.*[\"']\)reference-deploy_jar#coderesolvecode\([\"']\)@\1reference/targets/deploy_jar#resolve\2@g;
s@\(doc_url.*[\"']\)reference-global#remote_oauth_bearer_token\([\"']\)@\1reference/global-options#remote_oauth_bearer_token\2@g;
s@\(doc_url.*[\"']\)remote-caching-execution\([\"']\)@\1docs/using-pants/remote-caching-and-execution\2@g;
s@\(doc_url.*[\"']\)source-roots\([\"']\)@\1docs/using-pants/key-concepts/source-roots\2@g;
s@\(doc_url.*[\"']\)tagging-docker-images\([\"']\)@\1docs/docker/tagging-docker-images\2@g;
s@\(doc_url.*[\"']\)target-api-concepts\([\"']\)@\1docs/writing-plugins/the-target-api/concepts\2@g;
s@\(doc_url.*[\"']\)targets\([\"']\)@\1docs/using-pants/key-concepts/targets-and-build-files\2@g;
s@\(doc_url.*[\"']\)targets#field-default-values\([\"']\)@\1docs/using-pants/key-concepts/targets-and-build-files#field-default-values\2@g;
s@\(doc_url.*[\"']\)targets#dependencies-and-dependency-inference\([\"']\)@\1docs/using-pants/key-concepts/targets-and-build-files#dependencies-and-dependency-inference\2@g;
s@\(doc_url.*[\"']\)thrift-python\([\"']\)@\1docs/python/integrations/thrift\2@g;
s@\(doc_url.*[\"']\)troubleshooting\([\"']\)@\1docs/using-pants/troubleshooting-common-issues\2@g;
s@\(doc_url.*[\"']\)troubleshooting#import-errors-and-missing-dependencies\([\"']\)@\1docs/using-pants/troubleshooting-common-issues#import-errors-and-missing-dependencies\2@g;
s@\(doc_url.*[\"']\)upgrade-tips\([\"']\)@\1docs/releases/upgrade-tips\2@g;
"
```

</details>
@WorkerPants
Copy link
Member

I tried to automatically cherry-pick this change back to each relevant milestone, so that it is available in those older releases of Pants.

✔️ 2.20.x

Successfully opened #20588.


Thanks again for your contributions!

🤖 Beep Boop here's my run link

huonw added a commit that referenced this pull request Feb 21, 2024
This PR fixes #20427 by updating our use of `doc_url` to reflect the new
website structure. It does this semi-automatically, spread across
separate commits:

1. Update the `doc_url` function to reflect the new website structure,
including support for unversioned URLs like
https://www.pantsbuild.org/community/getting-help
2. Apply automated rewrites (see below for details)
3. Do one or two manual fix-ups

The automatic rewrites were done by:

1. finding the args passed to `doc_url`: `rg --only-matching
"doc_url\([^)]*\)" . | cut -f2 -d\( | cut -f1 -d\) | tr \' \" | sort -u`
2. put them into CSV file, then use
https://github.com/pantsbuild/pantsbuild.org/blob/92d5ce8d3442890c3fbeb0fade620b762ae2aa7e/old_site_redirects.js
to find the new URL for each one
3. transform the CSV into a long series of `s@...@...@` substitutions
for `sed`
4. apply them with `git ls-files '*.py' '*.rs' | xargs sed -i '' "..."`
5. do a basic check that the appropriate files that call `doc_url` were
updated: identify non-updated files via `comm -13 <(git diff --name-only
be42bc0^...be42bc0) <( rg --files-with-matches doc_url | sort)`.
The files without updates are expected, e.g.
- `src/python/pants/util/docutil.py` that defines `doc_url` itself (and
the test file etc.)
- `src/python/pants/backend/codegen/avro/target_types.py` that didn't
have an immediately obvious replacement, with more examples in #20584

This is marked for cherry-picking to 2.20.x. There _are_ redirects for
the URLs for 2.20, but it seems a bit annoying to rely on them, if we
don't have to.

<details>

```csv
old,new
"getting-help","community/getting-help"
"advanced-target-selection","docs/using-pants/advanced-target-selection"
"anonymous-telemetry","docs/using-pants/anonymous-telemetry"
"awslambda-python","docs/python/integrations/aws-lambda"
"enabling-backends","docs/using-pants/key-concepts/backends"
"google-cloud-function-python","docs/python/integrations/google-cloud-functions"
"installation","docs/getting-started/installing-pants"
"macros","docs/writing-plugins/macros"
"options","docs/using-pants/key-concepts/options"
"options#addremove-semantics","docs/using-pants/key-concepts/options#addremove-semantics"
"options#config-file-entries","docs/using-pants/key-concepts/options#config-file-entries"
"options#config-file-interpolation","docs/using-pants/key-concepts/options#config-file-interpolation"
"options#pantsrc-file","docs/using-pants/key-concepts/options#pantsrc-file"
"options#reading-individual-option-values-from-files","docs/using-pants/key-concepts/options#reading-individual-option-values-from-files"
"pex","docs/python/overview/pex"
"plugin-upgrade-guide","docs/writing-plugins/common-plugin-tasks/plugin-upgrade-guide"
"plugins-overview","docs/writing-plugins/overview"
"plugins-setup-py","docs/writing-plugins/common-plugin-tasks/custom-python-artifact-kwargs"
"protobuf-go","docs/go/integrations/protobuf"
"protobuf-python","docs/python/integrations/protobuf-and-grpc"
"python-check-goal","docs/python/goals/check"
"python-distributions","docs/python/overview/building-distributions"
"python-interpreter-compatibility","docs/python/overview/interpreter-compatibility"
"python-lockfiles#lockfiles-for-tools","docs/python/overview/lockfiles#lockfiles-for-tools"
"python-test-goal","docs/python/goals/test"
"python-test-goal#pytest-version-and-plugins","docs/python/goals/test#pytest-version-and-plugins"
"python-third-party-dependencies","docs/python/overview/third-party-dependencies"
"python-third-party-dependencies#local-requirements","docs/python/overview/third-party-dependencies#local-requirements"
"python-third-party-dependencies#multiple-lockfiles","docs/python/overview/lockfiles#multiple-lockfiles"
"python-third-party-dependencies#user-lockfiles","docs/python/overview/third-party-dependencies#user-lockfiles"
"reference-deploy_jar#coderesolvecode","reference/targets/deploy_jar#resolve"
"reference-global#remote_oauth_bearer_token","reference/global-options#remote_oauth_bearer_token"
"remote-caching-execution","docs/using-pants/remote-caching-and-execution"
"source-roots","docs/using-pants/key-concepts/source-roots"
"tagging-docker-images","docs/docker/tagging-docker-images"
"target-api-concepts","docs/writing-plugins/the-target-api/concepts"
"targets","docs/using-pants/key-concepts/targets-and-build-files"
"targets#field-default-values","docs/using-pants/key-concepts/targets-and-build-files#field-default-values"
"targets#dependencies-and-dependency-inference","docs/using-pants/key-concepts/targets-and-build-files#dependencies-and-dependency-inference"
"thrift-python","docs/python/integrations/thrift"
"troubleshooting","docs/using-pants/troubleshooting-common-issues"
"troubleshooting#import-errors-and-missing-dependencies","docs/using-pants/troubleshooting-common-issues#import-errors-and-missing-dependencies"
"upgrade-tips","docs/releases/upgrade-tips"
```

```python
import pandas as pd
print("\n".join(
  f"""s@\\(doc_url.*[\\"']\\){r.old}\\([\\"']\\)@\\1{r.new}\\2@g;"""
  for _, r in pd.read_csv("replacements.csv").iterrows()
))
```

```shell
git ls-files '*.py' '*.rs' | xargs sed -i '' "
s@\(doc_url.*[\"']\)getting-help\([\"']\)@\1community/getting-help\2@g;
s@\(doc_url.*[\"']\)advanced-target-selection\([\"']\)@\1docs/using-pants/advanced-target-selection\2@g;
s@\(doc_url.*[\"']\)anonymous-telemetry\([\"']\)@\1docs/using-pants/anonymous-telemetry\2@g;
s@\(doc_url.*[\"']\)awslambda-python\([\"']\)@\1docs/python/integrations/aws-lambda\2@g;
s@\(doc_url.*[\"']\)enabling-backends\([\"']\)@\1docs/using-pants/key-concepts/backends\2@g;
s@\(doc_url.*[\"']\)google-cloud-function-python\([\"']\)@\1docs/python/integrations/google-cloud-functions\2@g;
s@\(doc_url.*[\"']\)installation\([\"']\)@\1docs/getting-started/installing-pants\2@g;
s@\(doc_url.*[\"']\)macros\([\"']\)@\1docs/writing-plugins/macros\2@g;
s@\(doc_url.*[\"']\)options\([\"']\)@\1docs/using-pants/key-concepts/options\2@g;
s@\(doc_url.*[\"']\)options#addremove-semantics\([\"']\)@\1docs/using-pants/key-concepts/options#addremove-semantics\2@g;
s@\(doc_url.*[\"']\)options#config-file-entries\([\"']\)@\1docs/using-pants/key-concepts/options#config-file-entries\2@g;
s@\(doc_url.*[\"']\)options#config-file-interpolation\([\"']\)@\1docs/using-pants/key-concepts/options#config-file-interpolation\2@g;
s@\(doc_url.*[\"']\)options#pantsrc-file\([\"']\)@\1docs/using-pants/key-concepts/options#pantsrc-file\2@g;
s@\(doc_url.*[\"']\)options#reading-individual-option-values-from-files\([\"']\)@\1docs/using-pants/key-concepts/options#reading-individual-option-values-from-files\2@g;
s@\(doc_url.*[\"']\)pex\([\"']\)@\1docs/python/overview/pex\2@g;
s@\(doc_url.*[\"']\)plugin-upgrade-guide\([\"']\)@\1docs/writing-plugins/common-plugin-tasks/plugin-upgrade-guide\2@g;
s@\(doc_url.*[\"']\)plugins-overview\([\"']\)@\1docs/writing-plugins/overview\2@g;
s@\(doc_url.*[\"']\)plugins-setup-py\([\"']\)@\1docs/writing-plugins/common-plugin-tasks/custom-python-artifact-kwargs\2@g;
s@\(doc_url.*[\"']\)protobuf-go\([\"']\)@\1docs/go/integrations/protobuf\2@g;
s@\(doc_url.*[\"']\)protobuf-python\([\"']\)@\1docs/python/integrations/protobuf-and-grpc\2@g;
s@\(doc_url.*[\"']\)python-check-goal\([\"']\)@\1docs/python/goals/check\2@g;
s@\(doc_url.*[\"']\)python-distributions\([\"']\)@\1docs/python/overview/building-distributions\2@g;
s@\(doc_url.*[\"']\)python-interpreter-compatibility\([\"']\)@\1docs/python/overview/interpreter-compatibility\2@g;
s@\(doc_url.*[\"']\)python-lockfiles#lockfiles-for-tools\([\"']\)@\1docs/python/overview/lockfiles#lockfiles-for-tools\2@g;
s@\(doc_url.*[\"']\)python-test-goal\([\"']\)@\1docs/python/goals/test\2@g;
s@\(doc_url.*[\"']\)python-test-goal#pytest-version-and-plugins\([\"']\)@\1docs/python/goals/test#pytest-version-and-plugins\2@g;
s@\(doc_url.*[\"']\)python-third-party-dependencies\([\"']\)@\1docs/python/overview/third-party-dependencies\2@g;
s@\(doc_url.*[\"']\)python-third-party-dependencies#local-requirements\([\"']\)@\1docs/python/overview/third-party-dependencies#local-requirements\2@g;
s@\(doc_url.*[\"']\)python-third-party-dependencies#multiple-lockfiles\([\"']\)@\1docs/python/overview/lockfiles#multiple-lockfiles\2@g;
s@\(doc_url.*[\"']\)python-third-party-dependencies#user-lockfiles\([\"']\)@\1docs/python/overview/third-party-dependencies#user-lockfiles\2@g;
s@\(doc_url.*[\"']\)reference-deploy_jar#coderesolvecode\([\"']\)@\1reference/targets/deploy_jar#resolve\2@g;
s@\(doc_url.*[\"']\)reference-global#remote_oauth_bearer_token\([\"']\)@\1reference/global-options#remote_oauth_bearer_token\2@g;
s@\(doc_url.*[\"']\)remote-caching-execution\([\"']\)@\1docs/using-pants/remote-caching-and-execution\2@g;
s@\(doc_url.*[\"']\)source-roots\([\"']\)@\1docs/using-pants/key-concepts/source-roots\2@g;
s@\(doc_url.*[\"']\)tagging-docker-images\([\"']\)@\1docs/docker/tagging-docker-images\2@g;
s@\(doc_url.*[\"']\)target-api-concepts\([\"']\)@\1docs/writing-plugins/the-target-api/concepts\2@g;
s@\(doc_url.*[\"']\)targets\([\"']\)@\1docs/using-pants/key-concepts/targets-and-build-files\2@g;
s@\(doc_url.*[\"']\)targets#field-default-values\([\"']\)@\1docs/using-pants/key-concepts/targets-and-build-files#field-default-values\2@g;
s@\(doc_url.*[\"']\)targets#dependencies-and-dependency-inference\([\"']\)@\1docs/using-pants/key-concepts/targets-and-build-files#dependencies-and-dependency-inference\2@g;
s@\(doc_url.*[\"']\)thrift-python\([\"']\)@\1docs/python/integrations/thrift\2@g;
s@\(doc_url.*[\"']\)troubleshooting\([\"']\)@\1docs/using-pants/troubleshooting-common-issues\2@g;
s@\(doc_url.*[\"']\)troubleshooting#import-errors-and-missing-dependencies\([\"']\)@\1docs/using-pants/troubleshooting-common-issues#import-errors-and-missing-dependencies\2@g;
s@\(doc_url.*[\"']\)upgrade-tips\([\"']\)@\1docs/releases/upgrade-tips\2@g;
"
```

</details>

Co-authored-by: Huon Wilson <huon@exoflare.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

pants.util.docutil.doc_url should be updated for new docs site
4 participants