Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AIP-58: Add Airflow ObjectStore (AFS) #34729

Merged
merged 83 commits into from
Oct 27, 2023
Merged

AIP-58: Add Airflow ObjectStore (AFS) #34729

merged 83 commits into from
Oct 27, 2023

Commits on Oct 27, 2023

  1. IO

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    0c456ee View commit details
    Browse the repository at this point in the history

  2. Further work

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    d9c92f0 View commit details
    Browse the repository at this point in the history

  3. Add Airflow FS 

    This adds Airflow FS inspired by databricks fs (dbfs) and
partially ported from Apache Iceberg. It is based on fsspec,
which gives filelike compatibility
    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    bda96de View commit details
    Browse the repository at this point in the history

  4. Add fsspec dependencies

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    c064314 View commit details
    Browse the repository at this point in the history

  5. Move stuff to provider packages

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    29bf6b8 View commit details
    Browse the repository at this point in the history

  6. Add fsspec

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    ad22ed9 View commit details
    Browse the repository at this point in the history

  7. Use provider style plugins

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    58753fa View commit details
    Browse the repository at this point in the history

  8. Add plugin registration

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    0c4700c View commit details
    Browse the repository at this point in the history

  9. Move exception inline

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    f86c090 View commit details
    Browse the repository at this point in the history

  10. Clean ups

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    c7774bd View commit details
    Browse the repository at this point in the history

  11. Make FileIO work with connection ids

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    133fc4e View commit details
    Browse the repository at this point in the history

  12. Add simple mounts

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    ea5584a View commit details
    Browse the repository at this point in the history

  13. Add simple combinations

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    10d7582 View commit details
    Browse the repository at this point in the history

  14. Allow unmount to use str or Mount

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    6082452 View commit details
    Browse the repository at this point in the history

  15. Pre commit stuff - what a mess that creates :/

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    11a1f2d View commit details
    Browse the repository at this point in the history

  16. PY38 fixes

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    3d57e55 View commit details
    Browse the repository at this point in the history

  17. Address pre-commit

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    fbb151b View commit details
    Browse the repository at this point in the history

  18. Support contexts and PathLib concatenation

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    ba88b12 View commit details
    Browse the repository at this point in the history

  19. Add s3fs to devel

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    390c80d View commit details
    Browse the repository at this point in the history

  20. Use deterministic endpoints and generate fsid if not available

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    e8331cc View commit details
    Browse the repository at this point in the history

  21. Fix table

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    3d1595f View commit details
    Browse the repository at this point in the history

  22. Use PathLike object

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    48f47b4 View commit details
    Browse the repository at this point in the history

  23. Fix mypy and test

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    b2f5e26 View commit details
    Browse the repository at this point in the history

  24. Simplify implementaton

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    772b754 View commit details
    Browse the repository at this point in the history

  25. Use ObjectStoragePath directly

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    8a97977 View commit details
    Browse the repository at this point in the history

  26. Fix docstrings

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    0a46254 View commit details
    Browse the repository at this point in the history

  27. Check if samestore (maybe just switch to ObjectStorePath copy)

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    b6b92b2 View commit details
    Browse the repository at this point in the history

  28. Use class name instead of type

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    fdcd1ba View commit details
    Browse the repository at this point in the history

  29. Use shutil for copying between stores

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    b62778f View commit details
    Browse the repository at this point in the history

  30. Make sure to set alias only when not specified

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    45c8183 View commit details
    Browse the repository at this point in the history

  31. Use backing copy

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    1c1c2a9 View commit details
    Browse the repository at this point in the history

  32. Fix test

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    328eb00 View commit details
    Browse the repository at this point in the history

  33. Fix test

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    95da859 View commit details
    Browse the repository at this point in the history

  34. Implement caching for filesystems 

    Implement serde
Add extra tests
    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    6a1b525 View commit details
    Browse the repository at this point in the history

  35. Move FileTransferOperator to provider package

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    41785d0 View commit details
    Browse the repository at this point in the history

  36. Pin dependencies

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    44a03ae View commit details
    Browse the repository at this point in the history

  37. Pin aiobotocore until new release of fsspec

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    bd8d091 View commit details
    Browse the repository at this point in the history

  38. Address version name

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    6654d59 View commit details
    Browse the repository at this point in the history

  39. Don't copy paste too much

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    7df70f3 View commit details
    Browse the repository at this point in the history

  40. Use aws infrastructure for getting a session

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    cb6f442 View commit details
    Browse the repository at this point in the history

  41. Make sure endpoint_url is honored

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    1e40842 View commit details
    Browse the repository at this point in the history

  42. Remove s3fs from main and keep in provider

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    42e897a View commit details
    Browse the repository at this point in the history

  43. Use service config

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    c41dcfd View commit details
    Browse the repository at this point in the history

  44. remove s3fs when testing aws 

    remove common-io from 2.5 tests
    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    42908de View commit details
    Browse the repository at this point in the history

  45. Make sure prod can build

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    94ec2e3 View commit details
    Browse the repository at this point in the history

  46. Fix tests to not depend on s3fs

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    6a5de18 View commit details
    Browse the repository at this point in the history

  47. Readd s3fs to setup.py

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    9d7fcce View commit details
    Browse the repository at this point in the history

  48. Fix issues with docs

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    8b3b194 View commit details
    Browse the repository at this point in the history

  49. fix link

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    caf2253 View commit details
    Browse the repository at this point in the history

  50. Add example dag

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    841dbdc View commit details
    Browse the repository at this point in the history

  51. Optimize copy

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    96516af View commit details
    Browse the repository at this point in the history

  52. Extra

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    f085180 View commit details
    Browse the repository at this point in the history

  53. Regen images

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    6dc8c1a View commit details
    Browse the repository at this point in the history

  54. Fix docs

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    2 Configuration menu
    Copy the full SHA
    73a9f57 View commit details
    Browse the repository at this point in the history

  55. Update tests not to be dependent on s3fs

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    74ae073 View commit details
    Browse the repository at this point in the history

  56. Moved example test

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    b2fd6fc View commit details
    Browse the repository at this point in the history

  57. Add stat_result as a way to unify info and traditional stat_result

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    6dae720 View commit details
    Browse the repository at this point in the history

  58. Clean up

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    7b6a71e View commit details
    Browse the repository at this point in the history

  59. Add extra docs

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    1f125da View commit details
    Browse the repository at this point in the history

  60. Fix docs

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    91e0e94 View commit details
    Browse the repository at this point in the history

  61. Add words

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    acf81f5 View commit details
    Browse the repository at this point in the history

  62. Improve copying

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    671924c View commit details
    Browse the repository at this point in the history

  63. Upgrade fsspec and relax aiobotocore requirements

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    b1a4cbb View commit details
    Browse the repository at this point in the history

  64. Update docs/apache-airflow/core-concepts/objectstorage.rst 

    Co-authored-by: Hussein Awala <hussein@awala.fr>
    @bolkedebruin @hussein-awala
    bolkedebruin and hussein-awala committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    4dea223 View commit details
    Browse the repository at this point in the history

  65. Update docs/apache-airflow/core-concepts/objectstorage.rst 

    Co-authored-by: Hussein Awala <hussein@awala.fr>
    @bolkedebruin @hussein-awala
    bolkedebruin and hussein-awala committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    002de89 View commit details
    Browse the repository at this point in the history

  66. Revert "Upgrade fsspec and relax aiobotocore requirements" 

    This reverts commit d719e6333235fa0b185969b2ef5344a6d45dcc41.
    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    159c908 View commit details
    Browse the repository at this point in the history

  67. Relax aiobotocore

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    958eb51 View commit details
    Browse the repository at this point in the history

  68. Make copy work as expected

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    ec870c6 View commit details
    Browse the repository at this point in the history

  69. Revert "Relax aiobotocore" 

    This reverts commit 852cc7952f6a112988da72e966433e71db08fd30.
    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    a58000d View commit details
    Browse the repository at this point in the history

  70. Make rename work only within same store

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    b217eea View commit details
    Browse the repository at this point in the history

  71. Fix tests

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    855485a View commit details
    Browse the repository at this point in the history

  72. Fix test not te reuse alias

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    d07d044 View commit details
    Browse the repository at this point in the history

  73. Ensure templated fields for xcom

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    ed2f928 View commit details
    Browse the repository at this point in the history

  74. Improve handling of existing directories

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    cb8e4a4 View commit details
    Browse the repository at this point in the history

  75. Set aiobotocore to 2.7.0

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    980c3ee View commit details
    Browse the repository at this point in the history

  76. Allow larger versions of aiobotocore

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    1f26001 View commit details
    Browse the repository at this point in the history

  77. Update airflow/providers/amazon/aws/fs/s3.py 

    Co-authored-by: Andrey Anshin <Andrey.Anshin@taragol.is>
    @bolkedebruin @Taragolis
    bolkedebruin and Taragolis committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    8f8912b View commit details
    Browse the repository at this point in the history

  78. Add tests for s3fs

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    d7fc935 View commit details
    Browse the repository at this point in the history

  79. Add example dag

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    1af0dc3 View commit details
    Browse the repository at this point in the history

  80. Improve example

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    836131d View commit details
    Browse the repository at this point in the history

  81. Improve example

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    2e1cba6 View commit details
    Browse the repository at this point in the history

  82. Add tutorial and improve docs

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    88b9216 View commit details
    Browse the repository at this point in the history

  83. Add extra

    @bolkedebruin
    bolkedebruin committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    5099f45 View commit details
    Browse the repository at this point in the history