refactor: Added a generic FileStream (still in active development!)

[edgarrmondragon]

References:

• https://filesystem-spec.readthedocs.io/en/latest/api.html#base-classes

---

📚 Documentation preview 📚: https://meltano-sdk--2654.org.readthedocs.build/en/2654/

[codspeed-hq]

CodSpeed Performance Report

Merging #2654 will not alter performance

_{Comparing 2648-feat-add-a-generic-filestream-interface (e527cee) with main (56b496f)}

Summary

✅ 6 untouched benchmarks

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.50%. Comparing base (5c68b09) to head (e527cee).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2654      +/-   ##
==========================================
+ Coverage   90.21%   90.50%   +0.29%     
==========================================
  Files          58       62       +4     
  Lines        4895     4994      +99     
  Branches      964      974      +10     
==========================================
+ Hits         4416     4520     +104     
+ Misses        331      328       -3     
+ Partials      148      146       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[edgarrmondragon]

This is ready for review. Everything here is subject to change: naming conventions, implementation, abstractions so feel free to comment on those.

[edgarrmondragon]

I suggest reviewing this module in split view. All that's left after the refactor are the get_schema and read_file implementations.

[edgarrmondragon]

I suggest reviewing this module in split view.

CSV-specific settings were added and custom discovery was removed in favor of the default implementation.

[edgarrmondragon]

Using partitions allows us to track state for each individual file in merge mode.

[edgarrmondragon]

These are working now

FTP

{
    "filesystem": "ftp",
    "path": "fixtures/csv",
    "read_mode": "one_stream_per_file",
    "delimiter": "\t",
    "ftp": {
        "host": "127.0.0.1",
        "port": 21,
        "username": "my_ftp_user",
        "password": "my_ftp_password"
    }
}

SFTP

{
    "filesystem": "sftp",
    "path": "fixtures/csv",
    "read_mode": "one_stream_per_file",
    "delimiter": "\t",
    "sftp": {
        "host": "127.0.0.1",
        "port": 2022,
        "username": "my_ftp_user",
        "password": "my_ftp_password"
    }
}