Respect the ingestion limit if ingest_records is called multiple times

[stacimc]

Fixes

Fixes #[issue number] by @[issue author]
This is necessary pre-requisite work for refactoring Flickr (#585)

Description

Some of our provider ingesters iterate over a set of query params and call ingest_records for each. For example:

def ingest_records(self, **kwargs):
        for building in BUILDINGS:
            logger.info(f"Obtaining images of building {building}")
            super().ingest_records(building=building)

This causes a problem with ingestion_limit, an Airflow variable intended to restrict the number of records ingested (to speed up testing). Reaching the ingestion limit causes ingest_records to return early, but for DAGs like the above that call ingest_records multiple times, you'll have to hit the ingestion_limit once for every run of ingest_records.

This PR just updates the ProviderDataIngester to keep track of the number of records ingested across multiple runs of ingest_records, and respect the limit as soon as it has been reached.

Testing Instructions

just test. I verified the test failed before making the code changes, and passes now 😃

You can also try setting an ingestion_limit Airflow variable and then running finnish_museums_workflow, which has the mentioned problem. You should see in the logs that ingestion wasn't run for additional params.

Other tests:

• Try running a DAG that does not override ingest_records (example Cleveland) with an ingestion_limit set
• Try running DAGs without an ingestion limit

NOTE: The ingestion limit is still slightly fuzzy. Ingestion will be halted as soon as the number of records committed exceeds the limit, but the last batch it receives is fully processed (tldr if you have a limit of 100, you might see

Checklist

• My pull request has a descriptive title (not a vague title like Update index.md).
• My pull request targets the default branch of the repository (main) or a parent feature branch.
• My commit messages follow best practices.
• My code follows the established code style of the repository.
• I added or updated tests for the changes I made (if applicable).
• I added or updated documentation (if applicable).
• I tried running the project locally and verified that there are no visible errors.

Developer Certificate of Origin

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

[stacimc]

High priority because this is needed for Flickr (which will be calling ingest_records many times)

[zackkrida]

LGTM.

[AetherUnbound]

Looks good!