Optimise Export Memory Usage

[simonr44]

Description

This PR passes records to the export function in export_utils in small batches rather than all at once to minimise the memory consumption of the csv export process.

Motivation and Context

When exporting lots of records at one time, the php memory consumption can quickly exceed the default memory_limit
This change will process the records in small batches, consuming less memory.

Large exports of +/- 100k records can easily exceed 750MB peak memory consumption. With this change consumption remains below the php default value of 128MB ( Approx 74MB in testing, but will vary based on record contents )

How To Test This

Select all records on the list view and perform a bulk action export
Best performed on a test DB with many thousands of records.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Final checklist

• My code follows the code style of this project found here.
• My change requires a change to the documentation.
• I have read the How to Contribute guidelines.

[clemente-raposo]

LGTM

[chris001]

Would be great to have this on import also, if not already there, so that users wouldn't have to continue to import large number of records by the command line.

[johnM2401]

Hey @simonr44 !
After some testing, this does indeed seem to cut resource costs quite notably.
On a dataset of about 100k records locally, it seems to cut ~400mb of memory usage.

However, I've noticed one possible issue.

It seems as though it adds the Header row for each batch of records, back into the CSV.

I've grabbed a CSV from pre-fix and post-fix. (left and right respectively), using the same dataset.
However, the post-fix CSV appears to have 110 more rows:

After sorting A->Z on the "Name" header row, I noticed that there were now 110 extra rows that contained the header row items
See screenshot:

I imagine the CSV should only have the initial header row?
This might cause data issues if the CSV is ever re-imported to a CRM.
Could you have a look when you get a chance?

Thank you!