NestedLoopsJoin memory tracking may be insufficient

[alamb]

Is your feature request related to a problem or challenge?

Similarly to #7848, @metesynnada noted #8020 (comment) that it is possible for NestedLoopsJoin to generate a single (very) large RecordBatch. For certain pathalogical queries this may lead to DataFusion far exceeding its memory limits and erroring out

Describe the solution you'd like

Implement / adapt the same approach as @korowa did in #8020 (❤️ ) to incrementally create join output for joins that match many keys rather than doing it all at once.

Describe alternatives you've considered

No response

Additional context

No response

[yyy1000]

I'd like a try to help it. :)

[alamb]

THis one may be tricky, FWIW. The join code is not simple.

[yyy1000]

Aha, seems true. Maybe I can leave it here now and find some not so difficult. And I think I could fix it when I get more familiar with the code.❤️