Host summary generation for large inventories is too slow

[ryanpetrello]

ISSUE TYPE

• Bug Report

SUMMARY

Generate an inventory with 5000 hosts, and run a playbook.
Observe that the playbook_on_stats event takes a very long time (depending on total hosts, and how your inventory is structured, several minutes+) to show up.

This code is super duper crazy slow (mostly because it generates lots of insert and update queries for large inventories):

https://github.com/ansible/awx/blob/devel/awx/main/models/events.py#L482

@pytest.mark.django_db
def test_host_summary_generation():
    hostnames = [f'Host {i}' for i in range(5000)]
    inv = Inventory()
    inv.save()
    for h in hostnames:
        Host(name=h, inventory_id=inv.id).save()
    j = Job(inventory=inv)
    j.save()
    JobEvent.create_from_data(
        job_id=j.pk,
        parent_uuid='abc123',
        event='playbook_on_stats',
        event_data={'ok': dict((hostname, 5) for hostname in hostnames)}
    ).save()

    je = JobEvent.objects.first()
    je._update_host_summary_from_stats(hostnames)
    assert j.job_host_summaries.count() == len(hostnames)
    assert sorted([s.host_name for s in j.job_host_summaries.all()]) == sorted(hostnames)

[AlanCoding]

Sound like a rinse-and-repeat of what #6290 did, just for all the other non-notification playbook_on_stats processing.

[ryanpetrello]

@AlanCoding yep, exactly.

[ryanpetrello]

This probably isn't incredibly urgent/pressing, because this code has existed in this slow state for quite some time as far as I can tell.

[ryanpetrello]

Here's a playbook_on_stats event with 5000 hosts:

Almost an entire minute to process this event.

[ryanpetrello]

Here's a playbook_on_stats event with 5000 hosts After my PR:

[kdelee]

Given that event processing time is effected by general backlog of other events to process I was not able to get a very consistent measurement, but can say that processing events on jobs that act on thousands of hosts happens in a reasonable amount of time, and that lag in event processing is no longer showing to be correlated with inventory size.

Now strongest correlation is "when" the event occurred, e.g. if the system was backed up then, it is slow, if not, it can be fast.

Closing as it is a net improvement.