Replace EventContext fields prev_group and delta_ids with field state_group_deltas

[H-Shay]

Currently the class EventContext uses the fields prev_group and delta_ids to hold information about the state group previous to the event (or the state group previous to the group before the event) and the state delta between the previous state group and the new state group. This information was used to try and optimize storing the state at the event, as if the previous state group was known one could just store the state delta between the old state and the new.

However, this way of tracking the information could be very confusing. This PR attempts to make it less confusing, and replaces the fields prev_group and delta_ids with the field state_group_deltas in the hope that this formulation is a little clearer and easier to work with.

[clokep]

@H-Shay Is this pending my review? I think you had said in a meeting recently that you were going to look at this again, but want to ensure that I'm not blocking you!

[clokep]

We talked about this in person and @H-Shay is going to look into whether some of this code is even truly used in "real" situations or if the optimizations aren't needed. Once we figure that out we'll decide what to do with this PR.

Removing my review request for now.

[H-Shay]

Right so I think this is ready for another look.

For completeness, Erik and I logged how often the optimization that prev_group/state_group_deltas enables was hit and determined it was worth keeping. Thus I have attempted here to rework the PR to make the code clearer by reducing the amount of Nones flying around, better documentation, more documentation closer to the relevant code, and some encapsulation. Let me know what you think.

[clokep]

I think this is massively clearer without all the Nones floating around! I think most of my comments are minor refactoring.

[clokep]

It seems I never hit submit on this review. 😢 I think you've already updated a few of these things...

[erikjohnston]

I think this is back on @H-Shay's plate?

[H-Shay]

Sooo I think I've sorted the backwards/forwards compatibility issue: serializing produces a dict which de-serializing then converts back into an object.
The modes of possible incompatibility then are:

1. The event context is serialized by old code and de-serialized by new code: to address that the new de-serialization code checks for the existence of the key "state_group_deltas" in the dict and if it doesn't exist just uses an empty dict for that field in the object it is building and ignores the presence of the old keys in the dict
2. The event context is serialized by the new code and de-serialized by the old code: to address this I had the new code add two "dummy" keys set to None corresponding to the old fields we removed ("prev_group" and "delta_ids") so the old de-serialize code will use those to reconstruct the object and ignore the new key "state_group_deltas" in the dict.

I think this solves the problem? Let me know if my logic seems wacky. (We might want to also add in this case that there may be a slight performance penalty until all workers and master are on > v1.87.0)

[erikjohnston]

I think this solves the problem? Let me know if my logic seems wacky.

Sounds correct to me!

(We might want to also add in this case that there may be a slight performance penalty until all workers and master are on > v1.87.0)

TBH I wouldn't bother

[erikjohnston]

Woo! 🎉

We should add a future maintenance task to remove the backwards compat hack in a couple of releases.

[H-Shay]

Merged #15233 into develop, thanks for the reviews.

#15777 is the tracking issue for removing the backwards compat code in the future.