Use ijson to parse the response to /send_join, reducing memory usage.

[erikjohnston]

Instead of parsing the full response to /send_join into Python objects (which can be huge for large rooms) and then parsing that into events, we instead use ijson to stream parse the response directly into EventBase objects.

Question: Do we want to make this an optional dependency? It should be easy (just a case of adding a JsonParser subclass), though does mean we have more things to test

[clokep]

A general question -- how strict is ijson at parsing JSON (does it bypass some of the strictness we do:

synapse/synapse/util/__init__.py

Lines 59 to 60 in 03318a7

	# Create a custom decoder to reject Python extensions to JSON.
	json_decoder = json.JSONDecoder(parse_constant=_reject_invalid_json)

[richvdh]

generally lgtm, modulo a few thoughts.

As @clokep says: it would be good to check how it handles non-standard json (inf, NaN, etc).

[erikjohnston]

A general question -- how strict is ijson at parsing JSON (does it bypass some of the strictness we do:

synapse/synapse/util/__init__.py

Lines 59 to 60 in 03318a7

	# Create a custom decoder to reject Python extensions to JSON.
	json_decoder = json.JSONDecoder(parse_constant=_reject_invalid_json)

I tested this manually for all the ijson backends and they all reject NaN, Infinity and `-Infinity', so I think we're good on that front.

However, it will silently ignore if the state_events or auth keys are missing or have the wrong type, e.g. list(ijson.items('{"foo": 1}', "foo.item")) will produce [] rather than failing.

[clokep]

Seems reasonable. I wonder if there's other places we should be using this? (For some reason the receiving end of /send for federation jumps out to me.)

[erikjohnston]

Seems reasonable. I wonder if there's other places we should be using this? (For some reason the receiving end of /send for federation jumps out to me.)

Maybe, though I think /send_join is pretty much the only API (apart from deprecated /state) which is quite so big. /send only has a max of 50 events and 100 EDUs, so they're not that big 🤷