How high can we grow the size of group chat limit.

[nplasterer]

Up to 20k. How close can we get to that performantly?

[nplasterer]

Adding and removing large numbers of members is the concern.

[franziskuskiefer]

Do you have benchmarks for how long it would take? I have been working on benchmarking larger groups but it's hard to get something that is dominated by the setup or memory. For a 1000 member group for example I see 734ms to add a new member.

[insipx]

Hello @franziskuskiefer ! Doing benchmarking and have some early findings here: #793

I plan to have the full HTML report uploaded today but running into some bugs with all the runs.

Also, I'm benchmarking libxmtp e2e here which will probably yield slightly different results than the pure mls benchmarks. 734 ms for adding a member to a 1000 person group seems to check out, howevever. Right now bulk adding ~8000 people to an empty group adds up to ~8 seconds e2e libxmtp. There is a minor bug to sort out before benchmarking adding a single member to a large group, hoping to follow up with a benchmark for that soon once the bug is fixed.

[franziskuskiefer]

Great! This sounds roughly like what I'd expect.
I'm seeing a log of noise in my measurements and they highly fluctuate.
For example: In the 1000 person group, that's about 734 μs per member. But for 50 members I see 52 μs / member and 150 μs / member at a 4 member group.
So I don't fully trust them yet. I'm investigating trying to figure out what the real performance is.

[franziskuskiefer]

I had another look and as I expected was everything dominated by moving the memory around. Make sure you don't run into similar issues. Criterion is a little useless when handling large objects (or memory that's large, compared to the time one wants to measure). I was also a little surprised by the numbers. They really shouldn't have been that high.

Time in ms I'm see now for committing with a path are

10	20	50	100	200	500	1000
0.6	0.75	1.2	1.8	3	6	12

So 12ms, not 734ms for adding someone into a 1000 person group 😬

Extrapolating the data to 20k members this still seems to be acceptable performance (if no other bottlenecks kick in earlier).

[insipx]

this makes sense & is also great news for the MLS side, those numbers are awesome. I will have to review my benchmarks to try and make sure that I'm not accidentally benching memory allocations where I shouldn't be (In the current state I definitely am).

I also think that at least on libxmtp side benchmarks will be dominated by network overhead + there are definitely some optimizations to make internally to make things go faster, so i'm still pessimistic about getting libxmtp benchmarks to be much better than they are purely through modifying the benchmarking code.

I am very interested in the code you have for this too -- Also, curious about your opinions about criterion. Do you think we should stay away from it or does it still make sense to use for our benchmarks?

[franziskuskiefer]

I also think that at least on libxmtp side benchmarks will be dominated by network overhead + there are definitely some optimizations to make internally to make things go faster, so i'm still pessimistic about getting libxmtp benchmarks to be much better than they are purely through modifying the benchmarking code.

When we make sure that the benchmarks are somewhat aligned we should be able to estimate the overhead on the libxmtp side and see if it makes sense to reduce that.

I am very interested in the code you have for this too

The benchmarks are on a branch here. I'll get them PRed over the next days.

I've done a couple different things first to try to move the memory operations outside of the measured code. But since none of that worked (I think I did that a couple times before, but for some reason I had to re-lean this lesson) I did a simple benchmarking macro. Essentially, I'm trying not to move any memory around. There is always one clone. But that's only on something relatively small (one group and client state), such that I think that should be ok.
Another option would of course be to always generate fresh values for the benchmarks, then we could skip even that clone. But that would take forever to run for larger groups.

Also, curious about your opinions about criterion. Do you think we should stay away from it or does it still make sense to use for our benchmarks?

These are known issues. I filed something a couple years back to make sure it's on record.
I think criterion is great for quick benchmarking and rough numbers. But if you need more precise numbers, or if you have the issue (as we have in many cases for cryptographic algorithms or crypto protocols) that the memory needed is huge compared to the amount of time that you're measuring, I'd recommend using something custom where you have more control over the memory. Or at least measure the memory overhead separately to remove it from the measurements.