You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently the add attachment and add metadata subcommands rewrite the entire file, which takes time proportional to the size of the file. It would be way faster if we only rewrote the summary section. For this we need some tricky code to bump all the appropriate summary section indexes, depending on where the summary offsets for the metadata/attachment sections reside, but it should totally be possible.
One thing to be aware of - we will lose crash safety, vs the current tmpfile + move approach. Options to mitigate seem like a) copy the original file to a "original file" location, which has the downside of still being proportional to file size, or b) pairing this with a robust "reindex" subcommand that will regenerate a valid summary section from the presumed-valid original data section (we should not need to touch the data section). Option b) seems best IMO, to make the average case very fast.
The text was updated successfully, but these errors were encountered:
@jtbandes I don't think there's any reason that "reindex" needs to be in place, since the expectation would be that it would only get run in uncommon failures of this function. If we already have this in the recover command I think that's sufficient.
Currently the
add attachment
andadd metadata
subcommands rewrite the entire file, which takes time proportional to the size of the file. It would be way faster if we only rewrote the summary section. For this we need some tricky code to bump all the appropriate summary section indexes, depending on where the summary offsets for the metadata/attachment sections reside, but it should totally be possible.One thing to be aware of - we will lose crash safety, vs the current tmpfile + move approach. Options to mitigate seem like a) copy the original file to a "original file" location, which has the downside of still being proportional to file size, or b) pairing this with a robust "reindex" subcommand that will regenerate a valid summary section from the presumed-valid original data section (we should not need to touch the data section). Option b) seems best IMO, to make the average case very fast.
The text was updated successfully, but these errors were encountered: