WriteOutBytes improvements

[adarshsanjeev]

This PR generally improves the working of WriteOutBytes and WriteOutMedium. Some analysis of usage of TmpFileSegmentWriteOutMedium shows that they periodically get used for very small things. The overhead of creating a tmp file is actually very large. To improve the performance in these cases, this PR modifies TmpFileSegmentWriteOutMedium to return a heap-based WriteOutBytes that falls back to making a tmp file when it actually fills up.

• Increases the memory in FileWriteOutBytes and switches to using direct memory to improve performance.
• Refactor TmpFileSegmentWriteOutMedium to use a heap based WriteOutBytes before falling back to a temporary file.
• Modify FileWriteOutBytes's behavior on writing a value larger than its buffer. Previously, it used to throw an exception. This PR modifies it to write the value to the file directly instead, bypassing the buffer.
• Adds some additional logging around the usage of write out bytes.

---

This PR has:

• been self-reviewed.
  • using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added or updated version, license, or notice information in licenses.yaml
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

[clintropolis]

why not implement a new SegmentWriteOutMedium instead of changing the behavior of an existing one?

[imply-cheddar]

Because it has broad-reaching performance improving impacts? The alternative would be to create a new thing, update everything to use it and then delete the old, which is basically the same as changing in place. The reason to leave the old one would be if we wanted to maintain configurability to do the old thing, but I don't know why that would be useful here.

[clintropolis]

yea im for the strategy in general, just kind of worried about the slightly larger memory footprint here without any form of escape hatch. This looks like it can grow up to 16k (up from the 4k heap buffer of the old FileWriteOutBytes buffer that has been replaced with a direct buffer in this PR). Its probably cool, I'm just being nervous.

I was suggesting mostly just making a new thing and making it the default and leave the old thing just in case, but i can be convinced that this is close enough to not matter i think.

[imply-cheddar]

LGTM

[imply-cheddar]

Because it has broad-reaching performance improving impacts? The alternative would be to create a new thing, update everything to use it and then delete the old, which is basically the same as changing in place. The reason to leave the old one would be if we wanted to maintain configurability to do the old thing, but I don't know why that would be useful here.

[abhishekagarwal87]

Can you add please tests to cover the code changes? E.g. is there a test already that makes sure that filesCreated is changed correctly in TmpFileSegmentWriteOutMedium class.