feat: enhance binary array encoding, make it the default

[westonpace]

This adds support for the following things to the binary encoding:

• Nulls
• Large offsets
• Different types (e.g. String vs Binary)

In addition, I have changed the row limit on pages from u32 to u64. Partly, this is because string arrays larger that 2GB are not unheard of and it may be nice in some cases to store them in one page. Also pages are infrequent enough that an extra 4 bytes is trivial (actually, protobuf already stores u32 and u64 the same). Another reason is that, with some encodings (e.g. constant / compressed bitmap / etc.) it's possible to have 2^64 rows without occupying a large amount of space.

Note: this is technically a .proto change but in protobuf u32 -> u64 is backwards compatible so it is not a huge issue.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 95.42744% with 23 lines in your changes missing coverage. Please review.

Project coverage is 79.81%. Comparing base (c4def70) to head (4b79e33).
Report is 1 commits behind head on main.

Files	Patch %	Lines
...st/lance-encoding/src/encodings/physical/binary.rs	95.43%	8 Missing and 6 partials ⚠️
rust/lance-encoding/src/encodings/logical/list.rs	80.00%	2 Missing and 2 partials ⚠️
rust/lance-encoding/src/encoder.rs	71.42%	0 Missing and 2 partials ⚠️
rust/lance-encoding/src/decoder.rs	91.66%	1 Missing ⚠️
...ust/lance-encoding/src/encodings/logical/struct.rs	96.15%	0 Missing and 1 partial ⚠️
rust/lance-encoding/src/encodings/physical.rs	96.42%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2521      +/-   ##
==========================================
+ Coverage   79.45%   79.81%   +0.36%     
==========================================
  Files         208      207       -1     
  Lines       58863    58806      -57     
  Branches    58863    58806      -57     
==========================================
+ Hits        46768    46935     +167     
+ Misses       9357     9129     -228     
- Partials     2738     2742       +4     

Flag	Coverage Δ
unittests	79.81% <95.42%> (+0.36%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[raunaks13]

I think we can merge this as soon as #2508 is merged in