Bump hfh to 0.11.0

[lhoestq]

to fix errors like

requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://hub-ci.huggingface.co/api/datasets/__DUMMY_TRANSFORMERS_USER__/...

(e.g. from this failing CI)

0.11.0 is the current minimum version in transformers

around 5% of users are currently using versions <0.11.0

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006334 / 0.011353 (-0.005018)	0.004447 / 0.011008 (-0.006561)	0.099287 / 0.038508 (0.060779)	0.027426 / 0.023109 (0.004317)	0.322638 / 0.275898 (0.046740)	0.370501 / 0.323480 (0.047021)	0.004775 / 0.007986 (-0.003210)	0.003289 / 0.004328 (-0.001040)	0.076531 / 0.004250 (0.072280)	0.037485 / 0.037052 (0.000432)	0.335634 / 0.258489 (0.077145)	0.384031 / 0.293841 (0.090190)	0.031258 / 0.128546 (-0.097288)	0.011619 / 0.075646 (-0.064027)	0.326309 / 0.419271 (-0.092963)	0.042513 / 0.043533 (-0.001020)	0.340817 / 0.255139 (0.085678)	0.369846 / 0.283200 (0.086646)	0.084904 / 0.141683 (-0.056779)	1.481739 / 1.452155 (0.029584)	1.566593 / 1.492716 (0.073877)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.186424 / 0.018006 (0.168418)	0.400879 / 0.000490 (0.400389)	0.003520 / 0.000200 (0.003320)	0.000079 / 0.000054 (0.000024)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.023287 / 0.037411 (-0.014124)	0.097767 / 0.014526 (0.083241)	0.103271 / 0.176557 (-0.073286)	0.165414 / 0.737135 (-0.571722)	0.106437 / 0.296338 (-0.189901)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.422711 / 0.215209 (0.207502)	4.221382 / 2.077655 (2.143727)	1.906807 / 1.504120 (0.402687)	1.709595 / 1.541195 (0.168400)	1.720452 / 1.468490 (0.251962)	0.699477 / 4.584777 (-3.885300)	3.415840 / 3.745712 (-0.329873)	2.835669 / 5.269862 (-2.434192)	1.501775 / 4.565676 (-3.063901)	0.082896 / 0.424275 (-0.341379)	0.012855 / 0.007607 (0.005248)	0.514373 / 0.226044 (0.288329)	5.190000 / 2.268929 (2.921071)	2.302539 / 55.444624 (-53.142086)	1.963410 / 6.876477 (-4.913067)	2.020944 / 2.142072 (-0.121128)	0.805919 / 4.805227 (-3.999308)	0.150604 / 6.500664 (-6.350060)	0.065977 / 0.075469 (-0.009492)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.206487 / 1.841788 (-0.635300)	13.631513 / 8.074308 (5.557205)	13.800258 / 10.191392 (3.608866)	0.146914 / 0.680424 (-0.533509)	0.016454 / 0.534201 (-0.517747)	0.377752 / 0.579283 (-0.201532)	0.384312 / 0.434364 (-0.050052)	0.434912 / 0.540337 (-0.105425)	0.522507 / 1.386936 (-0.864429)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006328 / 0.011353 (-0.005025)	0.004406 / 0.011008 (-0.006602)	0.077951 / 0.038508 (0.039443)	0.026716 / 0.023109 (0.003607)	0.337303 / 0.275898 (0.061405)	0.372036 / 0.323480 (0.048556)	0.004800 / 0.007986 (-0.003185)	0.003153 / 0.004328 (-0.001175)	0.076823 / 0.004250 (0.072573)	0.035873 / 0.037052 (-0.001179)	0.340243 / 0.258489 (0.081754)	0.380183 / 0.293841 (0.086342)	0.032185 / 0.128546 (-0.096361)	0.011545 / 0.075646 (-0.064101)	0.086887 / 0.419271 (-0.332384)	0.041560 / 0.043533 (-0.001973)	0.338716 / 0.255139 (0.083577)	0.363080 / 0.283200 (0.079881)	0.088375 / 0.141683 (-0.053308)	1.499004 / 1.452155 (0.046850)	1.585904 / 1.492716 (0.093188)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.211645 / 0.018006 (0.193639)	0.403707 / 0.000490 (0.403218)	0.000415 / 0.000200 (0.000215)	0.000058 / 0.000054 (0.000004)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.024972 / 0.037411 (-0.012440)	0.097996 / 0.014526 (0.083470)	0.105941 / 0.176557 (-0.070616)	0.155521 / 0.737135 (-0.581615)	0.108246 / 0.296338 (-0.188092)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.442316 / 0.215209 (0.227107)	4.417977 / 2.077655 (2.340322)	2.078324 / 1.504120 (0.574205)	1.863678 / 1.541195 (0.322483)	1.917149 / 1.468490 (0.448659)	0.697628 / 4.584777 (-3.887149)	3.412810 / 3.745712 (-0.332902)	1.866473 / 5.269862 (-3.403389)	1.155923 / 4.565676 (-3.409754)	0.082831 / 0.424275 (-0.341444)	0.012367 / 0.007607 (0.004760)	0.540018 / 0.226044 (0.313974)	5.420472 / 2.268929 (3.151544)	2.508540 / 55.444624 (-52.936084)	2.166397 / 6.876477 (-4.710080)	2.153486 / 2.142072 (0.011414)	0.804860 / 4.805227 (-4.000367)	0.151178 / 6.500664 (-6.349486)	0.067870 / 0.075469 (-0.007599)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.310387 / 1.841788 (-0.531400)	13.908916 / 8.074308 (5.834608)	14.136895 / 10.191392 (3.945503)	0.139389 / 0.680424 (-0.541035)	0.016687 / 0.534201 (-0.517514)	0.379624 / 0.579283 (-0.199659)	0.382634 / 0.434364 (-0.051730)	0.439632 / 0.540337 (-0.100706)	0.524913 / 1.386936 (-0.862023)

[albertvillanova]

It is quite a big jump for a minimum requirement from 0.2.0 to 0.11.0.

Do you think this is really necessary and convenient? I would naively say that 5% of the users is not a negligible number...

Alternatively, we could try to find a different fix for the client error issue and add a deprecation warning to tell users to update their huggingface_hub version because it will not be supported in future releases.

Otherwise, if you think it is OK as it is, then go on!

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006365 / 0.011353 (-0.004988)	0.004457 / 0.011008 (-0.006551)	0.097989 / 0.038508 (0.059481)	0.027686 / 0.023109 (0.004577)	0.357412 / 0.275898 (0.081514)	0.368573 / 0.323480 (0.045093)	0.004859 / 0.007986 (-0.003127)	0.003262 / 0.004328 (-0.001066)	0.076487 / 0.004250 (0.072237)	0.035526 / 0.037052 (-0.001527)	0.332862 / 0.258489 (0.074373)	0.369334 / 0.293841 (0.075493)	0.030750 / 0.128546 (-0.097796)	0.011503 / 0.075646 (-0.064143)	0.323289 / 0.419271 (-0.095982)	0.042302 / 0.043533 (-0.001231)	0.334009 / 0.255139 (0.078870)	0.354150 / 0.283200 (0.070951)	0.082895 / 0.141683 (-0.058788)	1.499727 / 1.452155 (0.047572)	1.574123 / 1.492716 (0.081407)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.192583 / 0.018006 (0.174577)	0.408136 / 0.000490 (0.407646)	0.001272 / 0.000200 (0.001072)	0.000070 / 0.000054 (0.000015)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.022883 / 0.037411 (-0.014528)	0.095710 / 0.014526 (0.081185)	0.106545 / 0.176557 (-0.070011)	0.165784 / 0.737135 (-0.571352)	0.108594 / 0.296338 (-0.187744)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.429483 / 0.215209 (0.214274)	4.292338 / 2.077655 (2.214683)	1.917759 / 1.504120 (0.413639)	1.711489 / 1.541195 (0.170294)	1.735668 / 1.468490 (0.267178)	0.707602 / 4.584777 (-3.877175)	3.369643 / 3.745712 (-0.376070)	1.874517 / 5.269862 (-3.395344)	1.248560 / 4.565676 (-3.317117)	0.083247 / 0.424275 (-0.341028)	0.012606 / 0.007607 (0.004999)	0.519342 / 0.226044 (0.293297)	5.225462 / 2.268929 (2.956533)	2.433230 / 55.444624 (-53.011394)	2.006005 / 6.876477 (-4.870471)	2.093156 / 2.142072 (-0.048916)	0.809372 / 4.805227 (-3.995855)	0.151691 / 6.500664 (-6.348973)	0.066680 / 0.075469 (-0.008789)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.226283 / 1.841788 (-0.615505)	13.604338 / 8.074308 (5.530030)	13.953245 / 10.191392 (3.761853)	0.132904 / 0.680424 (-0.547520)	0.016420 / 0.534201 (-0.517781)	0.395316 / 0.579283 (-0.183967)	0.385003 / 0.434364 (-0.049361)	0.483303 / 0.540337 (-0.057034)	0.578459 / 1.386936 (-0.808477)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006218 / 0.011353 (-0.005135)	0.004451 / 0.011008 (-0.006557)	0.076892 / 0.038508 (0.038384)	0.027017 / 0.023109 (0.003908)	0.356976 / 0.275898 (0.081078)	0.396083 / 0.323480 (0.072603)	0.005510 / 0.007986 (-0.002476)	0.003265 / 0.004328 (-0.001063)	0.075771 / 0.004250 (0.071521)	0.037117 / 0.037052 (0.000064)	0.362181 / 0.258489 (0.103692)	0.401771 / 0.293841 (0.107931)	0.032062 / 0.128546 (-0.096484)	0.011453 / 0.075646 (-0.064194)	0.085773 / 0.419271 (-0.333498)	0.041679 / 0.043533 (-0.001854)	0.355120 / 0.255139 (0.099981)	0.390170 / 0.283200 (0.106970)	0.088210 / 0.141683 (-0.053473)	1.526434 / 1.452155 (0.074279)	1.586019 / 1.492716 (0.093302)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.196836 / 0.018006 (0.178830)	0.401161 / 0.000490 (0.400671)	0.002880 / 0.000200 (0.002680)	0.000080 / 0.000054 (0.000025)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.024445 / 0.037411 (-0.012966)	0.100187 / 0.014526 (0.085661)	0.106391 / 0.176557 (-0.070165)	0.159764 / 0.737135 (-0.577372)	0.109828 / 0.296338 (-0.186511)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.444228 / 0.215209 (0.229018)	4.420769 / 2.077655 (2.343114)	2.069437 / 1.504120 (0.565318)	1.862587 / 1.541195 (0.321392)	1.934627 / 1.468490 (0.466137)	0.699681 / 4.584777 (-3.885095)	3.352540 / 3.745712 (-0.393172)	2.613172 / 5.269862 (-2.656689)	1.445116 / 4.565676 (-3.120561)	0.083086 / 0.424275 (-0.341189)	0.012715 / 0.007607 (0.005108)	0.537450 / 0.226044 (0.311405)	5.403052 / 2.268929 (3.134123)	2.506703 / 55.444624 (-52.937921)	2.170198 / 6.876477 (-4.706279)	2.201909 / 2.142072 (0.059837)	0.799555 / 4.805227 (-4.005672)	0.150825 / 6.500664 (-6.349839)	0.067234 / 0.075469 (-0.008235)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.293097 / 1.841788 (-0.548691)	13.817133 / 8.074308 (5.742825)	14.247231 / 10.191392 (4.055839)	0.128422 / 0.680424 (-0.552002)	0.016541 / 0.534201 (-0.517660)	0.382466 / 0.579283 (-0.196817)	0.380560 / 0.434364 (-0.053804)	0.439061 / 0.540337 (-0.101276)	0.521865 / 1.386936 (-0.865071)

[lhoestq]

I also took the liberty of removing _hf_hub_fixes.py completely :)

Do you think this is really necessary and convenient? I would naively say that 5% of the users is not a negligible number...

I think it's ok. Most of them are using old versions of datasets anyway.

[lhoestq]

merging, but lmk if you have other concerns

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006810 / 0.011353 (-0.004543)	0.004683 / 0.011008 (-0.006325)	0.100889 / 0.038508 (0.062381)	0.030135 / 0.023109 (0.007026)	0.356407 / 0.275898 (0.080509)	0.389175 / 0.323480 (0.065695)	0.005358 / 0.007986 (-0.002627)	0.004760 / 0.004328 (0.000432)	0.075904 / 0.004250 (0.071654)	0.040341 / 0.037052 (0.003288)	0.357363 / 0.258489 (0.098874)	0.394185 / 0.293841 (0.100344)	0.031322 / 0.128546 (-0.097224)	0.011636 / 0.075646 (-0.064010)	0.327327 / 0.419271 (-0.091944)	0.042494 / 0.043533 (-0.001039)	0.338079 / 0.255139 (0.082940)	0.363388 / 0.283200 (0.080189)	0.087102 / 0.141683 (-0.054581)	1.505686 / 1.452155 (0.053531)	1.562112 / 1.492716 (0.069396)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.203630 / 0.018006 (0.185624)	0.425986 / 0.000490 (0.425496)	0.003786 / 0.000200 (0.003586)	0.000071 / 0.000054 (0.000017)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.024138 / 0.037411 (-0.013274)	0.101752 / 0.014526 (0.087226)	0.105436 / 0.176557 (-0.071121)	0.165385 / 0.737135 (-0.571750)	0.114510 / 0.296338 (-0.181828)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.447561 / 0.215209 (0.232352)	4.449212 / 2.077655 (2.371557)	2.169472 / 1.504120 (0.665352)	1.989025 / 1.541195 (0.447831)	2.036267 / 1.468490 (0.567776)	0.698647 / 4.584777 (-3.886130)	3.483281 / 3.745712 (-0.262431)	1.949306 / 5.269862 (-3.320555)	1.290313 / 4.565676 (-3.275363)	0.083079 / 0.424275 (-0.341196)	0.012759 / 0.007607 (0.005152)	0.540944 / 0.226044 (0.314899)	5.473391 / 2.268929 (3.204463)	2.632037 / 55.444624 (-52.812587)	2.327396 / 6.876477 (-4.549081)	2.428880 / 2.142072 (0.286808)	0.808918 / 4.805227 (-3.996309)	0.153283 / 6.500664 (-6.347381)	0.068325 / 0.075469 (-0.007145)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.212527 / 1.841788 (-0.629260)	14.306444 / 8.074308 (6.232136)	14.904980 / 10.191392 (4.713588)	0.142796 / 0.680424 (-0.537628)	0.016829 / 0.534201 (-0.517372)	0.384806 / 0.579283 (-0.194477)	0.390505 / 0.434364 (-0.043859)	0.441734 / 0.540337 (-0.098603)	0.526159 / 1.386936 (-0.860777)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006950 / 0.011353 (-0.004403)	0.004647 / 0.011008 (-0.006362)	0.078925 / 0.038508 (0.040417)	0.028081 / 0.023109 (0.004971)	0.343420 / 0.275898 (0.067522)	0.380567 / 0.323480 (0.057087)	0.005286 / 0.007986 (-0.002700)	0.004816 / 0.004328 (0.000487)	0.077332 / 0.004250 (0.073081)	0.042131 / 0.037052 (0.005078)	0.345371 / 0.258489 (0.086882)	0.390232 / 0.293841 (0.096392)	0.032395 / 0.128546 (-0.096152)	0.011669 / 0.075646 (-0.063978)	0.087649 / 0.419271 (-0.331622)	0.042465 / 0.043533 (-0.001068)	0.342863 / 0.255139 (0.087724)	0.368947 / 0.283200 (0.085748)	0.091725 / 0.141683 (-0.049958)	1.477435 / 1.452155 (0.025280)	1.563449 / 1.492716 (0.070733)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.208016 / 0.018006 (0.190010)	0.428387 / 0.000490 (0.427898)	0.000443 / 0.000200 (0.000243)	0.000060 / 0.000054 (0.000005)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.026963 / 0.037411 (-0.010449)	0.103854 / 0.014526 (0.089328)	0.109068 / 0.176557 (-0.067488)	0.160107 / 0.737135 (-0.577028)	0.112843 / 0.296338 (-0.183496)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.437161 / 0.215209 (0.221952)	4.396178 / 2.077655 (2.318523)	2.067597 / 1.504120 (0.563477)	1.875247 / 1.541195 (0.334053)	1.962451 / 1.468490 (0.493961)	0.701427 / 4.584777 (-3.883350)	3.459564 / 3.745712 (-0.286148)	1.959482 / 5.269862 (-3.310380)	1.191866 / 4.565676 (-3.373810)	0.083243 / 0.424275 (-0.341032)	0.012740 / 0.007607 (0.005133)	0.535236 / 0.226044 (0.309191)	5.351715 / 2.268929 (3.082786)	2.490868 / 55.444624 (-52.953756)	2.195680 / 6.876477 (-4.680797)	2.233854 / 2.142072 (0.091781)	0.809041 / 4.805227 (-3.996187)	0.151498 / 6.500664 (-6.349166)	0.068297 / 0.075469 (-0.007172)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.303596 / 1.841788 (-0.538192)	14.712746 / 8.074308 (6.638438)	14.778412 / 10.191392 (4.587020)	0.147093 / 0.680424 (-0.533331)	0.017105 / 0.534201 (-0.517096)	0.381687 / 0.579283 (-0.197596)	0.402435 / 0.434364 (-0.031929)	0.453538 / 0.540337 (-0.086800)	0.538866 / 1.386936 (-0.848070)