feat(rust, python): show column name if read_csv errors

[MarcoGorelli]

Is it OK to show the column name in the read_csv error?

It's not obvious from the error message if the column number is zero-indexed or one-indexed (at least, it wasn't to me when I encountered this, and I had hundreds of columns - in particular, it's not clear how to set the dtype of the failing column if you don't know its name)

[ritchie46]

This touches highly optimized code. I am a bit hesitant for this. I will take a look at this later when I have got time to take a good look with performance in mind.

[MarcoGorelli]

sure, no worries - I'll do some performance tests to check the impact

[MarcoGorelli]

I tried running some performance checks here: https://www.kaggle.com/code/marcogorelli/polars-timing/notebook?scriptVersionId=120311373

Reading a 113million-row dataset (4.5 GB):

pl.read_csv('/kaggle/input/100-million-data-csv/custom_1988_2020.csv')

here:

• 13.2 s ± 180 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

master:

• 13.3 s ± 193 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

pl.read_csv('/kaggle/input/100-million-data-csv/custom_1988_2020.csv', dtypes={'198801': pl.Boolean}, ignore_errors=True)

here:

• 12.5 s ± 127 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

master:

• 12.4 s ± 71.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

[ritchie46]

Right, I thoroughly looked at your changes and I agree that this should have no expected performance changes. Thanks a lot. Good error information is really valuable. 👍

[ritchie46]

As a side note. Next time when you benchmarks csv code, set rechunk=False this is a very expensive operation that doesn't relate to csv parsing at all.

[MarcoGorelli]

thanks, much appreciated!