Make pandas Index opt-in

[MarcoGorelli]

The issue here is

The DataFrame spec does not include an Index. Many Pandas APIs (e.g., groupby, concat) operate on the index by default.

Make Index an “opt-in” feature of these APIs.

Summary of the consortium meeting minutes:

• proposal is to have a default index which goes from 0 to len
• it handles things different when you do something like concat or filter out rows
• it will help other libraries.

[MarcoGorelli]

Write-up with some thoughts: https://hackmd.io/@mntOORP3TCesJJyvg-IdFQ/B1EqHfQfo

cc @jorisvandenbossche @shwina

[jorisvandenbossche]

Could you enabled commenting on the hackmd? (it's somewhere in the sharing settings I think)

[kkraus14]

-1

I've shared my opinion on this multiple times, but making this opt-in will effectively make it required for every project. A very large percentage of code will target this standard and test their usage of it via Pandas and will end up using the optional components that Pandas supports and avoid the components it doesn't support without a second thought. This will make all other libraries supporting this standard feel compelled to support indexes and other Pandas-supported optional components, which leaves us in a similar state to the existing DataFrame ecosystem.

The writeup seems to focus on compatibility with existing Pandas APIs and behaviors. Is that and should that really be a goal of this standard API work?

[rgommers]

@kkraus14 a bit of context is still missing in the issue description, this was about Pandas and not about a dataframe standard. No optional index in sight for the latter.

[jbrockmendel]

Then does this belong on the pandas tracker?

[MarcoGorelli]

@jorisvandenbossche sure, done (I think), are you able to comment now?

Apologies @kkraus14 , I shared this here because although it's about pandas, it's in the context of the dataframe standard. Perhaps the pandas issue tracker indeed have been a better place, I'll move it there

[MarcoGorelli]

I can't transfer this issue, so I've made a new one in the pandas tracker: pandas-dev/pandas#48880

[rgommers]

Thanks @MarcoGorelli, all. Let's close this issue then.