Implement sort=True for Index setops

[TomAugspurger]

#25063 made the 0.24.x sort=True behavior be enabled by sort=None. This allows us to use sort=True to mean "always sort".

So rather than raising

In [1]: import pandas as pd

In [2]: idx = pd.Index([1, 0])

In [3]: idx.union(idx, sort=True)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-92d6bafc02d6> in <module>
----> 1 idx.union(idx, sort=True)

~/sandbox/pandas/pandas/core/indexes/base.py in union(self, other, sort)
   2319         Index(['a', 'b', 'c', 'd', 1, 2, 3, 4], dtype='object')
   2320         """
-> 2321         self._validate_sort_keyword(sort)
   2322         self._assert_can_do_setop(other)
   2323

~/sandbox/pandas/pandas/core/indexes/base.py in _validate_sort_keyword(self, sort)
   2263         if sort not in [None, False]:
   2264             raise ValueError(
-> 2265                 "The 'sort' keyword only takes the values of "
   2266                 f"None or False; {sort} was passed."
   2267             )

ValueError: The 'sort' keyword only takes the values of None or False; True was passed.

We would instead return

Int64Index([0, 1], dtype='int64')

This is a (hopefully) exhaustive list of special cases not sorted when sort=None.

• union:
  • equal
  • length-0
  • incomparable
• intersection
  • equal
  • incomparable
• difference
  • incomparable
• symmetric_difference
  • incomparable

[TomAugspurger]

@jorisvandenbossche OK with moving off the 1.0 milestone?

[TomAugspurger]

Pushing off 1.0.

[jbrockmendel]

@TomAugspurger @jorisvandenbossche @jreback this issue was on the agenda for today's call but no one present knew what the reasoning was for having sort=None instead of sort=True in the first place. There are a bunch of comments in the tests to the effect of "# TODO: decide what sort=True means"

IIUC sort=True will differ from sort=None only in the handful of cases that currently go through a fastpath and avoid sorting like the [1, 0] example in the OP?

[mroeschke]

This is a psudo-blocker for #48553 as there are some test warnings involving join that hardcode sort, and passing through sort with how="outer"/"inner" hits Index.union/intersection which is not implemented when sort=True

pandas/tests/indexes/multi/test_missing.py::test_nan_stays_float
  /home/runner/work/pandas/pandas/pandas/tests/indexes/multi/test_missing.py:92: RuntimeWarning: The values in the array are unorderable. Pass `sort=False` to suppress this warning.
    idxm = idx0.join(idx1, how="outer")

pandas/tests/indexes/multi/test_missing.py::test_nan_stays_float
  /home/runner/work/pandas/pandas/pandas/tests/indexes/multi/test_missing.py:99: RuntimeWarning: The values in the array are unorderable. Pass `sort=False` to suppress this warning.
    dfm = df0 - df1

[jorisvandenbossche]

That's maybe rather a bug in the MultiIndex non-sort case? Because the warning says to pass sort=False to suppress this warning, but doing that doesn't actually solve it:

In [5]: idx0 = MultiIndex(levels=[["A", "B"], []], codes=[[1, 0], [-1, -1]], names=[0, 1])
   ...: idx1 = MultiIndex(levels=[["C"], ["D"]], codes=[[0], [0]], names=[0, 1])
   ...: idxm = idx0.join(idx1, how="outer")
<ipython-input-5-1d74c2084c24>:3: RuntimeWarning: The values in the array are unorderable. Pass `sort=False` to suppress this warning.
  idxm = idx0.join(idx1, how="outer")

In [6]: idxm = idx0.join(idx1, how="outer", sort=False)
<ipython-input-6-f0c3e74a3a0c>:3: RuntimeWarning: The values in the array are unorderable. Pass `sort=False` to suppress this warning.
  idxm = idx0.join(idx1, how="outer", sort=False)

[jorisvandenbossche]

no one present knew what the reasoning was for having sort=None instead of sort=True in the first place

The top post has a list of cases where the current default behaviour (sort=None) does not actually sort (and thus could potentially differ from a sort=True).
As a small example:

>>> idx1 = pd.Index(['c', 'b', 'a'])
>>> idx2 = pd.Index(['a', 'c', 'b'])

>>> idx1.union(idx2, sort=None)
Index(['a', 'b', 'c'], dtype='object')  # <-- sorted
>>> idx1.union(idx2, sort=False)
Index(['c', 'b', 'a'], dtype='object')

>>> idx1.union(idx1.copy(), sort=None)
Index(['c', 'b', 'a'], dtype='object')  # <-- not sorted
>>> idx1.union(idx1.copy(), sort=False)
Index(['c', 'b', 'a'], dtype='object')

So depending on the values, the default sort=None does sort or does not sort. That's already a good reason to not call this behaviour sort=True.

So we originally used sort=None as the default when adding this sort keyword to the Index setops methods, to allow adding sort=True in the future (but this part never happened in practice, I think).
In addition, at the time we also were thinking that we should actually move to sort=False as the default (Index.intersection already has this default).