Shuffling stage is too slow

[westonpace]

When training an index the shuffling stage runs too slowly. This seems to be because we are storing 1 batch per partition (even empty partitions need a batch) and there is a lot of concatenation. By storing a list array we can avoid both problems and the shuffling can actually completely quite quickly (in theory as fast as 6 minutes for 1B rows but might need many CPUs / RAM to achieve). At the very least we should be able to finish in 1-2 hours.

[westonpace]

5709bcd describes a potential fix

[westonpace]

Also need c59478b and 7f86689 (but this last one could use some tweaking, I think it creates too many sorted partition files)