Consistency of type deferral

[jthielen]

As hinted at by the varied language of "array interaction" in #3, there are several different interactions in practice in which handling/deferral needs to be considered. As far as I can recall, these are:

• Construction/wrapping
• Binary operations (addition, multiplication, etc.)
• __array_ufunc__
• __array_function__
• Modules of array functions (e.g., dask.array, but also future __array_module__ of NEP 37)

Current State

The consistency of type handling between these different forms of interaction is varied. For example, Pint Quantities are designed with these consistent (where they exist, as Pint doesn't have an array function module/namespace), whereas with Dask arrays are deliberately different (e.g., deferring to other types in binops and __array_*__, but assuming user intent to override the standard deferral in construction/wrapping and array functions) and (at least as I see it) xarray is accidentally different (see pydata/xarray#3950). The main sticking point in the discussions involved in each of these respective implementations can be summarized as "How much should we trust user to not 'break' the DAG through inconsistencies in type deferral?" or "How much should consistency be enforced?"

Specific Goals

• Resolve the question (both conceptually and in implementation): to what extent can we trust users to not "break" the type casting hierarchy when/if certain operations/interactions can partially introduce cyclicness to the otherwise acyclic directed graph?

Key Points Raised at Coordination Meeting

• Consensus (or at least absence of any objections) that all these forms of interaction should be consistent
  • Emphasis on clarity of messaging (i.e., "here's how I defer to another library" rather than "here's how I defer to other libraries with respect to interaction a, and here's how with interaction b, etc.")
  • Hopefully should ease implementation/allow shared utils
  • There is a desire for ability to opt-out/override for third-party/custom/unknown array types
• Big hurdle: how do we consistently support custom (unknown) libraries?

Suggested Paths Forward

Discussion

• Should this between-interaction consistency be up to each library to carry out within its own interaction implementation, or should deferral utils be shared (i.e., with Defining Pairwise Interactions #3)?
• As either a recommendation (if up to each library) or implementation (if shared util), how to best support custom/unknown array types?
  • Registration to collection of known types (i.e., inclusion in Defining Pairwise Interactions #3)?
  • Ability to opt-out/override? And if so how (i.e., full opt-out/override, so essentially opting out of Defining Pairwise Interactions #3, or optionally inconsistent)?
  • Something else?

Changes to Participating Libraries

• xarray should carry out changes in Consistent Handling of Type Casting Hierarchy xarray#3950
• Dask should carry out Dask Array's NEP-13 and NEP-18 protocols should use common utils dask/dask#6672, while also reconsidering intentional inconsistency in Add simple chunk type registry and defer as appropriate to upcast types dask/dask#6393 on outcome of discussions here
  • In particular, UserWarning when wrapping pint & dask arrays together xarray#5559 (comment) needs resolution