create list using over (implode) context expected behaviour? bug?

[Julian-J-S]

Problem description

Hi,
My goal is to create a list column from a over aggregation.

I wonder if the current behaviour of a plain pl.col("a").over("b") is expected because it does nothing (useless?) which is different from the groupby + agg context where it produces a list column.
If it is expected (would love to know why), then I would expect the implode to work but it does not.

Examples:

df = pl.DataFrame({
    "a": [1, 2, 3],
    "b": [4, 4, 6],
})

# << over >>
df.with_columns(
    # is this the expected behavior?? useless? Maybe this should already create a list like in agg context?
    a_over_b=pl.col("a").over("b"),

    # crashes! (imo either this or better the above should work and create a list)
    # a_implode_over_b=pl.col("a").implode().over("b"),

    # I want this but over b not over all
    a_implode=pl.col("a").implode(),
)
┌─────┬─────┬──────────┬───────────┐
│ a   ┆ b   ┆ a_over_b ┆ a_implode │
│ --- ┆ --- ┆ ---      ┆ ---       │
│ i64 ┆ i64 ┆ i64      ┆ list[i64] │
╞═════╪═════╪══════════╪═══════════╡
│ 1   ┆ 4   ┆ 1        ┆ [1, 2, 3] │
│ 2   ┆ 4   ┆ 2        ┆ [1, 2, 3] │
│ 3   ┆ 6   ┆ 3        ┆ [1, 2, 3] │
└─────┴─────┴──────────┴───────────┘

# << groupby + agg >>
df.groupby("b").agg(
    # plain col("a") already creates a list in agg context (makes sense!)
    agg_col=pl.col("a"),

    # implode creates an additional list
    agg_col_imploded=pl.col("a").implode(), # implode adds another list
)
┌─────┬───────────┬──────────────────┐
│ b   ┆ agg_col   ┆ agg_col_imploded │
│ --- ┆ ---       ┆ ---              │
│ i64 ┆ list[i64] ┆ list[list[i64]]  │
╞═════╪═══════════╪══════════════════╡
│ 6   ┆ [3]       ┆ [[3]]            │
│ 4   ┆ [1, 2]    ┆ [[1, 2]]         │
└─────┴───────────┴──────────────────┘

# << GOAL >>
┌─────┬─────┬───────────┐
│ a   ┆ b   ┆ goal      │
│ --- ┆ --- ┆ ---       │
│ i64 ┆ i64 ┆ list[i64] │
╞═════╪═════╪═══════════╡
│ 1   ┆ 4   ┆ [1, 2]    │
│ 2   ┆ 4   ┆ [1, 2]    │
│ 3   ┆ 6   ┆ [3]       │
└─────┴─────┴───────────┘

[cmdlineluser]

As for the goal, you can use mapping_strategy #8967 (comment)

df.with_columns(goal =
    pl.col("a").over("b", mapping_strategy="join")
)
# shape: (3, 3)
# ┌─────┬─────┬───────────┐
# │ a   ┆ b   ┆ goal      │
# │ --- ┆ --- ┆ ---       │
# │ i64 ┆ i64 ┆ list[i64] │
# ╞═════╪═════╪═══════════╡
# │ 1   ┆ 4   ┆ [1, 2]    │
# │ 2   ┆ 4   ┆ [1, 2]    │
# │ 3   ┆ 6   ┆ [3]       │
# └─────┴─────┴───────────┘

[ritchie46]

This is expected. Window functions map the list back to their position. Read mapping strategy

[ritchie46]

See more context here: #6487