Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

flatten should support Datatype::FixedSizeList #9138

Closed
universalmind303 opened this issue Feb 5, 2024 · 3 comments
Closed

flatten should support Datatype::FixedSizeList #9138

universalmind303 opened this issue Feb 5, 2024 · 3 comments
Labels
bug Something isn't working

Comments

@universalmind303
Copy link
Contributor

universalmind303 commented Feb 5, 2024

Describe the bug

flatten only supports List not FixedSizeList

To Reproduce

select flatten(some_fixed_size_list);

Error: Internal error: Not reachable, data_type should be List.
This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker

Expected behavior

flatten behaves the same on both List and FixedSizeList

Additional context

 BuiltinScalarFunction::Flatten => {
                fn get_base_type(data_type: &DataType) -> Result<DataType> {
                    match data_type {
                        DataType::List(field) => match field.data_type() {
                            DataType::List(_) => get_base_type(field.data_type()),
                            _ => Ok(data_type.to_owned()),
                        },
                        _ => internal_err!("Not reachable, data_type should be List"),
                    }
                }

                let data_type = get_base_type(&input_expr_types[0])?;
                Ok(data_type)
            }

probably just need to add a branch here for FixedSizeList

@universalmind303 universalmind303 added the bug Something isn't working label Feb 5, 2024
@Weijun-H
Copy link
Member

Weijun-H commented Feb 6, 2024

This issue can be fixed by #9108

@alamb
Copy link
Contributor

alamb commented Feb 8, 2024

This issue can be fixed by #9108

Thanks @Weijun-H -- I will put #9108 on my review queue

@jonahgao
Copy link
Member

Close it because it is already supported.

DataFusion CLI v38.0.0

> select flatten(arrow_cast(make_array([[1], [2]], [[3], [4]]), 'FixedSizeList(2, List(List(Int64)))')) as a;
+--------------+
| a            |
+--------------+
| [1, 2, 3, 4] |
+--------------+
1 row(s) fetched.
Elapsed 0.011 seconds.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants