[FEA] Add --meta parameter to explicitly specify the jsonl field dtypes #63
Labels
enhancement
New feature or request
Comments
|
I will work in the feature. |
|
PR #75. |
ayushdg
pushed a commit
that referenced
this issue
May 30, 2024
…ield dtypes (#75) * Add dtype support (optional) when reading jsonl files Signed-off-by: Miguel Martínez <miguelm@nvidia.com> Signed-off-by: Miguel Martínez <miguelusque@users.noreply.github.com> * Resolve merge conflit Signed-off-by: Miguel Martínez <miguelusque@users.noreply.github.com> * Resolve merge conflit Signed-off-by: Miguel Martínez <miguelusque@users.noreply.github.com> * Resolve merge conflit Signed-off-by: Miguel Martínez <miguelusque@users.noreply.github.com> * Resolve merge conflit Signed-off-by: Miguel Martínez <miguelusque@users.noreply.github.com> * Resolve merge conflit Signed-off-by: Miguel Martínez <miguelusque@users.noreply.github.com> * Resolve merge conflit Signed-off-by: Miguel Martínez <miguelusque@users.noreply.github.com> * Change input_meta type hint Signed-off-by: Miguel Martínez <miguelusque@users.noreply.github.com> * Change input_meta type hint Signed-off-by: Miguel Martínez <miguelusque@users.noreply.github.com> * Resolve merge conflit Signed-off-by: Miguel Martínez <miguelusque@users.noreply.github.com> * Resolve merge conflit Signed-off-by: Miguel Martínez <miguelusque@users.noreply.github.com> * Resolve merge conflit Signed-off-by: Miguel Martínez <miguelusque@users.noreply.github.com> * Resolve merge conflit Signed-off-by: Miguel Martínez <miguelusque@users.noreply.github.com> * Resolve merge conflit Signed-off-by: Miguel Martínez <miguelusque@users.noreply.github.com> * Resolve merge conflit Signed-off-by: Miguel Martínez <miguelusque@users.noreply.github.com> * Resolve merge conflit Signed-off-by: Miguel Martínez <miguelusque@users.noreply.github.com> * Resolve merge conflit Signed-off-by: Miguel Martínez <miguelusque@users.noreply.github.com> * Assign input_meta to the right variable Signed-off-by: Miguel Martínez <miguelusque@users.noreply.github.com> * Add warning when input_meta is used with non jsonl files. Signed-off-by: Miguel Martínez <miguelusque@users.noreply.github.com> * Explicitly check for None when validating input_meta Signed-off-by: Miguel Martínez <miguelusque@users.noreply.github.com> * Add input_meta test Signed-off-by: Miguel Martínez <miguelusque@users.noreply.github.com> * Add description to function Signed-off-by: Miguel Martínez <miguelusque@users.noreply.github.com> * Add test_meta_str Signed-off-by: Miguel Martínez <miguelusque@users.noreply.github.com> --------- Signed-off-by: Miguel Martínez <miguelm@nvidia.com> Signed-off-by: Miguel Martínez <miguelusque@users.noreply.github.com> Co-authored-by: Miguel Martínez <miguelusque@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
When reading jsonl files with Dask, the dataframe datatypes are inferred unless explicitly specified.
Inferring the data types can lead to several issues, such as incorrect type inference, degradation of performance and increased memory usage among others.
I think we could mitigate those issues if we would add a
--metaparameter, which would receive a dictionary of datatypes.That parameter would be optional, and be similar to the
--metaparameter available here: https://docs.dask.org/en/latest/generated/dask.dataframe.read_json.html.The text was updated successfully, but these errors were encountered: