Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CSV Import improvements #866

Open
abandoned-prototype opened this issue May 4, 2021 · 0 comments
Open

CSV Import improvements #866

abandoned-prototype opened this issue May 4, 2021 · 0 comments

Comments

@abandoned-prototype
Copy link
Collaborator

Having used the advanced csv import feature multiple times for various different departments now I have a few things I think we can improve on the tool.

  1. Automatically create job titles that do not exist yet, instead of failing. They can just be created in a random order, or maybe based on how many values we have per job title. By default all should be marked "sworn". An admin or area coordinator can then sort them in the correct order with the web ui (the order also doesn't seem to have any impact expect for the sort order in the selection menu).

  2. Instead of providing unit-id just provide unit names and create all the units that are provided in the data set. Currently we have no way of creating units via commands and relying on unit-ids means generating different csv files for the different environments (like local dev, staging and production).

  3. Add another flag for local development that will truncate the existing tables for the given department

  4. Either relax the condition on the department name column when having --force-create set or add a department name column to the csv download. Also make sure that all other fields of the csv download match fields expected by the csv import command with the goal that downloading the csv files and then importing locally gives you a basically exact copy of the production data for that department

  5. Dive deeper into different fields and decide if they can actually be optional or not. We have a lot of fields in our tables that are allowed to be NULL but if you actually set them NULL some script breaks because it can't handle None.
    For example

May 04 04:07:33 staging gunicorn[1350]:     {% for type, list in obj.links|groupby('link_type') %}
May 04 04:07:33 staging gunicorn[1350]:   File "[..]/lib/python3.5/site-packages/jinja2/filters.py", line 852, in do_groupby
May 04 04:07:33 staging gunicorn[1350]:     in groupby(sorted(value, key=expr), expr)]
May 04 04:07:33 staging gunicorn[1350]: TypeError: unorderable types: NoneType() < NoneType()

A result of not setting link_type.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant