Bug fixes and manual import schema customization. #941 #945 #946 #944
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zstumgoren
changed the title
…schema overrides * Add schema override option to manual_import command * Update Dataset.import_data and utils code to support schema overrides * Add test for schema override * Update docs to reflect schema override option
zstumgoren
changed the title
zstumgoren
changed the title
|
Updating PR to include addition of schema customization support for |
|
@JoeGermuska Could I trouble you to test a theory for me by trying both of the manual tests below and letting me know the results?
|
|
Your "Scenario 1" is what we did with the files you sent me, right? I just tested Scenario 2 with the original file and got the desired behavior. I made one comment on the docs in the PR issue, but otherwise, this tests out for me. |
|
Yep, scenario 1 is the CA data; Scenario 2 is the Cook data. |
* Note that override field names must precisely match field names in source data * Note unexpected behavior of type inference on dollar-sign prefixed fields * Flesh out workflow bits related to experimenting with slice of data
|
@JoeGermuska manual_import mgmt cmd docs are updated per our discussion on c89d607; your upstream changes to FAQ and reqs.txt (#941) have been integrated. Lmk if you have questions or need anything else in order to merge the PR. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses the openpyxl (#941) and db numeric type bugs (#945), and the addition of support for adding field-level search using the
manual importmgmt command (#946).All changes were tested on a local development version of PANDA, both manually (through web gui) and with the addition of new unittest (full test suite is still green).
Note that the migration updates three models that inherit from the
BaseUploadABC:DataUploadExportRelatedUploadAdditionally, I've verified in our production install that this patch works on a large upload exceeding the Postgres integer threshold; prior to this patch, that upload was failing with the error detailed here.
The openpyxl bug (#941) prevents a successful build of the stack, so combined with the db fix, it might be worth a new patch release (1.1.2) and announcement on the PANDA group. Changes in this PR are listed in the CHANGELOG under 1.1.2.
Let me know if you have questions or want a new PR with tweaks to the code/tests.