Enable reading metadata as a pandas DataFrame #743
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #743 +/- ##
==========================================
- Coverage 31.06% 31.05% -0.01%
==========================================
Files 41 41
Lines 5710 5731 +21
Branches 1389 1394 +5
==========================================
+ Hits 1774 1780 +6
- Misses 3861 3874 +13
- Partials 75 77 +2
Continue to review full report at Codecov.
|
This was referenced Jul 6, 2021
Allows metadata to be returned as a pandas DataFrame instead of a dictionary and uses this API to load metadata as a DataFrame in augur filter. Updates the corresponding metadata references in augur filter and related function calls to work with this new data structure. This change allows us to eventually switch to chunking metadata input to avoid reading it all into memory.
pylint complains about several stylistic issues with our code that are always low priority (e.g., import order). This commit ignores some of the most annoying warnings, so built-in linters (in emacs, etc.) are more useful for catching real issues.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of proposed changes
This PR modifies the current
read_metadatafunction inutils.pyand the associatedmetadata_file.pymodule to allow users to read metadata as a pandas DataFrame instead of as a Python dictionary. We have used pandas to read our metadata for a long time now, but we have converted the intermediate DataFrame into a dictionary for backwards compatibility across Augur. Since Python dictionaries use memory less efficiently than DataFrames for this type of data, we would like the option to return the intermediate DataFrame as the final representation of the metadata.Most changes in this PR involve updating the
filter.pymodule to use this new DataFrame interface for its downstream filtering logic. The most complicated change here involves a refactor of theget_numerical_datesutility function to support dates from a DataFrame. While we could rewrite this function to use less redundant logic, we can leave that work for a later PR.Note that this PR is the first step toward a refactor of
filter.pydescribed in #699. By using pandas DataFrames in the filter logic now, we can later rely on the DataFrame chunking interface to avoid reading all metadata into memory and still benefit from pandas'query, indexing, and data type-related functionality.Related issue(s)
Related to #699
Testing