Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added functionality to Datasource, to add an empty extract, and to filter the extract #74

Merged
merged 1 commit into from
Mar 19, 2024

Conversation

JustinGrilli
Copy link
Collaborator

@JustinGrilli JustinGrilli commented Mar 19, 2024

Summary

Our Astronomer / Airflow workers running updates on our datasources have been failing.
We discovered this was because the size of our datasources exceeded the 10GB storage limit for the Astronomer workers.

So, to help adjust the size of our Datasources while making updates, I am adding functionality to allow for manipulating the Datasource extract (.hyper file).

Changes

  • Added functionality to the Datasource class
    • empty_extract: To create an empty extract for the Datasource connection.
    • filter_extract: To delete data from the extract, based on a provided condition string.
  • Added --empty_extract (-E) and --filter_extract (-F) to the datasource CLI command args

Tests

tableau_utilities -l online -n Datasource -pn Project -f Datasource.tdsx datasource -E
tableau_utilities -n Datasource -pn Project -f Datasource.tdsx server_operate --publish datasource
  • Download datasource without including the extract, and create an empty extract
  • Publish the datasource (normally you would not be able to publish a datasource downloaded without an extract, now we can!)
tableau_utilities -l online -n Datasource -pn Project -f Datasource.tdsx --include_extract datasource -E
tableau_utilities -n Datasource -pn Project -f Datasource.tdsx server_operate --publish datasource
  • Download datasource including the extract, and overwrite extract with an empty extract
  • Publish the datasource
tableau_utilities -l online -n Datasource -pn Project -f Datasource.tdsx --include_extract datasource -F "\"CREATED_AT\" < '1/1/2024'"
tableau_utilities -n Datasource -pn Project -f Datasource.tdsx server_operate --publish datasource
  • Download datasource including the extract, and delete extract data from before 1/1/2024
  • Publish the datasource

@JustinGrilli JustinGrilli merged commit e8a0465 into main Mar 19, 2024
1 check passed
@JustinGrilli JustinGrilli deleted the add-tableau-file-functionality-empty-extract branch March 19, 2024 19:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant