-
Notifications
You must be signed in to change notification settings - Fork 60
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
37 migrate to create streaming table api from create streaming live table #39
37 migrate to create streaming table api from create streaming live table #39
Conversation
…able 2.Added expectations for silver layer as part of new create_streaming_table function params 3.Fixed unit and integrations tests with new config change for silver data quality expectations 4.Modidied demo config for silver transformations data quality expectations
2.Modified demo for silver dqe 3.Corrected dbfs api for file create
2.Added doc for silver data quality expectation json
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #39 +/- ##
==========================================
- Coverage 90.28% 89.90% -0.39%
==========================================
Files 8 8
Lines 803 832 +29
Branches 149 161 +12
==========================================
+ Hits 725 748 +23
- Misses 31 33 +2
- Partials 47 51 +4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
2.Corrected techsummit demo name
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approved changes
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
self.ws.dbfs.mkdirs(runner_conf.dbfs_tmp_path + f"/{self.base_dir}/") | ||
except Exception as e: | ||
print(f"Error in creating directory {runner_conf.dbfs_tmp_path + f'/{self.base_dir}/'}") | ||
print(e) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use logger, not print statements
@@ -726,7 +734,7 @@ def run(self, runner_conf: DLTMetaRunnerConf): | |||
print(e) | |||
finally: | |||
print("Cleaning up...") | |||
self.clean_up(runner_conf) | |||
# self.clean_up(runner_conf) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No commented code
create_streaming_live_table
to latestcreate_streaming_table
create_streaming_table
api takesexpectations
as arguments