Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add --ddl-file option and support offload to an existing empty table #149

Merged
merged 36 commits into from
Jul 9, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
59560a6
feat: Add --ddl-file option
nj1973 Apr 8, 2024
16d3175
feat: Add --ddl-file option
nj1973 Apr 9, 2024
9edbc7b
Merge remote-tracking branch 'origin/main' into 139-decouple-table-cr…
nj1973 Apr 9, 2024
1106434
feat: Prep for --ddl-file, move execute from config to operation
nj1973 Apr 9, 2024
a7c88af
Merge remote-tracking branch 'origin/main' into 139-decouple-table-cr…
nj1973 Apr 9, 2024
f75ca9c
feat: Add --ddl-file option for local filesystem
nj1973 Apr 10, 2024
4f0d519
feat: Add --ddl-file option
nj1973 Apr 11, 2024
0fd6666
feat: Add --ddl-file option
nj1973 Apr 11, 2024
92d44c6
feat: Add --ddl-file option
nj1973 Apr 12, 2024
0947495
Merge remote-tracking branch 'origin/main' into 139-decouple-table-cr…
nj1973 Apr 12, 2024
0470bda
feat: Decouple table creation and data loading
nj1973 Apr 15, 2024
cc73276
feat: Decouple table creation and data loading
nj1973 Apr 17, 2024
3abf5b3
feat: Decouple table creation and data loading
nj1973 Apr 17, 2024
2f9163a
Merge remote-tracking branch 'origin/main' into 139-decouple-table-cr…
nj1973 Apr 17, 2024
ae7b2f1
feat: Decouple table creation and data loading
nj1973 Apr 17, 2024
51f791a
chore: repo upgrade for 1.0.4
abb9979 Apr 17, 2024
ab44783
chore: repo upgrade for 1.0.4
abb9979 Apr 17, 2024
477a031
feat: Decouple table creation and data loading
nj1973 Apr 18, 2024
0dd77eb
feat: Decouple table creation and data loading
nj1973 Apr 25, 2024
2ff3429
Merge remote-tracking branch 'origin/main' into 139-decouple-table-cr…
nj1973 Apr 29, 2024
d32131e
feat: Decouple table creation and data loading
nj1973 May 10, 2024
0674398
feat: Decouple table creation and data loading
nj1973 May 10, 2024
3591321
feat: Decouple table creation and data loading
nj1973 May 10, 2024
2fe928e
feat: Add backend schema DDL to DDL file
nj1973 May 16, 2024
b806a76
feat: Better message when heap table already has data
nj1973 May 17, 2024
06a8c59
feat: Better message when heap table already has data
nj1973 May 17, 2024
593dcfb
feat: Add backend load database DDL to DDL file
nj1973 May 21, 2024
c260ce3
feat: Remove trailing spaces from backend table DDL
nj1973 May 22, 2024
127d0d8
Merge remote-tracking branch 'origin/main' into 139-decouple-table-cr…
nj1973 May 23, 2024
a7f895e
feat: Decouple table creation and data loading
nj1973 May 23, 2024
9f4b04d
feat: Change BigQUery external table creation to be with SQL and not …
nj1973 May 23, 2024
61cb152
Merge remote-tracking branch 'origin/main' into 139-decouple-table-cr…
nj1973 Jun 14, 2024
b56eac6
fix: Fix bug where load db dataset location is excluded from DDL file
nj1973 Jul 5, 2024
c308100
chore: Typo
nj1973 Jul 5, 2024
d5e9d64
Merge remote-tracking branch 'origin/main' into 139-decouple-table-cr…
nj1973 Jul 8, 2024
fa4f305
fix: Fix merge issues
nj1973 Jul 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix: Fix bug where load db dataset location is excluded from DDL file
  • Loading branch information
nj1973 committed Jul 5, 2024
commit b56eac6006d239c5788f0d99b9cf8a7071eb2147
7 changes: 3 additions & 4 deletions src/goe/offload/backend_table.py
Original file line number Diff line number Diff line change
Expand Up @@ -1802,11 +1802,10 @@ def create_conversion_view(self, column_tuples):
sync=True,
)

def create_load_db(self, location=None, with_terminator=False) -> list:
def create_load_db(self, with_terminator=False) -> list:
"""Generic code to create a load database, individual backends may have overrides."""
if self._db_api.load_db_transport_supported():
return self._create_load_db(
location=location, with_terminator=with_terminator
)
return self._create_load_db(with_terminator=with_terminator)
else:
return []

Expand Down
10 changes: 10 additions & 0 deletions src/goe/offload/bigquery/bigquery_backend_table.py
Original file line number Diff line number Diff line change
Expand Up @@ -498,6 +498,16 @@ def create_db(self, with_terminator=False) -> list:
with_terminator=with_terminator,
)

def create_load_db(self, with_terminator=False) -> list:
"""Create a load database."""
if self._db_api.load_db_transport_supported():
return self._create_load_db(
location=self._orchestration_config.bigquery_dataset_location,
with_terminator=with_terminator,
)
else:
return []

def default_udf_db_name(self):
"""By default we support UDF_DB but on BigQuery we use the data db as a fall back"""
return self._udf_db or self.db_name
Expand Down
10 changes: 10 additions & 0 deletions src/goe/offload/hadoop/hadoop_backend_table.py
Original file line number Diff line number Diff line change
Expand Up @@ -825,6 +825,16 @@ def create_db(self, with_terminator=False) -> list:
with_terminator=with_terminator,
)

def create_load_db(self, with_terminator=False) -> list:
"""Create a load database."""
if self._db_api.load_db_transport_supported():
return self._create_load_db(
location=self._get_load_db_hdfs_dir(),
with_terminator=with_terminator,
)
else:
return []

def default_udf_db_name(self):
"""By default we support UDF_DB but on Hadoop we use 'default' as a fall back"""
return self._udf_db or "default"
Expand Down