Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/decouple adapters from core #8906

Merged
merged 47 commits into from
Dec 22, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
469a9ac
remove dbt.contracts.connection imports from adapter module
MichelleArk Sep 18, 2023
c141148
Move events to common (#8676)
colin-rogers-dbt Sep 21, 2023
a89642a
fix import
MichelleArk Oct 25, 2023
dbd02e5
move types_pb2.py from events to common/events
MichelleArk Oct 25, 2023
f44d704
move agate_helper into common
MichelleArk Oct 25, 2023
1a82548
Add utils module (#8910)
colin-rogers-dbt Oct 26, 2023
b9a603e
remove usage of dbt.config.PartialProject from dbt/adapters (#8909)
MichelleArk Oct 26, 2023
200bcdc
Merge branch 'move-agate-client-to-common' into feature/decouple-adap…
MichelleArk Oct 26, 2023
7de8930
move agate_helper unit tests under tests/unit/common
MichelleArk Oct 26, 2023
af91666
move agate_helper into common (#8911)
MichelleArk Oct 26, 2023
333120b
Merge branch 'main' into feature/decouple-adapters-from-core
MichelleArk Nov 1, 2023
1260782
remove dbt.flags.MP_CONTEXT usage in dbt/adapters (#8931)
MichelleArk Nov 1, 2023
c4ff280
remove dbt.flags.LOG_CACHE_EVENTS usage in dbt/adapters (#8933)
MichelleArk Nov 1, 2023
4ee9504
Merge branch 'main' into feature/decouple-adapters-from-core
MichelleArk Nov 7, 2023
51b94b2
Refactor Base Exceptions (#8989)
colin-rogers-dbt Nov 10, 2023
1d0a3e9
Merge branch 'main' into feature/decouple-adapters-from-core
MichelleArk Nov 16, 2023
e56a5da
Remove usage of dbt.deprecations in dbt/adapters, enable core & adapt…
MichelleArk Nov 20, 2023
2ab0f7b
Decouple adapter constraints from core (#9054)
peterallenwebb Nov 21, 2023
615ad1f
move include/global_project to adapters (#8930)
MichelleArk Nov 27, 2023
220f56d
remove adapter.get_compiler (#9134)
MichelleArk Nov 28, 2023
8bad75c
Move adapter logger to adapters (#9165)
colin-rogers-dbt Nov 28, 2023
12bd1e8
Merge branch 'main' into feature/decouple-adapters-from-core
MichelleArk Nov 29, 2023
294ad82
delete accidentally merged types_pb2.py
MichelleArk Nov 29, 2023
373125e
Move the semver package to common and alter references. (#9166)
VersusFacit Nov 29, 2023
41ac915
Refactor EventManager setup and interaction (#9180)
colin-rogers-dbt Dec 1, 2023
1bca662
first pass: adapter migration script (#9160)
MichelleArk Dec 4, 2023
00f4920
Decouple macro generator from adapters (#9149)
MichelleArk Dec 5, 2023
e01eb30
Remove usage of dbt.contracts.relation in dbt/adapters (#9207)
MichelleArk Dec 6, 2023
ed8f5d3
Remove ResultNode usage from connections (#9211)
MichelleArk Dec 6, 2023
eb96e3d
Add RelationConfig Protocol for use in Relation.create_from (#9210)
MichelleArk Dec 6, 2023
45b3570
Merge main into feature/decouple-adapters-from-core (#9240)
colin-rogers-dbt Dec 6, 2023
c2734c5
Merge branch 'main' into feature/decouple-adapters-from-core
MichelleArk Dec 7, 2023
f1c2f06
Move BaseConfig to Common (#9224)
colin-rogers-dbt Dec 7, 2023
26ddaaf
Remove manifest from catalog and connection method signatures (#9242)
MichelleArk Dec 7, 2023
4d16524
Add MacroResolverProtocol, remove lazy loading of manifest in adapter…
MichelleArk Dec 7, 2023
c7b9b1a
pass context to MacroQueryStringSetter (#9248)
MichelleArk Dec 7, 2023
a68e427
add macro_context_generator on adapter (#9251)
MichelleArk Dec 11, 2023
61db33a
Pass mp_context to adapter factory (#9275)
colin-rogers-dbt Dec 13, 2023
0b5a4d3
Fix include for decoupling (#9286)
colin-rogers-dbt Dec 14, 2023
4bb1d4b
Fix include for decoupling (#9288)
colin-rogers-dbt Dec 14, 2023
308a7e9
move system client to common (#9294)
colin-rogers-dbt Dec 15, 2023
7e5a87a
Update materialized views to use RelationConfigs and remove refs to d…
colin-rogers-dbt Dec 15, 2023
c08af13
Add config field to RelationConfig (#9300)
colin-rogers-dbt Dec 18, 2023
34f77d9
merge main into feature/decouple-adapters-from-core (#9305)
colin-rogers-dbt Dec 20, 2023
ff52aef
resolve merge conflict on unparsed.py (#9309)
colin-rogers-dbt Dec 22, 2023
ab0e4c4
Resolve unparsed.py conflict (#9311)
colin-rogers-dbt Dec 22, 2023
ac3b17f
Merge branch 'main' into feature/decouple-adapters-from-core
ChenyuLInx Dec 22, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
6 changes: 6 additions & 0 deletions .changes/unreleased/Breaking Changes-20231127-114757.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Breaking Changes
body: Remove adapter.get_compiler interface
time: 2023-11-27T11:47:57.443202-05:00
custom:
Author: michelleark
Issue: "9148"
6 changes: 6 additions & 0 deletions .changes/unreleased/Breaking Changes-20231128-134356.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Breaking Changes
body: Move AdapterLogger to adapters folder
time: 2023-11-28T13:43:56.853925-08:00
custom:
Author: colin-rogers-dbt
Issue: "9151"
7 changes: 7 additions & 0 deletions .changes/unreleased/Breaking Changes-20231130-135348.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
kind: Breaking Changes
body: move event manager setup back to core, remove ref to global EVENT_MANAGER and
clean up event manager functions
time: 2023-11-30T13:53:48.645192-08:00
custom:
Author: colin-rogers-dbt
Issue: "9150"
6 changes: 6 additions & 0 deletions .changes/unreleased/Features-20230915-123733.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Features
body: 'Allow adapters to include package logs in dbt standard logging '
time: 2023-09-15T12:37:33.862862-07:00
custom:
Author: colin-rogers-dbt
Issue: "7859"
6 changes: 6 additions & 0 deletions .changes/unreleased/Features-20231026-110821.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Features
body: migrate utils to common and adapters folders
time: 2023-10-26T11:08:21.458709-07:00
custom:
Author: colin-rogers-dbt
Issue: "8924"
6 changes: 6 additions & 0 deletions .changes/unreleased/Features-20231026-123556.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Features
body: Move Agate helper client into common
time: 2023-10-26T12:35:56.538587-07:00
custom:
Author: MichelleArk
Issue: "8926"
6 changes: 6 additions & 0 deletions .changes/unreleased/Features-20231026-123913.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Features
body: remove usage of dbt.config.PartialProject from dbt/adapters
time: 2023-10-26T12:39:13.904116-07:00
custom:
Author: MichelleArk
Issue: "8928"
6 changes: 6 additions & 0 deletions .changes/unreleased/Features-20231107-135635.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Features
body: Remove legacy logger
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we don't use this on cloud anymore, do we need to double check?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MichelleArk checked and they have all moved to event logging

time: 2023-11-07T13:56:35.186648-08:00
custom:
Author: colin-rogers-dbt
Issue: "8027"
6 changes: 6 additions & 0 deletions .changes/unreleased/Under the Hood-20230831-164435.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Under the Hood
body: Added more type annotations.
time: 2023-08-31T16:44:35.737954-04:00
custom:
Author: peterallenwebb
Issue: "8537"
6 changes: 6 additions & 0 deletions .changes/unreleased/Under the Hood-20231026-184953.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Under the Hood
body: Remove usage of dbt.include.global_project in dbt/adapters
time: 2023-10-26T18:49:53.36449-04:00
custom:
Author: michelleark
Issue: "8925"
6 changes: 6 additions & 0 deletions .changes/unreleased/Under the Hood-20231101-102758.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Under the Hood
body: remove dbt.flags.MP_CONTEXT usage in dbt/adapters
time: 2023-11-01T10:27:58.790153-04:00
custom:
Author: michelleark
Issue: "8967"
6 changes: 6 additions & 0 deletions .changes/unreleased/Under the Hood-20231101-173124.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Under the Hood
body: 'Remove usage of dbt.flags.LOG_CACHE_EVENTS in dbt/adapters'
time: 2023-11-01T17:31:24.974093-04:00
custom:
Author: michelleark
Issue: "8969"
6 changes: 6 additions & 0 deletions .changes/unreleased/Under the Hood-20231107-135728.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Under the Hood
body: Remove use of dbt/core exceptions in dbt/adapter
time: 2023-11-07T13:57:28.683727-08:00
custom:
Author: colin-rogers-dbt MichelleArk
Issue: "8920"
7 changes: 7 additions & 0 deletions .changes/unreleased/Under the Hood-20231116-174251.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
kind: Under the Hood
body: Remove usage of dbt.deprecations in dbt/adapters, enable core & adapter-specific
event types and protos
time: 2023-11-16T17:42:51.005023-05:00
custom:
Author: michelleark
Issue: 8927 8918
7 changes: 7 additions & 0 deletions .changes/unreleased/Under the Hood-20231120-183214.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
kind: Under the Hood
body: Move column constraints into common/contracts, removing another dependency of
adapters on core.
time: 2023-11-20T18:32:14.859503-05:00
custom:
Author: peterallenwebb
Issue: "9024"
6 changes: 6 additions & 0 deletions .changes/unreleased/Under the Hood-20231128-170732.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Under the Hood
body: Move dbt.semver to dbt.common.semver and update references.
time: 2023-11-28T17:07:32.172421-08:00
custom:
Author: versusfacit
Issue: "9039"
6 changes: 6 additions & 0 deletions .changes/unreleased/Under the Hood-20231130-135432.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Under the Hood
body: Move lowercase utils method to common
time: 2023-11-30T13:54:32.561673-08:00
custom:
Author: colin-rogers-dbt
Issue: "9180"
6 changes: 6 additions & 0 deletions .changes/unreleased/Under the Hood-20231205-093544.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Under the Hood
body: Remove usages of dbt.clients.jinja in dbt/adapters
time: 2023-12-05T09:35:44.845352+09:00
custom:
Author: michelleark
Issue: "9205"
6 changes: 6 additions & 0 deletions .changes/unreleased/Under the Hood-20231205-120559.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Under the Hood
body: Remove usage of dbt.contracts in dbt/adapters
time: 2023-12-05T12:05:59.936775+09:00
custom:
Author: michelleark
Issue: "9208"
6 changes: 6 additions & 0 deletions .changes/unreleased/Under the Hood-20231205-165812.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Under the Hood
body: Remove usage of dbt.contracts.graph.nodes.ResultNode in dbt/adapters
time: 2023-12-05T16:58:12.932172+09:00
custom:
Author: michelleark
Issue: "9214"
6 changes: 6 additions & 0 deletions .changes/unreleased/Under the Hood-20231205-170725.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Under the Hood
body: Introduce RelationConfig Protocol, consolidate Relation.create_from
time: 2023-12-05T17:07:25.33861+09:00
custom:
Author: michelleark
Issue: "9215"
7 changes: 7 additions & 0 deletions .changes/unreleased/Under the Hood-20231205-185022.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
kind: Under the Hood
body: Move BaseConfig, Metadata and various other contract classes from model_config
to common/contracts/config
time: 2023-12-05T18:50:22.321229-08:00
custom:
Author: colin-rorgers-dbt
Issue: "8919"
6 changes: 6 additions & 0 deletions .changes/unreleased/Under the Hood-20231205-235830.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Under the Hood
body: remove manifest from adapter.set_relations_cache signature
time: 2023-12-05T23:58:30.920144+09:00
custom:
Author: michelleark
Issue: "9217"
6 changes: 6 additions & 0 deletions .changes/unreleased/Under the Hood-20231206-000343.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Under the Hood
body: ' remove manifest from adapter catalog method signatures'
time: 2023-12-06T00:03:43.824252+09:00
custom:
Author: michelleark
Issue: "9218"
6 changes: 6 additions & 0 deletions .changes/unreleased/Under the Hood-20231207-111554.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Under the Hood
body: Add MacroResolverProtocol, remove lazy loading of manifest in adapter.execute_macro
time: 2023-12-07T11:15:54.427818+09:00
custom:
Author: michelleark
Issue: "9244"
6 changes: 6 additions & 0 deletions .changes/unreleased/Under the Hood-20231207-224139.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Under the Hood
body: pass query header context to MacroQueryStringSetter
time: 2023-12-07T22:41:39.498024+09:00
custom:
Author: michelleark
Issue: 9249 9250
6 changes: 6 additions & 0 deletions .changes/unreleased/Under the Hood-20231208-004854.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Under the Hood
body: add macro_context_generator on adapter
time: 2023-12-08T00:48:54.506911+09:00
custom:
Author: michelleark
Issue: "9247"
6 changes: 6 additions & 0 deletions .changes/unreleased/Under the Hood-20231212-154842.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Under the Hood
body: pass mp_context to adapter factory as argument instead of import
time: 2023-12-12T15:48:42.866175-08:00
custom:
Author: colin-rogers-dbt
Issue: "9025"
6 changes: 6 additions & 0 deletions .changes/unreleased/Under the Hood-20231214-122134.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Under the Hood
body: have dbt-postgres use RelationConfig protocol for materialized views'
time: 2023-12-14T12:21:34.756973-08:00
custom:
Author: colin-rogers-dbt
Issue: "9292"
6 changes: 6 additions & 0 deletions .changes/unreleased/Under the Hood-20231214-164107.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Under the Hood
body: move system.py to common as dbt-bigquery relies on it to call gcloud
time: 2023-12-14T16:41:07.539814-08:00
custom:
Author: colin-rogers-dbt
Issue: "9293"
2 changes: 2 additions & 0 deletions .flake8
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,5 @@ ignore =
E741
E501 # long line checking is done in black
exclude = test/
per-file-ignores =
*/__init__.py: F401
2 changes: 1 addition & 1 deletion .gitattributes
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
core/dbt/include/index.html binary
core/dbt/task/docs/index.html binary
tests/functional/artifacts/data/state/*/manifest.json binary
core/dbt/docs/build/html/searchindex.js binary
core/dbt/docs/build/html/index.html binary
Expand Down
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Configuration for pre-commit hooks (see https://pre-commit.com/).
# Eventually the hooks described here will be run as tests before merging each PR.

exclude: ^(core/dbt/docs/build/|core/dbt/events/types_pb2.py)
exclude: ^(core/dbt/docs/build/|core/dbt/common/events/types_pb2.py|core/dbt/events/core_types_pb2.py|core/dbt/adapters/events/adapter_types_pb2.py)

# Force all unspecified python hooks to run python 3.8
default_language_version:
Expand Down
2 changes: 1 addition & 1 deletion ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Legacy tests are found in the 'test' directory:

The "tasks" map to top-level dbt commands. So `dbt run` => task.run.RunTask, etc. Some are more like abstract base classes (GraphRunnableTask, for example) but all the concrete types outside of task should map to tasks. Currently one executes at a time. The tasks kick off their “Runners” and those do execute in parallel. The parallelism is managed via a thread pool, in GraphRunnableTask.

core/dbt/include/index.html
core/dbt/task/docs/index.html
This is the docs website code. It comes from the dbt-docs repository, and is generated when a release is packaged.

## Adapters
Expand Down
11 changes: 10 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,16 @@ dev: dev_req ## Installs dbt-* packages in develop mode along with development d

.PHONY: proto_types
proto_types: ## generates google protobuf python file from types.proto
protoc -I=./core/dbt/events --python_out=./core/dbt/events ./core/dbt/events/types.proto
protoc -I=./core/dbt/common/events --python_out=./core/dbt/common/events ./core/dbt/common/events/types.proto

.PHONY: core_proto_types
core_proto_types: ## generates google protobuf python file from core_types.proto
protoc -I=./core/dbt/events --python_out=./core/dbt/events ./core/dbt/events/core_types.proto

.PHONY: adapter_proto_types
adapter_proto_types: ## generates google protobuf python file from core_types.proto
protoc -I=./core/dbt/adapters/events --python_out=./core/dbt/adapters/events ./core/dbt/adapters/events/adapter_types.proto


.PHONY: mypy
mypy: .env ## Runs mypy against staged changes for static type checking.
Expand Down
3 changes: 2 additions & 1 deletion core/MANIFEST.in
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
recursive-include dbt/include *.py *.sql *.yml *.html *.md .gitkeep .gitignore
recursive-include dbt/adapters/include *.py *.sql *.yml *.html *.md .gitkeep .gitignore
include dbt/py.typed
recursive-include dbt/task/docs *.html
2 changes: 1 addition & 1 deletion core/dbt/adapters/base/__init__.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# these are all just exports, #noqa them so flake8 will be happy

# TODO: Should we still include this in the `adapters` namespace?
from dbt.contracts.connection import Credentials # noqa: F401
from dbt.adapters.contracts.connection import Credentials # noqa: F401
from dbt.adapters.base.meta import available # noqa: F401
from dbt.adapters.base.connections import BaseConnectionManager # noqa: F401
from dbt.adapters.base.relation import ( # noqa: F401
Expand Down
2 changes: 1 addition & 1 deletion core/dbt/adapters/base/column.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
import re
from typing import Dict, ClassVar, Any, Optional

from dbt.exceptions import DbtRuntimeError
from dbt.common.exceptions import DbtRuntimeError


@dataclass
Expand Down
Loading