Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: fixes links to code examples #220

Merged
merged 2 commits into from
Jul 23, 2023
Merged

docs: fixes links to code examples #220

merged 2 commits into from
Jul 23, 2023

Conversation

willi-mueller
Copy link
Collaborator

Tell us what you do here

  • implementing verified source (please link a relevant issue labelled as verified source)
  • fixing a bug (please link a relevant bug report)
  • improving, documenting, or customizing existing sources (please link an issue or describe below)
  • anything else (please link an issue or describe below)

More PR info

This PR corrects dead links in the contributor documentation. I think I identified the currently correct line of code which is supposed to exemplify a concept.

However, if we link to a line number I'd propose to use permalinks instead of links to the master. Otherwise, they can become confusing soon.

AstrakhantsevaAA
AstrakhantsevaAA requested changes Jul 20, 2023
View reviewed changes
docs/BUILDING-BLOCKS.md Outdated
* [Incrementally dispatch github events to separate tables](https://github.com/dlt-hub/verified-sources/blob/master/sources/github/__init__.py#L70)
* [Read the participants for each deal using transformers and pipe operator](https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/__init__.py#L67)
* [Read the events for each ticket by attaching transformer to resource explicitly](https://github.com/dlt-hub/verified-sources/blob/master/sources/hubspot/__init__.py#L125)
* [Set `tags` column data type to complex to load them as JSON/struct](https://github.com/dlt-hub/verified-sources/blob/master/sources/zendesk/__init__.py#L108)
* Typical use of `merge` with incremental load for endpoints returning a list of updates to entities in [Shopify source](https://github.com/dlt-hub/verified-sources/blob/master/sources/shopify_dlt/__init__.py#L36).
* A `dlt` mega-combo in `pipedrive` source, where the deals from `deal` endpoint are [fed into](https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/__init__.py#L113) `deals_flow` resource to obtain events for a particular deal. [Both resources use `merge` write disposition and incremental load to get just the newest updates](https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/__init__.py#L103). [The `deals_flow` is dispatching different event types to separate tables with `dlt.mark.with_table_name`](https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/__init__.py#L135).
* An example of using JSONPath expression to get cursor value for incremental loading. In pipedrive some objects have `timestamp` property and others `update_time`. [The dlt.sources.incremental('update_time|modified') expression lets you bind the incremental to either](https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/recents.py#L39).
* An example of using JSONPath expression to get cursor value for incremental loading. In pipedrive some objects have `timestamp` property and others `update_time`. [The dlt.sources.incremental('update_time|modified') expression lets you bind the incremental to either](https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/__init__.py#L176).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you use this link please https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/helpers/pages.py#L46C5-L48C7

docs/BUILDING-BLOCKS.md Outdated
@@ -22,14 +22,14 @@ Concepts to grasp
* [How we distribute sources to our users](DISTRIBUTION.md)

Building blocks used right:
* [Create dynamic resources for tables by reflecting a whole database](https://github.com/dlt-hub/verified-sources/blob/master/sources/sql_database/sql_database.py#L50)
* [Create dynamic resources for tables by reflecting a whole database](https://github.com/dlt-hub/verified-sources/blob/master/sources/sql_database/__init__.py#L56)
* [Incrementally dispatch github events to separate tables](https://github.com/dlt-hub/verified-sources/blob/master/sources/github/__init__.py#L70)
* [Read the participants for each deal using transformers and pipe operator](https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/__init__.py#L67)
* [Read the events for each ticket by attaching transformer to resource explicitly](https://github.com/dlt-hub/verified-sources/blob/master/sources/hubspot/__init__.py#L125)
* [Set `tags` column data type to complex to load them as JSON/struct](https://github.com/dlt-hub/verified-sources/blob/master/sources/zendesk/__init__.py#L108)
* Typical use of `merge` with incremental load for endpoints returning a list of updates to entities in [Shopify source](https://github.com/dlt-hub/verified-sources/blob/master/sources/shopify_dlt/__init__.py#L36).
* A `dlt` mega-combo in `pipedrive` source, where the deals from `deal` endpoint are [fed into](https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/__init__.py#L113) `deals_flow` resource to obtain events for a particular deal. [Both resources use `merge` write disposition and incremental load to get just the newest updates](https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/__init__.py#L103). [The `deals_flow` is dispatching different event types to separate tables with `dlt.mark.with_table_name`](https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/__init__.py#L135).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

replace please
[The deals_flow is dispatching different event types to separate tables with dlt.mark.with_table_name] https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/__init__.py#L135

to https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/__init__.py#L105

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and this

[Both resources use merge write disposition and incremental load to get just the newest updates]
https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/__init__.py#L103

to https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/__init__.py#L77-L80

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and this

[fed into]
https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/__init__.py#L113

to https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/__init__.py#L90-L92

docs/BUILDING-BLOCKS.md Outdated
@@ -22,14 +22,14 @@ Concepts to grasp
* [How we distribute sources to our users](DISTRIBUTION.md)

Building blocks used right:
* [Create dynamic resources for tables by reflecting a whole database](https://github.com/dlt-hub/verified-sources/blob/master/sources/sql_database/sql_database.py#L50)
* [Create dynamic resources for tables by reflecting a whole database](https://github.com/dlt-hub/verified-sources/blob/master/sources/sql_database/__init__.py#L56)
* [Incrementally dispatch github events to separate tables](https://github.com/dlt-hub/verified-sources/blob/master/sources/github/__init__.py#L70)
* [Read the participants for each deal using transformers and pipe operator](https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/__init__.py#L67)
* [Read the events for each ticket by attaching transformer to resource explicitly](https://github.com/dlt-hub/verified-sources/blob/master/sources/hubspot/__init__.py#L125)
* [Set `tags` column data type to complex to load them as JSON/struct](https://github.com/dlt-hub/verified-sources/blob/master/sources/zendesk/__init__.py#L108)
* Typical use of `merge` with incremental load for endpoints returning a list of updates to entities in [Shopify source](https://github.com/dlt-hub/verified-sources/blob/master/sources/shopify_dlt/__init__.py#L36).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and this
[Shopify source]
https://github.com/dlt-hub/verified-sources/blob/master/sources/shopify_dlt/__init__.py#L36

to https://github.com/dlt-hub/verified-sources/blob/master/sources/shopify_dlt/__init__.py#L41

docs/BUILDING-BLOCKS.md Outdated
@@ -22,14 +22,14 @@ Concepts to grasp
* [How we distribute sources to our users](DISTRIBUTION.md)

Building blocks used right:
* [Create dynamic resources for tables by reflecting a whole database](https://github.com/dlt-hub/verified-sources/blob/master/sources/sql_database/sql_database.py#L50)
* [Create dynamic resources for tables by reflecting a whole database](https://github.com/dlt-hub/verified-sources/blob/master/sources/sql_database/__init__.py#L56)
* [Incrementally dispatch github events to separate tables](https://github.com/dlt-hub/verified-sources/blob/master/sources/github/__init__.py#L70)
* [Read the participants for each deal using transformers and pipe operator](https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/__init__.py#L67)
* [Read the events for each ticket by attaching transformer to resource explicitly](https://github.com/dlt-hub/verified-sources/blob/master/sources/hubspot/__init__.py#L125)
* [Set `tags` column data type to complex to load them as JSON/struct](https://github.com/dlt-hub/verified-sources/blob/master/sources/zendesk/__init__.py#L108)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to https://github.com/dlt-hub/verified-sources/blob/master/sources/zendesk/__init__.py#L254-L257

docs/BUILDING-BLOCKS.md Outdated
@@ -22,14 +22,14 @@ Concepts to grasp
* [How we distribute sources to our users](DISTRIBUTION.md)

Building blocks used right:
* [Create dynamic resources for tables by reflecting a whole database](https://github.com/dlt-hub/verified-sources/blob/master/sources/sql_database/sql_database.py#L50)
* [Create dynamic resources for tables by reflecting a whole database](https://github.com/dlt-hub/verified-sources/blob/master/sources/sql_database/__init__.py#L56)
* [Incrementally dispatch github events to separate tables](https://github.com/dlt-hub/verified-sources/blob/master/sources/github/__init__.py#L70)
* [Read the participants for each deal using transformers and pipe operator](https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/__init__.py#L67)
Copy link
Collaborator

@AstrakhantsevaAA AstrakhantsevaAA Jul 20, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/__init__.py#L85-L88

docs/BUILDING-BLOCKS.md Outdated
@@ -22,14 +22,14 @@ Concepts to grasp
* [How we distribute sources to our users](DISTRIBUTION.md)

Building blocks used right:
* [Create dynamic resources for tables by reflecting a whole database](https://github.com/dlt-hub/verified-sources/blob/master/sources/sql_database/sql_database.py#L50)
* [Create dynamic resources for tables by reflecting a whole database](https://github.com/dlt-hub/verified-sources/blob/master/sources/sql_database/__init__.py#L56)
* [Incrementally dispatch github events to separate tables](https://github.com/dlt-hub/verified-sources/blob/master/sources/github/__init__.py#L70)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to https://github.com/dlt-hub/verified-sources/blob/master/sources/github/__init__.py#L91-L95

docs/BUILDING-BLOCKS.md Outdated
* [Incrementally dispatch github events to separate tables](https://github.com/dlt-hub/verified-sources/blob/master/sources/github/__init__.py#L70)
* [Read the participants for each deal using transformers and pipe operator](https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/__init__.py#L67)
* [Read the events for each ticket by attaching transformer to resource explicitly](https://github.com/dlt-hub/verified-sources/blob/master/sources/hubspot/__init__.py#L125)
* [Set `tags` column data type to complex to load them as JSON/struct](https://github.com/dlt-hub/verified-sources/blob/master/sources/zendesk/__init__.py#L108)
* Typical use of `merge` with incremental load for endpoints returning a list of updates to entities in [Shopify source](https://github.com/dlt-hub/verified-sources/blob/master/sources/shopify_dlt/__init__.py#L36).
* A `dlt` mega-combo in `pipedrive` source, where the deals from `deal` endpoint are [fed into](https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/__init__.py#L113) `deals_flow` resource to obtain events for a particular deal. [Both resources use `merge` write disposition and incremental load to get just the newest updates](https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/__init__.py#L103). [The `deals_flow` is dispatching different event types to separate tables with `dlt.mark.with_table_name`](https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/__init__.py#L135).
* An example of using JSONPath expression to get cursor value for incremental loading. In pipedrive some objects have `timestamp` property and others `update_time`. [The dlt.sources.incremental('update_time|modified') expression lets you bind the incremental to either](https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/recents.py#L39).
* An example of using JSONPath expression to get cursor value for incremental loading. In pipedrive some objects have `timestamp` property and others `update_time`. [The dlt.sources.incremental('update_time|modified') expression lets you bind the incremental to either](https://github.com/dlt-hub/verified-sources/blob/master/sources/pipedrive/__init__.py#L176).
* If your source/resource needs google credentials, just use `dlt` built-in credentials as we do in [google sheets](https://github.com/dlt-hub/verified-sources/blob/master/sources/google_sheets/__init__.py#L26) and [google analytics](https://github.com/dlt-hub/verified-sources/blob/master/sources/google_analytics/__init__.py#L32). Also note how `credentials.to_native_credentials()` is used to initialize google api client.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[google sheets]https://github.com/dlt-hub/verified-sources/blob/master/sources/google_sheets/__init__.py#L26
to
[google sheets]https://github.com/dlt-hub/verified-sources/blob/master/sources/google_sheets/__init__.py#L27

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[google analytics](https://github.com/dlt-hub/verified-sources/blob/master/sources/google_analytics/__init__.py#L32
to
[google analytics](https://github.com/dlt-hub/verified-sources/blob/master/sources/google_analytics/__init__.py#L28

AstrakhantsevaAA
AstrakhantsevaAA reviewed Jul 20, 2023
View reviewed changes
Copy link
Collaborator

@AstrakhantsevaAA AstrakhantsevaAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this link is not valid as well
https://github.com/dlt-hub/verified-sources/blob/master/docs/BUILDING-BLOCKS.md?plain=1#L12C121-L12C266
use
https://dlthub.com/docs/general-usage/customising-pipelines/renaming_columns

@willi-mueller willi-mueller changed the title fixes docs links to code examples docs: fixes links to code examples Jul 22, 2023
@AstrakhantsevaAA AstrakhantsevaAA merged commit 95b8a58 into dlt-hub:master Jul 23, 2023
4 of 8 checks passed
@willi-mueller willi-mueller deleted the docs/fixes-links-in-building-blocks branch August 2, 2023 09:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AstrakhantsevaAA AstrakhantsevaAA AstrakhantsevaAA approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@willi-mueller @AstrakhantsevaAA