Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace the data integration teams with a process. #241

Open
esbran opened this issue Nov 29, 2021 · 0 comments
Open

Replace the data integration teams with a process. #241

esbran opened this issue Nov 29, 2021 · 0 comments
Assignees
Labels
backlog Backlog item documentation Improvements or additions to documentation
Milestone

Comments

@esbran
Copy link
Contributor

esbran commented Nov 29, 2021

What if we replace the data integration teams with a process.

  1. Search for existing data sources of same or similar use/type.
    1a) If an ingest already exists from the intended source, consider if the current pipeline can be reused or extended.
    1b) Establish a contract for multiple data products using the same pipeline, define dependencies, refresh requirements, quality requirements, classification.
  2. If a pipeline does not exist or it is considered to be substantially different from existing pipelines the cross functional team creates a new data integration product.
    2a) A critical consideration when creating new connections to source systems is handling the load on that system and ensuring that the additional load of the data egress is not affecting performance.
    2b) A design consideration is also that multiple pipelines increases the complexity for the overall data extraction from
    a source system.
  3. The pipelines must be monitored and data must be classified. Data policies can be applied to detect and act based on classification. Corporate regulations can be applied automatically or advised depending of criticality.

Without the data integration teams you remove a dependency on external teams and allow your cross functional team to be more self serviced. The cross functional teams has the SME's for the source systems. This simplifies enabling access and finding the right source data required to deliver the use-case.

Potential breaking points are the need for the teams to honor the source system limitations. This is only a potential problem if the cross functional team does not have representatives from the owners of the source system.

Originally posted by @esbran in #204 (comment)

@marvinbuss marvinbuss added documentation Improvements or additions to documentation backlog Backlog item labels Nov 29, 2021
@marvinbuss marvinbuss linked a pull request Nov 29, 2021 that will close this issue
@marvinbuss marvinbuss added this to the v1.3.0 milestone Dec 6, 2021
@marvinbuss marvinbuss removed a link to a pull request Dec 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backlog Backlog item documentation Improvements or additions to documentation
Projects
None yet
Development

No branches or pull requests

2 participants