Traders

• A modern data warehouse derived from a web-platform database.
• This tutorial project implemented locally using WSL ubuntu and postgresql as a dev-layer.
• The postgresql database "Traders" holds several schemas layers 'dataset' for the source database and for the target data warehouse.
• Some small tables from the source database have been replaced with seed constant data and others have been denormalized.
• Pre-commit sqlfluff rules implemented to align sql with postgres standard syntax.

Traders Data warehouse layers

1. Source [database] tables.
2. Staging layer contains dbt denormalized view models 'stg_'
3. Vault layer where data vault modelling is implemented'vlt_'
4. Semantic layer where the fact and dimension tables creating star schema models 'sem_'

Additional notes:

• When working with big data, it's crucial to utilize distributed computing storage with columnar reading capabilities, such as BigQuery or Synapse, to leverage partitioning for enhanced performance, reduced query latency, and cost optimization.

Advantages of Data Warehouse:

• Scalability: They can handle massive amounts of data.
• Performance: Fast BI reports and complex analysis.
• Consistency: Ensures data consistency across the organization.
• Flexibility: Data warehouses can be adapted to changing business needs.

Useful commands:

poetry shell 

poetry run pre-commit run sqlfluff-fix --files /RELATIVE_PATH