Fabric CI CD

• Using GitHub integration to employ source control for Fabric workspaces
• Using a single repository for content in all workspaces
• Each domains contains two environments (workspaces): dev and prod

Not supported

The following are not supported in deployment of workspace

• Task flow
• Schedules: creating schedules using rest api with service principal authentication is not yet supported (Create schedules with Fabric REST API)
• Lakehouse content
• Shortcuts
• Deleted items in source will not be deleted in target on deployment

Schedules

Workspace	Item type	Item name	Repeat	Time
reservations	pipeline	reservations-orchestrator	daily	02:00
WBS	pipeline	wbs-ingestion	daily	01:00
subscriptions	pipeline	subscriptions-ingestion	daily	01:30
applications	pipeline	application-ingestion	daily	01:00
pricesheet	pipeline	pricesheet-ingestion	daily	01:00
forex	pipeline	forex-ingestion	daily	02:00
cost	pipeline	hub-deployments	daily	02:00
cost	pipeline	services-ingestion	monthly	01:00, weekly on mondays
cost	pipeline	daily-actual-cost-ingestion	daily	09:00
cost	pipeline	daily-amortized-cost-ingestion	daily	09:00
cost	pipeline	monthly-actual-cost-ingestion	monthly	09:00, 6th of every month
cost	pipeline	monthly-amortized-cost-ingestion	monthly	09:00, 6th of every month
internal	semantic model	finops-internal-report-model	daily	10:30
reports	semantic model	finops-reports-model	daily	10:00
cost	pipeline	acdc-ingestion	weekly	23:00, weekly on sundays

Shortcuts

Workspace	Lakehouse	Shortcut Type	Source Workspace/SA	Lakehouse/Container	Reference	Shortcut Name
wbs	wbs_silver	table	FinOps Test	WBSLakehouse	SAPEmployees	---
wbs	wbs_silver	table	FinOps Test	WBSLakehouse	SAPTasks	---
subscriptions	subscriptions_silver	table	eip-control-room-azure-dev	Azure_Silver	resourceGroups	azure_resource_groups
subscriptions	subscriptions_silver	table	eip-control-room-azure-dev	Azure_Silver	subscriptionsTags	azure_subscription_tags
subscriptions	subscriptions_silver	table	eip-control-room-azure-dev	Azure_Silver	subscriptions	azure_subscriptions
subscriptions	subscriptions_silver	table	eip-control-room-azure-dev	Azure_Silver	tags	azure_tags
subscriptions	subscriptions_silver	table	wbs	wbs_gold	wbs_earliest	---
subscriptions	subscriptions_silver	files	wbs	wbs_gold	wbs	---
subscriptions	subscriptions_silver	files	applications	applications_silver	applications	---
cost	cost_silver	table	applications	application_silver	applications_earliest	---
cost	cost_silver	table	wbs	wbs_gold	azure_resource_groups_earliest	---
cost	cost_silver	table	pricesheet	pricesheet_silver	pricesheet_earliest	---
cost	cost_silver	table	subscriptions	subscriptions_gold	subscriptions_earliest	---
cost	cost_silver	table	wbs	wbs_gold	wbs_earliest	---
cost	cost_silver	files	applications	application_silver	applications	---
cost	cost_silver	table	pricesheet	pricesheet_silver	pricesheet	---
cost	cost_silver	table	subscription	subscriptions_gold	subscriptions	---
cost	cost_silver	table	wbs	wbs_gold	wbs	---
cost	hub_shortcuts	adlsv2	hub-automation	sql-activity-v2	sql-activity-v2	---
cost	hub_shortcuts	adlsv2	hub-automation	sql-config	sql-config	---
cost	hub_shortcuts	adlsv2	hub-automation	sql-failed	sql-failed	---
cost	hub_shortcuts	adlsv2	hub-automation	sql-hub-logs-v2	sql-hub-logs-v2	---
cost	hub_shortcuts	adlsv2	hub-automation	win-activity-v2	win-activity-v2	---
cost	hub_shortcuts	adlsv2	hub-automation	win-config	win-config	---
cost	hub_shortcuts	adlsv2	hub-automation	win-failed	win-failed	---
cost	hub_shortcuts	adlsv2	hub-automation	win-hub-logs-v2	win-hub-logs-v2	---
internal	source	table	cost	cost_platinum	amortized_3y	---
internal	source	table	cost	cost_platinum	benefit_purchases	---
internal	source	table	cost	cost_platinum	hub_deployments	---
internal	source	table	cost	cost_platinum	rouge_hub_deployments	---
internal	source	table	cost	reservations_gold	vm_recommendations	---
reports	source	table	cost	cost_platinum	amortized_1y	---
reports	source	table	cost	cost_platinum	amortized_3y	---
reports	source	table	applications	applications_silver	applications	---
reports	source	table	cost	cost_silver	regions	---
reports	source	table	subscriptions	subscriptions_gold	subscriptions	---
reports	source	table	wbs	wbs_gold	wbs	---
reports	source	table	cost	cost_silver	acdc_usage	---

Reservations

• Add deployment rules to notebook - change default to prod lakehouse
• Run reservation-orchestrator pipeline to create delta tables
• Setup schedule for reservation-orchestrator
• Setup dedicated pipeline to backfill lakehouse with historic files?
• Create task flow?
• Create permissions manually / automate in pipeline?

WBS

• Add deployment rules to notebook - change default to prod lakehouse
• Manually add SAP table shortcuts in silver lakehouse
• Run wbs-ingestion pipeline to create delta tables
• Setup schedule for wbs-ingestion
• Pull historic files from dev
• Manually create wbs_earliest table
• Grant SP read permission in gold lakehouse. Only users with either Admin, Member, or Contributor roles have access to read lakehouse data. OneLake data access can also be applied on top to provide more granular access on folder/table level (OneLake access docs)

Subscriptions

• Add deployment rules to notebook - change default to prod lakehouse
• Setup schedule for subscriptions-ingestion
• Manually add shortcuts
• Create azure_resource_groups_earliest delta table manually in subscriptions_gold lakehouse
• Create subscriptions_earliest delta table manually in subscriptions_gold lakehouse
• Run subscription-ingestion pipeline to create delta tables
• Copy historic files in bronze in dev to prod
• Run batch-validate-subscriptions-yearly
• Run batch-enrich-subscriptions-yearly

Applications

• Add deployment rules to notebook - change default to prod lakehouse
• Setup schedule for applications-ingestion
• Run subscription-ingestion pipeline to create delta tables
• Copy historic files in bronze from dev to prod
• Run batch-validate-applications-yearly to create historic files

Pricesheet

• Add deployment rules to notebook - change default to prod lakehouse
• Set schedule for pricesheet-ingestion
• Run pricesheet-ingestion to create delta tables
• Copy historic bronze files from dev to prod
• Copy historic silver files from dev to prod
• Manually create pricesheet_earliest in silver lakehouse

Forex

• Add deployment rules to notebook - change default to prod lakehouse
• Set schedule for forex-ingestion
• Run forex-ingestion
• Copy bronze/silver files from dev to prod

Cost

• Add deployment rules to notebook - change default to prod lakehouse
• Run service-ingestion once
• Run region-ingestion once
• Manually add shortcuts
• Copy bronze/silver/gold data from dev to prod
• Run daily-amortized-cost-ingestion once to initially create delta tables
• Run daily-actual-cost-ingestion once to initially create delta tables
• Run hub-deployments once to initially create delta tables
• Setup schedules

Reports

• Add deployment rules to semantic model - change warehouse connection string to prod
• Add shortcuts to lakehouse
• Refresh semantic model
• Setup scheduled refresh for semantic model

Internal

• Add deployment rules to semantic model - change warehouse connection string to prod
• Add shortcuts to lakehouse
• Refresh semantic model
• Setup scheduled refresh for semantic model