FinOps Data Product Requirements

Background

This documents serves as a high-level technical specification describing the requirements from the FinOps perspective for a data product solution.

The requirement for a data product solution is mainly based on the need of solving the following problems:

Cloud Wastage: Determine whether VMs are wasting cloud resources by analyzing CPU/Memory utilization metrics.
Advisor Recommendations: Benefit Azure Advisor Recommendations to compute potential savings by enabling the recommended best practices.

The data needed to address the problems above can be retrieved through the Azure Management API using the following endpoints:

Need: The FinOps team needs read access to the VM usage metrics and Advisor Recommendations at root management group level to gain insights into Cloud Wastage and Advisor recommendations across the entire organisation.

Restrictions

The endpoints described above require the requesting user/service princpal to be authorized with the appropriate Azure RBAC permissions at the root management group level. Sufficient permissions are included in the built-in roles Reader and Monitoring Reader, but these are rather privileged roles. An alternative to granting our product principal a privileged role at root management group level is for us to access the appropriate data through a data product.

Data needs

Below is a summary of a subset of data required by the FinOps team (based on the data made available through the discussed endpoints in the Azure Management API).

VM Usage Metrics

The following endpoint can be used to retrieve VM usage metrics data (based on Microsoft's documentation:

GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/virtualMachines/{vmname}/providers/microsoft.insights/metrics?api-version={api-version}&metricnames={commaSeparatedMetrcs}&timespan={fromDate}/{toDate}&interval={interval}&aggregation={aggregation}

The following query parameters would be needed to provide us with an appropriate result:

Parameter	Value	Note
metricnames	Percentage CPU	The following link provides an extensive list of available metrics for the Microsoft.Compute/virtualMachines resource type.
interval	PT1H	Get CPU percentage per hour
aggregation	average
timespan	e.g 2024-01-10T00:00:00Z/2024-01-11T00:00:00Z	Define a 24-hour interval, going from midnight to midnight

Using the query above would result in a response on the following format:

{
    "value": [
        {
            "timeseries": [
                {
                    "metadatavalues": [],
                    "data": [
                        {
                            "timeStamp": "2024-01-10T00:00:00Z",
                            "average": 0.44
                        },
                        {
                            "timeStamp": "2024-01-10T01:0:00Z",
                            "average": 0.31
                        },
                        {
                            "timeStamp": "2024-01-10T02:00:00Z",
                            "average": 0.29
                        },
                        {
                            "timeStamp": "2024-01-10T03:00:00Z",
                            "average": 0.29
                        },
                        {
                            "timeStamp": "2024-01-10T04:00:00Z",
                            "average": 0.285
                        } ]
                } ]
        } ]
}

The results from the VM usage metric requests would need to be stored in a table structure with the following format:

Column Name	Column Type
ResourceId	string
SubscriptionId	guid
ResourceGroupName	string
VirtualMachineName	string
Timestamp	datetime
CPUPercentage	float

Advisor Recommendations

The following endpoint can be used to retrieve Azure Advisor Recommendations_

GET https://management.azure.com/subscriptions/{subscriptionId}/providers/Microsoft.Advisor/recommendations?api-version={api-version}&$filter={filter-expression}

The following query parameters would be needed to provide us with an appropriate result:

Parameter	Value	Note
$filter	Category eq 'Cost'	Filter by category as we are only interested in cost recommendations

Using the query above would result in a response on the following format:

[
  {
    "value": [
      {
        "properties": {
          "category": "Cost",
          "impact": "High",
          "impactedField": "Microsoft.Subscriptions/subscriptions",
          "impactedValue": "1da97f87-c3d9-46fe-96b0-19bf3d312717",
          "lastUpdated": "2024-01-18T10:19:20.2721064Z",
          "recommendationTypeId": "89515250-1243-43d1-b4e7-f9437cedffd8",
          "shortDescription": {
            "problem": "Purchasing a savings plan for compute could unlock lower prices",
            "solution": "Purchasing a savings plan for compute could unlock lower prices"
          },
          "extendedProperties": {
            "subId": "1da97f87-c3d9-46fe-96b0-19bf3d312717",
            "sku": "Compute_Savings_Plan",
            "scope": "Single",
            "commitment": "0.653",
            "annualSavingsAmount": "2728",
            "savingsAmount": "227",
            "savingsCurrency": "NOK",
            "term": "P3Y",
            "lookbackPeriod": "30"
          },
          "resourceMetadata": {
            "resourceId": "/subscriptions/1da97f87-c3d9-46fe-96b0-19bf3d312717"
          }
        },
        "id": "/subscriptions/1da97f87-c3d9-46fe-96b0-19bf3d312717/providers/Microsoft.Advisor/recommendations/5456041a-21ba-48cb-83c3-f0e882a50ea9",
        "type": "Microsoft.Advisor/recommendations",
        "name": "5456041a-21ba-48cb-83c3-f0e882a50ea9"
      },
    ]
  }
]

The results from the VM usage metric requests would need to be stored in a table structure with the following format:

Column Name	Column Type
SubscriptionId	guid
Impact	string or enum(High, Medium, Low)
Date	datetime
RecommendationTypeId	guid
Problem	string
Solution	string
SKU	string
Scope	string
Commitment	float
AnnualSavingsAmount	float
SavingsAmount	float
Term	string or enum(P1Y, P3Y)
LookbackPeriod	int
Quantity	int
ReservedResourceType	string
Region	string
RecommendationIdString	string
RecommendationId	guid

Data Product Requirements

There are several ways the data above could be made accessible to the FinOps teams, but we will discuss two proposals for possible data product implementations: Blob Storage and Web API.

Proposal 1 - Blob Storage

Both Cloud Wastage and Advisor Recommendations data could be made available as blob storage in an Azure Storage account. It is two ways this could be implemented, and that is through either an export by the data product provider to the FinOps Storage account, or that the FinOps team is provided access to an external storage account for on-demand data retrieval.

Proposal 1.1 - Export to FinOps Storage Account

The high-level solution flow for export implementation would be:

Data product provider retrieve Cloud Wastage/Advisor Recommendations from Azure
Transform raw data to format required by the FinOps team
Push transformed data to FinOps storage account

Implementing the data product as an export feature would have following prerequisites/requirements:

Data product provider must retrieve Cloud Wastage/Advisor Recommendation data on a daily basis from Azure.
Data product provider must be granted write access to FinOps storage account in order to push data.

Proposal 1.2 - Import from external Storage Account

The high-level solution flow for import implementation would be:

Data product provider retrieve Cloud Wastage/Advisor Recommendations from Azure
Transform raw data to format required by the FinOps team
Persist transformed data in storage account administrated by the data product provider

Implementing the data product as an import feature would have following prerequisites/requirements:

Data product provider must retrieve Cloud Wastage/Advisor Recommendation data on a daily basis from Azure.
Data product provider must persist the Cloud Wastage data in an appropriate structure, e.g partitioned by year/month/day such that historic data can be pulled.
Data product provider must determine a retention period on Cloud Wastage/Advisor Recommendation data. Have far back in time should historic data be persisted.
Data product provider must grant FinOps team read access to storage account, such that data can be pulled on-demand

Proposal 2 - Web API

The Cloud Wastage/Advisor Recommendation data could also be made available through a Web API, such that the FinOps team can consume data on-demand. The data product implementation would then provide a set of endpoints that are similar to those that are accessible through the Azure Management API. E.g the Data product endpoints would be:

VM CPU utilization endpoint could look like

GET https://product-provider-base-url/vm-cpu-utilization?fromTimestamp={fromTimestamp}&toTimestamp={toTimestamp}

where metrics can be retrieved by a determined timestamp interval as indicated by the proposed query parameters. The above query would produce the following repsonse

{
  "values": [
    {
      "resourceId": "string",
      "subscriptionId": "guid",
      "resourceGroupName": "string",
      "virtualMachineName": "string",
      "timestamp": "datetime",
      "cpuPercentage": "float",
    }
  ]
}

Query parameters to scope VM metrics down to subscription, resource group or virtual machine level could also be included.

Advisor Recommendation endpoint could look like

GET https://product-provider-base-url/advisor-recommendations

We are not interested in looking into historic advisor recommendations, so we won't need to specify a timestamp interval for this endpoint. We are only interested in the current recommendations. The above query would produce the following repsonse

{
  "values": [
    {
      "subscriptionId": "guid",
      "Impact ": "string or enum(High, Medium, Low)",
      "Date ": "datetime",
      "RecommendationTypeId": "guid",
      "Problem": "string",
      "Solution": "string",
      "SKU": "string",
      "Scope": "string",
      "Commitment": "float",
      "AnnualSavingsAmount": "float",
      "Term": "string or enum(P1Y, P3Y)",
      "LookbackPeriod": "int",
      "Quantity": "int",
      "ReservedResourceType": "string",
      "Region": "string",
      "RecommendationIdString": "string",
      "RecommendationId": "guid",
    }
  ]
}

Query parameters to scope Advisor recommendations down to the subscription level could also be included.

Implementing the data product as a Web API would have following prerequisites/requirements:

Data product provider must retrieve Cloud Wastage/Advisor Recommendation data on a daily basis from Azure.
Data product provider must implement Azure Entra ID authentication and grant access to FinOps team
Data product provider must persist the Cloud Wastage data in an appropriate structure, e.g partitioned by year/month/day such that historic data can be pulled.
Data product provider must determine a retention period on Cloud Wastage/Advisor Recommendation data. Have far back in time should historic data be persisted.

Summary

	Blob Export	Blob Import	Web API
Data product provider needs write access to FinOps SA	✔	-	-
FinOps needs read access to Data product	-	✔	✔
Scheduled push from Data product provider	✔	-	-
On-demand pull by FinOps	-	✔	✔
Data product must persist historic records	-	✔	✔

Provide feedback

Saved searches

Use saved searches to filter your results more quickly