Skip to content

Commit

Permalink
Merge pull request #74 from RMI-PACTA/ci/simplify-pull-only-deploy
Browse files Browse the repository at this point in the history
Ci/simplify pull only deploy
  • Loading branch information
AlexAxthelm authored May 23, 2024
2 parents 05a5712 + 33bf9d1 commit eec61f9
Show file tree
Hide file tree
Showing 6 changed files with 510 additions and 202 deletions.
2 changes: 1 addition & 1 deletion DESCRIPTION
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Package: workflow.factset
Title: Extract Financial Data for use in PACTA
Version: 1.0.0.9002
Version: 1.0.0.9003
Authors@R:
c(person(given = "CJ",
family = "Yetman",
Expand Down
45 changes: 31 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,24 @@ This repo contains the `workflow.factset` R package, a Dockerfile to build an im

**QUICKSTART**: See ["Deploy"](#Deploy), below.

To deploy `main` branch **with a databse and running FactSet Data Loader**:

```sh
az deployment group create \
--resource-group "RMI-SP-PACTA-DEV" \
--template-file azure-deploy.with-db.json \
--parameters azure-deploy.with-db.rmi-pacta.parameters.json
```

To deploy `main` branch **Extracting PACTA files from an existing `FDS` database**:

```sh
az deployment group create \
--resource-group "RMI-SP-PACTA-DEV" \
--template-file azure-deploy.extract-only.json \
--parameters azure-deploy.extract-only.rmi-pacta.parameters.json
```

## `workflow.factset` R package

The `workflow.factset` package's purpose is to extract data from a database prepared by the FactSet DataFeed Loader application.
Expand Down Expand Up @@ -75,7 +93,7 @@ The `main` tag should be used, rather than `latest`.

## Azure ARM template

`azure-deply.json` is an [Azure ARM Template](https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/overview) which by default deploys:
`azure-deploy.json` is an [Azure ARM Template](https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/overview) which by default deploys:

* Azure Flexible Server for PostgreSQL
* `fds` database on that server
Expand Down Expand Up @@ -148,23 +166,22 @@ A parameter file with the values that the RMI-PACTA team uses for extracting dat
# change this value as needed.
RESOURCEGROUP="RMI-SP-PACTA-DEV"

# Users with access to the RMI-PACTA Azure subscription can run:
az deployment group create --resource-group "$RESOURCEGROUP" --template-file azure-deploy.json --parameters azure-deploy.rmi-pacta.parameters.json

# To deploy with DB and FDSLoader:
az deployment group create \
--resource-group "$RESOURCEGROUP" \
--template-file azure-deploy.with-db.json \
--parameters azure-deploy.with-db.rmi-pacta.parameters.json

# To deploy without DB and FDSLoader:
az deployment group create \
--resource-group "$RESOURCEGROUP" \
--template-file azure-deploy.extract-only.json \
--parameters azure-deploy.extract-only.rmi-pacta.parameters.json
```

For security, the RMI-PACTA parameters file makes heavy use of extracting secrets from an Azure Key vault, but an example file that passes parameters "in the clear" is available as [`azure-deploy.example.parameters.json`](azure-deploy.example.parameters.json)

Non RMI-PACTA users can define their own parameters and invoke the ARM Template with:

```sh
# Otherwise:
# Prompts for parameters without defaults
az deployment group create --resource-group "$RESOURCEGROUP" --template-file azure-deploy.json

# if you have created your own parameters file:
az deployment group create --resource-group "$RESOURCEGROUP" --template-file azure-deploy.json --parameters @azure-deploy.parameters.json
```
Non RMI-PACTA users can define their own parameters (examples in the `azure-deploy.example.parameters.json` file) and invoke the appropriate ARM Template

## Local Development

Expand Down
240 changes: 240 additions & 0 deletions azure-deploy.extract-only.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,240 @@
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "0.0.0.5",

"parameters": {

"containerGroupName": {
"type": "string",
"metadata": {
"description": "The name of the container group."
}
},

"dataTimestamp": {
"type": "string",
"metadata": {
"description": "The time to start the container group."
}
},
"issReportingYear": {
"type": "string",
"metadata": {
"description": "Reporting year to use for ISS data."
}
},

"identity": {
"type": "string",
"metadata": {
"description": "The ID of the user assigned identity to use for the container group."
}
},
"location": {
"type": "string",
"defaultValue": "[resourceGroup().location]",
"metadata": {
"description": "Location for all resources."
}
},
"restartPolicy": {
"type": "string",
"defaultValue": "Never",
"allowedValues": [
"Always",
"Never",
"OnFailure"
],
"metadata": {
"description": "The behavior of Azure runtime if container has stopped."
}
},
"starttime": {
"type": "string",
"defaultValue": "[utcNow()]",
"metadata": {
"description": "The time this template is deployed."
}
},

"PGHOST": {
"type": "string",
"metadata": {
"description": "Explicitly define a (full URL) hostname for PGHOST."
}
},
"PGPASSWORD": {
"type": "securestring",
"metadata": {
"description": "password to connect to database"
}
},

"imageTagWorkflow": {
"type": "string",
"defaultValue": "main",
"metadata": {
"description": "Image tag for the workflow container."
}
},

"logWorkspaceId": {
"type": "string",
"metadata": {
"description": "The ID for a Log Analytics Workspace."
}
},
"logWorkspaceKey": {
"type": "securestring",
"metadata": {
"description": "The key for a Log Analytics Workspace."
}
},

"storageAccountKeyExtracted": {
"type": "securestring",
"metadata": {
"description": "The storage account key for the storage account for extracted files."
}
},
"storageAccountNameExtracted": {
"type": "string",
"metadata": {
"description": "The storage account name for the storage account for extracted files."
}
},
"storageAccountShareExtracted": {
"type": "string",
"metadata": {
"description": "The file share name for the extracted files."
}
}

},

"variables": {

"PGDATABASE": "FDS",
"PGUSER": "postgres",
"containerregistry": "ghcr.io/rmi-pacta",
"machineCpuCoresLimitWorkflow": 2,
"machineCpuCoresRequestWorkflow": 2,
"machineMemoryInGBLimitWorkflow": 16,
"machineMemoryInGBRequestWorkflow": 16,
"mountPathExport": "/mnt/factset-extracted"

},

"functions": [],

"resources": [

{
"type": "Microsoft.ContainerInstance/containerGroups",
"apiVersion": "2021-09-01",
"name": "[parameters('containerGroupName')]",
"location": "[parameters('location')]",
"identity": {
"type": "UserAssigned",
"userAssignedIdentities": {
"[parameters('identity')]": {}
}
},
"properties": {
"diagnostics": {
"logAnalytics": {
"logType": "ContainerInstanceLogs",
"workspaceId": "[parameters('logWorkspaceId')]",
"workspaceKey": "[parameters('logWorkspaceKey')]"
}
},
"containers": [
{
"name": "workflow-factset",
"properties": {
"image": "[concat(variables('containerregistry'),'/workflow.factset:', parameters('imageTagWorkflow'))]",
"ports": [],
"resources": {
"limits": {
"cpu": "[variables('machineCpuCoresLimitWorkflow')]",
"memoryInGB": "[variables('machineMemoryInGBLimitWorkflow')]"
},
"requests": {
"cpu": "[variables('machineCpuCoresRequestWorkflow')]",
"memoryInGB": "[variables('machineMemoryInGBRequestWorkflow')]"
}
},
"environmentVariables": [
{
"name": "PGUSER",
"value": "[variables('PGUSER')]"
},
{
"name": "PGPASSWORD",
"secureValue": "[parameters('PGPASSWORD')]"
},
{
"name": "PGHOST",
"value": "[parameters('PGHOST')]"
},
{
"name": "PGDATABASE",
"value": "[variables('PGDATABASE')]"
},
{
"name": "DEPLOY_START_TIME",
"value": "[parameters('starttime')]"
},
{
"name": "MACHINE_CORES",
"value": "[variables('machineCpuCoresRequestWorkflow')]"
},
{
"name": "LOG_LEVEL",
"value": "TRACE"
},
{
"name": "EXPORT_DESTINATION",
"value": "[variables('mountPathExport')]"
},
{
"name": "DATA_TIMESTAMP",
"value": "[parameters('dataTimestamp')]"
},
{
"name": "ISS_REPORTING_YEAR",
"value": "[parameters('issReportingYear')]"
},
{
"name": "UPDATE_DB",
"value": "FALSE"
}
],
"volumeMounts": [
{
"name": "factset-extracted",
"mountPath": "[variables('mountPathExport')]"
}
]
}
}
],
"restartPolicy": "[parameters('restartPolicy')]",
"osType": "Linux",
"volumes": [
{
"name": "factset-extracted",
"azureFile": {
"shareName": "[parameters('storageAccountShareExtracted')]",
"readOnly": false,
"storageAccountName": "[parameters('storageAccountNameExtracted')]",
"storageAccountKey": "[parameters('storageAccountKeyExtracted')]"
}
}
]
}
}

],
"outputs": {}
}
62 changes: 62 additions & 0 deletions azure-deploy.extract-only.rmi-pacta.parameters.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {

"imageTagWorkflow": {
"value": "pr-73"
},
"dataTimestamp": {
"value": "20231231"
},
"issReportingYear": {
"value": "2022"
},

"identity": {
"value": "/subscriptions/feef729b-4584-44af-a0f9-4827075512f9/resourceGroups/RMI-SP-PACTA-DEV/providers/Microsoft.ManagedIdentity/userAssignedIdentities/pacta-runner-dev"
},

"storageAccountKeyExtracted": {
"reference": {
"keyVault": {
"id": "/subscriptions/feef729b-4584-44af-a0f9-4827075512f9/resourceGroups/RMI-SP-PACTA-DEV/providers/Microsoft.KeyVault/vaults/pacta-vault-dev"
},
"secretName": "rawdata-storageaccountkey"
}
},
"storageAccountNameExtracted": {
"value": "pactarawdata"
},
"storageAccountShareExtracted": {
"value": "factset-extracted"
},

"logWorkspaceId": {
"reference": {
"keyVault": {
"id": "/subscriptions/feef729b-4584-44af-a0f9-4827075512f9/resourceGroups/RMI-SP-PACTA-DEV/providers/Microsoft.KeyVault/vaults/pacta-vault-dev"
},
"secretName": "LogWorkspaceID-Dev"
}
},
"logWorkspaceKey": {
"reference": {
"keyVault": {
"id": "/subscriptions/feef729b-4584-44af-a0f9-4827075512f9/resourceGroups/RMI-SP-PACTA-DEV/providers/Microsoft.KeyVault/vaults/pacta-vault-dev"
},
"secretName": "LogWorkspaceKey-Dev"
}
},

"PGPASSWORD": {
"reference": {
"keyVault": {
"id": "/subscriptions/feef729b-4584-44af-a0f9-4827075512f9/resourceGroups/RMI-SP-PACTA-DEV/providers/Microsoft.KeyVault/vaults/pacta-vault-dev"
},
"secretName": "factset-database-password"
}
}

}
}
Loading

0 comments on commit eec61f9

Please sign in to comment.