Handles the migration of data related to prisoners in NOMIS to the new services
Currently one service is supported: Visit a Prisoner Service
The purpose of this service to bulk update a new service with data from NOMIS. This will handle the reading of data, possibly transformation of that data and pushing data to the new service. Since migration is a typically a process that takes many hours, this service relies heavily on SQS messaging to guarantee that the migration is completed, where any transient errors are automatically retried. Any persistent errors can be inspected and are retained in a SQS dead letter queue.
For running locally against docker instances of the following services:
- hmpps-auth
- hmpps-nomis-prisoner-api
- visit-scheduler
- hmpps-nomis-visits-mapping-service
- localstack
- run this application independently e.g. in IntelliJ
docker compose up --scale hmpps-prisoner-from-nomis-migration=0
or
docker-compose up --scale hmpps-prisoner-from-nomis-migration=0
Running all services including this service
docker compose up
or
docker-compose up
Though it is possible to run this service against both the test services deployed in the cloud and the SQS queue in AWS this is not recommended while the deployed version of this service is running in the Cloud Platform since there is no guarantee this local instance will read an incoming HMPPS domain event since other pods are also reading from the same queue.
However, if you do wish to do that the steps would be:
- Set all environment variables to match those in values-dev.yaml e.g.
API_BASE_URL_OAUTH=https://sign-in-dev.hmpps.service.justice.gov.uk/auth
- Set additional environment variables for personal client credentials you have that have the correct roles required to access the remotes services, the env names can be found in values.yaml
- Set additional environment variables for the SQS Queue secrets that can be found in the
hmpps-prisoner-from-nomis-migration-dev
namespace, again the env names can be found in values.yaml
A better hybrid solution which gives better control messaging would be similar to above but using the dev
profile and therefore localstack.
The first 2 of the 3 steps is required but instead of step 3
docker-compose up localstack migration-db
ordocker compose up localstack migration-db
(there is also docker-compose-local.yaml with just localstack and postgres defined )
Then run any of the bash
scripts at the root of this project to send events to the local topic
There a circumstances where you want to run this service end to end but without the consuming service being available, for example the consuming service has not be written yet. To emulate the publishing service we may provide a mock, for instance MockVisitsResource which consumes migrated data. Details of the configuration follows:
API_BASE_URL_VISITS=https://prisoner-to-nomis-update-dev.hmpps.service.justice.gov.uk
Since this services uses the HMPPS SQS library with defaults this has all the default endpoints for queue maintenance as documented in the SQS library.
For purging queues the queue name can be found in the health check and the required role is the default ROLE_QUEUE_ADMIN
.
With the kubernetes pods scaled to around 12, around 200,000
visits can be migrated per hour.
A migration can be started by calling the migration end point using a HMPPS Auth token with the role MIGRATE_VISITS
POST /migrate/visits
curl --location --request POST 'https://prisoner-nomis-migration.hmpps.service.justice.gov.uk/migrate/visits' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <token with MIGRATE_VISITS>' \
--data-raw '{
"prisonIds": ["HEI"],
"visitTypes": ["SCON"]
}'
POST body to optionally contain:
prisonIds
- a list of prison ids to migratevisitTypes
- a list of visit types to migrate. It will default toSCON
if not provided.fromDateTime
- only include visits created after this date. NB this is creation date not the actual visit date.endDateTime
- only include visits created before this date. NB this is creation date not the actual visit date.ignoreMissingRoom
- When true exclude visits without an associated room (visits created during the VSIP synchronisation process), defaults to false. Only required during testing when mapping records are manually deleted.
The from
and end
times are primarily used to ensure that only visits created after the last migration are migrated. This is a performance optimisation
since visits will never be migrated twice. Once it is marked as migrated it can only be migrated again by resetting the visit mapping table.
The response from this call will be an information JSON structure which echos the filter used but also includes:
migrationId
- the id of the migration that groups Application Insights events and is also written to the visit mapping tableestimatedCount
- the estimated number of visits that will be migrated. Since the last page of visits may include new visits created in NOMIS while the migration is running this number might change.
Given the health check page reports the number of messages in the main queue and DLQ, it is possible to see at a glance how far the migration has progressed from the /health
endpoint.
customEvents
| where cloud_RoleName == 'hmpps-prisoner-from-nomis-migration'
| where name startswith "nomis-migration"
| summarize count() by name
will show all significant visit migration events
nomis-migration-visits-started
- the single event for a migration which will contain the estimate count and the migration idnomis-migration-visit-migrated
- the event for each visit migrated. It will contain the visit ids and basic information about the visitnomis-migration-visit-mapping-failed
- indicates the visit was migrated but the mapping record could not be created. These events require manual intervention; see below.nomis-migration-visit-no-room-mapping
- indicates a visit was not migrated because the room it was in could not be mapped. These events require room mapping change and a rerun of the migration.
nomis-migration-visit-mapping-failed
requires the 2 IDs are extracted from application insights and manually added to the mapping table via the API
customEvents
| where cloud_RoleName == 'hmpps-prisoner-from-nomis-migration'
| where name == 'nomis-migration-visit-mapping-failed'
| extend migrationId_ = tostring(customDimensions.migrationId)
| extend nomisVisitId_ = tostring(customDimensions.nomisVisitId)
| extend vsipVisitId_ = tostring(customDimensions.vsipVisitId)
For each failure the mapping endpoint should be called to create the mapping record. e.g.
curl --location --request POST 'https://nomis-visits-mapping.hmpps.service.justice.gov.uk/mapping' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <token with ADMIN_NOMIS >' \
--data-raw '{
"mappingType": "MIGRATED",
"nomisId": 624145,
"vsipId": "6b111f4c-6593-412e-8ec1-685486962e09",
"label": "2022-02-14T09:58:45"
}'
Architecture decision records start here