CDP Instance Setup

This document outlines the steps necessary to finish initializing this CDP Instance.

Before You Begin

Install the command line tools that will help shorten the setup process

1. Install gcloud
2. Install pulumi
3. Install gsutil

Initial Repository Setup

There are additional tasks required after generating this repository.

1. Create the GitHub repository for this deployment to live in.

   Create a new Repository with the following parameters:

   • Set the repo name to: example
   • Set the repo owner to: CouncilDataProject
   • Set the repo visibility to: "Public"
   • Do not initialize with any of the extra options
   • Click "Create repository".

2. Login to both Google Cloud and Pulumi.

   IMPORTANT: During this process Pulumi will provide a token to use for authentication. Keep this token available for use in a later step.

   This step should be run while within the SETUP directory (cd SETUP).

   Run:

   make login

3. Initialize the basic project infrastructure.

   This step should be run while within the SETUP directory (cd SETUP)

   Run:

   make init

4. Create (or re-use) a Google Cloud billing account and attach it to the newly created project (cdp-example-nluypbsb).

   For more details on the cost of maintaining a CDP Instance, see our estimated cost breakdown.

5. Generate a Google Service Account JSON Key for your Google Cloud Project.

   This will create a directory called .keys within this SETUP directory and add a file called cdp-example-nluypbsb.json to it (i.e. .keys/cdp-example-nluypbsb). This file will be used later on.

   Run:

   make gen-key

6. Attach the Pulumi Access Token and the Google Service Account JSON as GitHub Repository Secrets.

   1. Pulumi Access Token -- Create a new secret
   • Set the name to: PULUMI_ACCESS_TOKEN
   • Set the value to: The token you kept from step #2
   • Click "Add secret"
   2. Google Service Account JSON -- Create a new secret
   • Set the name to: GOOGLE_CREDENTIALS
   • Set the value to: the contents of the file .keys/cdp-example-nluypbsb.json
   • Click "Add secret"

7. Initialize and push the local repository to GitHub.

   This step should be run while within the base directory of the repository (cd ..).

   To initialize the repo locally, run:

   git init
   git add -A
   git commit -m "Initial commit"
   git branch -M main

   To setup a connection to our GitHub repo, run either:

   git remote add origin https://github.com/CouncilDataProject/example.git

   Or (with SSH):

   git remote add origin git@github.com:CouncilDataProject/example.git

   Finally, to push this repo to GitHub, run:

    git push -u origin main

   Now refresh your repository's dashboard to ensure that all files were pushed.

8. Once the "Web App" GitHub Action Successfully Complete configure GitHub Pages.

   Go to your repository's GitHub Pages Configuration

   • Set the source to: "gh-pages"
   • Set the folder to: / (root)
   • Click "Save"

9. Once the "Infrastructure" GitHub Action Successfully Completes set the CORS policy for your Storage Bucket.

   This step should be run while within the SETUP directory (cd SETUP)

   Run:

   make set-cors

10. Once the "Infrastructure" GitHub Action Successfully Completes enable data-logging for the Google Speech-to-Text service.

    Direct Link to Enable

    If the above direct link doesn't work, follow the instructions from Google Documentation.

11. Once the "Infrastructure" GitHub Action Successfully Completes configure Firebase Security Rules.

    • Navigate to Firebase Console, login to the Google Account you used during step #2, select the cdp-example-nluypbsb Firebase project
      • Navigate to "Firestore Database", select the "Rules" tab, paste the following in:

        rules_version = '2';
        service cloud.firestore {
            match /databases/{database}/documents {
                match /{document=**} {
                    allow read;
                }
            }
        }
        

      • Click "Publish"
      • Navigate to "Storage", select the "Rules" tab, paste the following in:

        rules_version = '2';
        service firebase.storage {
            match /b/{bucket}/o {
                match /{allPaths=**} {
                    allow read;
                }
            }
        }
        

      • Click "Publish"

If all steps complete successful your web application will be viewable at: https://CouncilDataProject.github.io/example

Data Gathering Setup

Once your repository, infrastructure, and web application have been set up, you will need to write an event data gathering function.

Navigate and follow the instructions in the the file: python/cdp_example_backend/scraper.py.

As soon as you push your updates to your event gather function (get_events) to your GitHub repository, everything will be tested and configured for the next pipeline run. Events are gathered from this function every 6 hours from the default branch via a Github Action cron job. If you'd like to manually run event gathering, you can do so from within the Actions tab of your repo -> Event Gather -> Run workflow.

It is expected that the Event Index workflow will fail to start, as your database will not yet be populated with events to index.

There are some optional configurations for the data gathering pipeline which can be added to python/event-gather-config.json. No action is needed for a barebones pipeline run, but the optional parameters can be checked in the CDP pipeline config documentation. Note that google_credentials_file and get_events_function_path should not be modified and will populate automatically if you have followed the steps above.

Be sure to review the CDP Ingestion Model documentation for the object definition to return from your get_events function.

Once your function is complete and pushed to the main branch, feel free to delete this setup directory.

Other Documentation

For more documentation on adding data to your new CDP instance and maintainer or customizing your instance please see the "admin-docs" directory.