Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to :nonroot base image #160

Closed
mattmoor opened this issue May 3, 2020 · 0 comments · Fixed by #162
Closed

Switch to :nonroot base image #160

mattmoor opened this issue May 3, 2020 · 0 comments · Fixed by #162
Assignees

Comments

@mattmoor
Copy link
Collaborator

mattmoor commented May 3, 2020

Switch the default for defaultBaseImage to gcr.io/distroless/static:nonroot.

One of the goals of ko is to codify best practices, so users don't need to think about the containers, and one of those best practices is running as a non-root user. When ko started we didn't yet have the :nonroot distroless images, but now that we do we should migrate to that as the default.

The plan:

  1. When folks don't have .ko.yaml with defaultBaseImage set, emit a warning about the deprecation and impending change.
  2. Cut a 0.5 release
  3. Drop the warning and adopt :nonroot default.

To suppress this warning and preserve the legacy behavior

Add a file named .ko.yaml to the root of your repository with:

defaultBaseImage: gcr.io/distroless/static:latest

To suppress this warning and adopt the new behavior

Add a file named .ko.yaml to the root of your repository with:

defaultBaseImage: gcr.io/distroless/static:nonroot

Once 3. above lands, you can also install: go get github.com/google/ko/cmd/ko@master

@mattmoor mattmoor self-assigned this May 3, 2020
mattmoor added a commit to mattmoor/ko that referenced this issue May 3, 2020
mattmoor added a commit that referenced this issue May 4, 2020
* Add a deprecation notice when `defaultBaseImage` is not specified.

Related: #160

* Add a deprecation notice when `ko://` is not specified.

Related: #158

* Add a sample import path.
mattmoor added a commit to mattmoor/ko that referenced this issue May 4, 2020
mikehelmick added a commit to mikehelmick/exposure-notifications-server that referenced this issue May 18, 2020
google-prow-robot pushed a commit to google/exposure-notifications-server that referenced this issue May 18, 2020
randallmlough added a commit to randallmlough/exposure-notifications-server that referenced this issue Jun 26, 2020
* mark tests helper functions (google#317)

* Refactor apiconfig interfaces (google#313)

* Remove usused db tags in APIConfig

* Use struct{} instead of bool for allowed regions

* Move APIConfig back into model package

* Move Nonce calculation onto Publish model

There were too many inter-dependencies among the android and model/apiconfig packages. This makes android depend on model and moves things like nonce calculation into the model, since it depends on model-specific data anyway.

* Rename environment to config

* Merge and refactor apiconfig throughout

We had multiple layers of indirection around APIConfig. This refactors
from 6 layers to 2 layers. It's still not perfect, but it's certainly
better.

* Plumb through DeviceCheck/SafetyNet debug flags

* Resolve conflicts after the most painful rebase on the planet

* Fixes

* Load APIConfigs on boot

* Fix typo

* Move query back inline

* Group SafetyNet fields in struct

* Emit a metric on failed APIConfig loading

* Move VerifyOpts calculation into Android package

This resolves a circular reference when trying to use model inside the
android package.

* Cache APIConfig on a per-app basis

* Vet

* [WIP] add remaining config details to export config, denormalize on batch c… (google#274)

* add remaining config details to export config, denormalize on batch creation, and populate export files

* ensure a minimum export file length (google#304)

* ensure a minimum export file length

* add jitter to padded batch size

* allow for exports to have multiple signature configurations via SignatureInfo table

* small cleanup

* go fmt

* fix timestamp resolution on prow tests

* another timestamp fix, picky picky

* Rename APIConfig to AuthorizedApp (google#318)

* Add support for Azure KeyVault and HashiCorp Vault interfaces (google#314)

* Implement filesystem blobstore (google#331)

* Simplify context done checks (google#335)

The [doc for Context.Err](https://pkg.go.dev/context?tab=doc#Context.Err) says
that the result is non-nil only if context is done, and only the two context
errors can be returned. So we can simplify logic that tests for done-ness.

* minor simplifications (google#336)

Simplify some code and make it more idiomatic.

* internal/database: defer resetTestDB (google#338)

The existing code failed to reset the DB if t.Fatal was called.

* default max number of keys should be 15, per e2e testing w/ client team (google#339)

* Terraform docs move and improvement (google#340)

* move Terraform docs to be in docs

* add note about GOOGLE_APPLICATION_CREDENTIALS

* clearly mark cloud triggers as optional

* Incorporate Readme changes from @sethvargo in google#341

* further changes to deploying.md

Co-authored-by: Seth Vargo <seth@sethvargo.com>

* suppress ko warning, ko-build/ko#160 (google#344)

* Update Terraform setup from Mike's journey (google#341)

* Update Terraform setup from Mike's journey

* Note optional

* Remove unused variables

* Use separate service account for Cloud Run

* More updates

* Fix service account permissions

* Create AppEngine app in right region

* Use correct SA

* More Cloud SQL config

* Allow schema.yaml to work again after changes to docker containers. (google#301)

* Allow schema.yaml to work again after changes to docker containers.

* Move shell script to yaml file, skip including bash in the dockerfile, add ca-certificates so cloudsqlproxy can work.

* Don't display password in text output.

* log effective environment config on startup. Omit DB password (google#343)

* log effective environment config on startup. Omit DB password

* log DB config w/ deep copy instead of custom String

* Fix up auth, recurring diff (google#346)

* Unauth on out, not in

* Prevent recurring diff

Having a data source depend on a resource marks that data source as
always tainted. This means anything using that data source is also
always tainted. This fixes that.

* change federation fields/types (google#348)

* Remove verification authority name from DB and federation
Change transmission risk to int instead of enum in federation

* bow to the will of goimports

* document + validate transmission risk

* Combine database and model packages (google#347)

* move models into the database package

* rehome model packages

* update now internal refs for previous-model

* update imports

* gofmt

* fix up test imports

* fix locking error in appconfig caching (google#351)

* fix locking error in appconfig caching

* rearrange return + add documentation

* fix spellin'

* Fix typos. (google#363)

* Lower the spec for Cloud SQL to avoid excessive overprovisioning (google#364)

* gcloud run deploy now supports --no-traffic (google#356)

`--no-traffic` has graduated to GA for `--platform managed` in Cloud SDK 288.0.0 (2020-04-07)  
See [release notes](https://cloud.google.com/sdk/docs/release-notes#28800_2020-04-07)

I confirmed by running the following:
```
$ gcloud run deploy hello --image gcr.io/cloudrun/hello --platform managed --region europe-west1 --no-traffic
Deploying container to Cloud Run service [hello] in project [steren-serverless] region [europe-west1]
✓ Deploying... Done.                                                           
  ✓ Creating Revision...
  ✓ Routing traffic...
Done.
Service [hello] revision [hello-00028-fuf] has been deployed and is serving 0 percent of traffic
```

* Rename/prefix SafetyNet columns (google#352)

* Rename/prefix SafetyNet columns

* Rename migrations

* Migrate changes column

* create debug mode for exposure notifications upload so developers can… (google#366)

* create debug mode for exposure notifications upload so developers can see error messages for testing

* review comments

* Add cleanup-exposure and cleanup-export Terraform configs (google#354)

* Add missing comma in export SQL

* Add missing license headers

* Parallelize builds

This takes builds from ~11min to ~3min

* Add cleanup Cloud Run services in Terraform

* Don't set maxScale

* Terraform deployment edits (google#362)

* Terraform deployment edits

Cloud SQL high traffic info commented out - further context needed

* Remove comments that were accidentally checked in

* expand note for migration step

* Update cloudsql_disk_size_gb name

* Incorporate suggested edits from @sethvargo

* Incorporate feedback from @crwilcox

Co-authored-by: Christopher Wilcox <crwilcox@google.com>

* Break apart services into their own files (google#374)

* Break apart Cloud Run services into their own files

* Break apart storage and database

* Dont change name for now

* Moving util methods to the enclient lib. (google#358)

* Moving util methods to the enclient lib.
These methods will be reused for integration testing.

* Addressing PR comments.

* Addressing PR comments.

* Fixing error formatting pattern.

* Changing http request timeout.

* regen federation.pb.go, comments are out of date (google#379)

* Remove redundant text from README.md and index.md (google#380)

* Remove redundant text on README.md

* Remove redundant text and make it identical to /README.md

* Fixes for issue 252 (google#378)

* allow keys to be embargoed server side by a configurable amount, default of 2 hours

* don't accept keys that are still valid

* Incorporate Terraform changes (google#381)

* Rename database from contact-tracing to en

Fixes googleGH-377

* Make GCS bucket public

Fixes googleGH-373

* Create Cloud KMS KR and K, grant Exposure SA access

Fixes googleGH-372

* Bump minimum specs for Cloud Run services

Fixes googleGH-368

* Don't manage the full IAM policy for the Run instance

* Fix SA name description

* Re-run migrations when database changes

* Remove requirement for DC key_id and team_id to be secrets (google#383)

I've done a lot of reading and people generally don't consider these
secrets.

* Add scripts for build/deploy (google#385)

* Move ko into pre-built

* Use pre-built ko in CI

* Add ability to deploy

* Add ability to promote

* Missed one

* Make traffic percentage configurable

* add tool to generate export files (google#353)

* add tool to generate export files

* gather more data generation code into new util

make error handling more consistent

* resolve merge conflicts

* add flag to read keys in json format

* fix test

* gaaahhhh

* IOS Devicecheck edits (google#388)

* Add docs on sharing DC with 3ps

Fixes googleGH-369

* Wider page edits alongside edits to changes by @sethvargo

* Update docs/ios_device_check.md

Co-authored-by: Christopher Wilcox <crwilcox@google.com>

* Incorporating comments by @sethvargo

* Previously missed edit

Co-authored-by: Seth Vargo <seth@sethvargo.com>
Co-authored-by: Christopher Wilcox <crwilcox@google.com>

* Use same migrate container as local dev, share subs across build steps (google#389)

* Add quotes to make YAML linter happy

* Use pre-built migrate container

* Share subs between trigger and build

* Authorized App through interface (google#361)

* Authorized App through interface

Authorized App through interface

* change to calling interfaces models

* remove unrelated pb change

* continue calling it DB, not model

* split db test to util, move mode, database into authorizedapp

* split db test to util, move mode, database into authorizedapp

* remove unneeded alias

* Add full paths (google#390)

* Update tek-file documentation (google#391)

* Update tek-file documentation

* Update tools/export-generate/main.go

Co-authored-by: gurayAlsac <64565798+gurayAlsac@users.noreply.github.com>

Co-authored-by: gurayAlsac <64565798+gurayAlsac@users.noreply.github.com>

* Add script for running prod migrations (google#392)

* Rename schema -> migrate

* More quotes for the YAML gods

* Output database information

* Rename and have defaults for subs

* Make command a sub

* Add script for running prod migrations

* Allow per-app config of DeviceCheck and SafetyNet (google#393)

* introduce 'both' as platform option (google#396)

* internal/publish: don't pass a pointer to a pointer to json Unmarshal (google#398)

Pass the address of data to jsonutil.Unmarshal rather than the address
of a pointer to an uninitalised data.

* fixed the way to hold http.Client (google#399)

The Client's Transport typically has internal state (cached TCP connections), so Clients should be reused instead of created as needed. Clients are safe for concurrent use by multiple goroutines.

https://golang.org/pkg/net/http/#Client

* Added missing _REGION substitution for Cloud Build Triggers (google#402)

The `deploy.yaml` file uses the `_REGION` substitution for setting the region in various steps. When the `build-containers` trigger is pulled and doesn't have `_REGION` set, it errors out. This adds the missing `_REGION` substitution to the trigger with the value set by `var.region`

* Separate Export Model and DB (google#394)

* move db and model files

* Update References

* remove repeated name via new at jba@ recommendation

* be consistent, rename coredb, not exportdb

* move exportdb to handler

* clarify num-keys and tek-file flag interop (google#408)

* clarify num-keys and tek-file flag interop

* update generated federation pb

* Separate db and model for Exposure/Publish (google#407)

* move exposure db and model

* adjust refs

* remove testDB for exposure

* conform on publish to not have two words for the model/db

* Use more lenient Content-Type checking (google#409)

* Filesystem support (google#397)

* Making Blobstore configurable via environment variables.

* Simlifing the Blobstore initialization.

* Get Terraform out of the CI/CD business (google#406)

* Build containers separately

* Fail on errors requesting the secret

* Do not manage the lifecycle of Cloud Run services

* Remove unused secrets, more POLP

* Remove triggers

* Add ability to deploy single service

* Add ability to promote a single service

* Update docs + remote state

* Remove state.tf

* Fix GCR link

* Ignore name changes on Cloud SQL (google#411)

* Separate db/model federation (google#410)

* move files

* refs

* include change to separate db tests

* Admin Console - Implement AuthorizedApps editing (google#395)

* checkpoint

* initial review comments

* finish view / form for authorized apps

* complete admin console for authorized apps

* formatting

* review comments

* temporaryTracingKeys is now temporaryExposureKeys (google#416)

* Fix typo (google#417)

* fixed map allocation (google#418)

* add refrences to the admin console in the docs (google#412)

* Add Vault as a signing option (google#419)

* Add Vault as a signing option

* Add copyright

* Add tests for Vault secrets (google#421)

* Don't spew a ton of output on failure, only mount transit

* Add tests for Vault secrets

* Update from failing tests

* all: fix all staticcheck warnings (google#424)

I ran staticcheck 2020.1.4 against the repo and fixed all
the warnings.

* Fix database tests (google#428)

Database tests haven't been running since
google@f113c0d2#diff-f7ac49848dfd0ce5e8f3251a560f5a0f,
because of the package being renamed to no longer include a _test.go
suffix. Not shown in that diff is the inclusion of a TestMain, which
only runs if the file is actually a test (ends in the _test.go suffix).
This and other dependent database tests have been silently skipping.

This refactors the tests to use a new Docker container per instance
(where test implementers decide how they want to manage the instance),
and updates all the database tests to run in parallel. This results in a
net increase of 8s to the overall test suite runtime on my local
machine, but seems worth it given this is the 4th (5th?) time a
seemingly unrelated change as silently broken the database tests.

Folks can opt of out running the tests (either using -short or setting
an environment variable), but the default behavior is now to run the
tests. The only requirement is Docker (which was already a requirement
anyway).

* Move admin-console to gin framework (google#429)

* move admin console to gin framework

* tidy mods

* fix tabs

* unit tests for internal/cleanup  (google#427)

* comment for missing cleanup test

Signed-off-by: Abhilash Gnan <abhilashgnan@gmail.com>

* use testdb and initializers for blobstorage

Signed-off-by: Abhilash Gnan <abhilashgnan@gmail.com>

* use noop blobstore

Signed-off-by: Abhilash Gnan <abhilashgnan@gmail.com>

* undo local fix

Signed-off-by: Abhilash Gnan <abhilashgnan@gmail.com>

* fix migrations dir path for testDB

Signed-off-by: Abhilash Gnan <abhilashgnan@gmail.com>

* Revert "fix migrations dir path for testDB"

This reverts commit 2a1a7f8.

* basic tests for handler creators

Signed-off-by: Abhilash Gnan <abhilashgnan@gmail.com>

* Fix err strings and todo comment

Signed-off-by: Abhilash Gnan <abhilashgnan@gmail.com>

* all: add distributed tracing with OpenCensus (google#413)

Adds distributed tracing with OpenCensus' vendor agnostic
mechanisms that'll extract distributed traces and metrics
for:
* HTTP
* gRPC
* Custom instrumentation
and can trivially export them to any of the supported exporters
per https://opencensus.io/exporters/supported-exporters/go/

For now, we've just enabled either of:
* ocagent (OpenCensus Agent)
* stackdriver

which can be set using environment variable
```shell
OBSERVABILITY_EXPORTER="<EXPORTER>"
```
for example to use with Stackdriver aka Google Cloud Trace
```shell
export OBSERVABILITY_EXPORTER="stackdriver" PROJECT_ID="my-project"
```

to enable the exporter and observability, one just needs to import
```go
import (
	_ "github.com/google/exposure-notifications-server/internal/observability"
)
```

Fixes google#256

* Changed timestamp handling in the export-generate tool. (google#426)

* Changed timestamp handling in the export-generate tool.
The export's start and end timestamps now reflect the past 24h period.
If TEKs are supplied in a JSON file, their start intervals must be at least 24h old.
If random TEKs are generated by this tool, their parameters are now taken from the random keys that util.GenerateExposureKeys() generates, but their start interval is reduced by 144 (24h).

* Changed timestamp handling in the export-generate tool.
If TEKs are supplied in a JSON file, their end interval must not be in the future.

* Used constant publishmodel.MaxIntervalCount (==144)

* Adds Admin Console for Signature Info (google#435)

* move admin console to gin framework

* fix tabs

* admin console for editing of signature info objects

* fix SQL tabs

* fix presubmit failure

* fix tabs :shakesfist:

* Update and fix missing docs (google#439)

* Update and fix missing docs

Fixes googleGH-438

* Set depth from 2

* Document sslmode

Fixes google#432

* ci: reenable go vet (google#440)

* Break Monolith's realMain into an importable utility method for testing. (google#441)

* clarify verification key fields (google#442)

* clarify verification key fields

* clarify MCC

* update generated pb.go

* Cleanup issue templates, close stale (google#444)

* Unify docker and prow config

* Rename issue templates

* Provide more guidance in issue templates

* Add stale workflow

* Rename stale.yaml to stale.yml (google#446)

* Use correct default SSL mode (google#448)

* Add ExportConfig to admin console (google#449)

* move admin console to gin framework

* fix tabs

* admin console for editing of signature info objects

* fix SQL tabs

* Add ExportConfig editing to admin console

* add br on index page

* add render test for export config

* Adding method for extracting keys from the export file. (google#415)

* Adding method for extracting keys from the export file.

* Addressing PR comments.

* Addressing PR comments 2.

* Addressing PR comments 2.

* add deployable service to generate data (google#451)

* Fixing outdated comments. (google#452)

* Doc changes as docs/ is deployed self contained (google#458)

* relative links outside of docs folder are not allowed

* ToC on index is not separated by newlines

* relative contributing.md

* Add optional deployment of generate service (google#456)

* fix issues with index file (google#457)

This fixes two problems in the index files:
1) no longer limit available export files to those in same
   export config. During a config rollover this would have
   caused all old files to disappear from index
2) fixes google#453 by adding time constraint to lookup query

* proto: use full package path in go_option (google#462)

The official docs [1] for the go_option option specify
that the full go package import path should be used.
This, together with the paths=source_relative, results
in the same output paths.

[1]: https://developers.google.com/protocol-buffers/docs/reference/go-generated#invocation

* clarify batch field usage for iOS (google#463)

* Don't use magical "default" network (google#464)

Switch to lookup the full self-link. Required for cross-project.

* Update contribution docs (google#461)

* CONTRIBUTING.md: remove old test instructions

Database tests are now automatically starting
and stopping containers.

* internal/database: fix typo in NewtestDatabase

* Adding tool to read content of export files. (google#455)

* remove OCA (google#467)

* Exports from multiple regions (google#465)

* Exports from multiple regions

* Allow exportconfig to have a list of regions as part of the input
* Defaults to the single output region if not specfied
* Add multi input region config to admin console

/fixes google#445

* rename region to output region

* Add cleanup for Container Registry (google#468)

As we build more containers, GCR gets littered with a lot of images.
This runs a job (disabled by default) that cleans up old images, keeping
only the most recent 3.

* Drop Cloud SQL proxy and switch to VPC connections (google#472)

* Ensure containerregistry is enabled before spawning

* Generate longer db passwords

* Create secrets for all db TLS configuration

* Ensure KMS is enabled before creating keys

* Bump maxconns higher

* Enable verification

* Provide TLS configuration to all services

* Remove unnecessary beta providers

* Create VPC access connector

* Ensure Cloud Scheduler is enabled

* Connect to database over VPC

* Drop network datasource

The datasource results in an always-stale diff because it depends on a resource.

* Raise maxconns

* Tune scale parameters

* Allow more export but less concurrency

* Allow more concurrent connections to the database

* Allow more scale for exposure

* Update contribution instructions (google#473)

Rename reference to /pkg to /internal.

* for e2e testing, allow acceptance of keys valid till the end of the UTC day (google#474)

* update diagram verbiage (google#475)

* first draft of verification design doc (google#470)

* first draft of verification design doc

* address some comments

* address review comments

* minor edit about overrides

* wording change

* Return the created MonoConfig. (google#477)

An integration test runner that starts the monolith as part of the test
will need the information contained in the MonoConfig to find all the
endpoints.

* observability: bring back ocagent after updating to v0.0.7 (google#479)

Brings back the ocagent-exporter, after updating to
ocagent version v0.0.7 which uses newer dependencies of
the grpc-gateway, thus no longer needs the deprecated
"github.com/golang/protobuf/protoc-gen-go/generator"
package.

Fixes google#466
Reverts google#467

* check inbound platform in addition to authorizedapp settings (google#483)

* Export bucket is not used, remove it (google#486)

This appears to be a remenant of when the system wasn't as multi-tenant.
The bucket is now a per-row configuration in the database.

* all: fix staticcheck warnings (google#485)

* Allow per-service overrides (google#487)

* add shared API definitions for PHA verification protocol (google#478)

* Allow per-service location customization (google#488)

* Add multiple region vars to allow more deployment flexibility

Not all regions support all services; separate out different region
variables to support the main services but also keep the ability to
define a single region for deployment

Bugfix: scheduler_jobs need to use the app engine region

* Remove "clever" code to derive regional arguments

Replaces the region derivations with top-level variables and
extra documentation to sepcify dependencies

* Terraform variable rename (as required by review)

Also minor comment fixes

* Further comment fix. .gitignore *.tfvar files.

* Fix DB connection string to use "db_region" instead of "region"

Also add the different region variables to the outputs

* Simplify note

* Use in new services

Co-authored-by: Jae Hossell <jexh@criticalblue.com>

* Support runtime configuration of all current components (google#492)

* Allow runtime configuration of Key Manager

* Rename file

* Refactor remaining configs

* Add docs

* Rename file

* don't cache the export index (google#494)

For those blob storage implementations that support it, pass
in an optional bool for whether a created object should be
cacheable, and have the worker set that to false for index.txt.

* implement verification and processing of health authority diagnosis v… (google#493)

* implement verification and processing of health authority diagnosis verification certificates.

* time truncation for DB tests

* test timing again

* address review comments

* run goimports

* Fix nil pointer on environment setup. (google#499)

If the config structs, if there are pointers to sub-configs, those pointers are initialized to nil and aren't traversed by envconfig.

* Document why we process the envconfig (google#500)

* add health authority certificate bypass option to authorized app (google#501)

* Fix linting errors from experiment (google#498)

* Fix linting errors from experiment

* Return admin.Controller

* Re-unname

* apply gofmt (google#502)

* ci: stop formatting generated files (google#469)

Avoid formatting generated files by checking for
the official [1] generated file header before formatting.

Because everyone uses the same docker image to
generate protobuf files, this shouldn't cause much disruption.
Every time the docker image is updated the corresponding
regeneration will have to be made in the repo.

[1]: golang/go#13560 (comment)

* Provide a way to return config from the test database (google#503)

* Revert "ci: stop formatting generated files (google#469)" (google#506)

This reverts commit 208bab6.

* Add tests for setup (google#504)

* Test setup

* Add POC for testing a handler

* Undo trying to test a cmd

* Adding clear-export sql script (google#480)

* Adding clear-export sql script

* Using TRUNCATE instead of DELETE

* Remove DeviceCheck and SafetyNet (google#507)

* Remove DeviceCheck and SafetyNet

* t vs s

* Review comments

* Remove platform too

* Hook up health authority verification certificate validation (google#508)

- Add HTTP level testing to the publish API
 - starts in mem http server, sends request, verifies results in database
- Add additional tests to phaverify
- Add transmission risk override application

* Enable to use scripts/presubmit.sh as a pre-push git hook (google#511)

* Enable to use scripts/presubmit.sh as a pre-push git hook

The CONTRIBUTING.md file suggests the presubmit script can be used as a
pre-push git hook but it fails because the script relies on being stored
in a directory directly below the project root and the git hooks are
stored two levels below it.

The current patch allows to run the checks as a pre-push git hook while
still being able to run it by direct invocation.

* revert previous commit and remove recommendation

* Use noops in tests to avoid credential lookup (google#512)

Fixes googleGH-510

* Add support for AWS Secret Manager (google#515)

* Add support for AWS Secret Manager

* Add docs

* Tidy mod

* Add support for AWS KMS for signing (google#516)

* Add AWS KMS as a signer

* Always pass the signing options

The AWS KMS adapter doesn't check if this is nil

* Reduce code complexity and just return the error

* Tidy mod

* Add AWS S3 blobstore (google#517)

* plugged aws-s3 adapter (google#518)

* fixed s3 docstrings

* plugged-in AWSS3 blob adapter

* added AWS S3 to docs

* Add additional language to server requirement document (google#522)

* Add Azure Blobstore implementation (google#519)

* use kenvconfig to resolve secret manager config since it cannot contain secrets. (google#523)

* let setups tests pass if you have any secret:// in your env (google#524)

* Add Azure KeyVault signing support (google#527)

* add health authority configuration to admin console (google#520)

* add health authority configuration to admin console

* review comments

* tabs

* Does this make CI run faster? (google#528)

* ci: stop formatting generated files (google#509)

Avoid formatting generated files by checking for
the official [1] generated file header before formatting.

Because everyone uses the same docker image to
generate protobuf files, this shouldn't cause much disruption.
Every time the docker image is updated the corresponding
regeneration will have to be made in the repo.

[1]: golang/go#13560 (comment)

* Added path to export.proto (google#531)

Added path to export.proto as per this request: google@dadaebe#commitcomment-39632255

* Marked unused fetchType as deprecated in the federation proto. (google#534)

* Revert "ci: stop formatting generated files (google#509)" (google#533)

This reverts commit 6435ae8.

* increase padding test size. prow test environment has less entropy (it appears) (google#535)

* Fix bypass HA certificate. Wasn't setting field in form. (google#536)

* Ensure that repeated fields end up as empty arrays in DB and not arrays containing an empty string. (google#538)

* [master] Fix spelling errors (google#539)

Produced via:
  `github.com/client9/misspell`
/assign helmick
/cc helmick

* Make AppPackageName and BundleID case-insensitive (google#541)

* refactor API definitions to PKG instead of internal so they can be included via our go mod. (google#532)

* [master] Fix spelling errors (google#546)

Produced via:
  `github.com/client9/misspell`
/assign mikehelmick
/cc mikehelmick

* Update functional requirements (google#548)

* reflect latest thinking on diagnosis verification server

/fixes google#544

* Accept, but embargo currently valid keys (google#550)

* Accept, but embargo currently valid keys

Instead of rejecting keys that have started before the instant now, but aren't yet expired, accept those keys and emabargo them until they can no longer be used.
This is done by setting the createdAt time on that key.

There is an invariant here that there can only be one key in this state since we don't allow overlapping intervals and we don't allow keys to start in the future.

/fixes google#543

* update other pieces that touch this

* review comments

* Use custom envconfig resolver, decouple secrets (google#549)

* Rewrite envconfig

There are a number of issues with our current envconfig implementation.
First, we've got two packages that are named envconfig, and we regularly
have to switch between them. Second, we've hit a number of bugs due to
misunderstandings in how kenvconfig works.

* Have serverenv provide a closer

* More envconfig tweaks

* Move secret resolver into secrets

* Use new envconfig everywhere, with tests

* Add test for secret resolution

* Break apart into separate lib

* Use tagged version

* Fix reference to verification_protocol.md (google#553)

This change fixes a reference to the `verification_protocol.md` file.

* Move monolith into tools (google#554)

* Enable cloudresourcemanager.googleapis.com in Terraform (google#552)

* Enable cloudresourcemanager.googleapis.com in Terraform

* Require most recent provider versions

* Gracefully stop servers on interrupt (google#551)

* Gracefully stop servers on interrupt

* Server handles stop

* Remove unused error

* Factor out cache implementation to standalone package. (google#557)

* Factor out cache implementation to standalone package.

* Create write thru cache implementation
* Move database provider to it
* First part of google#555

Follow on PR will bring this to other parts of the codebase that use caching.

* review comments + concurrency test

* fix missed err return

* Minor changes to support e2e testing (google#559)

* Add Add() interface to authorizedapp provider

* Sleep for a full second before polling in the database

* Expose IP, Port, and Addr for server

* Add in-memory blobstore option

* Fix comment

* Add initial e2e framework (google#561)

* Removing time window used by worker to query for exposure batches available for export. (google#521)

* Adding worker's configuration to support export batch time shift.

* Renaming export config parameter.

* Removing time shift from the worker LeaseBatch argument.

* Add Navigation, theming, for docs (google#563)

* remove app_bundle_id & android_package in SigInfo (google#562)

* remove app_bundle_id & android_package in SigInfo

* rebased and fixed integration test

* Move to single cache impl. (google#560)

* Move to single cache impl.

* Move secret manager cache to internal/cache write through impl
* Start caching HealthAuthoirty configuration in verificaiton code

/fixes 555

* rename config env var name

* fix race condition

* reliably trigger race condition on cache miss

* Remove min latency handler from integ (google#566)

* Remove monolith (google#567)

We probably don't need this anymore

* Add interfaces for getting objects via the blobstore (google#565)

This is required for testing, but could also be useful as the system
expands. Added sample use in publish_test.go

* Add security policy (google#568)

* Create SECURITY.md

* Update SECURITY.md

* [master] Fix spelling errors (google#569)

Produced via:
  `github.com/client9/misspell`
/assign mikehelmick
/cc mikehelmick

* Switch to git-sha based deployments (google#570)

* Update envconfig to fix nested struct passing mutator funcs (google#571)

* Add more debugging information

* Upgrade envconfig

* Use correct generate variable in Terraform (google#572)

* Don't log passwords (google#574)

* Don't log passwords

* Add comment

* Fix data race in setup tests (google#577)

* Update deploy docs to note git-sha (google#576)

* Remove unused db struct tags (google#575)

* fix(signing): add configuration option for azure key vault (google#558)

* Add documentation for publishing, retrieving keys. (google#573)

* Add link to 'edit this page'

* Create publishing and retrieving getting-started pages

* further changes to docs

* move deploying under getting-started

* expand retrieving

* add a brief walkthrough on how to configure the admin console

* add a brief walkthrough on how to configure the admin console for retrieval

* some grammar fixes

* Update docs/getting-started/publishing-keys.md

Co-authored-by: Seth Vargo <seth@sethvargo.com>

* move docs, incorporate changes, extract prereqs to an include

* disambiguate key

* expand on what a TEK is

* Incorporate Guray's feedback

* Incorporate Guray's feedback

Co-authored-by: Seth Vargo <seth@sethvargo.com>

* Fixed ambiguity in docs, regarding batch generation. (google#505)

* Fixed ambiguity in docs

The original sentence made it seem like the batches were generated per device (the 'per device' sentence grammatically referenced the 'file generation'). Made this slightly more clear.

* Update docs/server_functional_requirements.md

Co-authored-by: Christopher Wilcox <crwilcox@google.com>

* README.md should link out to docsite (google#582)

* README.md should link out to docsite

* add missing file from previous merge

* relative links (google#583)

* add keyvault specific authorizer (google#585)

* Move test steps into make targets (google#584)

* Add Makefile for running short tests, full tests, spellcheck, staticcheck, and fmtcheck

* Update presubmit

* Fix easy staticcheck/vet errors

* Don't call t.Error/Fatal from a goroutine

* Stop formatting protos with goimports

* Better error

* Basshhhh

* Fix spelling errors

* Fetch first

* Capture more output

* Track tools in go modules

* Pre-install tools in image

* Set GOMAXPROCS

* Link to build

* Handle errors returned from Azure setup (google#601)

* Document authorizer

* Handle error returned from environment

* Build the URL instead of parsing

* Return nil on errors

* [master] golang format tools (google#600)

Produced via:
  `gofmt -s -w $(find -path './vendor' -prune -o -path './third_party' -prune -o -type f -name '*.go' -print)`
  `goimports -w $(find -name '*.go' | grep -v vendor | grep -v third_party | grep -v wire_gen.go)`
/assign mikehelmick
/cc mikehelmick

* Add example for issuing a verification certificate. (google#598)

* Add example for issuing a verification certificate.

/fixes 581

* review comments

* better documentation for signing portion

* remove struct wrapping azure authorizer (google#602)

* Remove gcr-cleaner (google#603)

* Use JWT 'kid' header to identify public key (google#605)

* Update design doc
* Update public API for custom claims set
* Update verificaiton + tests
* Update example certificate issuer

/fixes google#593

* add goimports formatting to generated pb.go files. apply immediatly after generation. (google#607)

* Add comments around aud specification. (google#610)

/fixes google#590

* clarify what is logged (google#611)

/fixes google#588

* point to new link for file format (google#612)

* point to new link for file format

* add link to export readme and proto

* modify functional requirements to call out that some fields are required for verification (google#615)

* URL encode database passwords (google#616)

Fixes googleGH-613

* Reduce function arity (google#614)

/fixes 312

* Fix folder where urlencode lives to fix build script (google#617)

* Avoid dividing by 0 (google#618)

This is unlikely to happen in production, but in testing with a low
export period, it can panic because anything%0 is a division by zero
error.

Refs: https://github.com/google/exposure-notifications-server/pull/599/files#r438477087

* Update verification flow doc (google#619)

* Don't stale privacy bugs (google#620)

* Export handlers cleanup (google#621)

I pulled the non-debug changes out of googleGH-609 since I'm not sure we agree
that the /debug endpoint should exist on export. However, these changes
are still worth including (especially the database check and logger
fixes).

* Add links to API Definitions to docs (google#624)

* Add debugger (google#625)

* Remove SM from AAs

* Order and sort batches

* Add SQL for all latest batch ends

* Add debugger service

* Add Terraform

* Decode into correct struct

* Grant permission to see other services

* Initialize maps

* Don't shadow service

* tvs

* Stop flakey test

* include end time in export filenames (google#604)

* move cache and base64utils to pkg so they can be imported into verification server codebase (google#628)

* package comments (google#629)

* Read all of stdin in urlencode (google#630)

* Handle null output region (google#626)

* Handle null output region

* Tidy mod

* Use pointer

* Stop logging

* move retry to pkg (google#631)

* add a prometheus metrics endpoint when using 'prometheus' exporter (google#634)

* Docs for Estimating Costs (google#633)

* Estimating Costs (WIP)

* cleaned up some of the math

* remove decimals in wide estimates

* remove assumption of where costs will come frome

* correct calculation for network

* links don't work in notes

* Clarify based on comments

* Adjust wrapping for note callouts (google#637)

* google#635 export-create-batches creating an empty index file (google#638)

* Adding Publish - Export integration test. (google#599)

* Publish integration test

* Adding e2e integration test.

* Test Code Cleanup

* Integrating test code with the in-process SUT.

* Addressing PR review comments.

* Fixing Ling errors.

* Fixing Ling errors.

* Making presubmit happy:

Fixing lint errors.
Fixing cmp + proto incompatibility.

* Fixing waiting params.

* Applying PR review comments.

* Applying PR review comments.

* Applying PR review comments.

* Added variable for vpc_access_connector max_throughput (google#644)

Fixes: google#639

Add the following to terraform.tfvars
vpc_access_connector_max_throughput = 300

* feat(observability): use envconfig to manage configuration for the observability exporter (google#641)

* Make secrets and key management public (google#645)

* Move secrets out of internal/

* Add note about public

* Move signing public, rename to keys

* Finish rename

* Rename main file

* Cleanup package comments

* fix(observability-exporter): return err from the observability exporter initialization and dont panic (google#646)

* Remove latency delay and return real responses (google#649)

* Remove latency delay and return real responses

Fixes googleGH-648

* Return error when unauthorized app

* Use correct syntax for default (google#651)

Fixes googleGH-650

* Adding integration test for cleanup published exposures (google#647)

* Adding Publish - Export integration test. (google#599)

* Remove deps of apimachinery

* Fixing staticcheck

Co-authored-by: Maxim Gulimonov <maximg@google.com>

* Use KeyVault SDK authorizer, which is more flexible (google#656)

* Using a more flexible authorizer for Azure KeyVault

* Ran go mody tidy && go mod verify

* Return error from secret cacher (google#657)

* Change key terminology to be more consistent, fixes google#556 (google#658)

* feat(secrets-manager): add ability to json-decode secret values and select the value of a map by specifying the key in the secret-name (google#660)

* Updating protobuf for v1.5 (google#659)

Bringing this in sync with the latest changes and clarifications for v1.5

Adds:
* Report Type
* Days since onset
* revised_keys to list revoked / revised keys

* Add cleanup test for testing cleanup keys export (google#664)

* Refactor cleanup exposure to make it reusable

* Add cleanup export test

* Create initial Mako project for performance and load tests. (google#666)

Co-authored-by: Assel Meher <asselmeher@gmail.com>
Co-authored-by: Seth Vargo <seth@sethvargo.com>
Co-authored-by: Mike Helmick <helmick@google.com>
Co-authored-by: Jonathan Amsterdam <jba@users.noreply.github.com>
Co-authored-by: Christopher Wilcox <crwilcox@google.com>
Co-authored-by: Nathan McKinley <nmckinley@google.com>
Co-authored-by: Artem Yakshin <neetkee@users.noreply.github.com>
Co-authored-by: Steren <steren@google.com>
Co-authored-by: Lawrence Latif <llatif@users.noreply.github.com>
Co-authored-by: Maxim Gulimonov <maximg@google.com>
Co-authored-by: gurayAlsac <64565798+gurayAlsac@users.noreply.github.com>
Co-authored-by: Michael Stillwell <mjs@beebo.org>
Co-authored-by: Dave Cheney <davecheney@github.com>
Co-authored-by: Kei Kamikawa <Code-Hex@users.noreply.github.com>
Co-authored-by: Ernie Kim <krazykid@gmail.com>
Co-authored-by: Hirochika Matsumoto <matsujika@gmail.com>
Co-authored-by: Johan Brandhorst <johan.brandhorst@gmail.com>
Co-authored-by: Mike Helmick <mike.helmick@gmail.com>
Co-authored-by: Abhilash Gnan <abhilashgnan@gmail.com>
Co-authored-by: Emmanuel T Odeke <emm.odeke@gmail.com>
Co-authored-by: mh- <michael@huebler.org>
Co-authored-by: Keith Goodman <stati0n@users.noreply.github.com>
Co-authored-by: Alberto Cortés <alcortesm@gmail.com>
Co-authored-by: Jae Hossell <jexh@criticalblue.com>
Co-authored-by: Asutosh <asutosh97@users.noreply.github.com>
Co-authored-by: Jason Collins <jasonco@google.com>
Co-authored-by: Matt Moore <mattmoor@vmware.com>
Co-authored-by: Nicholas M. Iodice <niiodice@microsoft.com>
Co-authored-by: Matt Tucker <ultimateboy@users.noreply.github.com>
Co-authored-by: Ivo Jansch <ivo@egeniq.com>
Co-authored-by: Erin Corson <frodopwns@gmail.com>
Co-authored-by: Erin Corson <erin.corson@microsoft.com>
Co-authored-by: Madhava Jay <me@madhavajay.com>
Co-authored-by: chaodaiG <45011425+chaodaiG@users.noreply.github.com>
Co-authored-by: Brad Nelson <flagxor@gmail.com>
mattmoor added a commit to mattmoor/ko that referenced this issue Sep 24, 2020
mattmoor added a commit that referenced this issue Sep 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant