spring-cloud-gcp-data-datastore-sample fails with "com.google.cloud.datastore.DatastoreException: The project XXX does not exist."

[Kehrlann]

Running the spring-cloud-gcp-data-datastore-sample project fails with: com.google.cloud.datastore.DatastoreException: The project XXX does not exist."

Setup

Running:

• Java 21
• This project at commit 7733fe5
• The following config:

spring.cloud.gcp.datastore.credentials.location=classpath:credentials.json
spring.cloud.gcp.datastore.database-id=my-db

• GCP API key with role Cloud Datastore User
  • Edit: even when using ACD with my account, I get the same error

Bug:

Running the following command:

./mvnw -pl spring-cloud-gcp-samples/spring-cloud-gcp-data-datastore-sample spring-boot:run

Fails with:

com.google.cloud.datastore.DatastoreException: The project <REDACTED> does not exist.
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:140) ~[google-cloud-datastore-2.21.1.jar:2.21.1]
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:125) ~[google-cloud-datastore-2.21.1.jar:2.21.1]
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.runQuery(HttpDatastoreRpc.java:202) ~[google-cloud-datastore-2.21.1.jar:2.21.1]
	at com.google.cloud.datastore.DatastoreImpl.lambda$runQuery$0(DatastoreImpl.java:242) ~[google-cloud-datastore-2.21.1.jar:2.21.1]
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:102) ~[gax-2.51.0.jar:2.51.0]
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76) ~[google-cloud-core-2.41.0.jar:2.41.0]
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) ~[google-cloud-core-2.41.0.jar:2.41.0]
	at com.google.cloud.datastore.DatastoreImpl.runQuery(DatastoreImpl.java:241) ~[google-cloud-datastore-2.21.1.jar:2.21.1]
	at com.google.cloud.datastore.QueryResultsImpl.sendRequest(QueryResultsImpl.java:89) ~[google-cloud-datastore-2.21.1.jar:2.21.1]
	at com.google.cloud.datastore.QueryResultsImpl.<init>(QueryResultsImpl.java:70) ~[google-cloud-datastore-2.21.1.jar:2.21.1]
	at com.google.cloud.datastore.DatastoreImpl.run(DatastoreImpl.java:208) ~[google-cloud-datastore-2.21.1.jar:2.21.1]
	at com.google.cloud.datastore.DatastoreImpl.run(DatastoreImpl.java:187) ~[google-cloud-datastore-2.21.1.jar:2.21.1]
	at com.google.cloud.spring.data.datastore.core.DatastoreTemplate.lambda$resolveDescendantProperties$18(DatastoreTemplate.java:852) ~[classes/:na]
        ...

Note: The first time the project is run, the error may be different, but subsequent runs all fail when loading the @Descendant List<Instrument> instruments from Singer.

The query that fails is the following:

partition_id {
  project_id: "<redacted>"
  database_id: "my-db"
}
query {
  kind {
    name: "instrument"
  }
  filter {
    property_filter {
      property {
        name: "__key__"
      }
      op: HAS_ANCESTOR
      value {
        key_value {
          partition_id {
            project_id: "<redacted>"
          }
          path {
            kind: "singers"
            name: "singer3"
          }
        }
      }
    }
  }
}
project_id: "<redacted>"
database_id: "my-db"

Note that the instruments are recorded in the datastore.

[ldetmer]

@Kehrlann I am able to run this test multiple times successfully against the latest commit.

Can you share a little more information? where are you setting the project ID, if its via a properties file could you please share?

Also to confirm if you log into your project via the console, you see a default database in your DataStore and it has some results if you click on it such as: name=Big Bland Band?

[Kehrlann]

Hey @ldetmer thanks for reaching out!

Edit: the sample works only with the default datastore. Any custom database-id and it fails for me.

When not using the default datastore:

• The project ID is set through the credentials file. I also tried setting it in the properties file, and I get the same result.
• When I log into the console, the DataStore has entities for Band (3), Singer (3) and Instruments (a lot, those are never deleted)
• Inserting data is fine.
• Simple querying by entity is fine
• But loading the Descendants specifically fails

[ldetmer]

There does seem to be an issue with application.properties getting overridden. If I run "gcloud config set project" to set my project ID, that gets me past your error, however it fails later when trying to run singerRepository.findSingersByFirstBand as the database id provided in application.properties gets overridden by the default value.

The request was for database '{db from application.properties}' but was attempting to access database '{default}'

I will have the team look into this and will update when we have more information.

[burkedavison]

Fix merged.