legacy cli json queries fail

[willscott]

curl "https://cid.contact/cid/bafybeidbjeqjovk2zdwh2dngy7tckid7l7qab5wivw2v5es4gphqxvsqqu"

fails with a 404

note that the x-ndjson variant succeeds

curl -H "Accept: application/x-ndjson" "https://cid.contact/cid/bafybeidbjeqjovk2zdwh2dngy7tckid7l7qab5wivw2v5es4gphqxvsqqu"

is okay

[masih]

Confirmed that this is an issue with the canary setup, where a percentage of queries are forwarded to the indexstar canary setup on cache misses.

The canary returns 404 every time when queried internally. The Accept header is a non issue here. What observed above is most likely caused by:

• first curl hitting cloudfront cache, which cached the 404 returned by canary.
• the second curl hitting cach miss at cloudfront (since response content-type is used as a cache key by cloudfront) and forwarded to the backend.
  • which so happens to hand on the non-canary ingress object that consistently returns 200.

Does that make sense?

@ischasny we need to find out:

• why canary returns 404, and
• if it's because it is behind then we should compensate for the false negatives by querying regular backends.