context deadline exceeded error in KIC

[anup-krai]

We are using Kong with KIC and Postgres DB.
Kong - 2.7.1
KIC - 2.2.1

We have two Kong deployments in two different data centers which are pointing to same Postgres DB.
DC1 - Kong and Postgres
DC2 - Kong

We are getting below error continuously in KIC in DC2 -

time="2022-12-05T08:20:05Z" level=error msg="could not update kong admin" error="11 errors occurred:\n\tfailed to sync all entities: context deadline exceeded\n\twhile processing event: {Update} service httpbin-service failed: making HTTP request: Put \"https://localhost:8444/services/65a74ed4-89d1-404c-9b65-fc81b37b46da\" : context deadline exceeded\n" subsystem=proxy-cache-resolver

Also we have below environment variable set in KIC -
proxy_timeout_seconds: 120
What we can do to resolve this error? Do we need to further increase the timeout values?

[pmalek]

Hi @anup-krai 👋

120 seems enough already but you can try increasing it. What I would focus on is the resource consumption of Kong GW deployment. Have you checked if it's not running near its limits (CPU and memory)?

[anup-krai]

We have enough CPU and memory allocation for both proxy and KIC containers. Their usage is very less as compared to the limits assigned.

[pmalek]

Do you mind sharing the levels of current CPU and memory for KIC as well as the limits set for those? Can you try accessing the /config endpoint of Kong GW at the admin API port to verify whether the size of the config hasn't ballooned to much? Or is it the same config as in the other environment that you're running and only that instance is causing issues/timeouts?

[anup-krai]

Below are the requests and limits for KIC -

cpuLimit: 2500m
memoryLimit: 2048Mi
cpuRequest: 500m
memoryRequest: 1024Mi

CPU and Memory usage is less then 10% for this.
This error is coming only in DC2 and it has same config as DC1. Also looks like /config endpoint is only available in DB less mode.

[pmalek]

Got it. Is this error intermittent or this is being logged constantly? Also: do you have some monitoring solutions that allow you to observe if KIC or Kong GW experiences spikes in CPU or memory usage that would correlate with those errors?

[anup-krai]

These errors are coming continuously every 2 mins. Also there are no irregular spike is CPU and memory usage of KIC and proxy container.