kubernetes-client could not work with java 8u252 #2212
Comments
|
Duplicate of #2145 |
|
@rohanKanojia I really appreciate for your quick response. I have noticed this issue of okhttp. However, i think it could not solve the problem. When i bump the fabric8 kubernetes-client to 4.9.1 and okhttp to 3.12.11, we still have the same exception with kubernetes version v1.17+. When we downgrade the kubernetes version to v1.16, it works well without any changes to the okhttp version. https://stackoverflow.com/questions/61565751/why-am-i-not-able-to-run-sparkpi-example-on-a-kubernetes-k8s-cluster |
|
OkHttp version was changed in 4.10.0, 4.9.1 still has the problem. Are you sure you're excluding in your pom the OkHttp dependency from kubernetes-client's dependency and including your own version? |
|
Yes, i use dependencyManagement to bump the okhttp version to 3.12.11. Actually we could find the stack trace is really different from that issue. We get Not only Flink, i think the Spark also comes into the same issue. |
|
We have the same problems as well in Strimzi - using okhttp 3.12.11 or 3.14.8 does not seem to solve the issue. Our exception is basically the same as @wangyang0918 pasted above. |
|
We are not able to reproduce. Can you try to bump |
|
@manusa What's the Kubernetes version you are using? It only happens in the K8s 1.7+ with jdk 8u252. |
|
In the Strimzi case, it is quite a bit complicated:
|
|
Sorry, didn't see your statement regarding k8s version. I can reproduce. |
|
@manusa I can confirm that even with Fabric8 4.10.1 I can still see this problem. Is there any way I can help with debuging this and finding the issue? |
CauseThe issue is related with the auto-selected Protocol used for each platform. When using JDK 8u252 the selected Protocol (in When using JDK 8u233 (<u252) the selected Protocol (in When using JDK 11 the selected Protocol is DescriptionWhen using > JDK 8u251, OkHttp "wrongly" detects the Platform as Jdk9Platform. This in turn enables Application-Layer Protocol Negotiation (ALPN) support which is not available out-of-the-box for JDK 8, thus enabling In order to prevent this we need to force Workaround for <= v4.10.1Set Config#http2Disable ( |
|
@manusa Thanks a lot for the investigation and fix. Could we also backport this to branch 4.9? We are going to upgrade the fabric8 kubernetes-client version to 4.9.x as @rohanKanojia 's suggestion.
|
|
Have you tried the workaround (e.g. There's no problem in backporting the fix but we're currently directing all efforts to cut a stable 4.10 release (so probably this will come first than a fixed 4.9.x). |
|
@manusa I have verified it could work after |
|
I can confirm as well that the workaround seems to work for me both locally on Kube 1.18.2 and on OCP 4.2. Thanks for looking into this @manusa and finding the solution ... great help for us! |
|
This was fixed in 3.12.11 and 4.6.0 a couple of weeks ago. https://square.github.io/okhttp/changelog/
|
|
The other report in your project which is for Azul's Zulu VM - #2145 When we saw that in OkHttp there was also a fix required in Zulu
|
|
Thx for your feedback. We are using the latest releases of OkHttp with your fixes ❤️. The problem is that You can check the below screenshot form a debugging session in Anyway, we managed to fix the issue by forcing |
|
Is there a reason you need to avoid HTTP/2? We have one existing issue with Keeppass client where HTTP/2 is a problem because they want the error from a smart HTTP/2 server that sends the HTTP/2 goaway before a request completes. But generally, I would have thought you would be happy that HTTP/2 is working out of the box in JDK 8. Or to put it another way, why not look into why HTTP/2 is failing in JDK 8 252? |
Not at all.
I completely agree on this. I'm a little bit lost here, is HTTP/2 supposed to be fully supported in JDK8u252? (I assume it is given your previous comment(s)) |
|
Yep - it should be working. This is with a test client (okurl). 10:35:17.443 OkHttp Platform: Jdk9Platform |
|
This is the error with Keeppass PhilippC/keepass2android#747 It's with a Go based server, and it's clearly an OkHttp bug but without a simple fix. IIRC in that case we are causing the pending 401, to become a "stream was reset: NO_ERROR". Maybe it's something similar. The fix there was for them to disable HTTP/2. |
|
Thx for you input! I'll try to find some time to see if at least we get the underlying cause for the failure in Kubernetes-Client. |
|
So do we have a concrete plan for v4.9.2? |
We're finishing with the fixes for bugs reported in 4.10.1, we'll probably release 4.10.2 in the next few days. This patched version should take care of any bugs and should be a direct replacement for 4.9.1. If after 4.10.2 release there are still some issues/bug, or if you find some strong reason (besides our previous recommendation) to not upgrade version, we can then release a back-ported patched 4.9.2. |
|
is there a release planned with this #2227 fix? Upgrade to okhttp 3.12.11 didn't help me. |
4.10.2 |
Actually I am not very sure whether 4.10.2 is stable enough for production. |
|
@manusa @rohanKanojia I share the same concern with @zhengcanbin. Bumping the version in downstream project is not very easy. So we want to make sure that whether 4.9.x is more stable that the latest version I think other projects may have the same situation. |
|
Downstream in Eclipse Che is interested in 4.9.x version. |
|
Just started release process for v4.9.2 |
|
Patched release with cherry- picked backport done: https://github.com/fabric8io/kubernetes-client/releases/tag/v4.9.2 Maven Central: https://repo1.maven.org/maven2/io/fabric8/kubernetes-client/4.9.2/ |
Thank you @manusa ! |
### What changes were proposed in this pull request? This PR aims to upgrade `kubernetes-client` library to bring the JDK8 related fixes. Please note that JDK11 works fine without any problem. - https://github.com/fabric8io/kubernetes-client/releases/tag/v4.9.2 - JDK8 always uses http/1.1 protocol (Prevent OkHttp from wrongly enabling http/2) ### Why are the changes needed? OkHttp "wrongly" detects the Platform as Jdk9Platform on JDK 8u251. - fabric8io/kubernetes-client#2212 - https://stackoverflow.com/questions/61565751/why-am-i-not-able-to-run-sparkpi-example-on-a-kubernetes-k8s-cluster Although there is a workaround `export HTTP2_DISABLE=true` and `Downgrade JDK or K8s`, we had better avoid this problematic situation. ### Does this PR introduce _any_ user-facing change? No. This will recover the failures on JDK 8u252. ### How was this patch tested? - [x] Pass the Jenkins UT (#28601 (comment)) - [x] Pass the Jenkins K8S IT with the K8s 1.13 (#28601 (comment)) - [x] Manual testing with K8s 1.17.3. (Below) **v1.17.6 result (on Minikube)** ``` KubernetesSuite: - Run SparkPi with no resources - Run SparkPi with a very long application name. - Use SparkLauncher.NO_RESOURCE - Run SparkPi with a master URL without a scheme. - Run SparkPi with an argument. - Run SparkPi with custom labels, annotations, and environment variables. - All pods have the same service account by default - Run extraJVMOptions check on driver - Run SparkRemoteFileTest using a remote data file - Run SparkPi with env and mount secrets. - Run PySpark on simple pi.py example - Run PySpark with Python2 to test a pyfiles example - Run PySpark with Python3 to test a pyfiles example - Run PySpark with memory customization - Run in client mode. - Start pod creation from template - PVs with local storage - Launcher client dependencies - Test basic decommissioning Run completed in 8 minutes, 27 seconds. Total number of tests run: 19 Suites: completed 2, aborted 0 Tests: succeeded 19, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` Closes #28601 from dongjoon-hyun/SPARK-K8S-CLIENT. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
### What changes were proposed in this pull request? This PR aims to upgrade `kubernetes-client` library to bring the JDK8 related fixes. Please note that JDK11 works fine without any problem. - https://github.com/fabric8io/kubernetes-client/releases/tag/v4.9.2 - JDK8 always uses http/1.1 protocol (Prevent OkHttp from wrongly enabling http/2) ### Why are the changes needed? OkHttp "wrongly" detects the Platform as Jdk9Platform on JDK 8u251. - fabric8io/kubernetes-client#2212 - https://stackoverflow.com/questions/61565751/why-am-i-not-able-to-run-sparkpi-example-on-a-kubernetes-k8s-cluster Although there is a workaround `export HTTP2_DISABLE=true` and `Downgrade JDK or K8s`, we had better avoid this problematic situation. ### Does this PR introduce _any_ user-facing change? No. This will recover the failures on JDK 8u252. ### How was this patch tested? - [x] Pass the Jenkins UT (#28601 (comment)) - [x] Pass the Jenkins K8S IT with the K8s 1.13 (#28601 (comment)) - [x] Manual testing with K8s 1.17.3. (Below) **v1.17.6 result (on Minikube)** ``` KubernetesSuite: - Run SparkPi with no resources - Run SparkPi with a very long application name. - Use SparkLauncher.NO_RESOURCE - Run SparkPi with a master URL without a scheme. - Run SparkPi with an argument. - Run SparkPi with custom labels, annotations, and environment variables. - All pods have the same service account by default - Run extraJVMOptions check on driver - Run SparkRemoteFileTest using a remote data file - Run SparkPi with env and mount secrets. - Run PySpark on simple pi.py example - Run PySpark with Python2 to test a pyfiles example - Run PySpark with Python3 to test a pyfiles example - Run PySpark with memory customization - Run in client mode. - Start pod creation from template - PVs with local storage - Launcher client dependencies - Test basic decommissioning Run completed in 8 minutes, 27 seconds. Total number of tests run: 19 Suites: completed 2, aborted 0 Tests: succeeded 19, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` Closes #28601 from dongjoon-hyun/SPARK-K8S-CLIENT. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 64ffc66) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
### What changes were proposed in this pull request? This PR aims to upgrade `kubernetes-client` library to bring the JDK8 related fixes. Please note that JDK11 works fine without any problem. - https://github.com/fabric8io/kubernetes-client/releases/tag/v4.9.2 - JDK8 always uses http/1.1 protocol (Prevent OkHttp from wrongly enabling http/2) ### Why are the changes needed? OkHttp "wrongly" detects the Platform as Jdk9Platform on JDK 8u251. - fabric8io/kubernetes-client#2212 - https://stackoverflow.com/questions/61565751/why-am-i-not-able-to-run-sparkpi-example-on-a-kubernetes-k8s-cluster Although there is a workaround `export HTTP2_DISABLE=true` and `Downgrade JDK or K8s`, we had better avoid this problematic situation. ### Does this PR introduce _any_ user-facing change? No. This will recover the failures on JDK 8u252. ### How was this patch tested? - [x] Pass the Jenkins UT (apache#28601 (comment)) - [x] Pass the Jenkins K8S IT with the K8s 1.13 (apache#28601 (comment)) - [x] Manual testing with K8s 1.17.3. (Below) **v1.17.6 result (on Minikube)** ``` KubernetesSuite: - Run SparkPi with no resources - Run SparkPi with a very long application name. - Use SparkLauncher.NO_RESOURCE - Run SparkPi with a master URL without a scheme. - Run SparkPi with an argument. - Run SparkPi with custom labels, annotations, and environment variables. - All pods have the same service account by default - Run extraJVMOptions check on driver - Run SparkRemoteFileTest using a remote data file - Run SparkPi with env and mount secrets. - Run PySpark on simple pi.py example - Run PySpark with Python2 to test a pyfiles example - Run PySpark with Python3 to test a pyfiles example - Run PySpark with memory customization - Run in client mode. - Start pod creation from template - PVs with local storage - Launcher client dependencies - Test basic decommissioning Run completed in 8 minutes, 27 seconds. Total number of tests run: 19 Suites: completed 2, aborted 0 Tests: succeeded 19, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` Closes apache#28601 from dongjoon-hyun/SPARK-K8S-CLIENT. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
This PR aims to upgrade `kubernetes-client` library to bring the JDK8 related fixes. Please note that JDK11 works fine without any problem. - https://github.com/fabric8io/kubernetes-client/releases/tag/v4.9.2 - JDK8 always uses http/1.1 protocol (Prevent OkHttp from wrongly enabling http/2) OkHttp "wrongly" detects the Platform as Jdk9Platform on JDK 8u251. - fabric8io/kubernetes-client#2212 - https://stackoverflow.com/questions/61565751/why-am-i-not-able-to-run-sparkpi-example-on-a-kubernetes-k8s-cluster Although there is a workaround `export HTTP2_DISABLE=true` and `Downgrade JDK or K8s`, we had better avoid this problematic situation. No. This will recover the failures on JDK 8u252. - [x] Pass the Jenkins UT (apache#28601 (comment)) - [x] Pass the Jenkins K8S IT with the K8s 1.13 (apache#28601 (comment)) - [x] Manual testing with K8s 1.17.3. (Below) **v1.17.6 result (on Minikube)** ``` KubernetesSuite: - Run SparkPi with no resources - Run SparkPi with a very long application name. - Use SparkLauncher.NO_RESOURCE - Run SparkPi with a master URL without a scheme. - Run SparkPi with an argument. - Run SparkPi with custom labels, annotations, and environment variables. - All pods have the same service account by default - Run extraJVMOptions check on driver - Run SparkRemoteFileTest using a remote data file - Run SparkPi with env and mount secrets. - Run PySpark on simple pi.py example - Run PySpark with Python2 to test a pyfiles example - Run PySpark with Python3 to test a pyfiles example - Run PySpark with memory customization - Run in client mode. - Start pod creation from template - PVs with local storage - Launcher client dependencies - Test basic decommissioning Run completed in 8 minutes, 27 seconds. Total number of tests run: 19 Suites: completed 2, aborted 0 Tests: succeeded 19, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` Closes apache#28601 from dongjoon-hyun/SPARK-K8S-CLIENT. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
|
@wangyang0918 Hello, I can still see the problem when I execute the following command. Is this normal? Thanks for your help. I use Java 11.0.8 on my laptop. I am using |
|
java version "1.8.0_152" Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T21:04:39Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"darwin/amd64"} The program finished with the following exception: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get] for kind: [Service] with name: [zhisheng-flink-session-cluster-rest] in namespace: [default] failed. |
When upgrading the jdk from 8u242 to 8u252, we find that the kubernetes-client could not work. It always throws the following exception.
Even when i bump the kubernetes-client version to 4.9.1, it still could not solve the problem. I know it might be a issue of okhttp or other dependencies of kubernetes-client. However, i could not find a clear clue for that.
The text was updated successfully, but these errors were encountered: