arm64 Docker images openly available #11820
Comments
asdf2014
added
Docker
|
https://github.com/arm64-compat/apache-druid I have implemented a workaround for the same, if you are using Mac with M1 chip processor, these images will greatly speed up your processes (from my experience). Note that images are only for development purpose. Not to be used in production. I haven't throughly tested the same, I would love to hear your comments. |
|
Woho~. Did you have do to anything special to get the arm build up and running? |
|
Nothing special as such, to summarise the changes I guess only 2 things are mainly required:
But in my repo I did a little bit more changes, I shifted the mvn build step from image to ci to make it utilise the maven build cache next time. And I did some related changes. Although the image that I pushed is not currently working I tested after I posted the comment, I repeated the same steps on my local and it was working fine. I will be able to figure out the issue and correct over the next 2 days. The UI is not loading and giving 404 instead to me it seems like the console didn't compile properly in the ci environment and nothing was reported while building, this is my initial guess, will drill down. |
|
OK, thanks for trying this out. I think it'll be awesome to get a arm build to run well on travis. |
|
@2bethere arm64-compat/apache-druid#1 I identified the issue and fixed the same. If you want you can give this image a try as well. I can also contribute to the Just we might have to decide on a base image. |
|
Yeah, on travis the console is skipped to improve build speed right now. We should probably try to push #11109 forward to get an arm build on docker hub. Double checking this is what you are asking for? |
|
Yes I may have mixed the commands while building on travis right now it is fixed in my image, I was asking if you would want to test out my image and see if everything works correctly or is there anyway I can contribute to the project to server arm compatible docker image. For me now I haven't seen any issues. |
|
ARM64 image now can be built on the master branch from both Linux and Mac M1/M2. But I don't know when the official ARM64 image will be provided on Docker Hub. |
|
Any update on this? |
|
Let's add 2024 to the mix. Any update on this? I'm pulling I see the docs show how to manually build, but is it that much of a hassle to host them? (legitimately curious) |
|
You can pull onofficial arm64 compatible image from https://github.com/arm64-compat/apache-druid let me know if you want me to upgrade the version for the same or help contribute |
|
Thanks, @anuragagarwal561994! It looks like there is "official support" now, you just have to build it yourself. Unfortunately I wasn't able to compile main on my M1 last night. This is the error I'm getting, if anyone is more familiar with Java. |
|
@dudo actually since it is more or less java, it is actually compiable multi platform. I will try to build it myself once, if there is more to look into, I will try to invest some time this week. Can you also tell me which version or tag you are trying to build |
|
@dudo I have built the latest version 28.0.1 https://github.com/arm64-compat/apache-druid/pkgs/container/apache%2Fdruid/177604533?tag=28.0.1 you can try and use it for your local / staging setup. Please refrain from using it in production as I am just a maintainer of this repo / oreganization but I can't actively test these images and their features myself, I hope I am able to make your life easier :) |
|
I second this. Would be very handy if we could run an "out-of-box" Druid image on AWS Graviton. Makes a lot of sense both for small and large scale setups, especially if sharing a K8s cluster with other ARM workloads. |
|
I have built locally the latest version 30.0.0 but when I use it I face issues with middleManager container which crashes during ingestion. |
|
Any specific issue you are seeing? Maybe some error logs will help. |
|
No issue visible as my container is no longer visible when it crashes. Just exit code 137 |
|
OK, can you do a docker logs #container_id and provide some details there? It's hard to know what's crashing. |
|
Here the last lines of logs |
|
I don't see any errors in the logs. What's the underlying machine you are testing this on? I assume this is native batch ingestion? (index_parallel task)? I can try to reproduce it. |
|
MSQ ingestion using a parquet file. I launch 3 MSQ ingestions at the same time. |
|
My machine is an Apple M2 Max |
|
Dope, will try a repro with docker. |
|
An alternative, is to build using buildx for amd64 platform for all druid containers. |
|
I think I got an arm64 build partially working locally in docker, but running into some service discovery issues. Will continue to chip away at this. I'm also trying to figure out how docker images are published to docker hub from this project. Will report back as I make progress. |
|
Hey @2bethere did you manage to make any progress here? |
Description
apache/druidrepository on Dockerhub storesamd64images only.I'd like to suggest generating official
arm64images.Motivation
arm64workloads are becoming more and more popular. If possible, Docker images should also be available for this architecture.The text was updated successfully, but these errors were encountered: