Support building Docker images #42

viesti · 2019-06-28T21:11:42Z

Adds support for building to Docker image directly via Jib. Uses the jib-core API and supports local Docker daemon, tarball and remote registries.

SevereOverfl0w

A few gotchas to watch out for

SevereOverfl0w · 2019-06-29T07:06:45Z

src/mach/pack/alpha/jib.clj

+(defn jib [{::tools-deps/keys [paths lib-map]} {:keys [image-name image-type tar-file base-image target-dir main]}]
+  (println "Building" image-name)
+  (-> (Jib/from base-image)
+      (.addLayer (into []


I think libraries need to be run through elodin, otherwise:

[foo/core "1.0.0"], and [bar/core "1.0.0"] both have the filename "core-1.0.0.jar", which means you miss one from the classpath (or get an exception, not sure what jib does)

Git dependencies are likely to use src as their folder name, this means conflicts again, but also that you have no identification of where they came from.

Thank you! Now I understand the need for elodin :)

The name is a bit silly, it's a bit of a joke referencing a book, the king killer Chronicle.

SevereOverfl0w · 2019-06-29T07:22:15Z

src/mach/pack/alpha/jib.clj

+                             paths)
+                     (.build)))
+      (.setEntrypoint (into-array String ["java"
+                                          "-cp" (str/join ":" (cons (str target-dir "/*")


Unfortunately, using "*" only utilizes .jar files, and ignores directories (which would be brought in by git dependencies).

This is a frustrating limitation I've run into before.

Nice, thanks for the tip! :)

viesti · 2019-06-29T09:40:09Z

Thanks for the comments, I'll fix these and test a bit more with libs with same artifact name and git libs.

I also thought that a progress bar (like docker build has) would be good especially while working with remote registries. Seems that this could be done with jib-core events. Thinking that progrock might be a good dependency-free library for rendering a progress bar.

SevereOverfl0w · 2019-06-29T09:51:00Z

No problem with a progress bar! Especially if this is a little slow.

So that paths to directories or libraries inside the container would not collide.

Status bar can be disabled via -q Verbose prints can be enable via -v

viesti · 2019-06-29T19:06:58Z

Added path munging for library jars, dirs and project dirs, and added a progress bar (can be disabled with -q) together with --verbose switch for logs provided by jib. Will do some testing next to see if things work :)

SevereOverfl0w

Looking great.

I think it might be worth adding support for -J and -O (either now or in the future) to this builder, what do you think?

viesti · 2019-06-30T11:42:10Z

Thanks! Some thoughts below (sorry for long post :) tl;dr, maybe -J and -O in the future).

At first I thought -J and -O would be great to support, but reading jib-faq seems to suggest using JAVA_TOOL_OPTIONS for JVM flags (via docker run -e JAVA_TOOL_OPTIONS=-X...). In past projects I've kept JVM options in deployment scripts, for example memory limits decided by RAM size of the target machine via Ansible, not in project.clj. Still thinking that -J and -Owould be nice for passing good defaults to the container (say minimum memory). I think this can be added later.

Another thing that I noticed in jib-faq was Java agents. The Maven/Gradle plugins have option to add files relative to container root, which helps when making a -javaagent:/path-to/agent.jar option. -e could work for this, but the container path is not user defined. Maybe a way to specify files and container paths could be added later.

Jib packages to /app directory, I'll change to this from /home/app to not give impression that we create a user.

A user coud be set via setUser and seems that distroless images have a nonroot user. This opens up a question on what options to support from JibContainerBuilder (maybe a Jib specific key config key in deps.edn or in separate config file/map), but this seems like furher development item.

SevereOverfl0w · 2019-06-30T12:01:00Z

I agree that I wouldn't primarily keep these settings in deps.edn. My general approach (and the reason pack doesn't have a pack.edn yet) is to create a ./pkg with my default flags embedded.

If we support JAVA_TOOL_OPTIONS then I'm more than happy. It was just an omission I was worried about.

I agree that -e needs to be different for this runner. Is the src:dest format that docker uses appropriate here?

Glad to see alignment with jib as much as possible. Thanks for the deep dive and reference.

viesti · 2019-06-30T13:12:24Z

Thanks for the guidance! I hope I'm not swamping you with this :)

Maybe we need both -e for extra classpath paths and another option for src:dest style files, for user defined container paths. I think the docker style [src:]dest could be neat. Maybe --include [src:]dest, which could be specified multiple times if tools.cli allows (later in pack.edn :)) and if the path is a directory, it would be recursively copied.

viesti · 2019-06-30T13:56:36Z

Went ahead and tried out --include, seems to work even recursively :).

SevereOverfl0w · 2019-06-30T15:21:02Z

Fantastic, there's something great here. This has been really easy to review, thank you for your contribution.

I'll have another look over before I merge, are you happy that this is finished now?

viesti · 2019-06-30T16:09:33Z

It has been a pleasure to work on this, thank you :)

Sounds great, I think I'm ok with the current state, would be good if someone else also tries this out to get feedback. Ideas discussed in the PR are good further development material :)

viesti · 2019-07-03T17:42:26Z

Hi!

I was thinking that a demo like this might be neat, maybe linked from readme or from a tweet, since it's a bit lengthy:

0% cat deps.edn
{:paths ["src"]
 :deps {ring/ring-jetty-adapter {:mvn/version "1.7.1"}}}
0% cat src/run_app/main.clj
(ns run-app.main
  (:require [ring.adapter.jetty :as jetty]))

(defn handler [_]
  {:status  200
   :headers {"Content-Type" "text/plain"}
   :body    "Hello World"})

(defn -main [& _]
  (jetty/run-jetty handler
                   {:port (or (some-> (System/getenv "PORT")
                                      (Integer/parseInt))
                              3000)}))
0% # First deploy is slower, since base layer needs to be pushed
0% clj -A:pack mach.pack.alpha.jib --image-name eu.gcr.io/tiuhti/run-app --image-type registry -m run-app.main
Building eu.gcr.io/tiuhti/run-app
[================================================= ]  98/100  00:37

0% gcloud beta run deploy run-app --image eu.gcr.io/tiuhti/run-app --region us-central1
Allow unauthenticated invocations to [run-app] (y/N)?  y

Deploying container to Cloud Run service [run-app] in project [tiuhti] region [us-central1]
✓ Deploying new service... Done.
  ✓ Creating Revision...
  ✓ Routing traffic...
  ✓ Setting IAM Policy...
Done.
Service [run-app] revision [run-app-00999eb5-debc-483a-9531-569dc2ff0f14] has been deployed and is serving traffic at https://run-app-uia3uyq3da-uc.a.run.app
0% curl https://run-app-uia3uyq3da-uc.a.run.app
Hello World

0% # Subsequent deploys are fast, since only changes in layers are transferred
0% sed -i "" -e 's/World/World\!/' src/run_app/main.clj
0% clj -A:pack mach.pack.alpha.jib --image-name eu.gcr.io/tiuhti/run-app --image-type registry -m run-app.main
Building eu.gcr.io/tiuhti/run-app
[===================================================] 103/100  00:06

0% gcloud beta run deploy run-app --image eu.gcr.io/tiuhti/run-app --region us-central1
Deploying container to Cloud Run service [run-app] in project [tiuhti] region [us-central1]
✓ Deploying... Done.
  ✓ Creating Revision...
  ✓ Routing traffic...
Done.
Service [run-app] revision [run-app-00002] has been deployed and is serving traffic at https://run-app-uia3uyq3da-uc.a.run.app
0% curl https://run-app-uia3uyq3da-uc.a.run.app
Hello World!

SevereOverfl0w · 2019-07-05T07:39:13Z

That demo would make a good https://asciinema.org/ and those link quite nicely in github readmes.

viesti and others added 4 commits June 28, 2019 14:11

Add Jib, package libs only for now

61e8334

Add project files, support tar, docker and registry

efa2bdd

Add documentation for Jib

2c2168f

Add GCR example

ec72af0

viesti mentioned this pull request Jun 28, 2019

Docker images via Jib #41

Closed

SevereOverfl0w reviewed Jun 29, 2019

View reviewed changes

viesti added 3 commits June 29, 2019 18:36

Use unique names for lib jars, dirs & project dirs

447943d

So that paths to directories or libraries inside the container would not collide.

Check for required arguments

f99b13e

Add status bar and verbose status prints

2077fca

Status bar can be disabled via -q Verbose prints can be enable via -v

SevereOverfl0w approved these changes Jun 30, 2019

View reviewed changes

Package to /app inside the container image

1d8d9b5

Add files relative to container root via --include

5bd2ae5

SevereOverfl0w approved these changes Jul 5, 2019

View reviewed changes

SevereOverfl0w merged commit 2769a62 into juxt:master Jul 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support building Docker images #42

Support building Docker images #42

viesti commented Jun 28, 2019

SevereOverfl0w left a comment

SevereOverfl0w Jun 29, 2019

viesti Jun 29, 2019

SevereOverfl0w Jun 29, 2019

SevereOverfl0w Jun 29, 2019

viesti Jun 29, 2019

viesti commented Jun 29, 2019

SevereOverfl0w commented Jun 29, 2019

viesti commented Jun 29, 2019

SevereOverfl0w left a comment

viesti commented Jun 30, 2019

SevereOverfl0w commented Jun 30, 2019

viesti commented Jun 30, 2019

viesti commented Jun 30, 2019

SevereOverfl0w commented Jun 30, 2019

viesti commented Jun 30, 2019

viesti commented Jul 3, 2019

SevereOverfl0w commented Jul 5, 2019

Support building Docker images #42

Support building Docker images #42

Conversation

viesti commented Jun 28, 2019

SevereOverfl0w left a comment

Choose a reason for hiding this comment

SevereOverfl0w Jun 29, 2019

Choose a reason for hiding this comment

viesti Jun 29, 2019

Choose a reason for hiding this comment

SevereOverfl0w Jun 29, 2019

Choose a reason for hiding this comment

SevereOverfl0w Jun 29, 2019

Choose a reason for hiding this comment

viesti Jun 29, 2019

Choose a reason for hiding this comment

viesti commented Jun 29, 2019

SevereOverfl0w commented Jun 29, 2019

viesti commented Jun 29, 2019

SevereOverfl0w left a comment

Choose a reason for hiding this comment

viesti commented Jun 30, 2019

SevereOverfl0w commented Jun 30, 2019

viesti commented Jun 30, 2019

viesti commented Jun 30, 2019

SevereOverfl0w commented Jun 30, 2019

viesti commented Jun 30, 2019

viesti commented Jul 3, 2019

SevereOverfl0w commented Jul 5, 2019