feat: allow injecting through stdin tar.gz on kaniko

[JordanGoasdoue]

Fixes #350

Description

feat: allow injecting through stdin tar.gz on kaniko

The idea is explained on the Issue.

Note_1:

I've decided to name this context tar://stdin because it's data coming from standard input, and we only want this in tar gz format.

Note_2:

As kaniko is running in container, it's not possible to differentiate terminal from pipe stdin.
Then, if it's run without piped data, with interactive mode activated and without tty, the process will wait for EOF signal to continue (Ctrl+D). I couldn't get away from this. But it can only happen if the context is tar://stdin and the kaniko is run in interactive mode and without any piped data.

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

• Includes unit tests
• Adds integration tests if needed.

See the contribution guide for more details.

Reviewer Notes

• The code flow looks good.
• Unit tests and or integration tests added.

Release Notes

Describe any changes here so maintainer can include it in the release notes, or delete this block.

• kaniko allow a new build context tar://stdin, to use with data catched from standard input only in .tar.gz format

[tejal29]

@JordanGoasdoue do you mean after?

[JordanGoasdoue]

@JordanGoasdoue do you mean after?

If you are talking about the order of PR to merge.

I meant here :
Start to merge this one first : #1115

Then Merge this second one once it's approuved and only when the first one is merged : #1139

This PR needed the previous one, because i've decided to put the STDIN case inside the local TAR one. Here instead of having a stdin build context by itself, stdin is a sub case of local tar.

[tejal29]

no worries, i think github will take care of merging correctly. I am going threough the process of verifying all approved candidate releases and merged the other one.

[tejal29]

This is great feature!!
I know in skaffold, we have an init container which blocks until a file is present. Behind the scenes, we do a docker cp in to running container the context.tar.
With the --interactive flag, we don't need to do this anymore. As mentioned in the linked issue #350 this is
also useful for other scenarios.

I really need time to test and run this in interactive mode in a k8 container.
If you are happy to provide us some steps or detailed logs i can get this in quicker.

@samos123 would you be up for testing this?

[tejal29]

Suggested change

	echo 'FROM alpine \nRUN echo "created from standard input"' > Dockerfile | tar -cf - Dockerfile | gzip -9 | docker run \
	echo -e 'FROM alpine \nRUN echo "created from standard input"' > Dockerfile | tar -cf - Dockerfile | gzip -9 | docker run \

[tejal29]

i ran this in docker and it looks good.

[JordanGoasdoue]

Thanks @tejal29 , This is how I got it to work with k8s after some tries :

I used kubernetes in docker and a local registry in docker too.

~/kaniko_stdin> ls
Dockerfile_STDIN  kaniko.yaml

my pod file kaniko.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: kaniko
spec:
  containers:
  - name: kaniko
    image: localhost:5000/executor:latest
    args: ["--dockerfile=Dockerfile_STDIN",
            "--context=tar://stdin",
            "--no-push"]
    stdin: true
  restartPolicy: Never

My Dockerfile Dockerfile_STDIN:

FROM alpine
RUN echo "created from standard input"

1. k apply -f kaniko.yaml
2. if you k logs your pod, it will be forever in To simulate EOF and exit, press 'Ctrl+D' state here but it's not a problem, I see this as a template POD to allow us to exec it with the right input on the next part 3)

~/kaniko_stdin> kl -f kaniko                                                                                            
INFO[0000] To simulate EOF and exit, press 'Ctrl+D'

3. tar -cf - Dockerfile_STDIN | gzip -9 | kubectl exec -i kaniko -- /kaniko/executor --context=tar://stdin --dockerfile=Dockerfile_STDIN --no-push

4. Return :

~/kaniko_stdin> tar -cf - Dockerfile_STDIN | gzip -9 | kubectl exec -i kaniko -- /kaniko/executor --context=tar://stdin --dockerfile=Dockerfile_STDIN --no-push
INFO[0000] To simulate EOF and exit, press 'Ctrl+D'     
INFO[0000] Resolved base name alpine to alpine          
INFO[0000] Resolved base name alpine to alpine          
INFO[0000] Retrieving image manifest alpine             
INFO[0002] Retrieving image manifest alpine             
INFO[0004] Built cross stage deps: map[]                
INFO[0004] Retrieving image manifest alpine             
INFO[0005] Retrieving image manifest alpine             
INFO[0007] Unpacking rootfs as cmd RUN echo "created from standard input" requires it. 
INFO[0011] Taking snapshot of full filesystem...        
INFO[0011] Resolving paths                              
INFO[0011] RUN echo "created from standard input"       
INFO[0011] cmd: /bin/sh                                 
INFO[0011] args: [-c echo "created from standard input"] 
created from standard input
INFO[0011] Taking snapshot of full filesystem...        
INFO[0011] Resolving paths                              
INFO[0011] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0011] Skipping push to container registry due to --no-push flag

I hope it will help you testing
Cheers

[tejal29]

Thanks @JordanGoasdoue i will give it a spin

[tejal29]

I tested the flow and everything looks great!

[gfvirga]

@JordanGoasdoue, we really appreciate that you got this feature supported and I've been using it in our pipelines. Do you happen to know if there is a way to make it more dynamic? I have a pipeline that creates 5 dockerbuilds at the same time and I need to create 5 kaniko pods with different names, because running kubectl exec on one pod is error'ing out.

[JordanGoasdoue]

Thank you @gfvirga,
I appreciate this is useful for you and your team indeed.

About your question, if I understand it well, I believe it is similar with the issue linked below :

#1118

The kaniko maintainer explains there that currently kaniko isn't designed yet to support re-usable pod, but it will hopefully in futur as a contributor already assigned the issue to him, but this is a big task because it needs to work for all possible use case.

I hope I've answered your question here.

[gfvirga]

Thank you for your response, I got what I need to work with the following:
$ tar -cf - * | gzip -9 | kubectl run $(RANDOMNAME) --rm --stdin=true --image=gcr.io/kaniko-project/executor:debug --restart=Never -- --dockerfile=Dockerfile -c=tar://stdin -d myregistry.com/testcontainer:kaniko

Now it generates a temporary pod, builds and deletes it the temporary pod. Love it!

[JordanGoasdoue]

Very nice @gfvirga, I'm glad it works well for you now, feel free to open a PR to update the README with this kubernetes part, I'm sure it will help others with same use case.