Add agentless mode to debug target pod when there isn't an agent running on target host #31

tkanng · 2019-05-18T05:23:46Z

Hi aylei, this PR adds agentless mode to debug target container when there isn't agent pod running on the targetHost. In agentless mode, we don't need to deploy agent daemonset any more.

In this mode, we first start agent pod on target node, and then start debugging container. Both agent pod and debug container will be deleted when debug process exists.The size of debug-agent image is about 30MB, so the preparation time for starting agent pod might not be too long when the network is OK.

I've tested some cases without agent daemonset deployed:

agentless mode when port-forward mode is on

root@iZhp37kmiszbkwzt5oh9csZ:/kq# ./kubectl-debug nginx  --agent-less=true  --port-foward=true
Agent Pod info : 
	Name:debug-agent-pod-izhp3gki207s34o84d5nluz
	Namespace:default
	HostPort:10027

2019/05/18 13:01:21 waiting for container running...
pod nginx PodIP 10.42.1.2, agentPodIP 10.10.251.242
wait for forward port to debug agent ready...
Forwarding from 127.0.0.1:10027 -> 10027
Handling connection for 10027
                             pulling image nicolaka/netshoot:latest... 
latest: Pulling from nicolaka/netshoot
Digest: sha256:5b1f5d66c4fa48a931ff54f2f34e5771eff2bc5e615fef441d5858e30e9bb921
Status: Image is up to date for nicolaka/netshoot:latest
starting debug container...
container created, open tty...
bash-5.0#

agentless mode when port-forward mode is off

root@iZhp37kmiszbkwzt5oh9csZ:/kq# ./kubectl-debug nginx  --agent-less=true 
Agent Pod info : 
	Name:debug-agent-pod-izhp3gki207s34o84d5nluz
	Namespace:default
	HostPort:10027

2019/05/18 13:00:44 waiting for container running...
pulling image nicolaka/netshoot:latest... 
latest: Pulling from nicolaka/netshoot
Digest: sha256:5b1f5d66c4fa48a931ff54f2f34e5771eff2bc5e615fef441d5858e30e9bb921
Status: Image is up to date for nicolaka/netshoot:latest
starting debug container...
container created, open tty...
bash-5.0#

agentless mode when fork mode is on

root@iZhp37kmiszbkwzt5oh9csZ:/kq# ./kubectl-debug nginx  --agent-less=true  --port-foward=true --fork=true
Agent Pod info : 
	Name:debug-agent-pod-izhp3gki207s34o84d5nluz
	Namespace:default
	HostPort:10027

2019/05/18 13:02:29 waiting for container running...
2019/05/18 13:02:35 waiting for container running...
pod nginx PodIP 10.42.1.48, agentPodIP 10.10.251.242
wait for forward port to debug agent ready...
Forwarding from 127.0.0.1:10027 -> 10027
Handling connection for 10027
                             pulling image nicolaka/netshoot:latest... 
latest: Pulling from nicolaka/netshoot
Digest: sha256:5b1f5d66c4fa48a931ff54f2f34e5771eff2bc5e615fef441d5858e30e9bb921
Status: Image is up to date for nicolaka/netshoot:latest
starting debug container...
container created, open tty...
bash-5.0#

In agentless mode ,it also supports changing agent port dynamically when port-forward mode is off, just like this：

root@iZhp37kmiszbkwzt5oh9csZ:/kq# ./kubectl-debug nginx  --agent-less=true  --fork=true --port=10000
Agent Pod info : 
	Name:debug-agent-pod-izhp3gki207s34o84d5nluz
	Namespace:default
	HostPort:10000

2019/05/18 13:05:12 waiting for container running...
2019/05/18 13:05:19 waiting for container running...
pulling image nicolaka/netshoot:latest... 
latest: Pulling from nicolaka/netshoot
Digest: sha256:5b1f5d66c4fa48a931ff54f2f34e5771eff2bc5e615fef441d5858e30e9bb921
Status: Image is up to date for nicolaka/netshoot:latest
starting debug container...
container created, open tty...
bash-5.0#

In port-forward mode, it doesn't support changing agent port dynamically, just like this:

root@iZhp37kmiszbkwzt5oh9csZ:/kq# ./kubectl-debug nginx  --agent-less=true  --port-foward=true --fork=true --port=10000
Agent Pod info : 
	Name:debug-agent-pod-izhp3gki207s34o84d5nluz
	Namespace:default
	HostPort:10000

2019/05/18 13:11:17 waiting for container running...
2019/05/18 13:11:23 waiting for container running...
pod nginx PodIP 10.42.1.52, agentPodIP 10.10.251.242
wait for forward port to debug agent ready...
Forwarding from 127.0.0.1:10000 -> 10000  
Handling connection for 10000
                             E0518 13:11:25.953111    3607 portforward.go:391] an error occurred forwarding 10000 -> 10000: error forwarding port 10000 to pod 7ca26882144aea50595d148212f1f4929b934bbb58ef0a8419d1eba73763c4ff, uid : exit status 1: 2019/05/18 05:11:25 socat[3991] E connect(5, AF=2 127.0.0.1:10000, 16): Connection refused
                                                                                                         delete forked pod.....
                                                                                                                               delete agent pod.....
                                                                                                                                                    end port-forward...
                                                                                                                                                                       error execute remote, error sending request: Post http://localhost:10000/api/v1/debug?command=%5B%22bash%22%5D&container=docker%3A%2F%2F9fec37c64da810b42c1c069e9e850ce06e8ddc37f582e43e5c89116007183ff1&image=nicolaka%2Fnetshoot%3Alatest: unexpected EOF
error: error sending request: Post http://localhost:10000/api/v1/debug?command=%5B%22bash%22%5D&container=docker%3A%2F%2F9fec37c64da810b42c1c069e9e850ce06e8ddc37f582e43e5c89116007183ff1&image=nicolaka%2Fnetshoot%3Alatest: unexpected EOF
root@iZhp37kmiszbkwzt5oh9csZ:/kq#

Agent pod listens on port 10027(inside container) and on port 10000(outside container), so kubectl-debug fails to connect pod's port 10000(inside container). In the future, we can fix this by adding some parameter to config agent listening port and config agent-pod yaml respectively. :)

…ing on the host.

Build debug-agent image from distroless image (aylei#30)

aylei · 2019-05-18T08:13:36Z

Looks awesome! I will give a review ASAP ;)

aylei

Thank you for your excellent work! It is generally looks good to me. I've left some comments to discuss the codes that we may want to refine it now.

aylei · 2019-05-20T03:49:08Z

pkg/plugin/cmd.go

@@ -386,6 +479,7 @@ func (o *DebugOptions) Run() error {
 	withCleanUp := func() error {
 		return interrupt.Chain(nil, func() {
 			if o.Fork {
+				fmt.Println("delete forked pod.....")


Prefer fmt.Sprintln(o, "delete forked pod.....")

Sorry, I made a typo in the comment. It should be fmt.Fprintln

aylei · 2019-05-20T03:49:20Z

pkg/plugin/cmd.go

@@ -399,6 +493,15 @@ func (o *DebugOptions) Run() error {
 					close(o.StopChannel)
 				}
 			}
+			// delete agent pod
+			if o.AgentLess && agentPod != nil {
+				fmt.Println("delete agent pod.....")


aylei · 2019-05-20T03:50:11Z

pkg/plugin/cmd.go

+	log.Println("waiting for container running...")
+	event, err := watch.UntilWithoutRetry(ctx, watcher, conditions.PodRunning)
+	if err != nil {
+		fmt.Printf("w:%v", err)


ditto, and what is w mean? maybe more specific

aylei · 2019-05-20T03:50:23Z

pkg/plugin/cmd.go

+		return nil, err
+	}
+
+	fmt.Printf("Agent Pod info : \n\tName:%s\n\tNamespace:%s\n\tHostPort:%d\n\n", agentPod.GetObjectMeta().GetName(), agentPod.GetObjectMeta().GetNamespace(), o.AgentPort)


aylei · 2019-05-20T03:51:17Z

pkg/plugin/cmd.go

@@ -149,6 +202,14 @@ func NewDebugCmd(streams genericclioptions.IOStreams) *cobra.Command {
 		"Debug agent daemonset name when using port-forward")
 	cmd.Flags().StringVar(&opts.DebugAgentNamespace, "daemonset-ns", opts.DebugAgentNamespace,
 		"Debug agent namespace, default to 'default'")
+	// flags used for agentless mode.
+	cmd.Flags().BoolVar(&opts.AgentLess, "agent-less", false, "Whether to turn on agentless mode. Agentless mode: debug target pod if there isn't an agent running on the target host.")


"agentless" is more accurate

And could we add a shorthand for this flag?

aylei · 2019-05-20T03:56:01Z

pkg/plugin/cmd.go

+	defaultAgentImage         = "aylei/debug-agent:latest"
+	defaultAgentPodNamePrefix = "debug-agent-pod-"
+	defaultAgentPodNamespace  = "default"
+	defaultAgentPodTemplate   = `


For type safety and flexibility, can we construct the Pod object via the client-go interface instead of YAML template?

tkanng · 2019-05-20T04:24:24Z

Thanks! I'll refine it ASAP! :)

tkanng · 2019-05-20T17:06:19Z

Hi aylei，the latest commit constructs the agent pod via client-go and refines the way to print message. But I'm not sure the way is good enough, so please help me check it again. By the way, I'm not skilled in Golang, so could you please tell me why fmt.Sprintln is better? Code clean or something else? Thanks a lot.

aylei

LGTM

aylei · 2019-05-21T03:40:11Z

@tkanng
Sorry, it should be fmt.Fprintln, but log is also make sense, AFAIK, the differences are:

fmt.Print*: print to os.Stdout.
fmt.Fprint*: print to the io.Writer the first parameter indicates, in our CLI application, the In, Out, Err stream is abstracted in IOStream for flexibility. And the IOStream is passed in by the caller in the NewDebugCmd function. So we prefer using fmt.Fprint(o, ...) in order to use the io.Writer passed in instead of os.Stdout
log.Print*: print to os.Stderr by default, which can be redirected. More over, log will carry timestamp and can be easily filtered using verbose level or other mechanism

So, generally, if some message is ought to notify the end user, it should use fmt.Fprint(writer, ...), and if it's the runtime information that helps user to better understand the process, it's better to use log.Print(...)

tkanng · 2019-05-21T05:33:24Z

Got it! Thank you very much!
The latest commit uses fmt.Fprint* to print message, instead of log. :)

aylei · 2019-05-21T11:06:01Z

Thank you!

tkanng and others added 4 commits May 13, 2019 12:26

Build debug-agent image from distroless image

379139e

Add agentless mode to debug target pod when there isn't an agent runn…

72d8bd0

…ing on the host.

Merge pull request #1 from aylei/master

42bf746

Build debug-agent image from distroless image (aylei#30)

fix typo

dc6e051

aylei reviewed May 20, 2019

View reviewed changes

Construct pod via client-go and refine the format of output message

8ffd9ef

aylei approved these changes May 21, 2019

View reviewed changes

Use fmt.Fprint* to notify end user

c62b5b9

aylei merged commit ef547de into aylei:master May 21, 2019

aylei mentioned this pull request May 22, 2019

Document the agentless mode #32

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add agentless mode to debug target pod when there isn't an agent running on target host #31

Add agentless mode to debug target pod when there isn't an agent running on target host #31

tkanng commented May 18, 2019

aylei commented May 18, 2019

aylei left a comment

aylei May 20, 2019

aylei May 21, 2019

aylei May 20, 2019

aylei May 20, 2019

aylei May 20, 2019

aylei May 20, 2019

aylei May 20, 2019

aylei May 20, 2019

tkanng commented May 20, 2019

tkanng commented May 20, 2019

aylei left a comment

aylei commented May 21, 2019

tkanng commented May 21, 2019

aylei commented May 21, 2019

Add agentless mode to debug target pod when there isn't an agent running on target host #31

Add agentless mode to debug target pod when there isn't an agent running on target host #31

Conversation

tkanng commented May 18, 2019

aylei commented May 18, 2019

aylei left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tkanng commented May 20, 2019

tkanng commented May 20, 2019

aylei left a comment

Choose a reason for hiding this comment

aylei commented May 21, 2019

tkanng commented May 21, 2019

aylei commented May 21, 2019