STRLCPY/cdebug

Nixery support + README.md
Ivan Velichko committed 2 years ago

3a30a8b7

1 parent a2991727

■ ■ ■ ■ ■ ■

README.md

1		-	# cdebug - experimental container debugger
	1	+	# cdebug - experimental container debugger (WIP)
	2	+
	3	+	A handy way of troubleshooting containers lacking a shell and/or debugging tools
	4	+	(e.g, scratch, slim, or distroless).
2	5
3		-	Mostly for troubleshooting slim & distroless containers that lack a shell and other debugging tools. The technique is based on the ideas from this [blog post](https://iximiuz.com/en/posts/docker-debug-slim-containers).
	6	+	The `cdebug exec` command is some sort of crossbreeding of `docker exec` and `kubectl debug` commands.
	7	+	You point the tool at a running container, say what toolkit image to use, and it starts
	8	+	a debugging "sidecar" container that _feels_ like a `docker exec` session into the target container:
	9	+
	10	+	- The root filesystem of the debugger _is_ the root filesystem of the target container.
	11	+	- The target container isn't recreated and/or restarted.
	12	+	- No extra volumes or copying of debugging tools is needed.
	13	+	- The debugging tools _are_ available in the target container.
4	14
5		-	Work in progres...
	15	+	Currently supported toolkit images:
	16	+
	17	+	- `busybox` - a good default choice
	18	+	- `nixery.dev/shell/...` - [a very powerful way to assemble images on the fly](https://nixery.dev/).
	19	+
	20	+	## How it works
	21	+
	22	+	The technique is based on the ideas from this [blog post](https://iximiuz.com/en/posts/docker-debug-slim-containers).
	23	+	Oversimplifying, the debugger container is started like:
	24	+
	25	+	```sh
	26	+	docker run [-it] \
	27	+	--network container:<target> \
	28	+	--pid container:<target> \
	29	+	--uts container:<target> \
	30	+	<toolkit-image>
	31	+	sh -c <<EOF
	32	+	ln -s /proc/$$/root/bin/ /proc/1/root/.cdebug
6	33
7		-	## Demo
	34	+	export PATH=$PATH:/.cdebug
	35	+	chroot /proc/1/root sh
	36	+	EOF
	37	+	```
8	38
9		-	The command is very similar to `docker exec`. You point it to the target container,
10		-	potentially ask the session to be interactive (`-it`), and specify the debugging
11		-	toolkit image (`busybox` or anything starting from `nixery.dev/shell`).
	39	+	The secret sauce is the symlink + PATH modification + chroot-ing.
12	40
13		-	Important: The target container isn't recreated and/or restarted. And no extra
14		-	volumes is needed.
	41	+	## Demo 1: An interactive shell with busybox
15	42
16		-	Notice how the debugger's shell actually has the original distroless rootfs as its root directory:
	43	+	First, a target container is needed. Let's use a distroless nodejs image for that:
17	44
18	45		```sh
19		-	$ docker run -d --rm \
	46	+	docker run -d --rm \
20	47		--name my-distroless gcr.io/distroless/nodejs \
21	48		-e 'setTimeout(() => console.log("Done"), 99999999)'
	49	+	```
22	50
23		-	$ go run main.go exec -it my-distroless
24		-	{"status":"Pulling from library/busybox","id":"latest"}
25		-	{"status":"Digest: sha256:9810966b5f712084ea05bf28fc8ba2c8fb110baa2531a10e2da52c1efc504698"}
26		-	{"status":"Status: Image is up to date for busybox:latest"}
27		-	+ rm -rf /proc/1/root/.cdebug
28		-	+ ln -s /proc/55/root/bin/ /proc/1/root/.cdebug
29		-	+ export 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/.cdebug'
30		-	+ chroot /proc/1/root sh
31		-	/ # ls -lah
	51	+	Now, let's start an interactive shell (using busybox) into the above container:
	52	+
	53	+	```sh
	54	+	cdebug exec -it my-distroless
	55	+	```
	56	+
	57	+	Exploring the filesystem shows that it's a rootfs of the nodejs container:
	58	+
	59	+	```sh
	60	+	/ $# ls -lah
32	61		total 60K
33	62		drwxr-xr-x 1 root root 4.0K Oct 17 23:49 .
34	63		drwxr-xr-x 1 root root 4.0K Oct 17 23:49 ..
35		-	lrwxrwxrwx 1 root root 18 Oct 17 23:49 .cdebug -> /proc/55/root/bin/
	64	+	�� lrwxrwxrwx 1 root root 18 Oct 17 23:49 .cdebug-c153d669 -> /proc/55/root/bin/
36	65		-rwxr-xr-x 1 root root 0 Oct 17 19:49 .dockerenv
37	66		drwxr-xr-x 2 root root 4.0K Jan 1 1970 bin
38	67		drwxr-xr-x 2 root root 4.0K Jan 1 1970 boot
		skipped 3 lines
42	71		drwxr-xr-x 1 root root 4.0K Jan 1 1970 lib
43	72		drwxr-xr-x 2 root root 4.0K Jan 1 1970 lib64
44	73		drwxr-xr-x 5 root root 4.0K Jan 1 1970 nodejs
45		-	dr-xr-xr-x 191 root root 0 Oct 17 19:49 proc
46		-	drwx------ 1 root root 4.0K Oct 17 19:55 root
47		-	drwxr-xr-x 2 root root 4.0K Jan 1 1970 run
48		-	drwxr-xr-x 2 root root 4.0K Jan 1 1970 sbin
49		-	dr-xr-xr-x 13 root root 0 Oct 17 19:49 sys
50		-	drwxrwxrwt 2 root root 4.0K Jan 1 1970 tmp
51		-	drwxr-xr-x 1 root root 4.0K Jan 1 1970 usr
52		-	drwxr-xr-x 1 root root 4.0K Jan 1 1970 var
	74	+	...
	75	+	```
	76	+
	77	+	Notice the 👉 emoji above - that's where the debugging tools live:
	78	+
	79	+	```sh
	80	+	/ $# echo $PATH
	81	+	/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/.cdebug-c153d669
	82	+	```
	83	+
	84	+	The process tree is also common:
	85	+
	86	+	```sh
	87	+	/ # ps auxf
	88	+	PID USER TIME COMMAND
	89	+	1 root 0:00 /nodejs/bin/node -e setTimeout(() => console.log("Done"),
	90	+	13 root 0:00 sh -c set -euo pipefail sleep 999999999 & SANDBOX_PID=$!
	91	+	19 root 0:00 sleep 999999999
	92	+	21 root 0:00 sh
	93	+	28 root 0:00 [sleep]
	94	+	39 root 0:00 [sleep]
	95	+	45 root 0:00 ps auxf
	96	+	```
	97	+
	98	+	## Demo 2: An interactive shell with advanced tools
	99	+
	100	+	If the tools provided by busybox aren't enough, you can bring your own tools with
	101	+	a ~~little~~ huge help of the [nixery](https://nixery.dev/) project:
	102	+
	103	+	```sh
	104	+	cdebug exec -it --image nixery.dev/shell/ps/findutils/tcpdump my-distroless
53	105		```
54	106

■ ■ ■ ■ ■ ■

cmd/exec/exec.go

1	1		package exec
2	2
3	3		import (
	4	+	"bytes"
4	5		"context"
5	6		"fmt"
	7	+	"html/template"
6	8		"io"
7	9		"strings"
8	10
		skipped 77 lines
86	88		return cmd
87	89		}
88	90
89		-	const chrootProgramBusybox = `
	91	+	var (
	92	+	// NOTE: Using $$ (current process PID) instead of ${SANDBOX_PID} breaks
	93	+	// (at least) the nixery program - the /proc symlinks become invalid
	94	+	// while chroot-ing and the operation cannot be finished (execve fails
	95	+	// with ENOENT, likely because of the missing/misplaced ELF interpreter).
	96	+	chrootProgramBusybox = template.Must(template.New("busybox-chroot").Parse(`
	97	+	set -euo pipefail
	98	+
	99	+	sleep 999999999 &
	100	+	SANDBOX_PID=$!
	101	+
	102	+	ln -s /proc/${SANDBOX_PID}/root/bin/ /proc/1/root/.cdebug-{{ .ID }}
	103	+
	104	+	export PATH=$PATH:/.cdebug-{{ .ID }}
	105	+
	106	+	chroot /proc/1/root sh
	107	+	`))
	108	+
	109	+	chrootProgramNixery = template.Must(template.New("nixery-chroot").Parse(`
90	110		set -euxo pipefail
91	111
92		-	rm -rf /proc/1/root/.cdebug
93		-	ln -s /proc/$$/root/bin/ /proc/1/root/.cdebug
94		-	export PATH=$PATH:/.cdebug
	112	+	sleep 999999999 &
	113	+	SANDBOX_PID=$!
	114	+
	115	+	rm -rf /proc/1/root/nix
	116	+	ln -s /proc/${SANDBOX_PID}/root/nix /proc/1/root/nix
	117	+
	118	+	ln -s /proc/${SANDBOX_PID}/root/bin /proc/1/root/.cdebug-{{ .ID }}
	119	+
	120	+	export PATH=$PATH:/.cdebug-{{ .ID }}
	121	+
95	122		chroot /proc/1/root sh
96		-	`
	123	+	`))
	124	+	)
97	125
98	126		func runDebugger(ctx context.Context, cli cmd.CLI, opts *options) error {
99	127		if err := cli.InputStream().CheckTty(opts.stdin, opts.tty); err != nil {
		skipped 12 lines
112	140		return err
113	141		}
114	142
	143	+	runID := shortID()
115	144		target := "container:" + opts.target
116	145		resp, err := client.ContainerCreate(
117	146		ctx,
118	147		&container.Config{
119	148		Image: opts.image,
120		-	Cmd: []string{"sh", "-c", chrootProgramBusybox},
	149	+	Cmd: []string{
	150	+	"sh",
	151	+	"-c",
	152	+	mustRenderTemplate(func() *template.Template {
	153	+	if strings.Contains(opts.image, "nixery") {
	154	+	return chrootProgramNixery
	155	+	}
	156	+	return chrootProgramBusybox
	157	+	}(), map[string]string{"ID": runID}),
	158	+	},
121	159		// AttachStdin: true,
122	160		OpenStdin: opts.stdin,
123	161		Tty: opts.tty,
		skipped 6 lines
130	168		},
131	169		nil,
132	170		nil,
133		-	debuggerName(opts.name),
	171	+	debuggerName(opts.name, runID),
134	172		)
135	173		if err != nil {
136	174		return fmt.Errorf("cannot create debugger container: %w", err)
		skipped 133 lines
270	308		return nil
271	309		}
272	310
273		-	func debuggerName(name string) string {
	311	+	func debuggerName(name string, runID string) string {
274	312		if len(name) > 0 {
275	313		return name
276	314		}
	315	+	return "cdebug-" + runID
	316	+	}
	317	+
	318	+	func shortID() string {
	319	+	return strings.Split(uuid.NewString(), "-")[0]
	320	+	}
277	321
278		-	return "cdebug-" + strings.Split(uuid.NewString(), "-")[0]
	322	+	func mustRenderTemplate(t *template.Template, data any) string {
	323	+	var buf bytes.Buffer
	324	+	if err := t.Execute(&buf, data); err != nil {
	325	+	panic(fmt.Errorf("cannot render template %q: %w", t.Name(), err))
	326	+	}
	327	+	return buf.String()
279	328		}
280	329

Nixery support + README.md