Projects STRLCPY cdebug Files
🤬
Revision indexing in progress... (search in this revision will be accurate after indexed)
Enable build support by adding .buildspec.yml
assets/images Loading last commit info...
cmd
e2e/portforward
pkg
.gitignore
.goreleaser.yaml
LICENSE
Makefile
README.md
go.mod
go.sum
main.go
README.md

cdebug - a swiss army knife of container debugging (WIP)

This project is looking for supporters 👀

With this tool you can:

  • Troubleshoot containers lacking shell and/or debugging tools
  • Forward unpublished or even localhost ports to your host system
  • Expose endpoints from the host system to containers & Kubernetes networks
  • Handily export image's and/or container's filesystem to local folders
  • and more :)

The following commands x runtimes are supported:

DockerContainerdKubernetesKubernetes CRIrunc
exec🛠️---
port-forward local----
port-forward remote🛠️-🛠️--
export-----

Installation

It's a statically linked Go binary, so you know the drill:

GOOS=linux
GOARCH=amd64

curl -Ls https://github.com/iximiuz/cdebug/releases/latest/download/cdebug_${GOOS}_${GOARCH}.tar.gz | tar xvz

sudo mv cdebug /usr/local/bin

At the moment, the following systems are (kinda sorta) supported:

  • linux/amd64
  • darwin/amd64
  • darwin/arm64

Commands

cdebug exec

Run an interactive shell in a scratch, slim, or distroless container, with ease:

cdebug exec -it <target-container>

The cdebug exec command is a crossbreeding of docker exec and kubectl debug commands. You point the tool at a running container, say what toolkit image to use, and it starts a debugging "sidecar" container that feels like a docker exec session to the target container:

  • The root filesystem of the debugger is the root filesystem of the target container.
  • The target container isn't recreated and/or restarted.
  • No extra volumes or copying of debugging tools is needed.
  • The debugging tools are available in the target container.

By default, the busybox:latest image is used for the debugger sidecar, but you can override it with the --image flag. Combining this with the superpower of Nix and Nixery, you can get all your favorite tools by simply listing them in the image name:

cdebug exec -it --image nixery.dev/shell/ps/vim/tshark <target-container>
How it works

The technique is based on the ideas from this blog post.

How: cdebug exec

Oversimplifying, the debugger container is started like:

docker run [-it] \
  --network container:<target> \
  --pid container:<target> \
  --uts container:<target> \
  <toolkit-image>
  sh -c <<EOF
ln -s /proc/$$/root/bin/ /proc/1/root/.cdebug

export PATH=$PATH:/.cdebug
chroot /proc/1/root sh
EOF

The secret sauce is the symlink + PATH modification + chroot-ing.

cdebug port-forward

Forward local ports to containers and vice versa. This command is another crossbreeding - this time it's kubectl port-forward and ssh -L|-R.

Currently, only local port forwarding (cdebug port-forward -L) is supported, but remote port forwarding is under active development.

Local port forwarding use cases (works for Docker Desktop too!):

  • Publish "unpublished" port 80 to a random port on the host: cdebug port-forward <target> -L 80
  • Expose container's localhost to the host system: cdebug port-forward <target> -L 127.0.0.1:5432
  • Proxy local traffic to a remote host via the target: cdebug port-forward <target> -L <LOCAL_HOST>:<LOCAL_PORT>:<REMOTE_HOST>:<REMOTE_PORT>
  • 🛠️ Expose a Kubernetes service to the host system: cdebug port-forward <target> -L 8888:my.svc.cluster.local:443

Remote port forwarding use cases:

  • Start a container/Pod forwarding traffic destined to its <IP>:<port> to a non-cluster endpoint reachable from the host system.
  • ...
How it works

Local port forwarding is implemented by starting an extra forwarder container in the target's network and publishing its ports to the host using the standard means (e.g., docker run --publish). The forwarder container itself runs something like:

socat TCP-LISTEN:<REMOTE_PORT>,fork TCP-CONNECT:<REMOTE_HOST>:<REMOTE_PORT>

How: cdebug port-forward -L (direct)

If the REMOTE_HOST doesn't belong to the target or it's the target's localhost, an extra sidecar container is started in the target's network namespace with another socat forwarding traffic from the target public interface to REMOTE_HOST:REMOTE_PORT.

How: cdebug port-forward -L (sidecar)

Remote port forwarding will use similar tricks but combined with more advanced reverse tunneling.

Demos

Below are a few popular scenarios formatted as reproducible demos.

A simple interactive shell to a distroless container

First, a target container is needed. Let's use a distroless nodejs image for that:

$ docker run -d --rm \
  --name my-distroless gcr.io/distroless/nodejs \
  -e 'setTimeout(() => console.log("Done"), 99999999)'

Now, let's start an interactive shell (using busybox) into the above container:

$ cdebug exec -it my-distroless

Exploring the filesystem shows that it's a rootfs of the nodejs container:

/ $# ls -lah
total 60K
drwxr-xr-x    1 root     root        4.0K Oct 17 23:49 .
drwxr-xr-x    1 root     root        4.0K Oct 17 23:49 ..
👉 lrwxrwxrwx 1 root     root          18 Oct 17 23:49 .cdebug-c153d669 -> /proc/55/root/bin/
-rwxr-xr-x    1 root     root           0 Oct 17 19:49 .dockerenv
drwxr-xr-x    2 root     root        4.0K Jan  1  1970 bin
drwxr-xr-x    2 root     root        4.0K Jan  1  1970 boot
drwxr-xr-x    5 root     root         340 Oct 17 19:49 dev
drwxr-xr-x    1 root     root        4.0K Oct 17 19:49 etc
drwxr-xr-x    3 nonroot  nonroot     4.0K Jan  1  1970 home
drwxr-xr-x    1 root     root        4.0K Jan  1  1970 lib
drwxr-xr-x    2 root     root        4.0K Jan  1  1970 lib64
drwxr-xr-x    5 root     root        4.0K Jan  1  1970 nodejs
...

Notice 👉 above - that's where the debugging tools live:

/ $# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/.cdebug-c153d669

The process tree of the debugger container is the process tree of the target:

/ $# ps auxf
PID   USER     TIME  COMMAND
    1 root      0:00 /nodejs/bin/node -e setTimeout(() => console.log("Done"),
   13 root      0:00 sh -c  set -euo pipefail  sleep 999999999 & SANDBOX_PID=$!
   19 root      0:00 sleep 999999999
   21 root      0:00 sh
   28 root      0:00 [sleep]
   39 root      0:00 [sleep]
   45 root      0:00 ps auxf

An interactive shell with code editor (vim)

If the tools provided by busybox aren't enough, you can bring your own tools with a little huge help of the nixery project:

$ cdebug exec -it --image nixery.dev/shell/vim my-distroless

An interactive shell with tshark and other advanced tools

Even more powerful exammple:

$ cdebug exec -it --image nixery.dev/shell/ps/findutils/tshark my-distroless

Publish "forgotten" port

Start an nginx container but don't expose its port 80:

$ docker run -d --name nginx-1 nginx:1.23

Forward local port 8080 to the nginx's 80:

$ cdebug port-forward nginx-1 -L 8080:80
$ curl localhost:8080

Expose localhost's ports

Start a containerized service that listens only on its localhost:

$ docker run -d --name svc-1 python:3-alpine python3 -m 'http.server' -b 127.0.0.1 8888

Tap into the above service:

$ cdebug port-forward svc-1 -L 127.0.0.1:8888
Pulling forwarder image...
latest: Pulling from shell/socat
Digest: sha256:b43b6cf8d22615616b13c744b8ff525f5f6c0ca6c11b37fa3832a951ebb3c20c
Status: Image is up to date for nixery.dev/shell/socat:latest
Forwarding 127.0.0.1:49176 to 127.0.0.1:8888 through 172.17.0.4:34128

$ curl localhost:49176
<!DOCTYPE HTML>
<html lang="en">
<head>
...

F.A.Q

Q: Running cdebug exec fails with rm: cannot remove '/proc/1/root/nix': Permission denied or ln: /proc/1/root/.cdebug-XXXXXXXX: Permission denied.

Chances are your target container has been started with elevated permissions while you're trying to run a non-privileged debugger sidecar. Try cdebug exec --privileged instead.

Similar tools

  • docker-slim debug - a PoC debug command for DockerSlim (contributed by D4N)
  • debug-ctr - a debugger that creates a new container out of the original container with the toolkit mounted in a volume (by Felipe Cruz Martinez)
  • docker-debug - much like cdebug exec but without the chroot trick.
  • docker-opener - a multi-purpose tool that in particular can run a shell session into your container (and if there is no shell inside, it'll bring its own busybox).
  • cntr - is "a replacement for docker exec that brings all your developers tools with you" by mounting the file system from one container (or the host) into the target container and creating a nested container with the help of a FUSE filesystem. Supports a huge range of runtimes (docker, podman, LXC/LXD, rkt, systemd-nspawn, containerd) because it operates on the OS level.
  • kdiag - a kubectl plugin to get shell access to scratch containers, stream logs from multiple pods simultaneously, and do reverse port forwarding to Kubernetes clusters.

TODO:

  • More exec flags (like in docker run): --cap-add, --cap-drop, --env, --volume, etc.
  • Helper command(s) suggesting nix(ery) packages
  • Non-docker runtimes (containerd, runc, k8s)
  • E2E Tests

Contributions

It's a pre-alpha with no sound design yet, so I may not be accepting all PRs. Sorry about that :)

Please wait...
Page is in error, reload to recover