Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Introduction

telemetry-blackbox

CI status on main OpenSSF Scorecard Line coverage across Rust and Python Latest release

A flight recorder for a simulated flight controller. Rust firmware runs a 1 kHz control loop and streams MAVLink telemetry over a serial link. A Rust gateway, Redpanda and a Spate pipeline carry every frame into ClickHouse, on a highly available Kubernetes cluster I built by hand.

Status: under construction, in the open. Progress is tracked in the milestones, and the docs site updates with every push.

Progress

Flight progress strips, one per leg. The flight log has the same detail as text.

Leg 0 Leg 1 Leg 2 Leg 3 Leg 4

I fly the route in evenings after work, so each leg has a budget in evenings and a due date to match. The cluster flies before the firmware because it carries the most risk. #62 has the reasoning.

Each leg is a strip in a rack, the way air traffic control tracks flights. The leg I’m flying sticks out, a late leg is outlined in red and a white tick marks where the plan says I should be. Headwind counts the work that crept in after takeoff, and a stamp unlocks at each milestone I care most about. The flight log has the detail.

Why I’m building it

I want to understand four things from the bottom up:

  • Kubernetes run on my own machines, including the parts that break.
  • Rust, both no_std on a microcontroller and async on a server.
  • Real-time embedded systems and the protocols between them.
  • Streaming data into ClickHouse with Spate.

A black box is a good way to learn all four at once. It has a hard deadline at one end, a database at the other and a simple test in between. Did every frame arrive?

The claim

Cut the serial link, kill the pipeline, drain a node or lose a control plane in the middle of a flight. Every frame still reaches ClickHouse. Some arrive twice, because delivery is at-least-once, and ClickHouse removes the duplicates by key. bbcheck measures the outcome after each drill, and the numbers go in Results.

How it fits together

Each part has one job.

PartBuilt withJob
firmware/Rust, no_std, EmbassyRuns the control loop on a hardware timer and sends telemetry over UART
gateway/Rust, tokioFinds frame boundaries, checks CRCs, spots sequence gaps and publishes to Redpanda
pipeline/Rust, SpateDecodes frames, routes them into tables and loads ClickHouse
deploy/cloud-init, kubeadm, YAMLBuilds the VMs and the cluster, then runs everything on it
tools/bbcheck/PythonProves what arrived: gaps, duplicates and latency

The cluster

Five VMs on one machine. Three control planes share a virtual IP, so the API stays up when one of them goes down. Two workers carry the workloads, and a node label decides which one owns the serial link.

Renode runs on the host, outside the cluster, because it stands in for a physical device. On real hardware the gateway would read /dev/ttyUSB0. Here it reads the TCP socket Renode exposes.

The life of one frame

The last step matters most. Spate only commits an offset once ClickHouse has confirmed the batch is durable, so a crash can repeat work but never skip it. That is where duplicates come from, and why the table removes them by (boot_id, system_id, seq).

The wire format

Telemetry uses MAVLink v2. Every frame carries a sequence number, which is what lets the gateway and bbcheck count gaps.

BytesFieldNotes
1STXAlways 0xFD in v2
1lenPayload length
1incompat_flagsBit 0 marks a signed frame
1compat_flagsSafe to ignore if unknown
1seqWraps at 255; a jump means lost frames
1sysidWhich vehicle
1compidWhich component on it
3msgid24-bit message ID
lenpayloadTrailing zero bytes are trimmed
2checksumCRC-16/MCRF4XX, seeded with the message’s CRC_EXTRA
13signatureOnly on signed frames

The firmware sends HEARTBEAT, ATTITUDE, HIGHRES_IMU and a custom LOOP_TIMING message that reports overruns and jitter.

Real-time, honestly

The control loop is hard real-time inside the simulation. Renode runs in deterministic virtual time, so loop timing reflects the firmware alone.

Everything after the serial port is soft real-time. I measure it as end-to-end latency percentiles, from the firmware’s timestamp to the row landing in ClickHouse.

Running it

Still being built. When it’s done, it will look like this:

mise install    # pinned tools: just, D2, uv, kubectl, Helm and more
just bench      # check everything is installed
just lint       # formatting and lints for the crates, firmware and bbcheck
just test       # the tests, plus a firmware build
just coverage   # line coverage across Rust and Python
just cluster    # five VMs and a highly available cluster
just deploy     # Redpanda, ClickHouse, Grafana, gateway and pipeline
just fly        # boot the firmware in Renode and start streaming
just check      # run bbcheck against ClickHouse
just docs       # build the docs site into book/
just audit      # scan the history for secrets

just on its own lists every recipe. You’ll also need Linux with KVM, about 11 GB of free memory, mise, Multipass, Renode, probe-rs, the ClickHouse client and Rust with the thumbv7em-none-eabihf target.

A released version

Every release puts multi-arch images of the gateway and pipeline on GitHub Container Registry. Once the firmware has a binary, it’s attached to the release too. The changelog says what each one changed.

docker pull ghcr.io/guy-with-a-why/telemetry-blackbox/gateway:latest
docker pull ghcr.io/guy-with-a-why/telemetry-blackbox/pipeline:latest
gh attestation verify oci://ghcr.io/guy-with-a-why/telemetry-blackbox/gateway:latest \
  -R guy-with-a-why/telemetry-blackbox

Each image carries a provenance attestation, so the last command proves it was built here, from a tagged commit.

Repository layout

firmware/        Embassy firmware for the STM32F4
renode/          the board Renode simulates, as a script
gateway/         serial link to Redpanda
pipeline/        Spate pipeline into ClickHouse
deploy/          VMs, cluster bootstrap and manifests
tools/bbcheck/   delivery and latency report
docs/adr/        decisions and the reasons behind them
docs/brand/      the logo, favicon, social preview and how to use them
docs/diagrams/   D2 sources and the SVGs they render
docs/runbooks/   drills, with real output
scripts/         small checks the justfile and CI call

Decisions

Each gets a short record in docs/adr/.

  1. MAVLink v2 over a custom framing.
  2. kubeadm over k3s or Talos.
  3. Redpanda over Apache Iggy or a home-made log.
  4. Deduplicating in ClickHouse over chasing exactly-once.
  5. just and mise over make, and a local registry for images.

Results

These fill in as the drills run.

DrillFrames sentMissingDuplicatesRecovery
Kill the pipelinependingpendingpendingpending
Drain the pipeline’s nodependingpendingpendingpending
Restart Redpandapendingpendingpendingpending
Cut the serial link for 60 spendingpendingpendingpending
Lose a control planependingpendingpendingpending
Scale the pipeline to two replicaspendingpendingpendingpending

Diagrams

Every diagram is a D2 file in docs/diagrams/. D2 lays it out, and each SVG follows the viewer’s light or dark theme. The animated ones step through boards declared in the same file.

To change one, edit the .d2 file and watch it live, then render the lot:

d2 --watch --layout=elk docs/diagrams/architecture.d2
just diagrams

Cite

These are the documents I need to build this, grouped by where I use them. The board Renode simulates is ST’s STM32F4DISCOVERY, which is built on the STM32F407VG, a Cortex-M4F.

Tooling (Leg 0)

Microcontroller (Leg 2)

Protocol (Legs 2 and 3)

Cluster (Leg 1)

Data path (Legs 1 and 3)

Diagrams

Licence

Dual-licensed under MIT or Apache 2.0, at your option. Any contribution you submit is licensed the same way, unless you say otherwise.

Flight log

19 of 43 issues closed (44%). 9 joined after takeoff. Stretch: 7 of 21.

Flight progress strips, one per leg. The lists below give the same detail as text.

Rebuilt whenever an issue changes. Last update: 2026-11-18 21:09 UTC.

Leg 0 · The bench

landed, due 2026-11-12

  • #40 Release versioned increments and publish them from CI (landed 2026-11-13)
  • #37 Decide on just, mise and a local registry (landed 2026-11-13)
  • #2 Scaffold the workspace and CI (landed 2026-11-12)
  • #1 Install the bench tools (landed 2026-11-12)
  • #44 Show build health and assurance badges in the README (landed 2026-11-12)
  • #39 Track progress with a flight strip (landed 2026-11-12)
  • #38 Check every push for secrets and blocked terms (landed 2026-11-12)
  • #35 Publish the docs site and keep it current (landed 2026-11-12)

Leg 1 · A cluster I built myself

in flight, due 2026-11-20

  • #14 Run Redpanda on local storage (landed 2026-11-18)
  • #13 Drill: snapshot and restore etcd (landed 2026-11-17)
  • #12 Drill: lose one control plane, then two (landed 2026-11-17)
  • #105 Bootstrap the cluster with one command (landed 2026-11-15)
  • #11 Bootstrap a highly available control plane by hand (landed 2026-11-15)
  • #82 Mirror quay.io and ghcr.io on the host, so Cilium and kube-vip pull once (landed 2026-11-14)
  • #75 Cache packages and images on the host, so a rebuild doesn't refetch them (landed 2026-11-14)
  • #10 Build five VMs with cloud-init (landed 2026-11-14)
  • #15 Run ClickHouse with raw, deduplicated and rolled-up tables
  • #16 Decide on kubeadm and on Redpanda
  • #17 Checkpoint: explain quorum, static Pods and MergeTree merges

Leg 2 · Firmware and the wire

started early, due 2026-11-24

  • #3 Boot Embassy firmware on an STM32F4 in Renode (landed 2026-11-14)
  • #62 Re-plan the route in evenings (landed 2026-11-14)
  • #56 Keep a daily blog on the docs site (landed 2026-11-13)
  • #4 Drive the control loop from a hardware timer at 1 kHz
  • #5 Estimate attitude from a simulated IMU
  • #6 Send MAVLink v2 telemetry over UART with DMA
  • #7 Read the telemetry on the host with pymavlink
  • #8 Decide on MAVLink v2 over a custom framing
  • #9 Checkpoint: explain the firmware and the frame format
  • #61 Test the firmware in Renode with Robot Framework

Leg 3 · Serial port to ClickHouse

not started, due 2026-11-30

  • #18 Find MAVLink frames in a noisy byte stream
  • #19 Publish frames to Redpanda without losing any
  • #20 Run the gateway on the node that owns the serial link
  • #21 Stream frames from Redpanda into ClickHouse with Spate
  • #22 Route attitude, IMU and timing into their own tables
  • #23 Fly end to end, from firmware to Grafana
  • #24 Decide to deduplicate in ClickHouse over chasing exactly-once
  • #25 Checkpoint: explain a frame's whole path
  • #107 Deliver the workloads from Git with Flux

Leg 4 · Break it and prove it

not started, due 2026-12-04

  • #26 Report gaps, duplicates and latency with bbcheck
  • #27 Drill: break the pipeline while it streams
  • #28 Build Grafana dashboards for the whole path
  • #29 Finish the README and record a demo
  • #30 Checkpoint: walk through the system and defend the decisions

Stretch

bonus

  • #116 Run only the CI a change needs, and merge stacks in one go (landed 2026-11-17)
  • #47 Scan the code with CodeQL (landed 2026-11-16)
  • #46 Keep pinned actions current with Dependabot (landed 2026-11-16)
  • #45 Choose a licence (landed 2026-11-16)
  • #95 Draw how the flight tracker works (landed 2026-11-15)
  • #90 Draw a narrow flight rack for phones (landed 2026-11-14)
  • #86 Redraw the flight strip as a rack, one strip per leg (landed 2026-11-14)
  • #31 Test firmware on the cluster from GitHub Actions
  • #32 Write an Apache Iggy source for Spate
  • #33 Add a CAN bus alongside MAVLink
  • #34 Queue work by priority with Kueue
  • #36 Stream metrics to a public Grafana dashboard
  • #102 Serve the metrics API with verified kubelet certificates
  • #124 Provision volumes on demand with local-path-provisioner
  • #130 Rebuild the platform on Talos, OpenTofu and Flux
  • #131 Decide on Talos, OpenTofu and Flux for the finished platform
  • #132 Provision the VMs with OpenTofu on libvirt
  • #133 Run Talos on the nodes
  • #134 Run the host services from Compose
  • #135 Move the cluster checks and etcd backups onto standard tools
  • #136 Turn the checks into Chainsaw tests

How this page is drawn

The diagram is wide, so here it is full size.

Cockpit voice recorder

A flight recorder has two halves. The flight log is the data recorder. This is the voice: a short post after each evening’s flying, on what I built and what broke, and a debrief when each leg lands. Posts start from _template.md and debriefs from _debrief.md.

Day 0: Laying out the bench

12 November 2026

What was done

No firmware today, and that was the plan. By the evening the repo was public and every push was checked for secrets and blocked terms. The docs site rebuilds whenever an issue changes, and a flight strip on it shows how far through the plan I am. The Rust workspace, bbcheck and CI were scaffolded by half nine, so Day 1 starts on firmware straight away.

Why was it done that way

The repo went public before any code, so the checks had to come first. gitleaks scans the whole history. A blocked-terms check covers files, commit messages and issue text, and GitHub’s push protection sits behind both. That’s cheap on day one and expensive to retrofit.

The diagrams are D2 because they stay as text I can diff and they render in light and dark themes. Tasks go through just, since they’re commands with arguments rather than builds.

Half an hour after takeoff I moved takeoff, so the issues I’d added that afternoon now count against me on the flight strip. Technically it is scope creep.

What challenges were faced

Two small things. Five minutes after the site went live, the dark logo turned out to be unreadable on the light theme, so that was the first fix.

The more interesting one was in the flight log. flight.py stitched gh’s pages of issues together by swapping ][ for ],[, which would also have mangled any issue title containing ][. Editing JSON as text is a shortcut I know better than to take. gh api --slurp already returns a single array, so the stitching went and a test with the title [a][b] keeps it that way (#43). Nobody hit it.

What’s next

Tomorrow is firmware. The riskiest part is getting Embassy and Renode to agree on the board, so that goes first. Every day of the plan has a fallback if it runs long.

Flight plan amended

14 November 2026

What was done

I’ve re-planned the route. The plan had five Days of nine hours each, 09:00 to 18:00, and I fly this in the evenings after a full working day. A Day was always going to be three or four evenings. The Days are now legs. Each has a budget in evenings and a due date that assumes weekday evenings only, and the last leg lands on 4 December. This timeline is documented in the table in #62.

Why was it done that way

A plan that’s late from its second day stops being a plan. By Monday the flight strip would have shown every waypoint red and told me nothing useful. Budgets in evenings make the strip honest again, and a weekend counts as time in hand.

The order changed too. The cluster now flies before the firmware, because it carries the most risk on the route and everything downstream runs on it. Five VMs have to share a 16 GB machine, and I bootstrap the control plane by hand. The firmware and the cluster don’t meet until the data path leg, so flying the riskier one first costs nothing and puts the slack where I need it.

What challenges were faced

Day 0 was planned at an hour. It took two evenings, about eight and a half hours. The bench itself was quick. Releases, README badges, this blog and a security policy all joined the scope on the way. Each was worth having and none of them fitted in the hour. The flight strip counted them as headwind, which is what it calls work added after takeoff. From here, new scope goes to Stretch unless it blocks a leg.

What’s next

Today #3 lands. The firmware already boots in Renode and blinks the board’s green LED, and I’m checking its memory map against the STM32F407VG datasheet before it merges. Once I’m happy with that the next focus will be the cluster, with four evenings to Friday 20 November. The biggest risk is memory, because I’m building the whole platform on one 16 GB laptop on purpose. If five VMs don’t fit in 16 GB, I drop to one control plane on a VM and run the HA drills on kind, which runs a whole multi-node cluster in Docker for a fraction of the memory.

Ground power: first boot in Renode

14 November 2026

What was done

The firmware boots. It’s an Embassy binary for the STM32F407VG, running on the STM32F4DISCOVERY board that Renode simulates, and it blinks the green LED on PD12 every 500 ms. just fly builds it, boots the board and shows Renode logging each change of the LED. That closes #3.

The binary is small: a 392-byte vector table, 3.4 KB of code and 96 bytes of RAM. cargo bloat puts 1.5 KB of the code in the main task, where Embassy’s chip setup and my blink loop are inlined, and 1.1 KB in the TIM2 timer that wakes it.

Why was it done that way

I wrote the memory map by hand rather than let Embassy generate it. It’s the contract between the linker and the silicon, so I want it in the repo where I can see it. It has two regions from ST’s RM0090 reference manual: 1 MiB of flash at 0x0800_0000, where the core boots, and 128 KiB of SRAM at 0x2000_0000.

The chip’s 64 KiB of core-coupled memory is left out on purpose. CCM is RAM wired straight to the CPU, so the DMA controllers can’t reach it. #6 sends telemetry over the UART by DMA from a static buffer, and if the linker put that buffer in CCM, the transfer would fail while the CPU side looked healthy. Leaving CCM out of the map rules that out, in Renode and on real silicon.

Embassy keeps time on TIM2, chosen explicitly so the other timers stay free for the 1 kHz loop in #4. The clocks stay on the 16 MHz internal oscillator, which asks the least of Renode’s partial clock model. Speed can wait until there’s a loop to measure.

What challenges were faced

Renode was the first wrong turn. The Debian package expects .NET 8 and two other dependencies on the host, and dpkg left it unconfigured. The portable build bundles its own runtime and was already working, so I removed the package.

The rest was reading the binary to check it matched the map. The stack starts at 0x2002_0000, the top of RAM, and the reset vector at 0x0800_0189 points at cortex-m-rt’s reset handler with the Thumb bit set. That handler zeroes 72 bytes of .bss and copies 24 bytes of .data before main, and the Embedonomicon explains why. The zeroing matters here. From #6 the MAVLink sequence counter lives in a static, and if it started at a random value, bbcheck would report missing frames that were never sent.

What’s next

The cluster flies next, with four evenings to Friday 20 November. The firmware returns with #4, which moves the loop onto a hardware timer at 1 kHz.

The biggest firmware risk is the gap between Renode and silicon. The CCM case is one example, and ST’s ES0182 errata sheet lists faults Renode won’t reproduce. My fallback is to tie timing claims to Renode’s virtual time, where they’re deterministic, and mark anything the simulation can’t prove as one for real hardware.

The hangar: five VMs and two caches

14 November 2026

What was done

The cluster has somewhere to live. just vms builds five Ubuntu VMs with Multipass, all from one cloud-init file: three control planes, edge-1 and data-1. Each node comes up with containerd and kubeadm 1.36, one minor behind the latest, so there’s an upgrade to practise later. A local registry carries my images in, and just vms-check proves every node is ready for kubeadm.

The first full build took 18 minutes. Caches on the host bring that down to 6m 09s once they’re warm.

Both pieces are up for review. #74 builds the VMs and closes #10. #78 adds the caches and is stacked on it.

Why was it done that way

I started on my laptop and moved within the hour. The VMs need 10.5 GB of memory, and the laptop had 6 GB free and a disk at 95%. My home server has 16 GB and room to spare, but only four cores for ten vCPUs. That’s fine at idle, and two vCPUs per node is the floor kubeadm checks for.

The control planes get 1.5 GB, under the 1.7 GB kubeadm’s preflight asks for, so #11 skips that one check. The figure is generic: etcd stays small in a five-node cluster, and the memory does more good on data-1, where Redpanda and ClickHouse will run.

I weighed the infrastructure-as-code options before writing any of it. Terraform earns its keep with a cloud API and state worth tracking, and Multipass has neither a first-party provider nor state I care about. Ansible suits long-lived hosts patched in place, and these VMs are cattle: if one drifts, I delete it and rebuild it in six minutes. So cloud-init sets each node up once at boot, and a short shell script drives Multipass.

All five share one Multipass zone. Multipass gives each zone its own subnet, with the host routing between them. kube-vip’s leader claims the API’s virtual IP by answering ARP for it, and ARP only reaches machines on the same subnet. Spread the control planes across zones and the address can’t move to a survivor when the leader fails.

When the first build took 18 minutes, I measured before changing anything. cloud-init’s own timings put most of each node’s 3m 20s in package downloads, and the server’s connection tops out at 1.1 MB/s. Each VM fetched the same 169 MB, so five of them pulled 845 MB of identical bytes. Parallel launches would only have split that connection five ways. A cache fixes that: the first VM fetches, and the rest read from the host. A registry.k8s.io mirror does the same for the control-plane images, which kubeadm will pull on three nodes in #11.

What challenges were faced

containerd was the first surprise. Each node finds my registry through a small file under certs.d, which says that images named localhost:5000 come from the host over plain HTTP. Ubuntu now ships containerd 2.2, where Kubernetes pulls go through a new transfer service that never reads that file. So every pull went to https://localhost:5000 on the node itself and failed. Pulling the same image two ways narrowed it down. ctr, handed the file directly, worked. crictl, which pulls the way the kubelet does, didn’t. Setting use_local_image_pull = true sends Kubernetes pulls back through the older path, which reads the file.

The apt cache took three attempts. Proxy mode refused the Kubernetes repository with a 403, because apt-cacher-ng reads the colons in pkgs.k8s.io paths as a port. Its shortcut paths failed in milliseconds with a 503. I traced its system calls and found it looking up a host called ubuntu in DNS. Putting the real host first in the path, as in http://cache:3142/pkgs.k8s.io/core:/stable:/v1.36/deb/, got past the 403 and the 503. The kubectl package then took 11.6 seconds through the cache the first time and 0.02 the second. The signing key still comes straight from upstream, so the cache can’t swap it.

What’s next

#11 is next: a highly available control plane, built by hand with kubeadm, kube-vip and Cilium. The virtual IP has to sit outside the range Multipass hands out, so that gets checked first.

The biggest risk is 1.5 GB per control plane. If one runs short of memory, I’ll move memory from data-1 before giving up a node. If that isn’t enough, the fallback is one control plane on a VM and the HA drills on kind.

Engine start: a control plane built by hand

15 November 2026

What was done

I bootstrapped the cluster this afternoon. It’s a Sunday, so it came out of the time in hand. I typed every kubeadm command myself on my home server, because that’s the point of #11. Forty-seven minutes after kubeadm init on cp-1, just cluster-check came back clean. Three control planes answer behind one virtual IP, 10.97.0.250, two workers have joined and Cilium carries the pod network.

A kubeadm bootstrap starts like an airliner’s engines. They come on one at a time in a fixed order, and the first runs on ground power. The cluster even keeps a record of it. Every Pod’s age counts from when it started, so reading the AGE column oldest first replays the afternoon. cp-1 came up at 12:53, cp-3 joined eight minutes later, then cp-2, then both workers. Cilium went on about fifteen minutes in.

This is how things looked a minute or two after that. The control plane was running on all three nodes, but every node was NotReady and CoreDNS was stuck in Pending. That’s expected. A node only reports Ready once a network plugin has written its config to /etc/cni/net.d, and Cilium writes that file when its agent starts. The agents were still waiting for their images.

The ages show more than the order. kube-vip-cp-1 is a minute younger than etcd-cp-1, because the one edit I made after init restarted it. On cp-3, kube-vip is younger than everything else on the node, because a node has no admin.conf until it has joined. A couple of hours later, all 34 Pods in kube-system were Running. The restarts on cp-1 are a story for further down.

Why was it done that way

kube-vip is the first engine, and it needs ground power. kubeadm init reaches the API through the virtual IP, so kube-vip has to hold that address while init runs. It claims the address by taking a Lease from the API server, and that needs credentials. Since Kubernetes 1.29, admin.conf gets its rights from a binding that init only creates once the API server is up. So on cp-1, kube-vip starts on super-admin.conf, which is in the system:masters group and skips RBAC entirely. When init finishes, I point kube-vip back at admin.conf and the cart comes off. The kube-vip docs cover the step.

Forgetting that last step would be easy, and nothing would tell me. kube-vip would carry on working with rights no binding grants and none can take away. That’s why cluster-check now looks for it. super-admin.conf lives only on cp-1, readable by root, and nothing runs on it now.

The whole control plane runs as static Pods, kube-vip included. A Deployment needs the API server to accept it, etcd to store it, the controller manager to create its Pods and the scheduler to place them. Those four are the control plane, so they can’t start that way. Instead, the kubelet runs whatever manifests sit in /etc/kubernetes/manifests, with no API involved, and keeps them running if the API goes down.

What kubectl shows for a static Pod is a mirror Pod, a read-only copy the kubelet posts to the API. Deleting it changes nothing on the node. I tried it on cp-2. The API server’s Pod came straight back with an age of nine seconds, the same container underneath and the node itself, Node/cp-2, as its owner. To change a static Pod, I edit its file, which is exactly what switching kube-vip to admin.conf was.

kubeadm also wrote the cluster’s trust into /etc/kubernetes/pki. There are three certificate authorities, each good for ten years, and leaf certificates that last a year. The main one, ca, signs for the API server, the other components and both admin kubeconfigs. etcd gets its own authority because it holds everything. Apart from etcd’s own members, only the API server and the health check carry a certificate etcd accepts. A leaked kubelet certificate would cost me a node and nothing more. The third authority covers the API server when it hands requests on to extension APIs, and sa.key signs the tokens ServiceAccounts carry. The PKI reference lists every file.

Mine expire on 15 November 2027. kubeadm upgrade renews the control plane’s certificates as it goes, and kubelets rotate their own, so a cluster upgraded once a year never sees one lapse.

etcd is stacked, one member on each control plane. External etcd would keep its failures apart from the control plane, but it needs three more VMs on a 16 GB box. Stacked ties each member to its control plane, and #12 will test what that costs. I went with stacked. Each API server talks only to its own member, on 127.0.0.1:2379, and the members agree through Raft, which counts a write once a majority has it.

Three is the smallest cluster that survives a failure. With two of three up, there’s still a majority, so writes carry on. Any two majorities share a member, so a new leader always has every committed write. A fourth member adds a vote but no tolerance, because the majority grows to three. Two members are worse than one. Either failure stops writes, and there’s twice the hardware to fail. The etcd FAQ has the arithmetic.

One call of my own sits under the pod network. Cilium’s default pod range, 10.0.0.0/8, covers the bridge the VMs live on. I could have given Cilium a range of its own, but then the pod range would live in two places. So Cilium takes each node’s range from Kubernetes, and 10.244.0.0/16 is set once, in kubeadm.yaml.

What challenges were faced

Before starting the failover drill, I read back through deploy/ and wasn’t sure kube-vip was set up properly. The manifest turned out to be fine. It matches what kube-vip v1.3.1 generates itself, apart from two settings that only matter when the address comes from DHCP or DNS. The real problem was simpler. There was no cluster to fail over. #11 had closed itself when #84 merged the evening before, with every box unticked, and the VMs had been rebuilt that evening without kubeadm ever running. I’d linked the PR in GitHub’s Development panel, and that closes an issue on merge, whatever the description says. I reopened it. Now I reference issues only in a pull request’s description, and I check that a pull request closes nothing it shouldn’t before it merges.

Going through the checks turned up a gap in my own proof. cluster-check confirmed one kube-vip held the address and never asked about the other two. A crashlooping standby would have passed, then failed the drill that matters. Nothing checked that cp-1 had come off super-admin.conf either. #100 now fails unless all three run kube-vip on admin.conf. It also fixes how I found kube-vip’s network interface. I took the third field of ip route get for the virtual IP, but on the node holding that IP the answer starts local 10.97.0.250 dev lo, and the third field is the word dev.

Cilium was slow to start. All five nodes pulled the same images through a quay.io mirror that had never seen them, on a line that tops out near 1 MB/s, and the agents waited over twenty minutes. The mirror pays off on the next rebuild.

The interesting bit came as Cilium finished. Three of its agents exited once as they started, and my home server was flat out. A few minutes later its load average read 10 on four threads. cp-1 stopped keeping up, and within twenty seconds it lost every leadership it held. etcd went first, at 13:36:37, when cp-3 won an election. Then kube-vip’s Lease went, then the controller manager’s and the scheduler’s. Each of those three stood down by exiting, and the kubelet started it again as a standby. Those are cp-1’s restarts in k9s.

kube-vip’s logs show the odd part. cp-3 took the address at 13:36:49, but cp-1 was so starved it only noticed its renewal had failed three seconds later. It then spent twelve seconds trying to hand the Lease back through its own slow API server, and only dropped the address at 13:37:09. For twenty seconds, two nodes answered for the same IP. The cluster came through it, and #12 now has a failover from the wild to measure itself against.

What’s next

#12 takes the control planes down on purpose, one and then two. With one down, etcd keeps its majority, and if that node held the address, kube-vip should move it within about fifteen seconds. With two down, writes stop. Every kube-vip renews its Lease through its own API server, so I expect the address to vanish, while Pods already on the workers keep serving. #13 follows with an etcd snapshot and restore. That leaves #12 to #17 for the five evenings to Friday.

The biggest risk is CPU. My home server is a 2012 laptop with an i5-3320M, battery still at 91%. At idle, with nothing of mine on it yet, the VMs hold it at about half its CPU. The three control planes take about 11% each and the workers around 5%. Memory comes second, with 4 GB free and 575 MiB already in swap. Redpanda and ClickHouse will want their share of both.

A longer Lease would make false alarms rarer, but it wouldn’t stop a starved node holding on to the address late. A bigger CPU share for the control-plane VMs would, so a busy data-1 can’t starve them. I’ll measure both in #12 and keep whichever the numbers back.

#11 closed this afternoon with every box ticked.

Quiz

Nine questions on the post, for anyone who wants to check it’s landed. Open one for my answer.

You delete kube-apiserver-cp-2 with kubectl. What happens on the node?

Nothing. I only deleted the mirror, the kubelet put it straight back and the container never stopped.

Why can't the control plane run as a Deployment?

A Deployment needs the API server, etcd, the scheduler and the controller manager to exist first. Those are the control plane, so the kubelet starts them from files on disk.

Why was every node NotReady until Cilium started?

The kubelet only reports Ready once a network plugin has written its config to /etc/cni/net.d. Cilium writes it when its agent starts.

Why does kube-vip start on super-admin.conf, and why move it off?

It has to take its Lease before admin.conf has any rights. super-admin.conf skips RBAC and nothing can revoke it, so it only stays until init finishes.

Why does etcd have its own certificate authority?

So a certificate from the main authority can’t open the database. Only the API server and the health check hold one that etcd accepts.

The certificates last a year. What keeps them from expiring?

kubeadm upgrade renews the control plane’s as it goes, and kubelets rotate their own. Upgrade at least once a year and nothing lapses.

Why run three etcd members and not four?

Three survives one failure. Four still only survives one, because the majority grows to three, and there’s more hardware to fail.

Two control planes go down. What happens to the virtual IP?

I expect it to vanish. Each kube-vip renews its Lease through its own API server, and without quorum nothing can renew it. #12 will tell me if I’m right.

cp-3 took the address at 13:36:49. Why could cp-1 still answer for it?

cp-1 was too starved to notice it had lost the Lease, then spent twelve seconds trying to hand it back through its own slow API server. Only then did it drop the address.

Autostart: the whole cluster in one command

15 November 2026

What was done

This afternoon I built the control plane by hand, and it took 47 minutes from kubeadm init to a clean check. This evening I made it one command. just up builds the five VMs, initialises cp-1, joins the other four nodes, installs Cilium and runs cluster-check. From nothing to a passing check took 20 minutes and 8 seconds, VMs included. Run it again on a healthy cluster and it skips every node in under a minute.

The engines still start in the same order, with cp-1 on ground power first. A jet’s autostart runs the same sequence a crew would, and the crew only has to press the button. #105 has the work and ADR 6 the reasoning.

Why was it done that way

If a sequence of actions is repeated by hand, and often, the system should do it instead. The drills in #12 and #13 break the cluster on purpose, so I’ll be rebuilding it several times a week.

Terraform is the usual way to stand up VMs, so I looked at it first. Its Multipass provider doesn’t support zones, and kube-vip needs all five VMs in one zone so they share a subnet. Ansible would have meant SSH keys and an inventory that copies my config, all for about ten commands. So kubeadm’s own config files carry the joins. The token and the certificate key are generated fresh on each run and set before init. The CA’s hash can’t be, because the CA doesn’t exist until init creates it, so the script reads it straight after. Every step checks before it acts, which is what makes a second run harmless.

What challenges were faced

The first full run hung at step 7, copying the kubeconfig off cp-1. multipass exec was writing straight into a file, and it sat at 99% CPU for twenty minutes without writing a byte. Every other call in the scripts reads the output into a variable first, so this one does too now. Rerunning on the half-built cluster finished the job and proved the skip path a little earlier than planned. Then I tore it all down and rebuilt it in one clean pass, so the 20 minutes is a single honest run.

What’s next

#12 takes the control planes down, one and then two, and #13 restores etcd from a snapshot. Either can now end in just up and twenty minutes.

Further out, what runs on the cluster should come from Git too. #107 puts Flux in charge of Redpanda, ClickHouse, the gateway, Spate and Grafana when Leg 3 brings them, so a rebuild restores the workloads as well as the cluster. Flux over Argo CD, because it’s a few small controllers and my workers have 2 GB and 4 GB. The risk is Flux undoing a drill’s damage as fast as I cause it, so the drill runbooks will suspend it first.

Deadstick: two control planes down

16 November 2026

What was done

Tonight I took the control planes down on purpose, one and then two, for #12. Losing cp-2 changed nothing: etcd kept a majority, two of three. cp-1 held the virtual IP all along, so this run never made kube-vip move it. Timing that move means stopping whichever control plane holds it.

Losing cp-3 as well took the majority with it. Writes stopped on the next probe. The API server kept answering cached reads for another 22 seconds, until kube-vip couldn’t renew its Lease and gave up the address. Yesterday’s post predicted that.

The echo Pods on the workers didn’t notice, and answered 79 of 79 probes. It’s a deadstick glide: nothing gets rescheduled or replaced, but everything already flying carries on, because running Pods and kube-proxy’s iptables rules don’t need the control plane. Starting cp-2 and cp-3 brought quorum back, and cluster-check passed once Cilium settled. The runbook has the output.

Why was it done that way

The drill needed a witness that doesn’t lean on what it’s watching. just drill-watch runs on the home server, outside the cluster, and logs every change in echo, the API and the address. It sends a consistent read, which needs quorum, and a cached read at resourceVersion=0, which doesn’t, so I can tell an API server that has lost etcd from one that has gone. Each VM goes down with multipass stop --force, which pulls the power, because a clean shutdown lets etcd hand over leadership first and real failures don’t. #111 has the script.

An infrastructure engineer I know asked me why all this runs on Bash instead of Terraform. Terraform reconciles a state file against an API, and its Multipass provider documents no zone option, which kube-vip needs. The drills break VMs by hand too, so the state would be wrong after every drill. The scripts check what’s there before they act, so the cluster is its own record. Terraform also stops at the machines, and the hard part is ten kubeadm commands in order across five of them. On real machines, or a hypervisor with an API such as Proxmox, Terraform for the machines and Ansible for kubeadm would earn their place, as ADR 6 says.

What challenges were faced

The first run ended early, for a reason that had nothing to do with etcd. With cp-2 and cp-3 down, I ran just cluster-check, and afterwards all five VMs were running. I hadn’t started any of them. Multipass’s log had cp-2 booting at 19:54:00 and cp-3 twenty seconds later. cluster-check finds a ready API server by running multipass exec on each control plane in turn. Without quorum cp-1 wasn’t ready, so it moved on to the stopped ones, and multipass exec starts a stopped VM first. Its help doesn’t say so. Its source does. The tool meant to report the outage ended it.

Both scripts now only touch running VMs, and the second run kept cp-2 and cp-3 down until I started them. #111 has the fix.

Afterwards I checked what happens when a worker goes instead. I cordoned edge-1 and deleted its echo Pod, and the replacement sat in Pending. echo’s spread rule allows at most one Pod more on any node than on the emptiest, and by default a node echo can’t run on still counts as empty. A second Pod on data-1, against none on edge-1, broke the rule, so the one place it could go was refused.

One line fixes it. nodeTaintsPolicy: Honor makes the rule count only nodes echo can run on, so a cordoned or dead worker and the control planes drop out of the sum. data-1 stops being measured against nodes it can’t use, and the replacement starts there. Without it, losing a worker would have cost echo half its Pods until the worker came back. The catch is that the scheduler never moves a Pod back, so once edge-1 returns, both stay on data-1 until something restarts one.

Applying that fix rolled echo out again and turned up a second gap: both new Pods landed on data-1. The rule still counted the old Pods on their way out, so data-1 never looked fuller than edge-1.

matchLabelKeys: [pod-template-hash] changes what gets counted: only Pods from the same rollout, which share a template hash. Pods from the last rollout stop steering the new ones, so every rollout ends with one Pod on each worker. Without it, the spread came down to the timing of the last rollout, and losing data-1 could have taken echo down entirely.

A re-run proved both fixes: the rollout left one Pod on each worker, and the cordon put the replacement on data-1 straight away. Neither gap shows while every node is healthy, which is what drills are for.

What’s next

#13 snapshots etcd, deletes a namespace and restores it. A restore rewinds the whole cluster, so every member has to start again from the same snapshot. The fallback is just up and twenty minutes. Leg 1 is due on Friday.

Quiz

Eleven questions on the post, for anyone who wants to check it’s landed. Open one for my answer.

cp-1's etcd was healthy and had every write. Why did it refuse to carry on alone?

It couldn’t tell dead peers from being cut off, and if cp-2 and cp-3 were alive, they were a majority that could keep writing. Two majorities always share a member, so only one side can write, and a write a majority holds survives into the next leader’s log. Raft explains how.

Writes stopped as soon as cp-3 went, but cached reads lasted 22 seconds. Why?

A plain read must be current, so it goes to etcd, which needs a majority. A read at resourceVersion=0 comes from the API server’s watch cache in memory, and that lasted until the address went.

Why does the virtual IP go when quorum does?

kube-vip renews its Lease through the API every 2 seconds and stands down after 10 without a renewal. Renewing and taking the Lease are both writes, so without quorum no kube-vip can hold the address.

The echo container crashes while the control plane is down. Does it come back?

Yes. The kubelet restarts it locally under restartPolicy: Always, with no API, and reports the new status once the API is back.

A worker dies while the control plane is down. What happens to its Pod and its traffic?

The Pod dies and nothing replaces it, because the ReplicaSet controller and the scheduler work through the API. Nobody removes it from the Service’s endpoints either, so about half the connections still go to it and time out.

One control plane stops. What does the cluster report, and when?

Its kubelet stops renewing its node Lease, which it does every 10 seconds, and after about 40 to 50 seconds the node goes NotReady with an unreachable taint. Ordinary Pods are evicted after 300 seconds, while DaemonSet Pods and static Pods stay with the node.

cp-2 came back before cp-3. Why could writes resume, and how does cp-3 catch up?

Two of three is a majority, so cp-1 and cp-2 elect a leader and commit a write once both have it. cp-3 is still a member, and the leader sends it what it missed as log entries, or as a snapshot if the log has moved on.

All three control planes run on one home server. What didn't the drill prove?

Anything that hits the host hits all three, such as CPU starvation or one slow disk under every etcd member. It also didn’t test a partition or a slow node, both harder than a clean stop. In production the three go in separate failure domains, and kubeadm’s HA topologies weigh where etcd sits.

With edge-1 cordoned, why did echo's replacement Pod sit in Pending?

The spread rule counted edge-1 and the control planes as nodes holding no echo Pods, so a second Pod on data-1 broke maxSkew: 1. nodeTaintsPolicy: Honor counts only nodes the Pod could run on.

Why did a rollout put both echo Pods on one worker?

The spread rule counted every Pod labelled app=echo, including the old ones still running. At each step, data-1 looked no fuller than edge-1. matchLabelKeys: [pod-template-hash] counts only Pods from the same rollout.

Why would Terraform's state go stale here?

The drills stop and delete VMs outside Terraform, so until the next apply its state describes a cluster that no longer exists.

Last known fix: etcd restored from a snapshot

17 November 2026

What was done

Tonight I took a snapshot of etcd, deleted a namespace and brought it back, for #13. Restoring a snapshot puts the whole cluster back at its last known fix, the moment the snapshot was taken, and anything that happened after it is gone.

just etcd-snapshot takes one from the first running control plane, copies it to my home server and checks the copy’s hash. It takes 9 seconds and comes to 15 MB.

I ran the drill twice. The first time I worked each step out as I went, from the kubeconfig to the restore flags, and reads failed for 19 minutes. I wrote the runbook from that run and followed it for the second run. Reads were back 93 seconds after the API servers stopped, and echo answered a minute later. The namespace came back with its old UID and creation time, so as far as Kubernetes knows, it never left.

The runbook has the steps and the output.

Why was it done that way

Velero is the obvious alternative. It backs up objects through the API and can restore one namespace on its own, which is all this drill asks for. It can’t help when the control plane itself is lost, and that’s the failure an etcd snapshot covers. The snapshot is the floor, so it comes first. Object backups can sit on top of it later.

etcd runs as three members, one on each control plane, and the workers run none. A member is a single etcd server with its own data, ID and peer address. The snapshot goes through crictl inside the etcd container, because I’ll need a snapshot most when the API is down. It reads one member’s own database, so it doesn’t need quorum either. The file holds every Secret in plain text, so it’s readable only by me.

Every member restores from the same file. One member restored alone would get a new cluster ID from the restore, and the other two would reject it. So every API server and etcd member stops, all three restore with one member list and they start as a new cluster that agrees from the first heartbeat. Each node keeps its old data directory until the cluster checks clean. If a member won’t start, all three go back to it, never one or two. The etcd recovery guide covers the mechanics.

The restore also moves the revision a billion ahead and marks everything older as compacted. The kubelets and controllers watch from the last revision they saw, which is newer than the snapshot. Without the jump, new writes would reuse numbers the watchers had already passed, so they’d never see them. With it, every watcher gets 410 Gone and lists everything again. The Kubernetes docs also restart the scheduler, the controller manager and the kubelets after a restore. I don’t, because the relist covers it, and the second run showed that’s enough here.

What challenges were faced

Before I could start, Multipass was down. Overnight, snapd refreshed it from 1.17.1 down to 1.17.0 and suspended all five VMs on the way. Their saved state pointed at the newer version’s firmware, so the older one crashed trying to resume them and left its DNS server holding port 53. Every restart after that failed on the port, which hid the real cause. Reverting to 1.17.1 brought all five back, and etcd recovered without a hand on it. Multipass now refreshes when I say so.

My first copy to a control plane went through multipass exec, and only 2.2 MB of the 15 arrived. etcdutl refused it, because every snapshot ends with a hash of its contents, so a torn copy fails safely. The runbook now copies with multipass transfer and checks the hash on every node before anything stops.

kubeadm left a trap too. It grew the cluster one join at a time, so cp-1’s etcd manifest lists only cp-1 as the initial cluster. Copy that into a restore and cp-1 starts a cluster of its own. The runbook spells out all three members.

What’s next

Redpanda on local storage is next, in #14, then ClickHouse in #15. The risk is data-1’s 4 GB holding both. Their memory limits are the first lever, and giving data-1 6 GB is the fallback, since the home server has the room. Leg 1 is due on Friday.

Quiz

Five questions on the post. Open one for my answer.

Why can't an etcd restore bring back one namespace on its own?

A snapshot is the whole database at one revision. Restoring it puts every object back to that moment, and everything written since is lost.

Why restore every member, rather than one and let the others catch up?

The restore gives that member a new cluster ID. The other two still have the old one, so they reject it.

What does --bump-revision protect against?

Watchers that saw revisions newer than the snapshot. The jump stops new writes reusing those numbers, and --mark-compacted makes the watchers get 410 Gone and relist.

Why take the snapshot through crictl and not the API?

The API is the thing most likely to be gone when I need a snapshot. crictl only needs the node and a running etcd container.

echo's Pods came back with their old names but new addresses. Why?

The restore brought back the Pod objects, still bound to their nodes. The kubelets had torn down their sandboxes when the namespace went, so they built new ones, and Cilium gave each a new address.

Holding pattern: Redpanda on data-1’s own disk

18 November 2026

What was done

Tonight Redpanda went onto the cluster, for #14. It’s where every frame will circle until ClickHouse is ready to take it, so it has to hold them through anything short of losing the disk. One broker runs as a StatefulSet in dev mode, on a 10 Gi volume on data-1, with a telemetry.raw topic of six partitions.

just redpanda deploys it and is safe to run again. A second run takes 15 seconds and changes nothing. just redpanda-check proves the issue’s first box. It writes 10,000 numbered records, deletes the Pod and reads every one back once the StatefulSet has replaced it. The broker was back on data-1 in 7 seconds with none missing, and 100,000 records came back the same way.

Redpanda’s image lives on Docker Hub, so the host gained a fourth pull-through mirror first, in #125. The broker and the check landed in #127 and #128.

Why was it done that way

The cluster had no StorageClass, so a claim had nowhere to go. A StorageClass is the menu an admin offers: who creates a volume, when a claim binds to it and what happens to the data afterwards. Mine is called local and has no provisioner. I create each volume by hand, as a local volume on one node’s disk, pinned to that node. Binding waits for the first Pod that uses the claim, so the scheduler picks a node before the claim picks a volume. With immediate binding, a claim could take a volume on a node the Pod can’t run on. The volume is set to Retain, so deleting the claim never deletes frames.

I passed over replicated storage such as Longhorn. There’s one data node, so it has nowhere to copy to, and with more nodes Redpanda and ClickHouse would replicate their own data more cheaply than a disk layer could. The price of local disks is that a stateful Pod can’t move. Draining data-1 stops Redpanda until the node is back, and the drain drill in #27 will show it. A dynamic provisioner is in Stretch as #124, for everything that doesn’t need a hand-made volume.

Redpanda is built on Seastar, the same framework as ScyllaDB. It runs one thread per core, claims its memory at startup and writes past the page cache. So --smp=1 --memory=512M gives it one core and 512 MB, and --overprovisioned stops it pinning and busy-polling data-1’s two shared vCPUs. The container’s limit is 1 GiB, because Seastar’s share doesn’t cover the binary or its thread stacks.

The bash behind the cluster is now about 610 lines. Talos, OpenTofu and Flux would bring that to one 60-line script plus about 350 lines of configuration, and I’ve filed that rebuild as #130 with six sub-issues. I’m not starting it yet. Building kubeadm by hand is the point of this leg, and the bash works, is linted and is about to stop changing, so replacing it now would be a rewrite rather than paying off debt. The Leg 4 drills are the safety net that will prove a new platform moved nothing, and Flux in #107 should come first, so the workloads are declarative before the layer underneath them changes. Writing it down now, with a trigger, makes it planned work. It also retired an issue that would have polished scripts #130 replaces.

What challenges were faced

The first deploy timed out. I’d rebuilt the cluster from a checkout on my home server that predated the mirror, so nothing on the bridge served Docker Hub. Redpanda’s 122 MB image came straight down my connection at about 2 minutes a node, which outlasted the script’s 2-minute wait for the topic Job. The broker and the Job had both finished fine. The wait is now 5 minutes, matching the rollout. just vms-check would have caught the missing mirror, and just up doesn’t run it yet.

The real mistake was durability. Dev mode turns write caching on, which acknowledges a write before it reaches the disk, so my first version turned it off with a bootstrap file and called the broker durable. A second pass over the manifests found that dev mode also starts the broker with --unsafe-bypass-fsync=true, so an ack no longer waited for the disk to confirm the write. Deleting a Pod leaves every cache below it intact, so the restart check passed 100,000 records, which proves the caches are very reliable right up until the power goes. The broker now starts with --unsafe-bypass-fsync=false, and its command line in the logs shows it. A restart check can’t see this class of fault. The Leg 4 drills should include stopping data-1 hard.

I also broke the deploy on purpose to see how it fails. A volume left Released after its claim was deleted now stops the script at once and prints the command that frees it, rather than ending on a 5-minute timeout. A topic Job that can’t reach the broker prints rpk’s own error within 25 seconds. That needed restartPolicy: Never, because with OnFailure the Job deletes its Pod when it gives up and the logs go with it.

What’s next

ClickHouse is next, on the same local class, in #15. The risk is data-1’s 40 GB disk. Redpanda refuses writes when free space falls under 5 GiB, and it measures that against the whole filesystem ClickHouse will share. ClickHouse’s retention is the first lever, and a bigger disk for data-1 is the fallback. Leg 1 lands on Friday, with the decisions in #16 and its debrief.

Quiz

Five questions on the post. Open one for my answer.

Why does telemetry.raw have six partitions, and what does a key decide?

A partition is an ordered log, and it’s the unit consumers share out. Six let up to six consumers in one group read in parallel. The key’s hash picks the partition, so every frame with the same key stays in order.

What's an offset, and who keeps it for a consumer group?

An offset is a record’s position in its partition. The group commits the offset it has processed to the broker, so a restarted consumer carries on from there. Committing only after ClickHouse confirms a batch is what makes delivery at-least-once.

Why does the local class wait for the first consumer before binding?

A local volume only exists on one node. Waiting lets the scheduler place the Pod first, so the claim binds to a volume the Pod can actually reach.

The claim was deleted and recreated. Why won't the volume bind again?

With Retain, the volume goes Released and keeps the old claim’s UID. Removing that UID makes it Available for the new claim, and the data is still there.

Write caching was off. Why could Redpanda still lose acknowledged frames?

Dev mode also skipped fsync, so an ack didn’t wait for the disk to confirm the write. A Pod restart keeps whatever is still cached, and a power cut doesn’t.

Decisions

Each decision gets a short record: the context, what I chose, what I turned down and what it costs. I write them as I make them, so this list grows over the four days. Each one starts from _template.md.

Records only ever add to the history. Once one is accepted I never rewrite or delete it. If I change my mind, a new record supersedes it and the old one’s status says so. Every decision and every reversal stays on file.

#DecisionStatus
1MAVLink v2 over a custom framingTo write
2kubeadm over k3s or TalosTo write
3Redpanda over Apache Iggy or a home-made logTo write
4Deduplicating in ClickHouse over chasing exactly-onceTo write
5just and mise over make, and a local registry for imagesAccepted
6A scripted kubeadm bootstrap over Terraform or AnsibleAccepted

0005. just and mise over make, and a local registry for images

  • Status: Accepted
  • Date: 2026-11-12
  • Issue: #37

Context

I started with a Makefile to render the diagrams. It didn’t last. Make is a build tool, not a task runner: recipes need tabs, every task needs .PHONY and shell variables need $$. Its strength is rebuilding files when their inputs change, and cargo and uv already do that for their own code.

The project runs from firmware up to the cluster, so it spans four ecosystems:

  1. Rust, for the host crates and the firmware.
  2. Python, for bbcheck and the scripts.
  3. D2 and mdBook, for the docs.
  4. kubectl and Helm, for the cluster.

Each has its own tooling and none of it covers the others. I need one entry point that sits above all four.

CI must run the same tools at the same versions as my laptop. Otherwise a check can pass in one place and fail in the other.

The constraints:

  • Everything runs on one laptop with about 10 GB of free memory, shared by five VMs and Renode.
  • Rust is pinned in rust-toolchain.toml, because rustup owns the toolchain and the firmware target.
  • Multipass, Renode, probe-rs and the ClickHouse client are installed by hand. They don’t fit a version manager.
  • Images built on the host have to reach containerd on five VMs, many times a day while I iterate.
  • I have four days, so the tooling has to be boring and quick to set up.

Decision

  • just runs every task. The justfile only calls programs and scripts, and the real logic lives in them.
  • mise pins the CLI tools in mise.toml. CI installs from the same file with jdx/mise-action.
  • A registry:2 on the host carries images into the cluster, and containerd on every VM trusts it through hosts.toml. #10 builds it.

Options I turned down

  • make. It was my first choice. For the reasons above, I replaced it before the first pull request.
  • cargo xtask. Tasks written in Rust suit a Rust-only repo. Here they would mostly wrap Python, D2 and Helm, and every change to a task means a compile.
  • asdf, or installing by hand. asdf needs a plugin per tool. Hand installs drift, and CI stops matching my laptop.
  • Importing images by hand. ctr images import has to run on each of five nodes. It’s easy to leave one stale, and pods don’t pull the way they would in production.
  • GHCR for every build. Each build would need a push, credentials and the network. GHCR stays the release channel (#40). The local registry is for the inner loop.

Consequences

  • There are two more tools to learn. mise install brings just, so setup is still one step.
  • The registry is one more process on the host, but it’s small next to five VMs.
  • The registry speaks plain HTTP on a private network. That’s fine on a laptop and wrong anywhere else.
  • Every task has one name, and CI runs the versions I run.
  • Pods pull images as they would from a real registry, so imagePullPolicy behaves normally.
  • I’d revisit this if the project moved to more than one machine, or if devcontainers or Nix took over the whole bench.

References

0006. A scripted kubeadm bootstrap over Terraform or Ansible

  • Status: Accepted
  • Date: 2026-11-15
  • Issue: #105

Context

I bootstrapped the cluster by hand for #11, following the bootstrap runbook. It took 47 minutes from kubeadm init to a passing just cluster-check. Doing it once by hand was the point. Doing it every time isn’t.

The drills that follow break the cluster on purpose. #12 stops two control planes, #13 restores etcd from a snapshot and the 1.36 to 1.37 upgrade can fail halfway. Any of them can end in a rebuild, and a missed step by hand, such as kube-vip left on super-admin.conf, costs an evening.

Part of the path is already scripted:

  1. just host runs the registry, the image mirrors and the apt cache.
  2. just vms builds the five VMs from cloud-init and skips any that exist. A warm rebuild takes 6m09s.
  3. just cluster-files renders the kubeadm and kube-vip files.
  4. just cluster-check proves the result.

What’s left is runbook steps 2 to 8. That’s about ten commands across five VMs, in order, passing a token, a certificate key and a CA hash from cp-1 to the others. The machines already exist, so this is orchestration, not provisioning.

The constraints:

  • One server runs five Multipass VMs in one zone, because kube-vip’s ARP mode needs one subnet.
  • I reach the VMs with multipass exec, not SSH.
  • Leg 1 is due on 20 November, and this has to land before #12.
  • Every line of the automation has to map to a runbook step.

Decision

  • just cluster-up runs runbook steps 2 to 8 through scripts/cluster-up.sh over multipass exec. just up chains vms, cluster-up and cluster-check into a rebuild from nothing.
  • kubeadm’s own configuration carries the joins. kubeadm.yaml takes a bootstrap token and a certificate key, generated on each run and never committed. A JoinConfiguration per role replaces the join flags, --ignore-preflight-errors=Mem included. The script reads the CA hash after init, because the CA doesn’t exist before it.
  • Each step checks before it acts. Init is skipped if cp-1 has admin.conf, and a join is skipped if the node has kubelet.conf.
  • The script copies admin.conf to the server and installs Cilium from there with Helm. just kubeconfig sets up my laptop’s copy and the tunnel.
  • The VMs stay on vms.sh. The upgrade and the drills stay manual.

Options I turned down

  • Terraform or OpenTofu for the VMs. Terraform earns its keep by reconciling state against an API. Here that’s one server’s Multipass, already driven by a script that skips what exists. The community provider documents no --zone option and was last updated in December 2025. Its state would also go stale whenever a drill has me delete a VM by hand.
  • Ansible. It’s agentless and the usual way to run kubeadm across machines. Here it needs SSH keys injected through cloud-init, an inventory that copies deploy/vms/config.sh and a Python toolchain, all for about ten commands.
  • Kubespray. It builds exactly this cluster, kube-vip and Cilium included. It also hides every step I’ve just learned inside a large Ansible project, and debugging it means debugging that project.
  • kubeadm from cloud-init. Each VM could init or join on first boot from shared secrets. The order between nodes would rest on retries, and a failed join would only show in that VM’s cloud-init log.
  • Flux or Argo CD at bootstrap. Both need a working pod network, so Cilium goes in directly either way. GitOps for the Leg 3 workloads is its own decision.
  • Staying manual. Fine for one bootstrap. Not for a cluster I plan to break several times a week.

Consequences

  • A rebuild is one command and a wait, so the drills can be as destructive as they need to be. The first clean just up took 20m08s from nothing to a passing cluster-check, VMs included. The bootstrap alone took 47 minutes by hand.
  • The runbook stays. It explains what the script does and why, and its recovery steps still apply when the script stops halfway.
  • The script and the runbook can drift apart. Each step in the script names the runbook step it runs, and cluster-check proves the result either way.
  • The token and certificate key sit in plain text in ~/cluster/ on the VMs. The token expires after 24 hours and the uploaded certificates after two. That’s fine on a private bridge and wrong anywhere else.
  • Helm has to be on the server. mise.toml already pins it.
  • I’d revisit this if the cluster moved to real machines or a hypervisor with an API, such as Proxmox. Then Terraform for the machines and Ansible for kubeadm earn their place, or Cluster API once there’s more than one cluster.

References

Runbooks

Every drill I run gets a runbook: the steps, what I expected and the real output. They fill in as the drills happen, and each one starts from _template.md.

RunbookDrillStatus
bootstrap.mdBootstrap the cluster by handRun on 15 Nov 2026
control-plane.mdLose one control plane, then twoRun on 16 Nov 2026
etcd.mdSnapshot and restore etcdRun on 17 Nov 2026
chaos.mdBreak the pipeline while it streamsTo run

Bootstrap the cluster by hand

How I bring up the highly available control plane on the five VMs, by hand, with kubeadm, kube-vip and Cilium. I first ran it on 15 November 2026, and it took 47 minutes from kubeadm init to a clean just cluster-check.

The VMs run on my home server. Steps 1 to 6 and step 9 run there, in the repo or on a VM through multipass shell. Steps 7 and 8 run on my laptop.

just cluster-up runs steps 1 to 8 in one go on the server, and just up rebuilds the VMs and the cluster from nothing. This runbook is what they do, and how to recover when they stop halfway. The script runs the joins in steps 5 and 6 from deploy/cluster/join-control-plane.yaml and join-worker.yaml in place of the flags.

Before I start

  • just vms has built all five VMs, and just vms-check passes.
  • The server’s checkout is on the branch I’m testing, with no stray edits.
  • I have two hours free. The certificate key from step 4 expires after two hours, and the join token after 24.

Steps

1. Write the files

just cluster-files

This renders deploy/cluster/ and copies kubeadm.yaml to cp-1 and kube-vip.yaml to all three control planes, under ~/cluster/. It fails if a VM has leased the virtual IP. kubeadm.yaml carries a fresh bootstrap token and certificate key, so the join commands init prints use them.

2. Put kube-vip on cp-1, on ground power

On cp-1:

sudo cp ~/cluster/kube-vip.yaml /etc/kubernetes/manifests/
sudo sed -i 's#path: /etc/kubernetes/admin.conf#path: /etc/kubernetes/super-admin.conf#' /etc/kubernetes/manifests/kube-vip.yaml

Since Kubernetes 1.29, admin.conf has no rights until init finishes, and kube-vip has to hold the virtual IP before then. So it borrows super-admin.conf. The sed only changes the hostPath line, because mountPath: has a capital P.

3. Initialise the first control plane

On cp-1:

sudo kubeadm init --config ~/cluster/kubeadm.yaml --upload-certs

I keep both join commands it prints. The control-plane one carries --certificate-key, which unlocks the shared certificates for two hours.

4. Take kube-vip off ground power

On cp-1, straight after init:

sudo sed -i 's#path: /etc/kubernetes/super-admin.conf#path: /etc/kubernetes/admin.conf#' /etc/kubernetes/manifests/kube-vip.yaml

The kubelet sees the file change and restarts kube-vip on admin.conf. super-admin.conf is in system:masters, which skips RBAC, so nothing should run on it once init is done. cluster-check fails if anything still does.

5. Join cp-2 and cp-3

On each, using the control-plane join command from step 3:

sudo kubeadm join 10.97.0.250:6443 --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane --certificate-key <key> \
  --ignore-preflight-errors=Mem
sudo cp ~/cluster/kube-vip.yaml /etc/kubernetes/manifests/

kube-vip goes in after the join. Before it, the node has no admin.conf. The control planes have 1.5 GB against kubeadm’s 1.7 GB floor, so the memory check is skipped. deploy/README.md has the reasoning.

6. Join the workers

On edge-1 and data-1, using the worker join command from step 3:

sudo kubeadm join 10.97.0.250:6443 --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash>

Every node now shows NotReady. That’s expected until step 9.

7. Copy the kubeconfig to my laptop

On my laptop:

ssh <home-server> multipass exec cp-1 -- sudo cat /etc/kubernetes/admin.conf > ~/.kube/config
ssh -fN -L 6443:10.97.0.250:6443 <home-server>

Then I change server: in ~/.kube/config to https://127.0.0.1:6443. The API server’s certificate names 127.0.0.1 for this tunnel.

just kubeconfig <home-server> does both, into ~/.kube/blackbox.conf.

8. Install Cilium

On my laptop:

helm install cilium cilium --repo https://helm.cilium.io --version 1.20.4 \
  -n kube-system -f deploy/cluster/cilium-values.yaml

On a cold mirror, the agents sit at Init:0/6 for a while. On the first run, all five nodes pulled the images through quay.io at about 1 MB/s, and it took over twenty minutes.

9. Prove it

On the server:

just cluster-check

What I saw on the first run

At T+17, every control-plane Pod was running, while Cilium waited for its images, CoreDNS was Pending and every node was NotReady:

NAME                               READY   STATUS              RESTARTS   AGE
cilium-2d88w                       0/1     Init:0/6            0          110s
cilium-envoy-fcxq9                 0/1     ContainerCreating   0          110s
cilium-operator-6c4c574b46-d8mtq   0/1     ContainerCreating   0          110s
coredns-589f44dc88-h6nzh           0/1     Pending             0          17m
etcd-cp-1                          1/1     Running             0          17m
etcd-cp-2                          1/1     Running             0          7m40s
etcd-cp-3                          1/1     Running             0          8m46s
kube-vip-cp-1                      1/1     Running             0          16m
kube-vip-cp-2                      1/1     Running             0          7m40s
kube-vip-cp-3                      1/1     Running             0          7m26s

That’s an excerpt of the 34 Pods. kube-vip-cp-1 is a minute younger than etcd-cp-1 because step 4 restarted it.

Once Cilium was up, cluster-check passed:

API on 10.97.0.250: ok, held by cp-3
kube-vip: running on 3 control planes, all reading admin.conf
nodes: 5 of 5 Ready
etcd: 3 of 3 members healthy
cilium: ready on 5 nodes

The address had moved to cp-3 on its own. At 13:36, with the server flat out, cp-1’s API server stopped answering in time and its leaders lost their Leases. cp-3 took over, as designed.

When it goes wrong

The certificate key has expired

A control-plane join fails after two hours. On a working control plane:

sudo kubeadm init phase upload-certs --upload-certs
sudo kubeadm token create --print-join-command --certificate-key <new key>

The first prints a new key, and the second prints a full control-plane join command that uses it.

The join token has expired

After 24 hours, a worker join fails. On a control plane:

sudo kubeadm token create --print-join-command

A join failed halfway

On the failed node:

sudo kubeadm reset -f

Then I join again. If it was a control plane that got as far as etcd, I remove its stale member first, from a healthy control plane:

kubectl -n kube-system exec etcd-cp-1 -- etcdctl \
  --endpoints https://127.0.0.1:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key /etc/kubernetes/pki/etcd/healthcheck-client.key \
  member list

Then member remove <ID> with the same flags.

cluster-check says kube-vip reads super-admin.conf

Step 4 was missed. I run it on the node that cluster-check names.

Cilium is stuck at Init

I check the events before anything else:

kubectl -n kube-system get events --sort-by=.lastTimestamp | grep -i cilium

Pulling image with no errors means a cold mirror. Waiting is the fix, and the next rebuild reads the images from the host.

Lose one control plane, then two

The cluster runs three control planes and three etcd members. With one gone, etcd keeps quorum and nothing changes. With two gone, etcd stops, the API goes with it and the workloads keep serving. This drill shows both.

I ran it twice on 16 November 2026. The first run ended early, for a reason that had nothing to do with etcd. The second run is the one that counts.

Everything runs on my home server, in two terminals.

Before I start

  • just cluster-check passes.

  • just drill-watch runs in the first terminal. I wait for Start the drill before I break anything, because it deploys echo first.

  • The steps run in the second terminal.

  • Both terminals append to one log, so neither overwrites the other:

    just drill-watch 2>&1 | tee -ai drill.log    # first terminal
    just cluster-check 2>&1 | tee -a drill.log   # every check below
    

    -i keeps tee alive through Ctrl-C, so the summary lands too. cluster-check prints its failures to stderr, hence 2>&1.

drill-watch prints a line whenever something changes: which control planes are up, which one holds the VIP and whether echo, a read, a cached read and a write succeed. deploy/README.md explains each probe.

Steps

1. Lose one control plane

multipass stop --force cp-2

--force pulls the power. A clean shutdown would let etcd hand over leadership politely, and real failures don’t.

Two of three members is still a majority, so etcd keeps writing. If cp-2 held the VIP, kube-vip on another control plane takes it once the 15 s Lease runs out. Every probe stays ok, apart from a gap while the VIP moves.

2. Check it

After a minute, so the node controller has noticed:

just cluster-check

It fails, and should: kube-vip isn’t ready on cp-2, four of five nodes are Ready and two of three etcd members are healthy. The API still answers on the VIP.

3. Lose a second

multipass stop --force cp-3

One of three is not a majority. etcd can’t elect a leader, so it refuses writes and consistent reads. cp-1’s API server stops being ready, but it still answers reads at resourceVersion=0 from its watch cache.

kube-vip renews its Lease through that API, every 2 s, and stands down if it can’t renew for 10 s. So the VIP disappears, and the cached reads go with it. The API doesn’t stay read-only. It goes away.

echo keeps answering. The Pods run on the workers, and kube-proxy’s iptables rules stay in the kernel. Nothing on the data path needs the control plane until something changes.

4. Check it

just cluster-check

It stops before it reaches the API:

no control plane's API server is ready (running: cp-1): kubeadm init hasn't run, or etcd has lost quorum

It only asks running VMs. The first run shows why that matters.

5. Bring both back

multipass start cp-2 cp-3

Once a second member is up, etcd has a majority again and elects a leader. The API comes back, kube-vip takes its Lease and the VIP returns. The drill-watch lines go back to ok.

6. Prove it

Ctrl-C in the first terminal prints how many probes of each kind passed. Then:

just cluster-check

Cilium on the two rebooted nodes takes a minute to be ready, so the first check may fail on it alone.

What I saw

First run, 16 November 2026

Stage one went as expected: cp-2 stopped and the cluster carried on without it.

Stage two lasted about a minute. cp-3 went at 19:53:13. 32 s later, the API dropped drill-watch’s connection, and from 19:53:50 the VIP didn’t answer at all:

E1116 19:53:50.592902 ... dial tcp 10.97.0.250:6443: connect: no route to host

That’s kube-vip standing down, as step 3 predicts. Then I ran just cluster-check, and afterwards all five VMs were running. I hadn’t started any of them. Multipass’s log, trimmed:

Nov 16 19:53:13 multipassd: started: qemu-img snapshot -l .../instances/cp-3/...
Nov 16 19:54:00 multipassd: started: qemu-system-x86_64 ... .../instances/cp-2/...
Nov 16 19:54:20 multipassd: started: qemu-system-x86_64 ... .../instances/cp-3/...

cluster-check looked for a ready API server by running multipass exec on each control plane in turn. cp-1 wasn’t ready, because etcd had no quorum, so it moved on to cp-2 and cp-3. multipass exec starts a stopped instance and retries. Its help doesn’t say so, but exec.cpp does. The check booted cp-2, waited 20 s for it, then booted cp-3. The tool meant to report the outage ended it.

drill-watch never reached its loop. It was still waiting for echo to roll out when I stopped cp-3, and it timed out once the API went.

Both scripts now only touch running VMs, and drill-watch reads the VIP holder from multipass list. It also says when to start.

Second run, 16 November 2026

With both fixes in:

20:16:10  +   1s  up cp-1,cp-2,cp-3  vip cp-1  echo ok  read ok  cache ok  write ok
20:16:20  +  11s  up cp-1,cp-3  vip cp-1  echo ok  read ok  cache ok  write ok
20:16:58  +  49s  up cp-1  vip cp-1  echo ok  read fail  cache ok  write fail
20:17:20  +  71s  up cp-1  vip none  echo ok  read fail  cache fail  write fail
20:19:46  + 217s  up cp-1,cp-2,cp-3  vip none  echo ok  read fail  cache fail  write fail

Losing cp-2 changed nothing. cp-1 held the VIP, so there was nothing to move.

Losing cp-3 stopped consistent reads and writes on the next probe. The API server kept serving cached reads for another 22 s, until kube-vip stood down and the VIP went. echo didn’t notice.

cluster-check stopped early and left cp-2 and cp-3 stopped:

no control plane's API server is ready (running: cp-1): kubeadm init hasn't run, or etcd has lost quorum

Both were back by 20:19:46. I stopped drill-watch before quorum returned:

echo: 79 of 79 passed
read: 29 of 79 passed
cache: 35 of 79 passed
write: 29 of 79 passed

echo answered every probe. Cached reads outlived consistent ones by six probes. With probes timing out, a round took about 3 s, not 1.

Then cluster-check:

API on 10.97.0.250: ok, held by cp-1
kube-vip: running on 3 control planes, all reading admin.conf
nodes: 5 of 5 Ready
etcd: 3 of 3 members healthy
cilium: ready on 3 of 5 nodes

etcd, the API and kube-vip were back. Cilium on cp-2 and cp-3 was still starting. Once it was ready:

API on 10.97.0.250: ok, held by cp-1
kube-vip: running on 3 control planes, all reading admin.conf
nodes: 5 of 5 Ready
etcd: 3 of 3 members healthy
cilium: ready on 5 nodes

The check after stage one is missing. drill-watch wrote through tee -i without -a, so it wrote over what the check had appended.

What else the drill found

echo is meant to keep one Pod on each worker, so losing either leaves one serving. Two checks after the drill showed its spread rule didn’t hold that.

A lost worker’s Pod couldn’t be replaced. I cordoned edge-1 and deleted its echo Pod. The replacement sat in Pending:

0/5 nodes are available: 1 node(s) didn't match pod topology spread constraints, 1 node(s) were unschedulable, 3 node(s) had untolerated taint(s).

The rule allows at most one Pod more on any node than on the emptiest. By default it counts every node, tainted ones included, so cordoned edge-1 and the three control planes each counted as holding none. A second Pod on data-1 broke the rule. A dead worker is tainted the same way. nodeTaintsPolicy: Honor leaves tainted nodes out of the count.

A rollout put both Pods on one worker. Applying that fix rolled echo out again, and both new Pods landed on data-1 (columns trimmed):

NAME                    READY   STATUS        RESTARTS   AGE   IP             NODE
echo-5976856569-r52d5   1/1     Terminating   0          18m   10.244.3.87    edge-1
echo-7bbd856f5b-ddzh8   1/1     Running       0          16s   10.244.5.176   data-1
echo-7bbd856f5b-v7z4q   1/1     Running       0          4s    10.244.5.2     data-1

The rule counts every Pod labelled app=echo, old ones included. A rollout replaces them one at a time, so at each step the old Pods still running made data-1 look no fuller than edge-1. matchLabelKeys: [pod-template-hash] counts only Pods from the same rollout.

The rule now reads:

topologySpreadConstraints:
  - maxSkew: 1                         # at most one Pod more than the emptiest node
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    nodeTaintsPolicy: Honor            # cordoned, dead and control-plane nodes don't count
    matchLabelKeys:
      - pod-template-hash              # only this rollout's Pods count
    labelSelector:
      matchLabels:
        app: echo

With both in place, a rollout left one Pod on each worker, and the same cordon put the replacement on data-1 straight away:

NAME                    READY   STATUS    RESTARTS   AGE   IP             NODE
echo-5cb4c8bc96-5brtt   1/1     Running   0          52s   10.244.5.133   data-1
echo-5cb4c8bc96-vblxj   1/1     Running   0          22s   10.244.5.18    data-1

The scheduler never moves a running Pod, so after the uncordon both stayed on data-1. kubectl -n drill rollout restart deployment/echo spreads them again.

Neither causes trouble while every node is healthy, which is why a drill is the place to find them.

Why three or five

etcd writes only when a majority of members agree. Quorum is ⌊n/2⌋ + 1.

MembersQuorumCan lose
110
220
321
431
532

Even sizes buy nothing. Four members survive one loss, like three, but give one more machine to fail, and every write waits for three acknowledgements instead of two. Two is worse than one, because losing either stops writes.

Partitions decide it. Four members can split two and two, and then neither side has a majority and the cluster stops. Split three or five in two and one side always keeps a majority. The majority rule itself is what stops a split brain: two halves can’t both accept writes, because only one of them can be a majority.

When it goes wrong

A control plane won’t rejoin etcd

--force is a power cut. etcd syncs every commit to disk, but a cut mid-write can leave its log or database torn. On restart, etcd-cp-N crashes in a loop and the kubelet keeps restarting it.

I remove the member and join the node again. That only works for cp-2 and cp-3: after a reset, cluster-up would run kubeadm init on cp-1 and build a new, empty cluster. If cp-1 is the broken one, I join it by hand with step 5 of the bootstrap runbook.

From a healthy control plane, I find the member’s ID with member list and remove it with member remove, using the flags in bootstrap.md. Then:

kubectl delete node cp-N
multipass exec cp-N -- sudo kubeadm reset -f
just cluster-up

cluster-up treats a control plane as joined once its Node is labelled, so the Node has to go before it will join cp-N again.

A node comes back on a new address

Each etcd member’s peer URL and certificate name the node’s IP. A new address means the others can’t reach it, or won’t trust it. Multipass never expires a lease, so it shouldn’t happen here, but on real hardware it’s the classic case. The fix is the same: remove, reset and join.

The VIP doesn’t come back

kube-vip needs a working API to take its Lease, so the VIP returns after quorum, not before. If etcd is healthy and the VIP is still missing, I read kube-vip’s log on a control plane:

multipass exec cp-1 -- sudo crictl logs --tail 50 \
  $(multipass exec cp-1 -- sudo crictl ps -a -q --latest --name kube-vip)

What I don’t do

With two members down, etcd --force-new-cluster on the survivor gets writes going again. The other two then come back with history the new cluster doesn’t share, and both have to be wiped and joined again. It’s for members that are never coming back, not ones that are slow.

Snapshot and restore etcd

etcd holds every object in the cluster. A snapshot is a copy of one member’s database, and restoring it puts the whole cluster back to that moment. This drill takes a snapshot, deletes a namespace and restores it.

I ran it twice on 17 November 2026. The first time I worked out each step as I went, and reads failed for 19 minutes. The second run follows these steps, and reads failed for under two minutes.

Everything runs on my home server, in two terminals.

Before I start

  • just cluster-check passes.

  • No control plane has a /var/lib/etcd.old left from an earlier drill. Step 5 keeps an existing one as its fallback.

  • just drill-watch runs in the first terminal. I wait for Start the drill, because it deploys echo and the drill namespace first.

  • The steps run in the second terminal, with the cluster’s kubeconfig:

    export KUBECONFIG=~/.kube/blackbox.conf
    

    ~/.kube/config on the server is an empty stub, and kubectl asks for a username without this.

  • Both terminals append to one log:

    just drill-watch 2>&1 | tee -ai drill.log    # first terminal
    just cluster-check 2>&1 | tee -a drill.log   # every check below
    

Steps

1. Take a snapshot

just etcd-snapshot

It saves the snapshot on the first running control plane, copies it to ~/.local/state/telemetry-blackbox/etcd/ and checks the copy’s hash. It prints the revision, which is the point the restore goes back to.

2. Delete the namespace

kubectl delete namespace drill

The namespace, its Pods and its ConfigMap go. echo fails, then reads and writes of the ConfigMap fail, because it no longer exists. The API is still up.

3. Copy the snapshot to every control plane

Multipass is a snap and can’t read dot-directories, so multipass transfer needs a copy in my home directory first:

snap=$(ls -t ~/.local/state/telemetry-blackbox/etcd/*.db | head -1) && sha256sum "$snap"
install -m 600 "$snap" ~/etcd-restore.db
for cp in cp-1 cp-2 cp-3; do
  multipass transfer ~/etcd-restore.db "$cp:/home/ubuntu/etcd-restore.db" &&
  multipass exec "$cp" -- chmod 600 /home/ubuntu/etcd-restore.db
done
for cp in cp-1 cp-2 cp-3; do echo "$cp $(multipass exec "$cp" -- sha256sum /home/ubuntu/etcd-restore.db)"; done
rm ~/etcd-restore.db

Every hash must match the first. I never restore from a copy that doesn’t. The copies hold every Secret, so only ubuntu can read them, and step 8 removes them.

4. Stop etcd and the API servers

for cp in cp-1 cp-2 cp-3; do
  multipass exec "$cp" -- sudo mv /etc/kubernetes/manifests/etcd.yaml \
    /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/
done

The kubelet stops a static Pod when its manifest leaves the directory. I wait until no IDs follow the names:

for cp in cp-1 cp-2 cp-3; do echo "$cp: $(multipass exec "$cp" -- sudo crictl ps -q --name '^(etcd|kube-apiserver)$')"; done

kube-vip can’t renew its Lease without an API, so the VIP goes too.

5. Restore every member

cluster='cp-1=https://10.97.0.54:2380,cp-2=https://10.97.0.213:2380,cp-3=https://10.97.0.153:2380'
declare -A ip=([cp-1]=10.97.0.54 [cp-2]=10.97.0.213 [cp-3]=10.97.0.153)
restore() {
  local cp=$1 image
  image=$(multipass exec "$cp" -- sudo awk '/image:/ { print $2 }' /etc/kubernetes/etcd.yaml)
  multipass exec "$cp" -- sudo sh -c 'test -d /var/lib/etcd.old || mv /var/lib/etcd /var/lib/etcd.old; rm -rf /var/lib/etcd' &&
  multipass exec "$cp" -- sudo ctr -n k8s.io run --rm \
    --mount type=bind,src=/var/lib,dst=/var/lib,options=rbind:rw \
    --mount type=bind,src=/home/ubuntu,dst=/home/ubuntu,options=rbind:ro \
    "$image" etcd-restore \
    etcdutl snapshot restore /home/ubuntu/etcd-restore.db \
      --data-dir /var/lib/etcd --name "$cp" \
      --initial-advertise-peer-urls "https://${ip[$cp]}:2380" \
      --initial-cluster "$cluster" \
      --bump-revision 1000000000 --mark-compacted
}
for cp in cp-1 cp-2 cp-3; do restore "$cp" || { echo "restore failed on $cp"; break; }; done

Each node prints restored snapshot. If one fails, restore cp-N runs it again on that node. It keeps the first etcd.old, so the data from before the restore survives a second attempt.

etcdutl isn’t installed on the nodes, so ctr runs it from the etcd image that’s already there. The tag comes from the moved manifest, so it follows upgrades.

Every node gets the same --initial-cluster. etcd derives the cluster ID and the member IDs from it, so all three agree on who’s in the cluster. The manifests are no help here: kubeadm grew the cluster one join at a time, so cp-1’s lists only cp-1. Each node’s own --name and peer URL say which member it is.

The snapshot is older than anything the API servers, kubelets and controllers last saw. Without --bump-revision, new writes would reuse revisions they’d already seen, so they’d miss them. With it, revisions only go forward, and --mark-compacted puts everything older below the compaction mark. Anything watching from an old resourceVersion gets 410 Gone and lists again.

The Kubernetes docs also restart the scheduler, the controller manager and the kubelets after a restore. I don’t, because the compaction mark makes every watcher list again, and the second run shows it’s enough here.

6. Start etcd, then the API servers

for cp in cp-1 cp-2 cp-3; do multipass exec "$cp" -- sudo mv /etc/kubernetes/etcd.yaml /etc/kubernetes/manifests/; done

I wait until all three members are healthy. This checks from cp-1:

multipass exec cp-1 -- sudo sh -c 'crictl exec $(crictl ps -q --name "^etcd$") etcdctl \
  --endpoints https://127.0.0.1:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key /etc/kubernetes/pki/etcd/healthcheck-client.key endpoint health --cluster'

Then the API servers:

for cp in cp-1 cp-2 cp-3; do multipass exec "$cp" -- sudo mv /etc/kubernetes/kube-apiserver.yaml /etc/kubernetes/manifests/; done

kube-vip takes its Lease again and the VIP returns.

7. Prove it

just cluster-check
kubectl get namespace drill -o jsonpath='{.metadata.uid} {.metadata.creationTimestamp}{"\n"}'
kubectl -n drill get pods -o wide

The namespace has its old UID and creation time, so it’s the same object, not a new one. The Pods have their old names. The kubelets build new sandboxes for them, and echo answers.

8. Clean up

Once cluster-check passes:

for cp in cp-1 cp-2 cp-3; do multipass exec "$cp" -- sudo rm -rf /var/lib/etcd.old /home/ubuntu/etcd-restore.db; done

What I saw

First run, 17 November 2026

I worked out each step as I ran it. Reads failed for 19 minutes, from 20:10:47 to 20:29:37:

20:09:54  +   1s  up cp-1,cp-2,cp-3  vip cp-3  echo ok  read ok  cache ok  write ok
20:10:42  +  49s  up cp-1,cp-2,cp-3  vip cp-3  echo fail  read ok  cache ok  write ok
20:10:47  +  54s  up cp-1,cp-2,cp-3  vip cp-3  echo fail  read fail  cache fail  write fail
20:17:24  + 451s  up cp-1,cp-2,cp-3  vip none  echo fail  read fail  cache fail  write fail
20:29:37  +1184s  up cp-1,cp-2,cp-3  vip none  echo fail  read ok  cache ok  write ok
20:29:40  +1187s  up cp-1,cp-2,cp-3  vip cp-3  echo fail  read ok  cache ok  write ok
20:30:17  +1224s  up cp-1,cp-2,cp-3  vip cp-3  echo ok  read ok  cache ok  write ok

Most of it went on working out each step: the kubeconfig, getting the snapshot onto the nodes and the restore flags.

Second run, 17 November 2026

The snapshot came from cp-1 at revision 1000382584. The first run had already bumped it by a billion.

+----------+------------+------------+------------+---------+
|   HASH   |  REVISION  | TOTAL KEYS | TOTAL SIZE | VERSION |
+----------+------------+------------+------------+---------+
| d31b1f50 | 1000382584 |        482 |      15 MB |   3.6.0 |
+----------+------------+------------+------------+---------+

VERSION is the storage format, not the binary, which is 3.6.8.

drill-watch:

20:38:13  +   1s  up cp-1,cp-2,cp-3  vip cp-3  echo ok  read ok  cache ok  write ok
20:38:47  +  35s  up cp-1,cp-2,cp-3  vip cp-3  echo fail  read ok  cache ok  write ok
20:38:55  +  43s  up cp-1,cp-2,cp-3  vip cp-3  echo fail  read fail  cache fail  write fail
20:39:43  +  91s  up cp-1,cp-2,cp-3  vip none  echo fail  read fail  cache fail  write fail
20:40:47  + 155s  up cp-1,cp-2,cp-3  vip none  echo fail  read ok  cache ok  write ok
20:40:50  + 158s  up cp-1,cp-2,cp-3  vip cp-3  echo fail  read ok  cache ok  write ok
20:41:53  + 221s  up cp-1,cp-2,cp-3  vip cp-3  echo ok  read ok  cache ok  write ok
TimeStepTook
20:38:40Delete the namespace14 s
20:39:05Copy the snapshot, three hashes match9 s
20:39:14Stop etcd and the API servers36 s
20:40:04Restore three members10 s
20:40:24Start etcd, three members healthy6 s
20:40:30Start the API servers, reads work through the VIP17 s

From stopping the API servers to reads working again took 93 seconds. Until the API servers stopped, from 20:39:14, the reads failed because the ConfigMap was gone, not the API. drill-watch can’t tell the two apart. multipass list reported the VIP 3 s after the API answered on it.

echo came back about a minute after the API. The Pods had their old names and new addresses, because the kubelets built new sandboxes for them (columns trimmed):

NAME                    READY   STATUS    RESTARTS   AGE   IP             NODE
echo-58849b6468-7cmc8   1/1     Running   1          32m   10.244.3.252   edge-1
echo-58849b6468-bdwz7   1/1     Running   1          32m   10.244.5.199   data-1

The namespace kept the UID and creation time it got in the first run, at 20:09:40. The ConfigMap’s resourceVersion read 2000383099, two bumps on.

echo: 33 of 102 passed
read: 69 of 102 passed
cache: 69 of 102 passed
write: 69 of 102 passed

Then cluster-check:

API on 10.97.0.250: ok, held by cp-3
kube-vip: running on 3 control planes, all reading admin.conf
nodes: 5 of 5 Ready
etcd: 3 of 3 members healthy
cilium: ready on 5 nodes

What a restore costs

A restore can’t bring back one namespace. It puts every object back to the snapshot. Anything written after it is lost: new Deployments, scaling, Secrets, Leases and Node heartbeats. Here that was a few minutes of a quiet cluster, so nothing mattered. On a busy cluster a restore is the last resort, not an undo.

The snapshot holds every Secret in plain text. etcd-snapshot leaves it readable only by me, and it never goes in the repo. Step 3’s copies on the control planes last until step 8.

Why every member

One member restored alone gets a new cluster ID from the restore. The other two still have the old one, so they reject it. Restoring all three from one file, with one --initial-cluster, makes a new cluster whose members agree from the start.

When it goes wrong

A copy is torn

My first attempt piped the snapshot into multipass exec. It arrived as 2.2 MB of 15 MB, and etcdutl refused it:

Error: snapshot missing hash but --skip-hash-check=false

Every snapshot ends in a hash of its contents, so a truncated copy fails safely. multipass transfer and the hash check in step 3 keep it from happening.

A member won’t start

If an etcd container crashes after step 6, I go back to the data from before the restore. I stop etcd and the API servers with step 4, run this, then start them with step 6:

for cp in cp-1 cp-2 cp-3; do multipass exec "$cp" -- sudo sh -c 'rm -rf /var/lib/etcd && mv /var/lib/etcd.old /var/lib/etcd'; done

That’s the cluster as it was before the restore, with the namespace still deleted. Always all three, never one or two: that’s members from two clusters.

A member has the wrong --initial-cluster

Copied from the manifest, cp-1 would know only itself and start a one-member cluster of its own. cp-2 and cp-3 would refuse it on the cluster ID. The fix is steps 4 to 6 again on all three, with the full list. Step 5 is safe to run twice.

What I don’t do

etcdctl snapshot restore no longer exists in etcd 3.6. etcdutl replaced it. I also don’t restore while any API server runs, because it would keep serving from its watch cache and write to a cluster that’s changing under it.

Brand

The logo is an attitude indicator inside a black box. It’s the instrument behind the ATTITUDE messages this project records, inside the box that records them.

Files

FileUse
logo-light.svg, logo-dark.svgThe full logo, for light and dark backgrounds
mark-light.svg, mark-dark.svgThe mark alone, where space is tight or the name is already nearby
favicon.svgThe browser tab icon. It switches theme itself
social-preview.pngThe 1280 × 640 link preview, uploaded in the repo’s settings

The wordmark is JetBrains Mono SemiBold converted to outlines, so it looks the same everywhere and needs no font installed. The files are plain SVG and open in Figma, Penpot or Inkscape.

Using it

Show the version that matches the reader’s theme. On GitHub and the docs site that means a <picture> element:

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="docs/brand/logo-dark.svg">
  <img alt="telemetry-blackbox" src="docs/brand/logo-light.svg" width="440">
</picture>
  • Minimum size: 16 px for the mark, 160 px wide for the full logo.
  • Clear space: half the mark’s height on every side.
  • Don’t recolour, stretch or rotate it, set the wordmark in another font, or put the light version on a dark background.
ColourHexWhere
Orange#f2662bThe sky, “blackbox” and accents
Ink#0f1720The box and the wordmark on light backgrounds
Box (dark)#1c2330The box on dark backgrounds, with a #4a5866 edge
Ground#2a3442Below the horizon
Slate#8b97a6The wordmark on dark backgrounds, chosen to stay legible (3:1) if it lands on a light page

Where it appears

PlaceWhenIssue
The README headerNow
The flight-progress stripNow#39
The repo’s social previewDay 4#29
The demo’s opening and closing cardsDay 4#29
The Grafana home dashboardDay 4#28
The docs site header and faviconStretch#35
The public Grafana dashboardStretch#36