Introduction
A flight recorder for a simulated flight controller. Rust firmware runs a 1 kHz control loop and streams MAVLink telemetry over a serial link. A Rust gateway, Redpanda and a Spate pipeline carry every frame into ClickHouse, on a highly available Kubernetes cluster I built by hand.
Status: under construction, in the open. Progress is tracked in the milestones, and the docs site updates with every push.
Progress
I fly the route in evenings after work, so each leg has a budget in evenings and a due date to match. The cluster flies before the firmware because it carries the most risk. #62 has the reasoning.
Each leg is a strip in a rack, the way air traffic control tracks flights. The leg I’m flying sticks out, a late leg is outlined in red and a white tick marks where the plan says I should be. Headwind counts the work that crept in after takeoff, and a stamp unlocks at each milestone I care most about. The flight log has the detail.
Why I’m building it
I want to understand four things from the bottom up:
- Kubernetes run on my own machines, including the parts that break.
- Rust, both
no_stdon a microcontroller and async on a server. - Real-time embedded systems and the protocols between them.
- Streaming data into ClickHouse with Spate.
A black box is a good way to learn all four at once. It has a hard deadline at one end, a database at the other and a simple test in between. Did every frame arrive?
The claim
Cut the serial link, kill the pipeline, drain a node or lose a control plane
in the middle of a flight. Every frame still reaches ClickHouse. Some arrive
twice, because delivery is at-least-once, and ClickHouse removes the
duplicates by key. bbcheck measures the outcome after each drill, and the
numbers go in Results.
How it fits together
Each part has one job.
| Part | Built with | Job |
|---|---|---|
firmware/ | Rust, no_std, Embassy | Runs the control loop on a hardware timer and sends telemetry over UART |
gateway/ | Rust, tokio | Finds frame boundaries, checks CRCs, spots sequence gaps and publishes to Redpanda |
pipeline/ | Rust, Spate | Decodes frames, routes them into tables and loads ClickHouse |
deploy/ | cloud-init, kubeadm, YAML | Builds the VMs and the cluster, then runs everything on it |
tools/bbcheck/ | Python | Proves what arrived: gaps, duplicates and latency |
The cluster
Five VMs on one machine. Three control planes share a virtual IP, so the API stays up when one of them goes down. Two workers carry the workloads, and a node label decides which one owns the serial link.
Renode runs on the host, outside the cluster, because it stands in for a
physical device. On real hardware the gateway would read /dev/ttyUSB0.
Here it reads the TCP socket Renode exposes.
The life of one frame
The last step matters most. Spate only commits an offset once ClickHouse has
confirmed the batch is durable, so a crash can repeat work but never skip it.
That is where duplicates come from, and why the table removes them by
(boot_id, system_id, seq).
The wire format
Telemetry uses MAVLink v2. Every frame carries a sequence number, which is
what lets the gateway and bbcheck count gaps.
| Bytes | Field | Notes |
|---|---|---|
| 1 | STX | Always 0xFD in v2 |
| 1 | len | Payload length |
| 1 | incompat_flags | Bit 0 marks a signed frame |
| 1 | compat_flags | Safe to ignore if unknown |
| 1 | seq | Wraps at 255; a jump means lost frames |
| 1 | sysid | Which vehicle |
| 1 | compid | Which component on it |
| 3 | msgid | 24-bit message ID |
len | payload | Trailing zero bytes are trimmed |
| 2 | checksum | CRC-16/MCRF4XX, seeded with the message’s CRC_EXTRA |
| 13 | signature | Only on signed frames |
The firmware sends HEARTBEAT, ATTITUDE, HIGHRES_IMU and a custom
LOOP_TIMING message that reports overruns and jitter.
Real-time, honestly
The control loop is hard real-time inside the simulation. Renode runs in deterministic virtual time, so loop timing reflects the firmware alone.
Everything after the serial port is soft real-time. I measure it as end-to-end latency percentiles, from the firmware’s timestamp to the row landing in ClickHouse.
Running it
Still being built. When it’s done, it will look like this:
mise install # pinned tools: just, D2, uv, kubectl, Helm and more
just bench # check everything is installed
just lint # formatting and lints for the crates, firmware and bbcheck
just test # the tests, plus a firmware build
just coverage # line coverage across Rust and Python
just cluster # five VMs and a highly available cluster
just deploy # Redpanda, ClickHouse, Grafana, gateway and pipeline
just fly # boot the firmware in Renode and start streaming
just check # run bbcheck against ClickHouse
just docs # build the docs site into book/
just audit # scan the history for secrets
just on its own lists every recipe. You’ll also need Linux with KVM, about
11 GB of free memory, mise, Multipass, Renode, probe-rs,
the ClickHouse client and Rust with the thumbv7em-none-eabihf target.
A released version
Every release puts multi-arch images of the gateway and pipeline on GitHub Container Registry. Once the firmware has a binary, it’s attached to the release too. The changelog says what each one changed.
docker pull ghcr.io/guy-with-a-why/telemetry-blackbox/gateway:latest
docker pull ghcr.io/guy-with-a-why/telemetry-blackbox/pipeline:latest
gh attestation verify oci://ghcr.io/guy-with-a-why/telemetry-blackbox/gateway:latest \
-R guy-with-a-why/telemetry-blackbox
Each image carries a provenance attestation, so the last command proves it was built here, from a tagged commit.
Repository layout
firmware/ Embassy firmware for the STM32F4
renode/ the board Renode simulates, as a script
gateway/ serial link to Redpanda
pipeline/ Spate pipeline into ClickHouse
deploy/ VMs, cluster bootstrap and manifests
tools/bbcheck/ delivery and latency report
docs/adr/ decisions and the reasons behind them
docs/brand/ the logo, favicon, social preview and how to use them
docs/diagrams/ D2 sources and the SVGs they render
docs/runbooks/ drills, with real output
scripts/ small checks the justfile and CI call
Decisions
Each gets a short record in docs/adr/.
- MAVLink v2 over a custom framing.
- kubeadm over k3s or Talos.
- Redpanda over Apache Iggy or a home-made log.
- Deduplicating in ClickHouse over chasing exactly-once.
- just and mise over make, and a local registry for images.
Results
These fill in as the drills run.
| Drill | Frames sent | Missing | Duplicates | Recovery |
|---|---|---|---|---|
| Kill the pipeline | pending | pending | pending | pending |
| Drain the pipeline’s node | pending | pending | pending | pending |
| Restart Redpanda | pending | pending | pending | pending |
| Cut the serial link for 60 s | pending | pending | pending | pending |
| Lose a control plane | pending | pending | pending | pending |
| Scale the pipeline to two replicas | pending | pending | pending | pending |
Diagrams
Every diagram is a D2 file in docs/diagrams/. D2 lays
it out, and each SVG follows the viewer’s light or dark theme. The animated
ones step through boards declared in the same file.
To change one, edit the .d2 file and watch it live, then render the lot:
d2 --watch --layout=elk docs/diagrams/architecture.d2
just diagrams
Cite
These are the documents I need to build this, grouped by where I use them. The board Renode simulates is ST’s STM32F4DISCOVERY, which is built on the STM32F407VG, a Cortex-M4F.
Tooling (Leg 0)
- The just manual and the mise documentation.
Microcontroller (Leg 2)
- STM32F407VG datasheet: pinout, memory map and electrical limits.
- RM0090 reference manual: the timers, USART, DMA and clocks, register by register.
- PM0214 Cortex-M4 programming manual: the NVIC, interrupt priorities, SysTick and the FPU.
- ES0182 errata: known silicon faults. Renode won’t reproduce them, but real hardware would.
- UM1472 Discovery kit user manual: the board around the chip.
- Renode’s STM32F4 Discovery platform and the Renode documentation.
- The Embassy book and the embassy-stm32 API docs.
Protocol (Legs 2 and 3)
- MAVLink serialisation: the v2 frame byte by byte.
- MAVLink CRC: the checksum and
CRC_EXTRA. - MAVLink common messages:
HEARTBEAT,ATTITUDEandHIGHRES_IMU.
Cluster (Leg 1)
- Multipass documentation.
- Creating highly available clusters with kubeadm.
- kubeadm configuration (v1beta4), for the init and join files.
- kube-vip as a static Pod.
- Cilium on a kubeadm cluster.
- etcd disaster recovery.
- Upgrading kubeadm clusters.
- The Distribution registry and containerd registry hosts, for the local image registry.
Data path (Legs 1 and 3)
- Redpanda on Kubernetes.
- ClickHouse ReplacingMergeTree.
- Spate’s connector contracts and its delivery invariants.
Diagrams
Licence
Dual-licensed under MIT or Apache 2.0, at your option. Any contribution you submit is licensed the same way, unless you say otherwise.
Flight log
19 of 43 issues closed (44%). 9 joined after takeoff. Stretch: 7 of 21.
Rebuilt whenever an issue changes. Last update: 2026-11-18 21:09 UTC.
Leg 0 · The bench
landed, due 2026-11-12
- #40 Release versioned increments and publish them from CI (landed 2026-11-13)
- #37 Decide on just, mise and a local registry (landed 2026-11-13)
- #2 Scaffold the workspace and CI (landed 2026-11-12)
- #1 Install the bench tools (landed 2026-11-12)
- #44 Show build health and assurance badges in the README (landed 2026-11-12)
- #39 Track progress with a flight strip (landed 2026-11-12)
- #38 Check every push for secrets and blocked terms (landed 2026-11-12)
- #35 Publish the docs site and keep it current (landed 2026-11-12)
Leg 1 · A cluster I built myself
in flight, due 2026-11-20
- #14 Run Redpanda on local storage (landed 2026-11-18)
- #13 Drill: snapshot and restore etcd (landed 2026-11-17)
- #12 Drill: lose one control plane, then two (landed 2026-11-17)
- #105 Bootstrap the cluster with one command (landed 2026-11-15)
- #11 Bootstrap a highly available control plane by hand (landed 2026-11-15)
- #82 Mirror quay.io and ghcr.io on the host, so Cilium and kube-vip pull once (landed 2026-11-14)
- #75 Cache packages and images on the host, so a rebuild doesn't refetch them (landed 2026-11-14)
- #10 Build five VMs with cloud-init (landed 2026-11-14)
- #15 Run ClickHouse with raw, deduplicated and rolled-up tables
- #16 Decide on kubeadm and on Redpanda
- #17 Checkpoint: explain quorum, static Pods and MergeTree merges
Leg 2 · Firmware and the wire
started early, due 2026-11-24
- #3 Boot Embassy firmware on an STM32F4 in Renode (landed 2026-11-14)
- #62 Re-plan the route in evenings (landed 2026-11-14)
- #56 Keep a daily blog on the docs site (landed 2026-11-13)
- #4 Drive the control loop from a hardware timer at 1 kHz
- #5 Estimate attitude from a simulated IMU
- #6 Send MAVLink v2 telemetry over UART with DMA
- #7 Read the telemetry on the host with pymavlink
- #8 Decide on MAVLink v2 over a custom framing
- #9 Checkpoint: explain the firmware and the frame format
- #61 Test the firmware in Renode with Robot Framework
Leg 3 · Serial port to ClickHouse
not started, due 2026-11-30
- #18 Find MAVLink frames in a noisy byte stream
- #19 Publish frames to Redpanda without losing any
- #20 Run the gateway on the node that owns the serial link
- #21 Stream frames from Redpanda into ClickHouse with Spate
- #22 Route attitude, IMU and timing into their own tables
- #23 Fly end to end, from firmware to Grafana
- #24 Decide to deduplicate in ClickHouse over chasing exactly-once
- #25 Checkpoint: explain a frame's whole path
- #107 Deliver the workloads from Git with Flux
Leg 4 · Break it and prove it
not started, due 2026-12-04
- #26 Report gaps, duplicates and latency with bbcheck
- #27 Drill: break the pipeline while it streams
- #28 Build Grafana dashboards for the whole path
- #29 Finish the README and record a demo
- #30 Checkpoint: walk through the system and defend the decisions
Stretch
bonus
- #116 Run only the CI a change needs, and merge stacks in one go (landed 2026-11-17)
- #47 Scan the code with CodeQL (landed 2026-11-16)
- #46 Keep pinned actions current with Dependabot (landed 2026-11-16)
- #45 Choose a licence (landed 2026-11-16)
- #95 Draw how the flight tracker works (landed 2026-11-15)
- #90 Draw a narrow flight rack for phones (landed 2026-11-14)
- #86 Redraw the flight strip as a rack, one strip per leg (landed 2026-11-14)
- #31 Test firmware on the cluster from GitHub Actions
- #32 Write an Apache Iggy source for Spate
- #33 Add a CAN bus alongside MAVLink
- #34 Queue work by priority with Kueue
- #36 Stream metrics to a public Grafana dashboard
- #102 Serve the metrics API with verified kubelet certificates
- #124 Provision volumes on demand with local-path-provisioner
- #130 Rebuild the platform on Talos, OpenTofu and Flux
- #131 Decide on Talos, OpenTofu and Flux for the finished platform
- #132 Provision the VMs with OpenTofu on libvirt
- #133 Run Talos on the nodes
- #134 Run the host services from Compose
- #135 Move the cluster checks and etcd backups onto standard tools
- #136 Turn the checks into Chainsaw tests
How this page is drawn
The diagram is wide, so here it is full size.
Cockpit voice recorder
A flight recorder has two halves. The flight log is the data recorder. This is the voice: a short post after each evening’s flying, on what I built and what broke, and a debrief when each leg lands. Posts start from _template.md and debriefs from _debrief.md.
| Date | Post |
|---|---|
| 12 Nov 2026 | Day 0: Laying out the bench |
| 14 Nov 2026 | Flight plan amended |
| 14 Nov 2026 | Ground power: first boot in Renode |
| 14 Nov 2026 | The hangar: five VMs and two caches |
| 15 Nov 2026 | Engine start: a control plane built by hand |
| 15 Nov 2026 | Autostart: the whole cluster in one command |
| 16 Nov 2026 | Deadstick: two control planes down |
| 17 Nov 2026 | Last known fix: etcd restored from a snapshot |
| 18 Nov 2026 | Holding pattern: Redpanda on data-1’s own disk |
Day 0: Laying out the bench
12 November 2026
What was done
No firmware today, and that was the plan. By the evening the repo was public and every push was checked for secrets and blocked terms. The docs site rebuilds whenever an issue changes, and a flight strip on it shows how far through the plan I am. The Rust workspace, bbcheck and CI were scaffolded by half nine, so Day 1 starts on firmware straight away.
Why was it done that way
The repo went public before any code, so the checks had to come first. gitleaks scans the whole history. A blocked-terms check covers files, commit messages and issue text, and GitHub’s push protection sits behind both. That’s cheap on day one and expensive to retrofit.
The diagrams are D2 because they stay as text I can diff and they render in light and dark themes. Tasks go through just, since they’re commands with arguments rather than builds.
Half an hour after takeoff I moved takeoff, so the issues I’d added that afternoon now count against me on the flight strip. Technically it is scope creep.
What challenges were faced
Two small things. Five minutes after the site went live, the dark logo turned out to be unreadable on the light theme, so that was the first fix.
The more interesting one was in the flight log. flight.py stitched gh’s pages of issues together by swapping ][ for ],[, which would also have mangled any issue title containing ][. Editing JSON as text is a shortcut I know better than to take. gh api --slurp already returns a single array, so the stitching went and a test with the title [a][b] keeps it that way (#43). Nobody hit it.
What’s next
Tomorrow is firmware. The riskiest part is getting Embassy and Renode to agree on the board, so that goes first. Every day of the plan has a fallback if it runs long.
Flight plan amended
14 November 2026
What was done
I’ve re-planned the route. The plan had five Days of nine hours each, 09:00 to 18:00, and I fly this in the evenings after a full working day. A Day was always going to be three or four evenings. The Days are now legs. Each has a budget in evenings and a due date that assumes weekday evenings only, and the last leg lands on 4 December. This timeline is documented in the table in #62.
Why was it done that way
A plan that’s late from its second day stops being a plan. By Monday the flight strip would have shown every waypoint red and told me nothing useful. Budgets in evenings make the strip honest again, and a weekend counts as time in hand.
The order changed too. The cluster now flies before the firmware, because it carries the most risk on the route and everything downstream runs on it. Five VMs have to share a 16 GB machine, and I bootstrap the control plane by hand. The firmware and the cluster don’t meet until the data path leg, so flying the riskier one first costs nothing and puts the slack where I need it.
What challenges were faced
Day 0 was planned at an hour. It took two evenings, about eight and a half hours. The bench itself was quick. Releases, README badges, this blog and a security policy all joined the scope on the way. Each was worth having and none of them fitted in the hour. The flight strip counted them as headwind, which is what it calls work added after takeoff. From here, new scope goes to Stretch unless it blocks a leg.
What’s next
Today #3 lands. The firmware already boots in Renode and blinks the board’s green LED, and I’m checking its memory map against the STM32F407VG datasheet before it merges. Once I’m happy with that the next focus will be the cluster, with four evenings to Friday 20 November. The biggest risk is memory, because I’m building the whole platform on one 16 GB laptop on purpose. If five VMs don’t fit in 16 GB, I drop to one control plane on a VM and run the HA drills on kind, which runs a whole multi-node cluster in Docker for a fraction of the memory.
Ground power: first boot in Renode
14 November 2026
What was done
The firmware boots. It’s an Embassy binary for the STM32F407VG, running on the STM32F4DISCOVERY board that Renode simulates, and it blinks the green LED on PD12 every 500 ms. just fly builds it, boots the board and shows Renode logging each change of the LED. That closes #3.
The binary is small: a 392-byte vector table, 3.4 KB of code and 96 bytes of RAM. cargo bloat puts 1.5 KB of the code in the main task, where Embassy’s chip setup and my blink loop are inlined, and 1.1 KB in the TIM2 timer that wakes it.
Why was it done that way
I wrote the memory map by hand rather than let Embassy generate it. It’s the contract between the linker and the silicon, so I want it in the repo where I can see it. It has two regions from ST’s RM0090 reference manual: 1 MiB of flash at 0x0800_0000, where the core boots, and 128 KiB of SRAM at 0x2000_0000.
The chip’s 64 KiB of core-coupled memory is left out on purpose. CCM is RAM wired straight to the CPU, so the DMA controllers can’t reach it. #6 sends telemetry over the UART by DMA from a static buffer, and if the linker put that buffer in CCM, the transfer would fail while the CPU side looked healthy. Leaving CCM out of the map rules that out, in Renode and on real silicon.
Embassy keeps time on TIM2, chosen explicitly so the other timers stay free for the 1 kHz loop in #4. The clocks stay on the 16 MHz internal oscillator, which asks the least of Renode’s partial clock model. Speed can wait until there’s a loop to measure.
What challenges were faced
Renode was the first wrong turn. The Debian package expects .NET 8 and two other dependencies on the host, and dpkg left it unconfigured. The portable build bundles its own runtime and was already working, so I removed the package.
The rest was reading the binary to check it matched the map. The stack starts at 0x2002_0000, the top of RAM, and the reset vector at 0x0800_0189 points at cortex-m-rt’s reset handler with the Thumb bit set. That handler zeroes 72 bytes of .bss and copies 24 bytes of .data before main, and the Embedonomicon explains why. The zeroing matters here. From #6 the MAVLink sequence counter lives in a static, and if it started at a random value, bbcheck would report missing frames that were never sent.
What’s next
The cluster flies next, with four evenings to Friday 20 November. The firmware returns with #4, which moves the loop onto a hardware timer at 1 kHz.
The biggest firmware risk is the gap between Renode and silicon. The CCM case is one example, and ST’s ES0182 errata sheet lists faults Renode won’t reproduce. My fallback is to tie timing claims to Renode’s virtual time, where they’re deterministic, and mark anything the simulation can’t prove as one for real hardware.
The hangar: five VMs and two caches
14 November 2026
What was done
The cluster has somewhere to live. just vms builds five Ubuntu VMs with Multipass, all from one cloud-init file: three control planes, edge-1 and data-1. Each node comes up with containerd and kubeadm 1.36, one minor behind the latest, so there’s an upgrade to practise later. A local registry carries my images in, and just vms-check proves every node is ready for kubeadm.
The first full build took 18 minutes. Caches on the host bring that down to 6m 09s once they’re warm.
Both pieces are up for review. #74 builds the VMs and closes #10. #78 adds the caches and is stacked on it.
Why was it done that way
I started on my laptop and moved within the hour. The VMs need 10.5 GB of memory, and the laptop had 6 GB free and a disk at 95%. My home server has 16 GB and room to spare, but only four cores for ten vCPUs. That’s fine at idle, and two vCPUs per node is the floor kubeadm checks for.
The control planes get 1.5 GB, under the 1.7 GB kubeadm’s preflight asks for, so #11 skips that one check. The figure is generic: etcd stays small in a five-node cluster, and the memory does more good on data-1, where Redpanda and ClickHouse will run.
I weighed the infrastructure-as-code options before writing any of it. Terraform earns its keep with a cloud API and state worth tracking, and Multipass has neither a first-party provider nor state I care about. Ansible suits long-lived hosts patched in place, and these VMs are cattle: if one drifts, I delete it and rebuild it in six minutes. So cloud-init sets each node up once at boot, and a short shell script drives Multipass.
All five share one Multipass zone. Multipass gives each zone its own subnet, with the host routing between them. kube-vip’s leader claims the API’s virtual IP by answering ARP for it, and ARP only reaches machines on the same subnet. Spread the control planes across zones and the address can’t move to a survivor when the leader fails.
When the first build took 18 minutes, I measured before changing anything. cloud-init’s own timings put most of each node’s 3m 20s in package downloads, and the server’s connection tops out at 1.1 MB/s. Each VM fetched the same 169 MB, so five of them pulled 845 MB of identical bytes. Parallel launches would only have split that connection five ways. A cache fixes that: the first VM fetches, and the rest read from the host. A registry.k8s.io mirror does the same for the control-plane images, which kubeadm will pull on three nodes in #11.
What challenges were faced
containerd was the first surprise. Each node finds my registry through a small file under certs.d, which says that images named localhost:5000 come from the host over plain HTTP. Ubuntu now ships containerd 2.2, where Kubernetes pulls go through a new transfer service that never reads that file. So every pull went to https://localhost:5000 on the node itself and failed. Pulling the same image two ways narrowed it down. ctr, handed the file directly, worked. crictl, which pulls the way the kubelet does, didn’t. Setting use_local_image_pull = true sends Kubernetes pulls back through the older path, which reads the file.
The apt cache took three attempts. Proxy mode refused the Kubernetes repository with a 403, because apt-cacher-ng reads the colons in pkgs.k8s.io paths as a port. Its shortcut paths failed in milliseconds with a 503. I traced its system calls and found it looking up a host called ubuntu in DNS. Putting the real host first in the path, as in http://cache:3142/pkgs.k8s.io/core:/stable:/v1.36/deb/, got past the 403 and the 503. The kubectl package then took 11.6 seconds through the cache the first time and 0.02 the second. The signing key still comes straight from upstream, so the cache can’t swap it.
What’s next
#11 is next: a highly available control plane, built by hand with kubeadm, kube-vip and Cilium. The virtual IP has to sit outside the range Multipass hands out, so that gets checked first.
The biggest risk is 1.5 GB per control plane. If one runs short of memory, I’ll move memory from data-1 before giving up a node. If that isn’t enough, the fallback is one control plane on a VM and the HA drills on kind.
Engine start: a control plane built by hand
15 November 2026
What was done
I bootstrapped the cluster this afternoon. It’s a Sunday, so it came out of the time in hand. I typed every kubeadm command myself on my home server, because that’s the point of #11. Forty-seven minutes after kubeadm init on cp-1, just cluster-check came back clean. Three control planes answer behind one virtual IP, 10.97.0.250, two workers have joined and Cilium carries the pod network.
A kubeadm bootstrap starts like an airliner’s engines. They come on one at a time in a fixed order, and the first runs on ground power. The cluster even keeps a record of it. Every Pod’s age counts from when it started, so reading the AGE column oldest first replays the afternoon. cp-1 came up at 12:53, cp-3 joined eight minutes later, then cp-2, then both workers. Cilium went on about fifteen minutes in.
This is how things looked a minute or two after that. The control plane was running on all three nodes, but every node was NotReady and CoreDNS was stuck in Pending. That’s expected. A node only reports Ready once a network plugin has written its config to /etc/cni/net.d, and Cilium writes that file when its agent starts. The agents were still waiting for their images.
The ages show more than the order. kube-vip-cp-1 is a minute younger than etcd-cp-1, because the one edit I made after init restarted it. On cp-3, kube-vip is younger than everything else on the node, because a node has no admin.conf until it has joined. A couple of hours later, all 34 Pods in kube-system were Running. The restarts on cp-1 are a story for further down.
Why was it done that way
kube-vip is the first engine, and it needs ground power. kubeadm init reaches the API through the virtual IP, so kube-vip has to hold that address while init runs. It claims the address by taking a Lease from the API server, and that needs credentials. Since Kubernetes 1.29, admin.conf gets its rights from a binding that init only creates once the API server is up. So on cp-1, kube-vip starts on super-admin.conf, which is in the system:masters group and skips RBAC entirely. When init finishes, I point kube-vip back at admin.conf and the cart comes off. The kube-vip docs cover the step.
Forgetting that last step would be easy, and nothing would tell me. kube-vip would carry on working with rights no binding grants and none can take away. That’s why cluster-check now looks for it. super-admin.conf lives only on cp-1, readable by root, and nothing runs on it now.
The whole control plane runs as static Pods, kube-vip included. A Deployment needs the API server to accept it, etcd to store it, the controller manager to create its Pods and the scheduler to place them. Those four are the control plane, so they can’t start that way. Instead, the kubelet runs whatever manifests sit in /etc/kubernetes/manifests, with no API involved, and keeps them running if the API goes down.
What kubectl shows for a static Pod is a mirror Pod, a read-only copy the kubelet posts to the API. Deleting it changes nothing on the node. I tried it on cp-2. The API server’s Pod came straight back with an age of nine seconds, the same container underneath and the node itself, Node/cp-2, as its owner. To change a static Pod, I edit its file, which is exactly what switching kube-vip to admin.conf was.
kubeadm also wrote the cluster’s trust into /etc/kubernetes/pki. There are three certificate authorities, each good for ten years, and leaf certificates that last a year. The main one, ca, signs for the API server, the other components and both admin kubeconfigs. etcd gets its own authority because it holds everything. Apart from etcd’s own members, only the API server and the health check carry a certificate etcd accepts. A leaked kubelet certificate would cost me a node and nothing more. The third authority covers the API server when it hands requests on to extension APIs, and sa.key signs the tokens ServiceAccounts carry. The PKI reference lists every file.
Mine expire on 15 November 2027. kubeadm upgrade renews the control plane’s certificates as it goes, and kubelets rotate their own, so a cluster upgraded once a year never sees one lapse.
etcd is stacked, one member on each control plane. External etcd would keep its failures apart from the control plane, but it needs three more VMs on a 16 GB box. Stacked ties each member to its control plane, and #12 will test what that costs. I went with stacked. Each API server talks only to its own member, on 127.0.0.1:2379, and the members agree through Raft, which counts a write once a majority has it.
Three is the smallest cluster that survives a failure. With two of three up, there’s still a majority, so writes carry on. Any two majorities share a member, so a new leader always has every committed write. A fourth member adds a vote but no tolerance, because the majority grows to three. Two members are worse than one. Either failure stops writes, and there’s twice the hardware to fail. The etcd FAQ has the arithmetic.
One call of my own sits under the pod network. Cilium’s default pod range, 10.0.0.0/8, covers the bridge the VMs live on. I could have given Cilium a range of its own, but then the pod range would live in two places. So Cilium takes each node’s range from Kubernetes, and 10.244.0.0/16 is set once, in kubeadm.yaml.
What challenges were faced
Before starting the failover drill, I read back through deploy/ and wasn’t sure kube-vip was set up properly. The manifest turned out to be fine. It matches what kube-vip v1.3.1 generates itself, apart from two settings that only matter when the address comes from DHCP or DNS. The real problem was simpler. There was no cluster to fail over. #11 had closed itself when #84 merged the evening before, with every box unticked, and the VMs had been rebuilt that evening without kubeadm ever running. I’d linked the PR in GitHub’s Development panel, and that closes an issue on merge, whatever the description says. I reopened it. Now I reference issues only in a pull request’s description, and I check that a pull request closes nothing it shouldn’t before it merges.
Going through the checks turned up a gap in my own proof. cluster-check confirmed one kube-vip held the address and never asked about the other two. A crashlooping standby would have passed, then failed the drill that matters. Nothing checked that cp-1 had come off super-admin.conf either. #100 now fails unless all three run kube-vip on admin.conf. It also fixes how I found kube-vip’s network interface. I took the third field of ip route get for the virtual IP, but on the node holding that IP the answer starts local 10.97.0.250 dev lo, and the third field is the word dev.
Cilium was slow to start. All five nodes pulled the same images through a quay.io mirror that had never seen them, on a line that tops out near 1 MB/s, and the agents waited over twenty minutes. The mirror pays off on the next rebuild.
The interesting bit came as Cilium finished. Three of its agents exited once as they started, and my home server was flat out. A few minutes later its load average read 10 on four threads. cp-1 stopped keeping up, and within twenty seconds it lost every leadership it held. etcd went first, at 13:36:37, when cp-3 won an election. Then kube-vip’s Lease went, then the controller manager’s and the scheduler’s. Each of those three stood down by exiting, and the kubelet started it again as a standby. Those are cp-1’s restarts in k9s.
kube-vip’s logs show the odd part. cp-3 took the address at 13:36:49, but cp-1 was so starved it only noticed its renewal had failed three seconds later. It then spent twelve seconds trying to hand the Lease back through its own slow API server, and only dropped the address at 13:37:09. For twenty seconds, two nodes answered for the same IP. The cluster came through it, and #12 now has a failover from the wild to measure itself against.
What’s next
#12 takes the control planes down on purpose, one and then two. With one down, etcd keeps its majority, and if that node held the address, kube-vip should move it within about fifteen seconds. With two down, writes stop. Every kube-vip renews its Lease through its own API server, so I expect the address to vanish, while Pods already on the workers keep serving. #13 follows with an etcd snapshot and restore. That leaves #12 to #17 for the five evenings to Friday.
The biggest risk is CPU. My home server is a 2012 laptop with an i5-3320M, battery still at 91%. At idle, with nothing of mine on it yet, the VMs hold it at about half its CPU. The three control planes take about 11% each and the workers around 5%. Memory comes second, with 4 GB free and 575 MiB already in swap. Redpanda and ClickHouse will want their share of both.
A longer Lease would make false alarms rarer, but it wouldn’t stop a starved node holding on to the address late. A bigger CPU share for the control-plane VMs would, so a busy data-1 can’t starve them. I’ll measure both in #12 and keep whichever the numbers back.
#11 closed this afternoon with every box ticked.
Quiz
Nine questions on the post, for anyone who wants to check it’s landed. Open one for my answer.
You delete kube-apiserver-cp-2 with kubectl. What happens on the node?
Nothing. I only deleted the mirror, the kubelet put it straight back and the container never stopped.
Why can't the control plane run as a Deployment?
A Deployment needs the API server, etcd, the scheduler and the controller manager to exist first. Those are the control plane, so the kubelet starts them from files on disk.
Why was every node NotReady until Cilium started?
The kubelet only reports Ready once a network plugin has written its config to /etc/cni/net.d. Cilium writes it when its agent starts.
Why does kube-vip start on super-admin.conf, and why move it off?
It has to take its Lease before admin.conf has any rights. super-admin.conf skips RBAC and nothing can revoke it, so it only stays until init finishes.
Why does etcd have its own certificate authority?
So a certificate from the main authority can’t open the database. Only the API server and the health check hold one that etcd accepts.
The certificates last a year. What keeps them from expiring?
kubeadm upgrade renews the control plane’s as it goes, and kubelets rotate their own. Upgrade at least once a year and nothing lapses.
Why run three etcd members and not four?
Three survives one failure. Four still only survives one, because the majority grows to three, and there’s more hardware to fail.
Two control planes go down. What happens to the virtual IP?
I expect it to vanish. Each kube-vip renews its Lease through its own API server, and without quorum nothing can renew it. #12 will tell me if I’m right.
cp-3 took the address at 13:36:49. Why could cp-1 still answer for it?
cp-1 was too starved to notice it had lost the Lease, then spent twelve seconds trying to hand it back through its own slow API server. Only then did it drop the address.
Autostart: the whole cluster in one command
15 November 2026
What was done
This afternoon I built the control plane by hand, and it took 47 minutes from kubeadm init to a clean check. This evening I made it one command. just up builds the five VMs, initialises cp-1, joins the other four nodes, installs Cilium and runs cluster-check. From nothing to a passing check took 20 minutes and 8 seconds, VMs included. Run it again on a healthy cluster and it skips every node in under a minute.
The engines still start in the same order, with cp-1 on ground power first. A jet’s autostart runs the same sequence a crew would, and the crew only has to press the button. #105 has the work and ADR 6 the reasoning.
Why was it done that way
If a sequence of actions is repeated by hand, and often, the system should do it instead. The drills in #12 and #13 break the cluster on purpose, so I’ll be rebuilding it several times a week.
Terraform is the usual way to stand up VMs, so I looked at it first. Its Multipass provider doesn’t support zones, and kube-vip needs all five VMs in one zone so they share a subnet. Ansible would have meant SSH keys and an inventory that copies my config, all for about ten commands. So kubeadm’s own config files carry the joins. The token and the certificate key are generated fresh on each run and set before init. The CA’s hash can’t be, because the CA doesn’t exist until init creates it, so the script reads it straight after. Every step checks before it acts, which is what makes a second run harmless.
What challenges were faced
The first full run hung at step 7, copying the kubeconfig off cp-1. multipass exec was writing straight into a file, and it sat at 99% CPU for twenty minutes without writing a byte. Every other call in the scripts reads the output into a variable first, so this one does too now. Rerunning on the half-built cluster finished the job and proved the skip path a little earlier than planned. Then I tore it all down and rebuilt it in one clean pass, so the 20 minutes is a single honest run.
What’s next
#12 takes the control planes down, one and then two, and #13 restores etcd from a snapshot. Either can now end in just up and twenty minutes.
Further out, what runs on the cluster should come from Git too. #107 puts Flux in charge of Redpanda, ClickHouse, the gateway, Spate and Grafana when Leg 3 brings them, so a rebuild restores the workloads as well as the cluster. Flux over Argo CD, because it’s a few small controllers and my workers have 2 GB and 4 GB. The risk is Flux undoing a drill’s damage as fast as I cause it, so the drill runbooks will suspend it first.
Deadstick: two control planes down
16 November 2026
What was done
Tonight I took the control planes down on purpose, one and then two, for #12. Losing cp-2 changed nothing: etcd kept a majority, two of three. cp-1 held the virtual IP all along, so this run never made kube-vip move it. Timing that move means stopping whichever control plane holds it.
Losing cp-3 as well took the majority with it. Writes stopped on the next probe. The API server kept answering cached reads for another 22 seconds, until kube-vip couldn’t renew its Lease and gave up the address. Yesterday’s post predicted that.
The echo Pods on the workers didn’t notice, and answered 79 of 79 probes. It’s a
deadstick glide: nothing gets rescheduled or replaced, but everything already
flying carries on, because running Pods and
kube-proxy’s iptables rules
don’t need the control plane. Starting cp-2 and cp-3 brought quorum back, and
cluster-check passed once Cilium settled. The
runbook has the output.
Why was it done that way
The drill needed a witness that doesn’t lean on what it’s watching.
just drill-watch runs on the home server, outside the cluster, and logs every
change in echo, the API and the address. It sends a consistent read, which needs
quorum, and a cached read at resourceVersion=0, which doesn’t, so I can tell
an API server that has lost etcd from one that has gone. Each VM goes down with
multipass stop --force, which pulls the power, because a clean shutdown lets
etcd hand over leadership first and real failures don’t.
#111 has the
script.
An infrastructure engineer I know asked me why all this runs on Bash instead of Terraform. Terraform reconciles a state file against an API, and its Multipass provider documents no zone option, which kube-vip needs. The drills break VMs by hand too, so the state would be wrong after every drill. The scripts check what’s there before they act, so the cluster is its own record. Terraform also stops at the machines, and the hard part is ten kubeadm commands in order across five of them. On real machines, or a hypervisor with an API such as Proxmox, Terraform for the machines and Ansible for kubeadm would earn their place, as ADR 6 says.
What challenges were faced
The first run ended early, for a reason that had nothing to do with etcd. With
cp-2 and cp-3 down, I ran just cluster-check, and afterwards all five VMs were
running. I hadn’t started any of them. Multipass’s log had cp-2 booting at
19:54:00 and cp-3 twenty seconds later. cluster-check finds a ready API server
by running multipass exec on each control plane in turn. Without quorum cp-1
wasn’t ready, so it moved on to the stopped ones, and multipass exec starts a
stopped VM first. Its help doesn’t say so. Its source does. The tool meant to
report the outage ended it.
Both scripts now only touch running VMs, and the second run kept cp-2 and cp-3 down until I started them. #111 has the fix.
Afterwards I checked what happens when a worker goes instead. I cordoned edge-1 and deleted its echo Pod, and the replacement sat in Pending. echo’s spread rule allows at most one Pod more on any node than on the emptiest, and by default a node echo can’t run on still counts as empty. A second Pod on data-1, against none on edge-1, broke the rule, so the one place it could go was refused.
One line fixes it. nodeTaintsPolicy: Honor makes the rule count only nodes
echo can run on, so a cordoned or dead worker and the control planes drop out of
the sum. data-1 stops being measured against nodes it can’t use, and the
replacement starts there. Without it, losing a worker would have cost echo half
its Pods until the worker came back. The catch is that the scheduler never moves
a Pod back, so once edge-1 returns, both stay on data-1 until something restarts
one.
Applying that fix rolled echo out again and turned up a second gap: both new Pods landed on data-1. The rule still counted the old Pods on their way out, so data-1 never looked fuller than edge-1.
matchLabelKeys: [pod-template-hash] changes what gets counted: only Pods from
the same rollout, which share a template hash. Pods from the last rollout stop
steering the new ones, so every rollout ends with one Pod on each worker.
Without it, the spread came down to the timing of the last rollout, and losing
data-1 could have taken echo down entirely.
A re-run proved both fixes: the rollout left one Pod on each worker, and the cordon put the replacement on data-1 straight away. Neither gap shows while every node is healthy, which is what drills are for.
What’s next
#13 snapshots
etcd, deletes a namespace and restores it. A restore rewinds the whole cluster,
so every member has to
start again from the same snapshot.
The fallback is just up and twenty minutes. Leg 1 is due on Friday.
Quiz
Eleven questions on the post, for anyone who wants to check it’s landed. Open one for my answer.
cp-1's etcd was healthy and had every write. Why did it refuse to carry on alone?
It couldn’t tell dead peers from being cut off, and if cp-2 and cp-3 were alive, they were a majority that could keep writing. Two majorities always share a member, so only one side can write, and a write a majority holds survives into the next leader’s log. Raft explains how.
Writes stopped as soon as cp-3 went, but cached reads lasted 22 seconds. Why?
A plain read must be current, so it goes to etcd, which needs a majority. A read
at resourceVersion=0 comes from the API server’s watch cache in memory, and
that lasted until the address went.
Why does the virtual IP go when quorum does?
kube-vip renews its Lease through the API every 2 seconds and stands down after 10 without a renewal. Renewing and taking the Lease are both writes, so without quorum no kube-vip can hold the address.
The echo container crashes while the control plane is down. Does it come back?
Yes. The kubelet restarts it locally under
restartPolicy: Always,
with no API, and reports the new status once the API is back.
A worker dies while the control plane is down. What happens to its Pod and its traffic?
The Pod dies and nothing replaces it, because the ReplicaSet controller and the scheduler work through the API. Nobody removes it from the Service’s endpoints either, so about half the connections still go to it and time out.
One control plane stops. What does the cluster report, and when?
Its kubelet stops renewing its node Lease, which it does every 10 seconds, and after about 40 to 50 seconds the node goes NotReady with an unreachable taint. Ordinary Pods are evicted after 300 seconds, while DaemonSet Pods and static Pods stay with the node.
cp-2 came back before cp-3. Why could writes resume, and how does cp-3 catch up?
Two of three is a majority, so cp-1 and cp-2 elect a leader and commit a write once both have it. cp-3 is still a member, and the leader sends it what it missed as log entries, or as a snapshot if the log has moved on.
All three control planes run on one home server. What didn't the drill prove?
Anything that hits the host hits all three, such as CPU starvation or one slow disk under every etcd member. It also didn’t test a partition or a slow node, both harder than a clean stop. In production the three go in separate failure domains, and kubeadm’s HA topologies weigh where etcd sits.
With edge-1 cordoned, why did echo's replacement Pod sit in Pending?
The spread rule counted edge-1 and the control planes as nodes holding no echo
Pods, so a second Pod on data-1 broke maxSkew: 1. nodeTaintsPolicy: Honor
counts only nodes the Pod could run on.
Why did a rollout put both echo Pods on one worker?
The spread rule counted every Pod labelled app=echo, including the old ones
still running. At each step, data-1 looked no fuller than edge-1.
matchLabelKeys: [pod-template-hash] counts only Pods from the same rollout.
Why would Terraform's state go stale here?
The drills stop and delete VMs outside Terraform, so until the next apply its state describes a cluster that no longer exists.
Last known fix: etcd restored from a snapshot
17 November 2026
What was done
Tonight I took a snapshot of etcd, deleted a namespace and brought it back, for #13. Restoring a snapshot puts the whole cluster back at its last known fix, the moment the snapshot was taken, and anything that happened after it is gone.
just etcd-snapshot takes one from the first running control plane, copies it
to my home server and checks the copy’s hash. It takes 9 seconds and comes to
15 MB.
I ran the drill twice. The first time I worked each step out as I went, from the kubeconfig to the restore flags, and reads failed for 19 minutes. I wrote the runbook from that run and followed it for the second run. Reads were back 93 seconds after the API servers stopped, and echo answered a minute later. The namespace came back with its old UID and creation time, so as far as Kubernetes knows, it never left.
The runbook has the steps and the output.
Why was it done that way
Velero is the obvious alternative. It backs up objects through the API and can restore one namespace on its own, which is all this drill asks for. It can’t help when the control plane itself is lost, and that’s the failure an etcd snapshot covers. The snapshot is the floor, so it comes first. Object backups can sit on top of it later.
etcd runs as three members, one on each control plane, and the workers run
none. A member is a single etcd server with its own data, ID and peer address.
The snapshot goes through crictl inside the etcd container, because I’ll need
a snapshot most when the API is down. It reads one member’s own database, so it
doesn’t need quorum either. The file holds every Secret in plain text, so it’s
readable only by me.
Every member restores from the same file. One member restored alone would get a new cluster ID from the restore, and the other two would reject it. So every API server and etcd member stops, all three restore with one member list and they start as a new cluster that agrees from the first heartbeat. Each node keeps its old data directory until the cluster checks clean. If a member won’t start, all three go back to it, never one or two. The etcd recovery guide covers the mechanics.
The restore also moves the revision a billion ahead and marks everything older as compacted. The kubelets and controllers watch from the last revision they saw, which is newer than the snapshot. Without the jump, new writes would reuse numbers the watchers had already passed, so they’d never see them. With it, every watcher gets 410 Gone and lists everything again. The Kubernetes docs also restart the scheduler, the controller manager and the kubelets after a restore. I don’t, because the relist covers it, and the second run showed that’s enough here.
What challenges were faced
Before I could start, Multipass was down. Overnight, snapd refreshed it from 1.17.1 down to 1.17.0 and suspended all five VMs on the way. Their saved state pointed at the newer version’s firmware, so the older one crashed trying to resume them and left its DNS server holding port 53. Every restart after that failed on the port, which hid the real cause. Reverting to 1.17.1 brought all five back, and etcd recovered without a hand on it. Multipass now refreshes when I say so.
My first copy to a control plane went through multipass exec, and only 2.2 MB
of the 15 arrived. etcdutl refused it, because every snapshot ends with a hash
of its contents, so a torn copy fails safely. The runbook now copies with
multipass transfer and checks the hash on every node before anything stops.
kubeadm left a trap too. It grew the cluster one join at a time, so cp-1’s etcd manifest lists only cp-1 as the initial cluster. Copy that into a restore and cp-1 starts a cluster of its own. The runbook spells out all three members.
What’s next
Redpanda on local storage is next, in #14, then ClickHouse in #15. The risk is data-1’s 4 GB holding both. Their memory limits are the first lever, and giving data-1 6 GB is the fallback, since the home server has the room. Leg 1 is due on Friday.
Quiz
Five questions on the post. Open one for my answer.
Why can't an etcd restore bring back one namespace on its own?
A snapshot is the whole database at one revision. Restoring it puts every object back to that moment, and everything written since is lost.
Why restore every member, rather than one and let the others catch up?
The restore gives that member a new cluster ID. The other two still have the old one, so they reject it.
What does --bump-revision protect against?
Watchers that saw revisions newer than the snapshot. The jump stops new writes
reusing those numbers, and --mark-compacted makes the watchers get 410 Gone
and relist.
Why take the snapshot through crictl and not the
API?
The API is the thing most likely to be gone when I need a snapshot. crictl
only needs the node and a running etcd container.
echo's Pods came back with their old names but new addresses. Why?
The restore brought back the Pod objects, still bound to their nodes. The kubelets had torn down their sandboxes when the namespace went, so they built new ones, and Cilium gave each a new address.
Holding pattern: Redpanda on data-1’s own disk
18 November 2026
What was done
Tonight Redpanda went onto the cluster, for
#14. It’s
where every frame will circle until ClickHouse is ready to take it, so it has to
hold them through anything short of losing the disk. One broker runs as a
StatefulSet in dev mode, on a 10 Gi volume on data-1, with a telemetry.raw
topic of six partitions.
just redpanda deploys it and is safe to run again. A second run takes 15
seconds and changes nothing. just redpanda-check proves the issue’s first
box. It writes 10,000 numbered records, deletes the Pod and reads every one
back once the StatefulSet has replaced it. The broker was back on data-1 in 7
seconds with none missing, and 100,000 records came back the same way.
Redpanda’s image lives on Docker Hub, so the host gained a fourth pull-through mirror first, in #125. The broker and the check landed in #127 and #128.
Why was it done that way
The cluster had no StorageClass, so a claim had nowhere to go. A
StorageClass is the menu an admin offers: who creates a volume, when a claim binds to it and
what happens to the data afterwards. Mine is called local and has no
provisioner. I create each volume by hand, as a
local volume on one node’s disk, pinned to that
node. Binding waits for the
first Pod that uses the claim, so the scheduler picks a node before the claim
picks a volume. With immediate binding,
a claim could take a volume on a node the Pod can’t run on. The volume is set to
Retain, so deleting the claim never
deletes frames.
I passed over replicated storage such as Longhorn. There’s one data node, so it has nowhere to copy to, and with more nodes Redpanda and ClickHouse would replicate their own data more cheaply than a disk layer could. The price of local disks is that a stateful Pod can’t move. Draining data-1 stops Redpanda until the node is back, and the drain drill in #27 will show it. A dynamic provisioner is in Stretch as #124, for everything that doesn’t need a hand-made volume.
Redpanda is built on Seastar, the same framework as
ScyllaDB. It runs one
thread per core, claims its memory at startup and writes past the page cache.
So --smp=1 --memory=512M gives it one core and 512 MB, and --overprovisioned
stops it pinning and busy-polling data-1’s two shared vCPUs. The container’s
limit is 1 GiB, because Seastar’s share doesn’t cover the binary or its thread
stacks.
The bash behind the cluster is now about 610 lines. Talos, OpenTofu and Flux would bring that to one 60-line script plus about 350 lines of configuration, and I’ve filed that rebuild as #130 with six sub-issues. I’m not starting it yet. Building kubeadm by hand is the point of this leg, and the bash works, is linted and is about to stop changing, so replacing it now would be a rewrite rather than paying off debt. The Leg 4 drills are the safety net that will prove a new platform moved nothing, and Flux in #107 should come first, so the workloads are declarative before the layer underneath them changes. Writing it down now, with a trigger, makes it planned work. It also retired an issue that would have polished scripts #130 replaces.
What challenges were faced
The first deploy timed out. I’d rebuilt the cluster from a checkout on my home
server that predated the mirror, so nothing on the bridge served Docker Hub.
Redpanda’s 122 MB image came straight down my connection at about 2 minutes a
node, which outlasted the script’s 2-minute wait for the topic Job. The broker
and the Job had both finished fine. The wait is now 5 minutes, matching the
rollout. just vms-check would have caught the missing mirror, and just up
doesn’t run it yet.
The real mistake was durability.
Dev mode turns write
caching on, which acknowledges a write before it reaches the disk, so my first
version turned it off with a
bootstrap file
and called the broker durable. A second pass over the
manifests found that dev mode also starts the broker with
--unsafe-bypass-fsync=true, so an ack no longer waited for the disk to
confirm the write. Deleting a Pod leaves every cache below it intact, so the
restart check passed 100,000 records, which proves the caches are very reliable
right up until the power goes. The broker now starts with
--unsafe-bypass-fsync=false, and its command line in the logs shows it. A restart check can’t see this class of fault. The Leg 4
drills should include stopping data-1 hard.
I also broke the deploy on purpose to see how it fails. A volume left
Released after its claim was deleted now stops the script at once and prints
the command that frees it, rather than ending on a 5-minute timeout. A topic Job
that can’t reach the broker prints rpk’s own error within 25 seconds. That
needed restartPolicy: Never, because with OnFailure the Job deletes its Pod
when it
gives up
and the logs go with it.
What’s next
ClickHouse is next, on the same local class, in
#15. The risk
is data-1’s 40 GB disk. Redpanda
refuses writes when free
space falls under 5 GiB, and it measures that against the whole filesystem ClickHouse will share.
ClickHouse’s retention is the first lever, and a bigger disk for data-1 is the
fallback. Leg 1 lands on Friday, with the decisions in
#16 and its
debrief.
Quiz
Five questions on the post. Open one for my answer.
Why does telemetry.raw have six partitions, and what does
a key decide?
A partition is an ordered log, and it’s the unit consumers share out. Six let up to six consumers in one group read in parallel. The key’s hash picks the partition, so every frame with the same key stays in order.
What's an offset, and who keeps it for a consumer group?
An offset is a record’s position in its partition. The group commits the offset it has processed to the broker, so a restarted consumer carries on from there. Committing only after ClickHouse confirms a batch is what makes delivery at-least-once.
Why does the local class wait for the first consumer
before binding?
A local volume only exists on one node. Waiting lets the scheduler place the Pod first, so the claim binds to a volume the Pod can actually reach.
The claim was deleted and recreated. Why won't the volume bind again?
With Retain, the volume goes Released and keeps the old claim’s UID. Removing
that UID makes it Available for the new claim, and the data is still there.
Write caching was off. Why could Redpanda still lose acknowledged frames?
Dev mode also skipped fsync, so an ack didn’t wait for the disk to confirm the write. A Pod restart keeps whatever is still cached, and a power cut doesn’t.
Decisions
Each decision gets a short record: the context, what I chose, what I turned
down and what it costs. I write them as I make them, so this list grows over
the four days. Each one starts from _template.md.
Records only ever add to the history. Once one is accepted I never rewrite or delete it. If I change my mind, a new record supersedes it and the old one’s status says so. Every decision and every reversal stays on file.
| # | Decision | Status |
|---|---|---|
| 1 | MAVLink v2 over a custom framing | To write |
| 2 | kubeadm over k3s or Talos | To write |
| 3 | Redpanda over Apache Iggy or a home-made log | To write |
| 4 | Deduplicating in ClickHouse over chasing exactly-once | To write |
| 5 | just and mise over make, and a local registry for images | Accepted |
| 6 | A scripted kubeadm bootstrap over Terraform or Ansible | Accepted |
0005. just and mise over make, and a local registry for images
- Status: Accepted
- Date: 2026-11-12
- Issue: #37
Context
I started with a Makefile to render the diagrams. It didn’t last. Make is a
build tool, not a task runner: recipes need tabs, every task needs .PHONY
and shell variables need $$. Its strength is rebuilding files when their
inputs change, and cargo and uv already do that for their own code.
The project runs from firmware up to the cluster, so it spans four ecosystems:
- Rust, for the host crates and the firmware.
- Python, for bbcheck and the scripts.
- D2 and mdBook, for the docs.
- kubectl and Helm, for the cluster.
Each has its own tooling and none of it covers the others. I need one entry point that sits above all four.
CI must run the same tools at the same versions as my laptop. Otherwise a check can pass in one place and fail in the other.
The constraints:
- Everything runs on one laptop with about 10 GB of free memory, shared by five VMs and Renode.
- Rust is pinned in
rust-toolchain.toml, because rustup owns the toolchain and the firmware target. - Multipass, Renode, probe-rs and the ClickHouse client are installed by hand. They don’t fit a version manager.
- Images built on the host have to reach containerd on five VMs, many times a day while I iterate.
- I have four days, so the tooling has to be boring and quick to set up.
Decision
justruns every task. The justfile only calls programs and scripts, and the real logic lives in them.misepins the CLI tools inmise.toml. CI installs from the same file withjdx/mise-action.- A
registry:2on the host carries images into the cluster, and containerd on every VM trusts it throughhosts.toml. #10 builds it.
Options I turned down
- make. It was my first choice. For the reasons above, I replaced it before the first pull request.
- cargo xtask. Tasks written in Rust suit a Rust-only repo. Here they would mostly wrap Python, D2 and Helm, and every change to a task means a compile.
- asdf, or installing by hand. asdf needs a plugin per tool. Hand installs drift, and CI stops matching my laptop.
- Importing images by hand.
ctr images importhas to run on each of five nodes. It’s easy to leave one stale, and pods don’t pull the way they would in production. - GHCR for every build. Each build would need a push, credentials and the network. GHCR stays the release channel (#40). The local registry is for the inner loop.
Consequences
- There are two more tools to learn.
mise installbringsjust, so setup is still one step. - The registry is one more process on the host, but it’s small next to five VMs.
- The registry speaks plain HTTP on a private network. That’s fine on a laptop and wrong anywhere else.
- Every task has one name, and CI runs the versions I run.
- Pods pull images as they would from a real registry, so
imagePullPolicybehaves normally. - I’d revisit this if the project moved to more than one machine, or if devcontainers or Nix took over the whole bench.
References
0006. A scripted kubeadm bootstrap over Terraform or Ansible
- Status: Accepted
- Date: 2026-11-15
- Issue: #105
Context
I bootstrapped the cluster by hand for #11, following
the bootstrap runbook. It took 47 minutes from
kubeadm init to a passing just cluster-check. Doing it once by hand was
the point. Doing it every time isn’t.
The drills that follow break the cluster on purpose. #12 stops two control
planes, #13 restores etcd from a snapshot and the 1.36 to 1.37 upgrade can fail
halfway. Any of them can end in a rebuild, and a missed step by hand, such as
kube-vip left on super-admin.conf, costs an evening.
Part of the path is already scripted:
just hostruns the registry, the image mirrors and the apt cache.just vmsbuilds the five VMs from cloud-init and skips any that exist. A warm rebuild takes 6m09s.just cluster-filesrenders the kubeadm and kube-vip files.just cluster-checkproves the result.
What’s left is runbook steps 2 to 8. That’s about ten commands across five VMs, in order, passing a token, a certificate key and a CA hash from cp-1 to the others. The machines already exist, so this is orchestration, not provisioning.
The constraints:
- One server runs five Multipass VMs in one zone, because kube-vip’s ARP mode needs one subnet.
- I reach the VMs with
multipass exec, not SSH. - Leg 1 is due on 20 November, and this has to land before #12.
- Every line of the automation has to map to a runbook step.
Decision
just cluster-upruns runbook steps 2 to 8 throughscripts/cluster-up.shovermultipass exec.just upchainsvms,cluster-upandcluster-checkinto a rebuild from nothing.- kubeadm’s own configuration carries the joins.
kubeadm.yamltakes a bootstrap token and a certificate key, generated on each run and never committed. AJoinConfigurationper role replaces the join flags,--ignore-preflight-errors=Memincluded. The script reads the CA hash after init, because the CA doesn’t exist before it. - Each step checks before it acts. Init is skipped if cp-1 has
admin.conf, and a join is skipped if the node haskubelet.conf. - The script copies
admin.confto the server and installs Cilium from there with Helm.just kubeconfigsets up my laptop’s copy and the tunnel. - The VMs stay on
vms.sh. The upgrade and the drills stay manual.
Options I turned down
- Terraform or OpenTofu for the VMs. Terraform earns its keep by
reconciling state against an API. Here that’s one server’s Multipass,
already driven by a script that skips what exists. The community provider
documents no
--zoneoption and was last updated in December 2025. Its state would also go stale whenever a drill has me delete a VM by hand. - Ansible. It’s agentless and the usual way to run kubeadm across
machines. Here it needs SSH keys injected through cloud-init, an inventory
that copies
deploy/vms/config.shand a Python toolchain, all for about ten commands. - Kubespray. It builds exactly this cluster, kube-vip and Cilium included. It also hides every step I’ve just learned inside a large Ansible project, and debugging it means debugging that project.
- kubeadm from cloud-init. Each VM could init or join on first boot from shared secrets. The order between nodes would rest on retries, and a failed join would only show in that VM’s cloud-init log.
- Flux or Argo CD at bootstrap. Both need a working pod network, so Cilium goes in directly either way. GitOps for the Leg 3 workloads is its own decision.
- Staying manual. Fine for one bootstrap. Not for a cluster I plan to break several times a week.
Consequences
- A rebuild is one command and a wait, so the drills can be as destructive as
they need to be. The first clean
just uptook 20m08s from nothing to a passingcluster-check, VMs included. The bootstrap alone took 47 minutes by hand. - The runbook stays. It explains what the script does and why, and its recovery steps still apply when the script stops halfway.
- The script and the runbook can drift apart. Each step in the script names
the runbook step it runs, and
cluster-checkproves the result either way. - The token and certificate key sit in plain text in
~/cluster/on the VMs. The token expires after 24 hours and the uploaded certificates after two. That’s fine on a private bridge and wrong anywhere else. - Helm has to be on the server.
mise.tomlalready pins it. - I’d revisit this if the cluster moved to real machines or a hypervisor with an API, such as Proxmox. Then Terraform for the machines and Ansible for kubeadm earn their place, or Cluster API once there’s more than one cluster.
References
- kubeadm configuration (v1beta4)
- Creating highly available clusters with kubeadm
- terraform-provider-multipass
- Multipass availability zones
- Kubespray
Runbooks
Every drill I run gets a runbook: the steps, what I expected and the real
output. They fill in as the drills happen, and each one starts from
_template.md.
| Runbook | Drill | Status |
|---|---|---|
bootstrap.md | Bootstrap the cluster by hand | Run on 15 Nov 2026 |
control-plane.md | Lose one control plane, then two | Run on 16 Nov 2026 |
etcd.md | Snapshot and restore etcd | Run on 17 Nov 2026 |
chaos.md | Break the pipeline while it streams | To run |
Bootstrap the cluster by hand
How I bring up the highly available control plane on the five VMs, by hand,
with kubeadm, kube-vip and Cilium. I first ran it on 15 November 2026, and it
took 47 minutes from kubeadm init to a clean just cluster-check.
The VMs run on my home server. Steps 1 to 6 and step 9 run there, in
the repo or on a VM through multipass shell. Steps 7 and 8 run on my laptop.
just cluster-up runs steps 1 to 8 in one go on the server, and just up
rebuilds the VMs and the cluster from nothing. This runbook is what they do,
and how to recover when they stop halfway. The script runs the joins in steps
5 and 6 from deploy/cluster/join-control-plane.yaml and join-worker.yaml
in place of the flags.
Before I start
just vmshas built all five VMs, andjust vms-checkpasses.- The server’s checkout is on the branch I’m testing, with no stray edits.
- I have two hours free. The certificate key from step 4 expires after two hours, and the join token after 24.
Steps
1. Write the files
just cluster-files
This renders deploy/cluster/ and copies kubeadm.yaml to cp-1 and
kube-vip.yaml to all three control planes, under ~/cluster/. It fails if a
VM has leased the virtual IP. kubeadm.yaml carries a fresh bootstrap token
and certificate key, so the join commands init prints use them.
2. Put kube-vip on cp-1, on ground power
On cp-1:
sudo cp ~/cluster/kube-vip.yaml /etc/kubernetes/manifests/
sudo sed -i 's#path: /etc/kubernetes/admin.conf#path: /etc/kubernetes/super-admin.conf#' /etc/kubernetes/manifests/kube-vip.yaml
Since Kubernetes 1.29, admin.conf has no rights until init finishes, and
kube-vip has to hold the virtual IP before then. So it borrows
super-admin.conf. The sed only changes the hostPath line, because
mountPath: has a capital P.
3. Initialise the first control plane
On cp-1:
sudo kubeadm init --config ~/cluster/kubeadm.yaml --upload-certs
I keep both join commands it prints. The control-plane one carries
--certificate-key, which unlocks the shared certificates for two hours.
4. Take kube-vip off ground power
On cp-1, straight after init:
sudo sed -i 's#path: /etc/kubernetes/super-admin.conf#path: /etc/kubernetes/admin.conf#' /etc/kubernetes/manifests/kube-vip.yaml
The kubelet sees the file change and restarts kube-vip on admin.conf.
super-admin.conf is in system:masters, which skips RBAC, so nothing should
run on it once init is done. cluster-check fails if anything still does.
5. Join cp-2 and cp-3
On each, using the control-plane join command from step 3:
sudo kubeadm join 10.97.0.250:6443 --token <token> \
--discovery-token-ca-cert-hash sha256:<hash> \
--control-plane --certificate-key <key> \
--ignore-preflight-errors=Mem
sudo cp ~/cluster/kube-vip.yaml /etc/kubernetes/manifests/
kube-vip goes in after the join. Before it, the node has no admin.conf.
The control planes have 1.5 GB against kubeadm’s 1.7 GB floor, so the memory
check is skipped. deploy/README.md has the reasoning.
6. Join the workers
On edge-1 and data-1, using the worker join command from step 3:
sudo kubeadm join 10.97.0.250:6443 --token <token> \
--discovery-token-ca-cert-hash sha256:<hash>
Every node now shows NotReady. That’s expected until step 9.
7. Copy the kubeconfig to my laptop
On my laptop:
ssh <home-server> multipass exec cp-1 -- sudo cat /etc/kubernetes/admin.conf > ~/.kube/config
ssh -fN -L 6443:10.97.0.250:6443 <home-server>
Then I change server: in ~/.kube/config to https://127.0.0.1:6443. The
API server’s certificate names 127.0.0.1 for this tunnel.
just kubeconfig <home-server> does both, into ~/.kube/blackbox.conf.
8. Install Cilium
On my laptop:
helm install cilium cilium --repo https://helm.cilium.io --version 1.20.4 \
-n kube-system -f deploy/cluster/cilium-values.yaml
On a cold mirror, the agents sit at Init:0/6 for a while. On the first run,
all five nodes pulled the images through quay.io at about 1 MB/s, and it took
over twenty minutes.
9. Prove it
On the server:
just cluster-check
What I saw on the first run
At T+17, every control-plane Pod was running, while Cilium waited for its images, CoreDNS was Pending and every node was NotReady:
NAME READY STATUS RESTARTS AGE
cilium-2d88w 0/1 Init:0/6 0 110s
cilium-envoy-fcxq9 0/1 ContainerCreating 0 110s
cilium-operator-6c4c574b46-d8mtq 0/1 ContainerCreating 0 110s
coredns-589f44dc88-h6nzh 0/1 Pending 0 17m
etcd-cp-1 1/1 Running 0 17m
etcd-cp-2 1/1 Running 0 7m40s
etcd-cp-3 1/1 Running 0 8m46s
kube-vip-cp-1 1/1 Running 0 16m
kube-vip-cp-2 1/1 Running 0 7m40s
kube-vip-cp-3 1/1 Running 0 7m26s
That’s an excerpt of the 34 Pods. kube-vip-cp-1 is a minute younger than
etcd-cp-1 because step 4 restarted it.
Once Cilium was up, cluster-check passed:
API on 10.97.0.250: ok, held by cp-3
kube-vip: running on 3 control planes, all reading admin.conf
nodes: 5 of 5 Ready
etcd: 3 of 3 members healthy
cilium: ready on 5 nodes
The address had moved to cp-3 on its own. At 13:36, with the server flat out, cp-1’s API server stopped answering in time and its leaders lost their Leases. cp-3 took over, as designed.
When it goes wrong
The certificate key has expired
A control-plane join fails after two hours. On a working control plane:
sudo kubeadm init phase upload-certs --upload-certs
sudo kubeadm token create --print-join-command --certificate-key <new key>
The first prints a new key, and the second prints a full control-plane join command that uses it.
The join token has expired
After 24 hours, a worker join fails. On a control plane:
sudo kubeadm token create --print-join-command
A join failed halfway
On the failed node:
sudo kubeadm reset -f
Then I join again. If it was a control plane that got as far as etcd, I remove its stale member first, from a healthy control plane:
kubectl -n kube-system exec etcd-cp-1 -- etcdctl \
--endpoints https://127.0.0.1:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt \
--cert /etc/kubernetes/pki/etcd/healthcheck-client.crt \
--key /etc/kubernetes/pki/etcd/healthcheck-client.key \
member list
Then member remove <ID> with the same flags.
cluster-check says kube-vip reads super-admin.conf
Step 4 was missed. I run it on the node that cluster-check names.
Cilium is stuck at Init
I check the events before anything else:
kubectl -n kube-system get events --sort-by=.lastTimestamp | grep -i cilium
Pulling image with no errors means a cold mirror. Waiting is the fix, and
the next rebuild reads the images from the host.
Lose one control plane, then two
The cluster runs three control planes and three etcd members. With one gone, etcd keeps quorum and nothing changes. With two gone, etcd stops, the API goes with it and the workloads keep serving. This drill shows both.
I ran it twice on 16 November 2026. The first run ended early, for a reason that had nothing to do with etcd. The second run is the one that counts.
Everything runs on my home server, in two terminals.
Before I start
-
just cluster-checkpasses. -
just drill-watchruns in the first terminal. I wait forStart the drillbefore I break anything, because it deploys echo first. -
The steps run in the second terminal.
-
Both terminals append to one log, so neither overwrites the other:
just drill-watch 2>&1 | tee -ai drill.log # first terminal just cluster-check 2>&1 | tee -a drill.log # every check below-ikeepsteealive through Ctrl-C, so the summary lands too.cluster-checkprints its failures to stderr, hence2>&1.
drill-watch prints a line whenever something changes: which control planes
are up, which one holds the VIP and whether echo, a read, a cached read and a
write succeed. deploy/README.md explains each probe.
Steps
1. Lose one control plane
multipass stop --force cp-2
--force pulls the power. A clean shutdown would let etcd hand over
leadership politely, and real failures don’t.
Two of three members is still a majority, so etcd keeps writing. If cp-2 held
the VIP, kube-vip on another control plane takes it once the 15 s Lease runs
out. Every probe stays ok, apart from a gap while the VIP moves.
2. Check it
After a minute, so the node controller has noticed:
just cluster-check
It fails, and should: kube-vip isn’t ready on cp-2, four of five nodes are Ready and two of three etcd members are healthy. The API still answers on the VIP.
3. Lose a second
multipass stop --force cp-3
One of three is not a majority. etcd can’t elect a leader, so it refuses
writes and consistent reads. cp-1’s API server stops being ready, but it
still answers reads at resourceVersion=0 from its watch cache.
kube-vip renews its Lease through that API, every 2 s, and stands down if it can’t renew for 10 s. So the VIP disappears, and the cached reads go with it. The API doesn’t stay read-only. It goes away.
echo keeps answering. The Pods run on the workers, and kube-proxy’s iptables rules stay in the kernel. Nothing on the data path needs the control plane until something changes.
4. Check it
just cluster-check
It stops before it reaches the API:
no control plane's API server is ready (running: cp-1): kubeadm init hasn't run, or etcd has lost quorum
It only asks running VMs. The first run shows why that matters.
5. Bring both back
multipass start cp-2 cp-3
Once a second member is up, etcd has a majority again and elects a leader.
The API comes back, kube-vip takes its Lease and the VIP returns. The
drill-watch lines go back to ok.
6. Prove it
Ctrl-C in the first terminal prints how many probes of each kind passed. Then:
just cluster-check
Cilium on the two rebooted nodes takes a minute to be ready, so the first check may fail on it alone.
What I saw
First run, 16 November 2026
Stage one went as expected: cp-2 stopped and the cluster carried on without it.
Stage two lasted about a minute. cp-3 went at 19:53:13. 32 s later, the
API dropped drill-watch’s connection, and from 19:53:50 the VIP didn’t
answer at all:
E1116 19:53:50.592902 ... dial tcp 10.97.0.250:6443: connect: no route to host
That’s kube-vip standing down, as step 3 predicts. Then I ran
just cluster-check, and afterwards all five VMs were running. I hadn’t
started any of them. Multipass’s log, trimmed:
Nov 16 19:53:13 multipassd: started: qemu-img snapshot -l .../instances/cp-3/...
Nov 16 19:54:00 multipassd: started: qemu-system-x86_64 ... .../instances/cp-2/...
Nov 16 19:54:20 multipassd: started: qemu-system-x86_64 ... .../instances/cp-3/...
cluster-check looked for a ready API server by running multipass exec on
each control plane in turn. cp-1 wasn’t ready, because etcd had no quorum, so
it moved on to cp-2 and cp-3. multipass exec starts a stopped instance and
retries. Its help doesn’t say so, but exec.cpp does. The check booted cp-2,
waited 20 s for it, then booted cp-3. The tool meant to report the outage
ended it.
drill-watch never reached its loop. It was still waiting for echo to roll
out when I stopped cp-3, and it timed out once the API went.
Both scripts now only touch running VMs, and drill-watch reads the VIP
holder from multipass list. It also says when to start.
Second run, 16 November 2026
With both fixes in:
20:16:10 + 1s up cp-1,cp-2,cp-3 vip cp-1 echo ok read ok cache ok write ok
20:16:20 + 11s up cp-1,cp-3 vip cp-1 echo ok read ok cache ok write ok
20:16:58 + 49s up cp-1 vip cp-1 echo ok read fail cache ok write fail
20:17:20 + 71s up cp-1 vip none echo ok read fail cache fail write fail
20:19:46 + 217s up cp-1,cp-2,cp-3 vip none echo ok read fail cache fail write fail
Losing cp-2 changed nothing. cp-1 held the VIP, so there was nothing to move.
Losing cp-3 stopped consistent reads and writes on the next probe. The API server kept serving cached reads for another 22 s, until kube-vip stood down and the VIP went. echo didn’t notice.
cluster-check stopped early and left cp-2 and cp-3 stopped:
no control plane's API server is ready (running: cp-1): kubeadm init hasn't run, or etcd has lost quorum
Both were back by 20:19:46. I stopped drill-watch before quorum returned:
echo: 79 of 79 passed
read: 29 of 79 passed
cache: 35 of 79 passed
write: 29 of 79 passed
echo answered every probe. Cached reads outlived consistent ones by six probes. With probes timing out, a round took about 3 s, not 1.
Then cluster-check:
API on 10.97.0.250: ok, held by cp-1
kube-vip: running on 3 control planes, all reading admin.conf
nodes: 5 of 5 Ready
etcd: 3 of 3 members healthy
cilium: ready on 3 of 5 nodes
etcd, the API and kube-vip were back. Cilium on cp-2 and cp-3 was still starting. Once it was ready:
API on 10.97.0.250: ok, held by cp-1
kube-vip: running on 3 control planes, all reading admin.conf
nodes: 5 of 5 Ready
etcd: 3 of 3 members healthy
cilium: ready on 5 nodes
The check after stage one is missing. drill-watch wrote through tee -i
without -a, so it wrote over what the check had appended.
What else the drill found
echo is meant to keep one Pod on each worker, so losing either leaves one serving. Two checks after the drill showed its spread rule didn’t hold that.
A lost worker’s Pod couldn’t be replaced. I cordoned edge-1 and deleted its echo Pod. The replacement sat in Pending:
0/5 nodes are available: 1 node(s) didn't match pod topology spread constraints, 1 node(s) were unschedulable, 3 node(s) had untolerated taint(s).
The rule allows at most one Pod more on any node than on the emptiest. By
default it counts every node, tainted ones included, so cordoned edge-1 and the
three control planes each counted as holding none. A second Pod on data-1 broke
the rule. A dead worker is tainted the same way. nodeTaintsPolicy: Honor
leaves tainted nodes out of the count.
A rollout put both Pods on one worker. Applying that fix rolled echo out again, and both new Pods landed on data-1 (columns trimmed):
NAME READY STATUS RESTARTS AGE IP NODE
echo-5976856569-r52d5 1/1 Terminating 0 18m 10.244.3.87 edge-1
echo-7bbd856f5b-ddzh8 1/1 Running 0 16s 10.244.5.176 data-1
echo-7bbd856f5b-v7z4q 1/1 Running 0 4s 10.244.5.2 data-1
The rule counts every Pod labelled app=echo, old ones included. A rollout
replaces them one at a time, so at each step the old Pods still running made
data-1 look no fuller than edge-1. matchLabelKeys: [pod-template-hash] counts
only Pods from the same rollout.
The rule now reads:
topologySpreadConstraints:
- maxSkew: 1 # at most one Pod more than the emptiest node
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
nodeTaintsPolicy: Honor # cordoned, dead and control-plane nodes don't count
matchLabelKeys:
- pod-template-hash # only this rollout's Pods count
labelSelector:
matchLabels:
app: echo
With both in place, a rollout left one Pod on each worker, and the same cordon put the replacement on data-1 straight away:
NAME READY STATUS RESTARTS AGE IP NODE
echo-5cb4c8bc96-5brtt 1/1 Running 0 52s 10.244.5.133 data-1
echo-5cb4c8bc96-vblxj 1/1 Running 0 22s 10.244.5.18 data-1
The scheduler never moves a running Pod, so after the uncordon both stayed on
data-1. kubectl -n drill rollout restart deployment/echo spreads them again.
Neither causes trouble while every node is healthy, which is why a drill is the place to find them.
Why three or five
etcd writes only when a majority of members agree. Quorum is ⌊n/2⌋ + 1.
| Members | Quorum | Can lose |
|---|---|---|
| 1 | 1 | 0 |
| 2 | 2 | 0 |
| 3 | 2 | 1 |
| 4 | 3 | 1 |
| 5 | 3 | 2 |
Even sizes buy nothing. Four members survive one loss, like three, but give one more machine to fail, and every write waits for three acknowledgements instead of two. Two is worse than one, because losing either stops writes.
Partitions decide it. Four members can split two and two, and then neither side has a majority and the cluster stops. Split three or five in two and one side always keeps a majority. The majority rule itself is what stops a split brain: two halves can’t both accept writes, because only one of them can be a majority.
When it goes wrong
A control plane won’t rejoin etcd
--force is a power cut. etcd syncs every commit to disk, but a cut
mid-write can leave its log or database torn. On restart, etcd-cp-N crashes
in a loop and the kubelet keeps restarting it.
I remove the member and join the node again. That only works for cp-2 and
cp-3: after a reset, cluster-up would run kubeadm init on cp-1 and build
a new, empty cluster. If cp-1 is the broken one, I join it by hand with step
5 of the bootstrap runbook.
From a healthy control plane, I find the member’s ID with member list and
remove it with member remove, using the flags in
bootstrap.md. Then:
kubectl delete node cp-N
multipass exec cp-N -- sudo kubeadm reset -f
just cluster-up
cluster-up treats a control plane as joined once its Node is labelled, so
the Node has to go before it will join cp-N again.
A node comes back on a new address
Each etcd member’s peer URL and certificate name the node’s IP. A new address means the others can’t reach it, or won’t trust it. Multipass never expires a lease, so it shouldn’t happen here, but on real hardware it’s the classic case. The fix is the same: remove, reset and join.
The VIP doesn’t come back
kube-vip needs a working API to take its Lease, so the VIP returns after quorum, not before. If etcd is healthy and the VIP is still missing, I read kube-vip’s log on a control plane:
multipass exec cp-1 -- sudo crictl logs --tail 50 \
$(multipass exec cp-1 -- sudo crictl ps -a -q --latest --name kube-vip)
What I don’t do
With two members down, etcd --force-new-cluster on the survivor gets writes
going again. The other two then come back with history the new cluster
doesn’t share, and both have to be wiped and joined again. It’s for members
that are never coming back, not ones that are slow.
Snapshot and restore etcd
etcd holds every object in the cluster. A snapshot is a copy of one member’s database, and restoring it puts the whole cluster back to that moment. This drill takes a snapshot, deletes a namespace and restores it.
I ran it twice on 17 November 2026. The first time I worked out each step as I went, and reads failed for 19 minutes. The second run follows these steps, and reads failed for under two minutes.
Everything runs on my home server, in two terminals.
Before I start
-
just cluster-checkpasses. -
No control plane has a
/var/lib/etcd.oldleft from an earlier drill. Step 5 keeps an existing one as its fallback. -
just drill-watchruns in the first terminal. I wait forStart the drill, because it deploys echo and thedrillnamespace first. -
The steps run in the second terminal, with the cluster’s kubeconfig:
export KUBECONFIG=~/.kube/blackbox.conf~/.kube/configon the server is an empty stub, and kubectl asks for a username without this. -
Both terminals append to one log:
just drill-watch 2>&1 | tee -ai drill.log # first terminal just cluster-check 2>&1 | tee -a drill.log # every check below
Steps
1. Take a snapshot
just etcd-snapshot
It saves the snapshot on the first running control plane, copies it to
~/.local/state/telemetry-blackbox/etcd/ and checks the copy’s hash. It prints
the revision, which is the point the restore goes back to.
2. Delete the namespace
kubectl delete namespace drill
The namespace, its Pods and its ConfigMap go. echo fails, then reads and writes of the ConfigMap fail, because it no longer exists. The API is still up.
3. Copy the snapshot to every control plane
Multipass is a snap and can’t read dot-directories, so multipass transfer
needs a copy in my home directory first:
snap=$(ls -t ~/.local/state/telemetry-blackbox/etcd/*.db | head -1) && sha256sum "$snap"
install -m 600 "$snap" ~/etcd-restore.db
for cp in cp-1 cp-2 cp-3; do
multipass transfer ~/etcd-restore.db "$cp:/home/ubuntu/etcd-restore.db" &&
multipass exec "$cp" -- chmod 600 /home/ubuntu/etcd-restore.db
done
for cp in cp-1 cp-2 cp-3; do echo "$cp $(multipass exec "$cp" -- sha256sum /home/ubuntu/etcd-restore.db)"; done
rm ~/etcd-restore.db
Every hash must match the first. I never restore from a copy that doesn’t. The
copies hold every Secret, so only ubuntu can read them, and step 8 removes
them.
4. Stop etcd and the API servers
for cp in cp-1 cp-2 cp-3; do
multipass exec "$cp" -- sudo mv /etc/kubernetes/manifests/etcd.yaml \
/etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/
done
The kubelet stops a static Pod when its manifest leaves the directory. I wait until no IDs follow the names:
for cp in cp-1 cp-2 cp-3; do echo "$cp: $(multipass exec "$cp" -- sudo crictl ps -q --name '^(etcd|kube-apiserver)$')"; done
kube-vip can’t renew its Lease without an API, so the VIP goes too.
5. Restore every member
cluster='cp-1=https://10.97.0.54:2380,cp-2=https://10.97.0.213:2380,cp-3=https://10.97.0.153:2380'
declare -A ip=([cp-1]=10.97.0.54 [cp-2]=10.97.0.213 [cp-3]=10.97.0.153)
restore() {
local cp=$1 image
image=$(multipass exec "$cp" -- sudo awk '/image:/ { print $2 }' /etc/kubernetes/etcd.yaml)
multipass exec "$cp" -- sudo sh -c 'test -d /var/lib/etcd.old || mv /var/lib/etcd /var/lib/etcd.old; rm -rf /var/lib/etcd' &&
multipass exec "$cp" -- sudo ctr -n k8s.io run --rm \
--mount type=bind,src=/var/lib,dst=/var/lib,options=rbind:rw \
--mount type=bind,src=/home/ubuntu,dst=/home/ubuntu,options=rbind:ro \
"$image" etcd-restore \
etcdutl snapshot restore /home/ubuntu/etcd-restore.db \
--data-dir /var/lib/etcd --name "$cp" \
--initial-advertise-peer-urls "https://${ip[$cp]}:2380" \
--initial-cluster "$cluster" \
--bump-revision 1000000000 --mark-compacted
}
for cp in cp-1 cp-2 cp-3; do restore "$cp" || { echo "restore failed on $cp"; break; }; done
Each node prints restored snapshot. If one fails, restore cp-N runs it
again on that node. It keeps the first etcd.old, so the data from before the
restore survives a second attempt.
etcdutl isn’t installed on the nodes, so ctr runs it from the etcd image
that’s already there. The tag comes from the moved manifest, so it follows
upgrades.
Every node gets the same --initial-cluster. etcd derives the cluster ID and
the member IDs from it, so all three agree on who’s in the cluster. The
manifests are no help here: kubeadm grew the cluster one join at a time, so
cp-1’s lists only cp-1. Each node’s own --name and peer URL say which member
it is.
The snapshot is older than anything the API servers, kubelets and controllers
last saw. Without --bump-revision, new writes would reuse revisions they’d
already seen, so they’d miss them. With it, revisions only go forward, and
--mark-compacted puts everything older below the compaction mark. Anything
watching from an old resourceVersion gets 410 Gone and lists again.
The Kubernetes docs also restart the scheduler, the controller manager and the kubelets after a restore. I don’t, because the compaction mark makes every watcher list again, and the second run shows it’s enough here.
6. Start etcd, then the API servers
for cp in cp-1 cp-2 cp-3; do multipass exec "$cp" -- sudo mv /etc/kubernetes/etcd.yaml /etc/kubernetes/manifests/; done
I wait until all three members are healthy. This checks from cp-1:
multipass exec cp-1 -- sudo sh -c 'crictl exec $(crictl ps -q --name "^etcd$") etcdctl \
--endpoints https://127.0.0.1:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt \
--cert /etc/kubernetes/pki/etcd/healthcheck-client.crt \
--key /etc/kubernetes/pki/etcd/healthcheck-client.key endpoint health --cluster'
Then the API servers:
for cp in cp-1 cp-2 cp-3; do multipass exec "$cp" -- sudo mv /etc/kubernetes/kube-apiserver.yaml /etc/kubernetes/manifests/; done
kube-vip takes its Lease again and the VIP returns.
7. Prove it
just cluster-check
kubectl get namespace drill -o jsonpath='{.metadata.uid} {.metadata.creationTimestamp}{"\n"}'
kubectl -n drill get pods -o wide
The namespace has its old UID and creation time, so it’s the same object, not a new one. The Pods have their old names. The kubelets build new sandboxes for them, and echo answers.
8. Clean up
Once cluster-check passes:
for cp in cp-1 cp-2 cp-3; do multipass exec "$cp" -- sudo rm -rf /var/lib/etcd.old /home/ubuntu/etcd-restore.db; done
What I saw
First run, 17 November 2026
I worked out each step as I ran it. Reads failed for 19 minutes, from 20:10:47 to 20:29:37:
20:09:54 + 1s up cp-1,cp-2,cp-3 vip cp-3 echo ok read ok cache ok write ok
20:10:42 + 49s up cp-1,cp-2,cp-3 vip cp-3 echo fail read ok cache ok write ok
20:10:47 + 54s up cp-1,cp-2,cp-3 vip cp-3 echo fail read fail cache fail write fail
20:17:24 + 451s up cp-1,cp-2,cp-3 vip none echo fail read fail cache fail write fail
20:29:37 +1184s up cp-1,cp-2,cp-3 vip none echo fail read ok cache ok write ok
20:29:40 +1187s up cp-1,cp-2,cp-3 vip cp-3 echo fail read ok cache ok write ok
20:30:17 +1224s up cp-1,cp-2,cp-3 vip cp-3 echo ok read ok cache ok write ok
Most of it went on working out each step: the kubeconfig, getting the snapshot onto the nodes and the restore flags.
Second run, 17 November 2026
The snapshot came from cp-1 at revision 1000382584. The first run had already bumped it by a billion.
+----------+------------+------------+------------+---------+
| HASH | REVISION | TOTAL KEYS | TOTAL SIZE | VERSION |
+----------+------------+------------+------------+---------+
| d31b1f50 | 1000382584 | 482 | 15 MB | 3.6.0 |
+----------+------------+------------+------------+---------+
VERSION is the storage format, not the binary, which is 3.6.8.
drill-watch:
20:38:13 + 1s up cp-1,cp-2,cp-3 vip cp-3 echo ok read ok cache ok write ok
20:38:47 + 35s up cp-1,cp-2,cp-3 vip cp-3 echo fail read ok cache ok write ok
20:38:55 + 43s up cp-1,cp-2,cp-3 vip cp-3 echo fail read fail cache fail write fail
20:39:43 + 91s up cp-1,cp-2,cp-3 vip none echo fail read fail cache fail write fail
20:40:47 + 155s up cp-1,cp-2,cp-3 vip none echo fail read ok cache ok write ok
20:40:50 + 158s up cp-1,cp-2,cp-3 vip cp-3 echo fail read ok cache ok write ok
20:41:53 + 221s up cp-1,cp-2,cp-3 vip cp-3 echo ok read ok cache ok write ok
| Time | Step | Took |
|---|---|---|
| 20:38:40 | Delete the namespace | 14 s |
| 20:39:05 | Copy the snapshot, three hashes match | 9 s |
| 20:39:14 | Stop etcd and the API servers | 36 s |
| 20:40:04 | Restore three members | 10 s |
| 20:40:24 | Start etcd, three members healthy | 6 s |
| 20:40:30 | Start the API servers, reads work through the VIP | 17 s |
From stopping the API servers to reads working again took 93 seconds. Until
the API servers stopped, from 20:39:14, the reads failed because the ConfigMap
was gone, not the API. drill-watch can’t tell the two apart. multipass list
reported the VIP 3 s after the API answered on it.
echo came back about a minute after the API. The Pods had their old names and new addresses, because the kubelets built new sandboxes for them (columns trimmed):
NAME READY STATUS RESTARTS AGE IP NODE
echo-58849b6468-7cmc8 1/1 Running 1 32m 10.244.3.252 edge-1
echo-58849b6468-bdwz7 1/1 Running 1 32m 10.244.5.199 data-1
The namespace kept the UID and creation time it got in the first run, at 20:09:40. The ConfigMap’s resourceVersion read 2000383099, two bumps on.
echo: 33 of 102 passed
read: 69 of 102 passed
cache: 69 of 102 passed
write: 69 of 102 passed
Then cluster-check:
API on 10.97.0.250: ok, held by cp-3
kube-vip: running on 3 control planes, all reading admin.conf
nodes: 5 of 5 Ready
etcd: 3 of 3 members healthy
cilium: ready on 5 nodes
What a restore costs
A restore can’t bring back one namespace. It puts every object back to the snapshot. Anything written after it is lost: new Deployments, scaling, Secrets, Leases and Node heartbeats. Here that was a few minutes of a quiet cluster, so nothing mattered. On a busy cluster a restore is the last resort, not an undo.
The snapshot holds every Secret in plain text. etcd-snapshot leaves it
readable only by me, and it never goes in the repo. Step 3’s copies on the
control planes last until step 8.
Why every member
One member restored alone gets a new cluster ID from the restore. The other two
still have the old one, so they reject it. Restoring all three from one file,
with one --initial-cluster, makes a new cluster whose members agree from the
start.
When it goes wrong
A copy is torn
My first attempt piped the snapshot into multipass exec. It arrived as 2.2 MB
of 15 MB, and etcdutl refused it:
Error: snapshot missing hash but --skip-hash-check=false
Every snapshot ends in a hash of its contents, so a truncated copy fails
safely. multipass transfer and the hash check in step 3 keep it from
happening.
A member won’t start
If an etcd container crashes after step 6, I go back to the data from before the restore. I stop etcd and the API servers with step 4, run this, then start them with step 6:
for cp in cp-1 cp-2 cp-3; do multipass exec "$cp" -- sudo sh -c 'rm -rf /var/lib/etcd && mv /var/lib/etcd.old /var/lib/etcd'; done
That’s the cluster as it was before the restore, with the namespace still deleted. Always all three, never one or two: that’s members from two clusters.
A member has the wrong --initial-cluster
Copied from the manifest, cp-1 would know only itself and start a one-member cluster of its own. cp-2 and cp-3 would refuse it on the cluster ID. The fix is steps 4 to 6 again on all three, with the full list. Step 5 is safe to run twice.
What I don’t do
etcdctl snapshot restore no longer exists in etcd 3.6. etcdutl replaced it.
I also don’t restore while any API server runs, because it would keep serving
from its watch cache and write to a cluster that’s changing under it.
Brand
The logo is an attitude indicator inside a black box. It’s the instrument
behind the ATTITUDE messages this project records, inside the box that
records them.
Files
| File | Use |
|---|---|
logo-light.svg, logo-dark.svg | The full logo, for light and dark backgrounds |
mark-light.svg, mark-dark.svg | The mark alone, where space is tight or the name is already nearby |
favicon.svg | The browser tab icon. It switches theme itself |
social-preview.png | The 1280 × 640 link preview, uploaded in the repo’s settings |
The wordmark is JetBrains Mono SemiBold converted to outlines, so it looks the same everywhere and needs no font installed. The files are plain SVG and open in Figma, Penpot or Inkscape.
Using it
Show the version that matches the reader’s theme. On GitHub and the docs site
that means a <picture> element:
<picture>
<source media="(prefers-color-scheme: dark)" srcset="docs/brand/logo-dark.svg">
<img alt="telemetry-blackbox" src="docs/brand/logo-light.svg" width="440">
</picture>
- Minimum size: 16 px for the mark, 160 px wide for the full logo.
- Clear space: half the mark’s height on every side.
- Don’t recolour, stretch or rotate it, set the wordmark in another font, or put the light version on a dark background.
| Colour | Hex | Where |
|---|---|---|
| Orange | #f2662b | The sky, “blackbox” and accents |
| Ink | #0f1720 | The box and the wordmark on light backgrounds |
| Box (dark) | #1c2330 | The box on dark backgrounds, with a #4a5866 edge |
| Ground | #2a3442 | Below the horizon |
| Slate | #8b97a6 | The wordmark on dark backgrounds, chosen to stay legible (3:1) if it lands on a light page |
Where it appears
| Place | When | Issue |
|---|---|---|
| The README header | Now | |
| The flight-progress strip | Now | #39 |
| The repo’s social preview | Day 4 | #29 |
| The demo’s opening and closing cards | Day 4 | #29 |
| The Grafana home dashboard | Day 4 | #28 |
| The docs site header and favicon | Stretch | #35 |
| The public Grafana dashboard | Stretch | #36 |