A flight recorder for a simulated flight controller. Rust firmware runs a 1 kHz control loop and streams MAVLink telemetry over a serial link. A Rust gateway, Redpanda and a Spate pipeline carry every frame into ClickHouse, on a highly available Kubernetes cluster I built by hand.
Status: under construction, in the open. Progress is tracked in the milestones, and the docs site updates with every push.
Progress
I fly the route in evenings after work, so each leg has a budget in evenings and a due date to match. The cluster flies before the firmware because it carries the most risk. #62 has the reasoning.
Each leg is a strip in a rack, the way air traffic control tracks flights. The leg I’m flying sticks out, a late leg is outlined in red and a white tick marks where the plan says I should be. Headwind counts the work that crept in after takeoff, and a stamp unlocks at each milestone I care most about. The flight log has the detail.
Why I’m building it
I want to understand four things from the bottom up:
- Kubernetes run on my own machines, including the parts that break.
- Rust, both
no_stdon a microcontroller and async on a server. - Real-time embedded systems and the protocols between them.
- Streaming data into ClickHouse with Spate.
A black box is a good way to learn all four at once. It has a hard deadline at one end, a database at the other and a simple test in between. Did every frame arrive?
The claim
Cut the serial link, kill the pipeline, drain a node or lose a control plane
in the middle of a flight. Every frame still reaches ClickHouse. Some arrive
twice, because delivery is at-least-once, and ClickHouse removes the
duplicates by key. bbcheck measures the outcome after each drill, and the
numbers go in Results.
How it fits together
Each part has one job.
| Part | Built with | Job |
|---|---|---|
firmware/ | Rust, no_std, Embassy | Runs the control loop on a hardware timer and sends telemetry over UART |
gateway/ | Rust, tokio | Finds frame boundaries, checks CRCs, spots sequence gaps and publishes to Redpanda |
pipeline/ | Rust, Spate | Decodes frames, routes them into tables and loads ClickHouse |
deploy/ | cloud-init, kubeadm, YAML | Builds the VMs and the cluster, then runs everything on it |
tools/bbcheck/ | Python | Proves what arrived: gaps, duplicates and latency |
The cluster
Five VMs on one machine. Three control planes share a virtual IP, so the API stays up when one of them goes down. Two workers carry the workloads, and a node label decides which one owns the serial link.
Renode runs on the host, outside the cluster, because it stands in for a
physical device. On real hardware the gateway would read /dev/ttyUSB0.
Here it reads the TCP socket Renode exposes.
The life of one frame
The last step matters most. Spate only commits an offset once ClickHouse has
confirmed the batch is durable, so a crash can repeat work but never skip it.
That is where duplicates come from, and why the table removes them by
(boot_id, system_id, seq).
The wire format
Telemetry uses MAVLink v2. Every frame carries a sequence number, which is
what lets the gateway and bbcheck count gaps.
| Bytes | Field | Notes |
|---|---|---|
| 1 | STX | Always 0xFD in v2 |
| 1 | len | Payload length |
| 1 | incompat_flags | Bit 0 marks a signed frame |
| 1 | compat_flags | Safe to ignore if unknown |
| 1 | seq | Wraps at 255; a jump means lost frames |
| 1 | sysid | Which vehicle |
| 1 | compid | Which component on it |
| 3 | msgid | 24-bit message ID |
len | payload | Trailing zero bytes are trimmed |
| 2 | checksum | CRC-16/MCRF4XX, seeded with the message’s CRC_EXTRA |
| 13 | signature | Only on signed frames |
The firmware sends HEARTBEAT, ATTITUDE, HIGHRES_IMU and a custom
LOOP_TIMING message that reports overruns and jitter.
Real-time, honestly
The control loop is hard real-time inside the simulation. Renode runs in deterministic virtual time, so loop timing reflects the firmware alone.
Everything after the serial port is soft real-time. I measure it as end-to-end latency percentiles, from the firmware’s timestamp to the row landing in ClickHouse.
Running it
Still being built. When it’s done, it will look like this:
mise install # pinned tools: just, D2, uv, kubectl, Helm and more
just bench # check everything is installed
just lint # formatting and lints for the crates, firmware and bbcheck
just test # the tests, plus a firmware build
just coverage # line coverage across Rust and Python
just cluster # five VMs and a highly available cluster
just deploy # Redpanda, ClickHouse, Grafana, gateway and pipeline
just fly # boot the firmware in Renode and start streaming
just check # run bbcheck against ClickHouse
just docs # build the docs site into book/
just audit # scan the history for secrets
just on its own lists every recipe. You’ll also need Linux with KVM, about
11 GB of free memory, mise, Multipass, Renode, probe-rs,
the ClickHouse client and Rust with the thumbv7em-none-eabihf target.
A released version
Every release puts multi-arch images of the gateway and pipeline on GitHub Container Registry. Once the firmware has a binary, it’s attached to the release too. The changelog says what each one changed.
docker pull ghcr.io/guy-with-a-why/telemetry-blackbox/gateway:latest
docker pull ghcr.io/guy-with-a-why/telemetry-blackbox/pipeline:latest
gh attestation verify oci://ghcr.io/guy-with-a-why/telemetry-blackbox/gateway:latest \
-R guy-with-a-why/telemetry-blackbox
Each image carries a provenance attestation, so the last command proves it was built here, from a tagged commit.
Repository layout
firmware/ Embassy firmware for the STM32F4
renode/ the board Renode simulates, as a script
gateway/ serial link to Redpanda
pipeline/ Spate pipeline into ClickHouse
deploy/ VMs, cluster bootstrap and manifests
tools/bbcheck/ delivery and latency report
docs/adr/ decisions and the reasons behind them
docs/brand/ the logo, favicon, social preview and how to use them
docs/diagrams/ D2 sources and the SVGs they render
docs/runbooks/ drills, with real output
scripts/ small checks the justfile and CI call
Decisions
Each gets a short record in docs/adr/.
- MAVLink v2 over a custom framing.
- kubeadm over k3s or Talos.
- Redpanda over Apache Iggy or a home-made log.
- Deduplicating in ClickHouse over chasing exactly-once.
- just and mise over make, and a local registry for images.
Results
These fill in as the drills run.
| Drill | Frames sent | Missing | Duplicates | Recovery |
|---|---|---|---|---|
| Kill the pipeline | pending | pending | pending | pending |
| Drain the pipeline’s node | pending | pending | pending | pending |
| Restart Redpanda | pending | pending | pending | pending |
| Cut the serial link for 60 s | pending | pending | pending | pending |
| Lose a control plane | pending | pending | pending | pending |
| Scale the pipeline to two replicas | pending | pending | pending | pending |
Diagrams
Every diagram is a D2 file in docs/diagrams/. D2 lays
it out, and each SVG follows the viewer’s light or dark theme. The animated
ones step through boards declared in the same file.
To change one, edit the .d2 file and watch it live, then render the lot:
d2 --watch --layout=elk docs/diagrams/architecture.d2
just diagrams
Cite
These are the documents I need to build this, grouped by where I use them. The board Renode simulates is ST’s STM32F4DISCOVERY, which is built on the STM32F407VG, a Cortex-M4F.
Tooling (Leg 0)
- The just manual and the mise documentation.
Microcontroller (Leg 2)
- STM32F407VG datasheet: pinout, memory map and electrical limits.
- RM0090 reference manual: the timers, USART, DMA and clocks, register by register.
- PM0214 Cortex-M4 programming manual: the NVIC, interrupt priorities, SysTick and the FPU.
- ES0182 errata: known silicon faults. Renode won’t reproduce them, but real hardware would.
- UM1472 Discovery kit user manual: the board around the chip.
- Renode’s STM32F4 Discovery platform and the Renode documentation.
- The Embassy book and the embassy-stm32 API docs.
Protocol (Legs 2 and 3)
- MAVLink serialisation: the v2 frame byte by byte.
- MAVLink CRC: the checksum and
CRC_EXTRA. - MAVLink common messages:
HEARTBEAT,ATTITUDEandHIGHRES_IMU.
Cluster (Leg 1)
- Multipass documentation.
- Creating highly available clusters with kubeadm.
- kubeadm configuration (v1beta4), for the init and join files.
- kube-vip as a static Pod.
- Cilium on a kubeadm cluster.
- etcd disaster recovery.
- Upgrading kubeadm clusters.
- The Distribution registry and containerd registry hosts, for the local image registry.
Data path (Legs 1 and 3)
- Redpanda on Kubernetes.
- ClickHouse ReplacingMergeTree.
- Spate’s connector contracts and its delivery invariants.
Diagrams
Licence
Dual-licensed under MIT or Apache 2.0, at your option. Any contribution you submit is licensed the same way, unless you say otherwise.