Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

telemetry-blackbox

CI status on main OpenSSF Scorecard Line coverage across Rust and Python Latest release

A flight recorder for a simulated flight controller. Rust firmware runs a 1 kHz control loop and streams MAVLink telemetry over a serial link. A Rust gateway, Redpanda and a Spate pipeline carry every frame into ClickHouse, on a highly available Kubernetes cluster I built by hand.

Status: under construction, in the open. Progress is tracked in the milestones, and the docs site updates with every push.

Progress

Flight progress strips, one per leg. The flight log has the same detail as text.

Leg 0 Leg 1 Leg 2 Leg 3 Leg 4

I fly the route in evenings after work, so each leg has a budget in evenings and a due date to match. The cluster flies before the firmware because it carries the most risk. #62 has the reasoning.

Each leg is a strip in a rack, the way air traffic control tracks flights. The leg I’m flying sticks out, a late leg is outlined in red and a white tick marks where the plan says I should be. Headwind counts the work that crept in after takeoff, and a stamp unlocks at each milestone I care most about. The flight log has the detail.

Why I’m building it

I want to understand four things from the bottom up:

  • Kubernetes run on my own machines, including the parts that break.
  • Rust, both no_std on a microcontroller and async on a server.
  • Real-time embedded systems and the protocols between them.
  • Streaming data into ClickHouse with Spate.

A black box is a good way to learn all four at once. It has a hard deadline at one end, a database at the other and a simple test in between. Did every frame arrive?

The claim

Cut the serial link, kill the pipeline, drain a node or lose a control plane in the middle of a flight. Every frame still reaches ClickHouse. Some arrive twice, because delivery is at-least-once, and ClickHouse removes the duplicates by key. bbcheck measures the outcome after each drill, and the numbers go in Results.

How it fits together

Each part has one job.

PartBuilt withJob
firmware/Rust, no_std, EmbassyRuns the control loop on a hardware timer and sends telemetry over UART
gateway/Rust, tokioFinds frame boundaries, checks CRCs, spots sequence gaps and publishes to Redpanda
pipeline/Rust, SpateDecodes frames, routes them into tables and loads ClickHouse
deploy/cloud-init, kubeadm, YAMLBuilds the VMs and the cluster, then runs everything on it
tools/bbcheck/PythonProves what arrived: gaps, duplicates and latency

The cluster

Five VMs on one machine. Three control planes share a virtual IP, so the API stays up when one of them goes down. Two workers carry the workloads, and a node label decides which one owns the serial link.

Renode runs on the host, outside the cluster, because it stands in for a physical device. On real hardware the gateway would read /dev/ttyUSB0. Here it reads the TCP socket Renode exposes.

The life of one frame

The last step matters most. Spate only commits an offset once ClickHouse has confirmed the batch is durable, so a crash can repeat work but never skip it. That is where duplicates come from, and why the table removes them by (boot_id, system_id, seq).

The wire format

Telemetry uses MAVLink v2. Every frame carries a sequence number, which is what lets the gateway and bbcheck count gaps.

BytesFieldNotes
1STXAlways 0xFD in v2
1lenPayload length
1incompat_flagsBit 0 marks a signed frame
1compat_flagsSafe to ignore if unknown
1seqWraps at 255; a jump means lost frames
1sysidWhich vehicle
1compidWhich component on it
3msgid24-bit message ID
lenpayloadTrailing zero bytes are trimmed
2checksumCRC-16/MCRF4XX, seeded with the message’s CRC_EXTRA
13signatureOnly on signed frames

The firmware sends HEARTBEAT, ATTITUDE, HIGHRES_IMU and a custom LOOP_TIMING message that reports overruns and jitter.

Real-time, honestly

The control loop is hard real-time inside the simulation. Renode runs in deterministic virtual time, so loop timing reflects the firmware alone.

Everything after the serial port is soft real-time. I measure it as end-to-end latency percentiles, from the firmware’s timestamp to the row landing in ClickHouse.

Running it

Still being built. When it’s done, it will look like this:

mise install    # pinned tools: just, D2, uv, kubectl, Helm and more
just bench      # check everything is installed
just lint       # formatting and lints for the crates, firmware and bbcheck
just test       # the tests, plus a firmware build
just coverage   # line coverage across Rust and Python
just cluster    # five VMs and a highly available cluster
just deploy     # Redpanda, ClickHouse, Grafana, gateway and pipeline
just fly        # boot the firmware in Renode and start streaming
just check      # run bbcheck against ClickHouse
just docs       # build the docs site into book/
just audit      # scan the history for secrets

just on its own lists every recipe. You’ll also need Linux with KVM, about 11 GB of free memory, mise, Multipass, Renode, probe-rs, the ClickHouse client and Rust with the thumbv7em-none-eabihf target.

A released version

Every release puts multi-arch images of the gateway and pipeline on GitHub Container Registry. Once the firmware has a binary, it’s attached to the release too. The changelog says what each one changed.

docker pull ghcr.io/guy-with-a-why/telemetry-blackbox/gateway:latest
docker pull ghcr.io/guy-with-a-why/telemetry-blackbox/pipeline:latest
gh attestation verify oci://ghcr.io/guy-with-a-why/telemetry-blackbox/gateway:latest \
  -R guy-with-a-why/telemetry-blackbox

Each image carries a provenance attestation, so the last command proves it was built here, from a tagged commit.

Repository layout

firmware/        Embassy firmware for the STM32F4
renode/          the board Renode simulates, as a script
gateway/         serial link to Redpanda
pipeline/        Spate pipeline into ClickHouse
deploy/          VMs, cluster bootstrap and manifests
tools/bbcheck/   delivery and latency report
docs/adr/        decisions and the reasons behind them
docs/brand/      the logo, favicon, social preview and how to use them
docs/diagrams/   D2 sources and the SVGs they render
docs/runbooks/   drills, with real output
scripts/         small checks the justfile and CI call

Decisions

Each gets a short record in docs/adr/.

  1. MAVLink v2 over a custom framing.
  2. kubeadm over k3s or Talos.
  3. Redpanda over Apache Iggy or a home-made log.
  4. Deduplicating in ClickHouse over chasing exactly-once.
  5. just and mise over make, and a local registry for images.

Results

These fill in as the drills run.

DrillFrames sentMissingDuplicatesRecovery
Kill the pipelinependingpendingpendingpending
Drain the pipeline’s nodependingpendingpendingpending
Restart Redpandapendingpendingpendingpending
Cut the serial link for 60 spendingpendingpendingpending
Lose a control planependingpendingpendingpending
Scale the pipeline to two replicaspendingpendingpendingpending

Diagrams

Every diagram is a D2 file in docs/diagrams/. D2 lays it out, and each SVG follows the viewer’s light or dark theme. The animated ones step through boards declared in the same file.

To change one, edit the .d2 file and watch it live, then render the lot:

d2 --watch --layout=elk docs/diagrams/architecture.d2
just diagrams

Cite

These are the documents I need to build this, grouped by where I use them. The board Renode simulates is ST’s STM32F4DISCOVERY, which is built on the STM32F407VG, a Cortex-M4F.

Tooling (Leg 0)

Microcontroller (Leg 2)

Protocol (Legs 2 and 3)

Cluster (Leg 1)

Data path (Legs 1 and 3)

Diagrams

Licence

Dual-licensed under MIT or Apache 2.0, at your option. Any contribution you submit is licensed the same way, unless you say otherwise.