Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Holding pattern: Redpanda on data-1’s own disk

18 November 2026

What was done

Tonight Redpanda went onto the cluster, for #14. It’s where every frame will circle until ClickHouse is ready to take it, so it has to hold them through anything short of losing the disk. One broker runs as a StatefulSet in dev mode, on a 10 Gi volume on data-1, with a telemetry.raw topic of six partitions.

just redpanda deploys it and is safe to run again. A second run takes 15 seconds and changes nothing. just redpanda-check proves the issue’s first box. It writes 10,000 numbered records, deletes the Pod and reads every one back once the StatefulSet has replaced it. The broker was back on data-1 in 7 seconds with none missing, and 100,000 records came back the same way.

Redpanda’s image lives on Docker Hub, so the host gained a fourth pull-through mirror first, in #125. The broker and the check landed in #127 and #128.

Why was it done that way

The cluster had no StorageClass, so a claim had nowhere to go. A StorageClass is the menu an admin offers: who creates a volume, when a claim binds to it and what happens to the data afterwards. Mine is called local and has no provisioner. I create each volume by hand, as a local volume on one node’s disk, pinned to that node. Binding waits for the first Pod that uses the claim, so the scheduler picks a node before the claim picks a volume. With immediate binding, a claim could take a volume on a node the Pod can’t run on. The volume is set to Retain, so deleting the claim never deletes frames.

I passed over replicated storage such as Longhorn. There’s one data node, so it has nowhere to copy to, and with more nodes Redpanda and ClickHouse would replicate their own data more cheaply than a disk layer could. The price of local disks is that a stateful Pod can’t move. Draining data-1 stops Redpanda until the node is back, and the drain drill in #27 will show it. A dynamic provisioner is in Stretch as #124, for everything that doesn’t need a hand-made volume.

Redpanda is built on Seastar, the same framework as ScyllaDB. It runs one thread per core, claims its memory at startup and writes past the page cache. So --smp=1 --memory=512M gives it one core and 512 MB, and --overprovisioned stops it pinning and busy-polling data-1’s two shared vCPUs. The container’s limit is 1 GiB, because Seastar’s share doesn’t cover the binary or its thread stacks.

The bash behind the cluster is now about 610 lines. Talos, OpenTofu and Flux would bring that to one 60-line script plus about 350 lines of configuration, and I’ve filed that rebuild as #130 with six sub-issues. I’m not starting it yet. Building kubeadm by hand is the point of this leg, and the bash works, is linted and is about to stop changing, so replacing it now would be a rewrite rather than paying off debt. The Leg 4 drills are the safety net that will prove a new platform moved nothing, and Flux in #107 should come first, so the workloads are declarative before the layer underneath them changes. Writing it down now, with a trigger, makes it planned work. It also retired an issue that would have polished scripts #130 replaces.

What challenges were faced

The first deploy timed out. I’d rebuilt the cluster from a checkout on my home server that predated the mirror, so nothing on the bridge served Docker Hub. Redpanda’s 122 MB image came straight down my connection at about 2 minutes a node, which outlasted the script’s 2-minute wait for the topic Job. The broker and the Job had both finished fine. The wait is now 5 minutes, matching the rollout. just vms-check would have caught the missing mirror, and just up doesn’t run it yet.

The real mistake was durability. Dev mode turns write caching on, which acknowledges a write before it reaches the disk, so my first version turned it off with a bootstrap file and called the broker durable. A second pass over the manifests found that dev mode also starts the broker with --unsafe-bypass-fsync=true, so an ack no longer waited for the disk to confirm the write. Deleting a Pod leaves every cache below it intact, so the restart check passed 100,000 records, which proves the caches are very reliable right up until the power goes. The broker now starts with --unsafe-bypass-fsync=false, and its command line in the logs shows it. A restart check can’t see this class of fault. The Leg 4 drills should include stopping data-1 hard.

I also broke the deploy on purpose to see how it fails. A volume left Released after its claim was deleted now stops the script at once and prints the command that frees it, rather than ending on a 5-minute timeout. A topic Job that can’t reach the broker prints rpk’s own error within 25 seconds. That needed restartPolicy: Never, because with OnFailure the Job deletes its Pod when it gives up and the logs go with it.

What’s next

ClickHouse is next, on the same local class, in #15. The risk is data-1’s 40 GB disk. Redpanda refuses writes when free space falls under 5 GiB, and it measures that against the whole filesystem ClickHouse will share. ClickHouse’s retention is the first lever, and a bigger disk for data-1 is the fallback. Leg 1 lands on Friday, with the decisions in #16 and its debrief.

Quiz

Five questions on the post. Open one for my answer.

Why does telemetry.raw have six partitions, and what does a key decide?

A partition is an ordered log, and it’s the unit consumers share out. Six let up to six consumers in one group read in parallel. The key’s hash picks the partition, so every frame with the same key stays in order.

What's an offset, and who keeps it for a consumer group?

An offset is a record’s position in its partition. The group commits the offset it has processed to the broker, so a restarted consumer carries on from there. Committing only after ClickHouse confirms a batch is what makes delivery at-least-once.

Why does the local class wait for the first consumer before binding?

A local volume only exists on one node. Waiting lets the scheduler place the Pod first, so the claim binds to a volume the Pod can actually reach.

The claim was deleted and recreated. Why won't the volume bind again?

With Retain, the volume goes Released and keeps the old claim’s UID. Removing that UID makes it Available for the new claim, and the data is still there.

Write caching was off. Why could Redpanda still lose acknowledged frames?

Dev mode also skipped fsync, so an ack didn’t wait for the disk to confirm the write. A Pod restart keeps whatever is still cached, and a power cut doesn’t.