A ring buffer in flash: how an off-grid gateway loses no data while it sleeps

There is no power socket in an apiary. It is a trivial sentence, and it decides most of the architecture of bjtOS. Every so often it also forces you to reopen decisions that looked settled: that is what happened when I tackled gateway buffering, that is the box that collects the data from all the hives in an apiary and ships it to the cloud.

The result is a non-volatile ring buffer in flash: a circular buffer written to permanent memory, which survives power-off and reboots. In this piece I tell why it is needed, how it is built, and the thing that surprised me most while designing it: that the widespread instinct of “do not write to flash too often, you will wear it out” is only half true, and the missing half is exactly the one you need in order to decide.

The problem: the most capable piece is also the hungriest

The gateway has two very different brains inside it.

One is a small Linux computer. It knows how to talk to the cellular network, encrypt, manage queues, retry when something goes wrong. It is convenient, and it is also hungry: powered around the clock, it asks for almost ten watt-hours over a winter night.

The other is a microcontroller with the radio, the one listening to the hive nodes. It draws about forty milliwatts, an order of magnitude less, and it can stay on forever without weighing on the energy budget.

The arithmetic is quickly done. If I want a gateway powered by a reasonably sized solar panel, as I described when writing about how long a battery-powered hive lasts, the computer cannot stay on all night. It has to be off nearly always, and wake up two or three times between sunset and dawn to send everything in one batch.

Fine. But the hive sensors do not know the computer is asleep. Radio messages keep arriving all night long. Somebody has to take note.

Why volatile memory is not enough

The first obvious answer is to keep the data in RAM on the microcontroller, which is always on anyway. It is also the wrong answer, and you get there only by thinking about what an off-grid night is really like.

At night the panel produces nothing. The battery sags, especially in the cold, and especially when the cellular modem draws current to latch onto the network. A brownout, that is a voltage dip that restarts the microcontroller, is not an exotic accident to file under remote possibilities: it is the normal condition the system has to withstand.

And a reboot wipes the RAM. A whole night of data gone, silently, with nobody noticing until the next morning.

So the buffer has to live in non-volatile memory, that is in flash: a memory that keeps what it has written even with the power disconnected.

Program and erase are not the same operation

Here comes the part that changed the way I designed the whole thing.

Anyone working in embedded carries around a conditioned reflex: do not write to flash too much, you will wear it out. It is a healthy piece of advice, but if you stop there you end up designing around the problem instead of through it.

In NOR flash, the kind integrated in the microcontrollers we use, there are two clearly distinct operations:

• Program, the actual write. It sets to zero the bits you need, it is done in small chunks, even many times inside the same sector, as long as you write on still-virgin space. It does not cause appreciable wear.
• Erase, the deletion. It brings an entire sector, typically 4 KB, back to all ones. It is this operation that has limited endurance, on the order of a hundred thousand cycles per sector.

The practical consequence flips the instinct on its head. Writing every single message as soon as it arrives is not a waste: it is a program, it costs little, and it makes the buffer resilient to power loss, because if the microcontroller dies a millisecond later the datum is already there. What has to be counted is not the number of writes, but the number of laps the buffer makes around the memory, because it is at every lap that you pay an erase on the recycled sectors.

It is a distinction that lets you pick the more cautious behavior, that is write immediately and always, without paying for it in hardware lifetime.

A ring, two indices, and the ability to rebuild itself

The buffer structure is the oldest one in the trade: a circular buffer, or ring buffer. A head that advances as it writes, a tail that advances as it reads, a ring of slots on which both travel. When the head reaches the end it starts again from the beginning, reusing space that has already been read. It is a fixed-size FIFO: it never grows, it never fragments, and it has an upper bound on occupancy known from the start. On a device that has to work on its own for years, predictability is worth more than elegance.

Every record written carries a minimal header: a marker to recognize it, the length, and a CRC, that is a check code that tells whether the content is intact. They are there for the worst moment, which is the reboot.

At power-up the microcontroller trusts nothing: it rescans the partition and rebuilds head and tail from scratch, reading the records one by one. If power dropped halfway through a write, that record fails the CRC check, is discarded, and the tail stays consistent. The buffer repairs itself, without needing anyone to have left a note about what it was doing an instant before dying.

On top of raw flash you can lay a filesystem designed for microcontrollers, which brings along wear leveling, that is spreading the wear across all sectors, and power-loss resilience already solved. You pay a little extra writing for metadata, which at our volumes is irrelevant, and in exchange you avoid rewriting by hand a class of bugs others have already paid for. Managing the raw partition manually stays the fallback option, if one day a more deterministic efficiency were needed.

The numbers, which are the reassuring part

Take an apiary with twenty hives, our typical case study.

A record fits in about forty bytes, so a hundred of them fit in a 4 KB sector. A fourteen-hour winter night, at a dense transmission rate, produces something like five hundred and sixty messages: a little over twenty kilobytes.

Twenty kilobytes. On a partition to which we dedicate five hundred to a thousand of them.

So why so wide, if the peak is so small? Not for the normal night. For the night when the modem does not latch onto the network. In that case the computer wakes up, tries, fails, and the buffer is not drained: the data stays there and sets off again at the following window. The margin is not there to hold the traffic, it is there to buy days of autonomy when cellular coverage is missing.

On wear: about thirteen hundred records a day make thirteen sectors a day. On a one megabyte partition, that is two hundred and fifty-six sectors, the ring makes a lap every twenty days at most. That comes to fewer than twenty erases per sector per year, against an endurance of a hundred thousand cycles. The constraint, quite simply, is not this one: we are on a scale the hardware will never see. You check it once with a spreadsheet and then you stop looking at it.

It is worth saying because it is the kind of calculation that is always worth doing up front: it saved me from optimizing against a problem that does not exist, and it let me choose the safest behavior with no guilt.

The rule that makes it all honest: you delete only after confirmation

The last piece is the one that turns a buffer into a guarantee.

When the computer wakes up, the microcontroller does not hand it the data and then get rid of it. It passes the data by reading it from the tail, and the tail advances only when confirmation arrives that the data has been accepted on the other side, in the cloud. No deletion before confirmation. Ever.

The result is at-least-once delivery: in the worst case the same message arrives twice, and downstream it is recognized as a duplicate and dropped, because every message carries a stable identifier with it. The acceptable worst case is the duplicate. Loss is not.

The two queues therefore work in series, and each has its own job. The one in flash on the microcontroller accumulates over the long run and survives the computer being off. The one on the computer, which in our system already exists and is field-proven, handles the single session towards the cloud with its reconnections and its retries. It is the same philosophy the whole bjtOS transport chain was born with, which I told about in I built the road before the sensors: the datum is never lost, it accumulates until there is a way to deliver it.

What is still open

I would be dishonest to tell it as a done deal. This is a design that is written, sized and argued, not yet code running in an apiary. What is missing I know precisely, and it is the most instructive part:

• the drain protocol, that is how the two brains talk to each other during the transfer, with the session start and end markers and the tail advancing tied to the confirmation;
• the time, which today the microcontroller receives from the computer, which in turn takes it from the network. With the computer off it has to keep time on its own between one wake-up and the next, and realign when the other comes back on;
• the urgent path: an anti-theft alarm cannot wait for the night window, it has to power the computer on immediately. And for the receiver to recognize it without opening the message payload, you need a priority bit readable in the header, which has to be coordinated with the node firmware and therefore with the versioned wire contract;
• the watchdog: what happens if the computer powers on and never confirms, because it got stuck at boot or while latching onto the network.

The criterion: what happens at the worst moment

The underlying choice, though, has been made, and it holds up to a simple criterion I often use when designing: what happens at the worst moment?

The worst moment here is a February night, with a low battery, the modem finding no signal and the voltage wobbling. At that moment, somewhere, there has to be a notebook with everything already written on it, one that does not erase itself when the lights go out.

It is the same principle guiding everything else in the system: a digitally native hive is designed starting from the constraints, not from the sensors. If you are interested in the big picture there is how a smart hive works, and on what we are building you will find the pages dedicated to the technology and to the product.