Backward compatibility by design: the wire contract of a low-power IoT system
In a fleet of connected hives you cannot update everything at once: old and new nodes coexist for weeks. How a versioned, append-only wire contract makes backward compatibility structural instead of a patch.
In almost any system that ends up in the field, “I update everything and start from scratch” is impossible to say. With a web app it is a deploy: you press a button and from that instant everyone is on the new version. With a fleet of hives scattered across the countryside it does not work that way. The nodes are small, battery-powered, often kilometers from the nearest person; you update them when enough bandwidth is available, and in the meantime, for days or weeks, you have old and new firmware in the air at the same time, speaking to the same sky.
This is not an incident to manage: it is the normal condition. And once you have accepted it, it changes the whole way you design communication between devices. In this piece I tell how bjtOS holds together a heterogeneous fleet without breaking anything, and why backward compatibility, for us, is not a patch added afterwards but a property built into the protocol from the very first byte.
The real problem: updating what cannot be updated all at once
Let me put it in concrete terms. I have a hive installed in March that sends its data regularly. In July I improve the firmware: I add a measurement, fix a behavior, change how I pack the data. New hives leave the factory with the July version. The March one keeps transmitting its version, and will keep doing so until the conditions to update it are met.
In the meantime the system has to work for all of them. It is not acceptable for the March hive to disappear from the charts because the server has learned to speak only the July language. And the opposite is not acceptable either: that to introduce something new I have to wait until I have updated the last node in the installed base before I can use it on the new ones.
The solution: you have to treat the message format as a versioned contract, with precise rules on how it may evolve, exactly as you do with a database schema you cannot migrate cold, or with a public API that has clients you do not control.
The contract: a binary format with the version up front
A hive node transmits over radio at very low power. Every byte in the air costs energy and transmission time, and energy is the resource that decides whether the node makes it to the end of winter alive. So no verbose formats: the telemetry message is binary, compact, fixed-structure. Every field occupies a known position and a known width, and the receiver knows how to read it because it knows the layout in advance, not because the message is self-describing.
A format like this is extremely efficient but also rigid: if the receiver and transmitter do not agree byte for byte on how the message is built, they read garbage. And this is where the first building block of backward compatibility comes in: the very first byte of the message is the schema version number. Before interpreting any data at all, the receiver reads that byte and knows which grammar the rest was written in. It does not have to guess, it does not have to infer from the length: the message itself tells it, in a single byte.
This turns a rigid format into one that is extensible in a controlled way. The version byte is the pivot around which the whole coexistence between different firmware generations turns.
Two axes of version, never to be confused
Here is the distinction that, once I brought it into focus, made everything else simple. In the system two different notions of version coexist, and keeping them separate is half the work.
The first is the wire schema version: that byte up front. It changes only and exclusively when the byte layout changes, that is when I add or move a field. It is a rare and deliberate event, because it touches the contract between the devices.
The second is the node firmware version: the release of the software running on board. It changes far more often, at every release, even for a bugfix that does not alter the message format by a comma. This version also travels inside the message, but as any other piece of data, a couple of bytes at the tail, not as part of the grammar.
Why is it important not to confuse them? Because most of my updates change the node’s behavior without touching the format. If I tied every release to a schema change, I would force the whole downstream chain to update for nothing. By keeping the two axes independent, a firmware bugfix requires not a single change on the server: only a number changes, which the server records and displays, and life goes on. The wire contract moves only when it really needs to.
The jump we paid for once, and the rule that came out of it
I would be dishonest to tell it as if it were born perfect. There was, at the beginning, an invasive schema change: I had to widen the field that identifies the node, and widening a field in the middle of the message means shifting everything that comes after it. It is the kind of change that an old decoder cannot absorb, because from that byte onward it reads everything out of place.
That jump we paid for once, and it taught us the rule we have followed ever since: you only extend at the tail. Every new schema version adds its fields at the end of the message, never in the middle. The offsets of everything that existed before stay identical. The node identity, the basic telemetry: they are always in the same bytes, version after version.
The consequence is powerful. A decoder that knows an older version, put in front of a newer message, correctly reads the whole part that concerns it and simply ignores the tail it does not know how to interpret. And a new decoder, in front of an old message, knows that tail is not there. Append-only makes the two worlds compatible without either having to know about the other. It is the same principle by which robust file formats and protocols meant to last evolve: adding does not break, moving does.
The flag word: evolving capabilities at zero cost
There is a second mechanism, more subtle, working on a different axis. Not all nodes have the same capabilities: some carry a certain sensor, others do not; some carry it but at a given moment it is in error. If every different configuration required a different schema, I would have an unmanageable explosion of versions.
The solution is a flag word inside the message: a small group of bits in which each bit declares whether a certain field is valid in this frame. An absent or faulty sensor holds its bit at zero, and the corresponding value is simply not emitted and not read. Two nodes with different equipment send the exact same schema, with different flags.
This decouples one thing from another that is usually confused: “which sensors that node has” from “which protocol version it speaks”. I can have the whole fleet on the same wire contract while individual nodes differ in what they can measure. And when someday I want to add a value produced only by some units, often not even a new schema version is needed: it is enough to turn on a bit that was unused until then. The evolution of capabilities, below that threshold, costs zero in terms of compatibility.
Backward compatibility by design: the decoders and automatic routing
On the server side, all of this translates into an architecture that is simple and, in my opinion, elegant. There is no single parser trying to understand every message. There is a family of decoders, one per schema version, each one knowing exactly one grammar and validating rigorously what it receives, starting from the expected length and the version byte.
Above them there is an automatic router: it reads the version byte of the incoming message and hands it to the right decoder. A frame from the March version and one from the July version enter the same flow, indistinct, and are each dispatched to their own interpreter. The fleet can be as heterogeneous as you like: as long as each version has its registered decoder, all messages are understood.
Adding a new version, at this point, is an additive operation in the code too: you write a new decoder, typically reusing almost everything from the previous one since the difference is only the tail, and you register it. The existing decoders are not touched. The old one keeps working identically, the new one sits beside it.
An operational detail worth mentioning, because it is the point where theory meets the field: introducing a new version is not a switch you flip at time zero. You first enable the new decoder on the server, so it is ready to accept both versions, and then you start reflashing the nodes, one at a time, at your own pace. At no instant of that transition, which can last weeks, is a single datum lost. There is no breaking window, because the server already speaks both languages before the first node changes its own.
Identity and version are different things
A principle I keep separate with discipline: a node’s identity has nothing to do with its version. Every node receives at production time a unique identifier, assigned centrally, which is its plate forever. From that identifier I also derive the readable serial number printed on the label, but it is the same number in another guise, not a parallel identity.
That identity is fixed. The firmware on top of it goes up and down at every release, the wire schema changes when it must, but the plate remains. It is what lets me say, looking at the console, “this node, which is always the same one, in March was running one version and since September runs another”. If I tied identity to firmware, every update would look like a new device and I would lose the history. By keeping them separate, the version becomes an observed attribute of a stable entity, not part of its definition.
One source, many binaries
The same philosophy applies upstream, in the firmware. I do not have a code branch for every node variant: I have a single source from which I produce different binaries by choosing, at compile time, a profile and a handful of options. The board revision, the individual unit’s identity, the power profile, even which sensor variant to fit: they are all decisions taken at compile time through build options, not divergent branches to keep aligned by hand.
This keeps complexity in one place. The logic lives once; the differences are parameters. A profile meant for the test bench, with the service screen on and verbose logs, and a production one that turns off everything not needed and does not spend a microamp on useless printouts, are the same program with a different switch. Less duplicated code means fewer places where a bug can hide and survive across versions.
The gateway has no version: fleet observability
Here is the statement that throws off anyone expecting a single number. The hive node has a firmware version, clear and in-band. The gateway that collects the messages and carries them to the cloud, instead, has no version: it has several, because it is a stack of independent pieces. There is the radio receiver firmware, there is the software agent that bridges to the internet, there is the system image everything runs on. Each evolves on its own and has its own number.
That is why version tracking happens over two different channels. The node version travels inside every telemetry message: it is in the data, I know it at every frame. The gateway versions, instead, arrive out of band, in a periodic status message the gateway publishes on its own, which the server merges into its registry keeping the last known value of each piece. The result, on the console, is not “the system version” but a matrix: for every device, who it is, what runs where, and since when I last saw it.
This is not pedantry. In a real base, distributed and updated in waves, the question “which nodes are still running the old version?” is operational: it tells me where I still have to go, what I can already take for granted, what behavior to expect from each hive. Without that matrix, a rollout is blind.
The return path is versioned too
So far I have talked about the flow from node to cloud, but there is also the opposite direction. After transmitting, the node keeps a listening window open for a moment, and the gateway can answer it: an acknowledgment of receipt, the name the hive should display itself with, and for some time now also the current time, so the node keeps its clock synced without needing any other hardware.
And this return channel follows exactly the same discipline as the outbound one: it is versioned too, and in a tolerant way. There came the moment when I added the time to the reply message, that is a new version of the return format. A node with older firmware, which does not expect that field, keeps reading without trouble the part it knows and ignores the rest, because here too the initial part of the message is shared across versions. The same idea, applied in both directions: add at the tail, and whoever cannot read the tail does not even notice it.
The discipline that holds it all together
None of these mechanisms works on its own without a bit of operational hygiene around it. The rules I follow are few and firm.
The first: the version is incremented at every reflash with changes. If I change something on board and do not raise the number, in the field I can no longer tell the new node from the old, and I have lost the only tool I have to understand the state of the fleet. The version number is worth as much as the change it represents.
The second: the new decoder is enabled before touching the nodes. The server must be able to read the new language before anyone starts speaking it, never after.
The third: data migrations are designed so they do not touch the history. When I widened the node identity, the telemetry history did not move by a single row, because it is anchored to the hive’s identity, not the node’s. The two things are separate on purpose, and in a moment like that the separation pays back all the annoyance of having maintained it.
Heterogeneity as a choice, not an accident
If I had to sum it up in one sentence: I stopped treating a uniform fleet as the goal, and started treating a heterogeneous one as the starting point. I do not design for the mythical day when all hives run the same version, because that day never arrives in a living system that grows. I design so that different versions coexist well, indefinitely, without any one breaking the others and without my having to coordinate a global update to move a single step.
The versioned wire contract, append-only, the flag word, the decoders that sit side by side, identity kept separate from version, the matrix instead of the single number: they are all faces of the same choice. Backward compatibility is not something you add when you need it. It is something that, if you do not build it in from the very first byte, you can no longer recover afterwards.
If you are interested in the big picture of what is inside a connected hive, I have told how a smart hive works, and on why certain energy choices come before everything else there is the piece on how long a battery-powered hive lasts. The rest of what we are building on is in the pages dedicated to the technology and to the product.