Watching the Time In Between: Designing an IoT Cold-Chain Compliance Engine

Food businesses have to prove they keep their fridges cold. Under HACCP the proof is a scheduled check — typically twice a day — where someone reads a thermometer and writes the value down. The first thing we automated was that step: a Bluetooth temperature probe in the fridge fills in the scheduled check so nobody has to walk over with a clipboard.

That turned out to be the least valuable thing a probe can do. A probe reports every minute. Using it to fill in two boxes a day throws away almost all of its readings, and it means that when a fridge dies at 23:50 the store manager finds out at the morning check, hours and one spoiled inventory later. The value of the probe is not filling in the checks. It is watching the time in between.

This post is about how we redesigned the system to do that: a compliance engine that evaluates every probe continuously, opens a managed deviation only when something is actually wrong, escalates when nobody looks, and leaves a history an inspector can read — while staying flat in cost as the fleet grows by orders of magnitude. The design was written as a series of Architecture Decision Records, and I will name the alternatives we rejected, because that is where the design actually lives.

What Was There Before

Before touching anything we wrote down what production actually did, because a plan that starts from an imagined baseline is a plan for a different system. What ran was this:

• A chain of one-shot schedules, each armed for the time of the next scheduled check, re-arming itself after firing.
• On each trigger, one read: the latest reading for the probe, with no condition on how old it was. That value was written into the scheduled check.
• The comparison with the limits happened on that single point. Any out-of-range reading opened the "under monitoring" stage — and, thanks to a bug, so did readings in range. Compliant fridges were polluting the very history an inspector would read.
• No freshness check. A probe switched off a week ago still returned its last reading, recorded as a compliant check. Wrong, but at least visible.
• No polling, no window, no moving average anywhere in the code. The continuous evaluation some documents described did not exist.

The first decision was about separating levels. The old design fused three things with different natures: raw telemetry (a reading a minute), the scheduled recordings that are the backbone of HACCP self-control, and the management of deviations. The new one keeps them apart, each with its own retention and its own store: telemetry lives for days, recordings and deviation events for years.

The Shape of the System on AWS

Before the individual decisions, the map. The pipeline has six layers, and most of the left half already existed: probes, gateways, IoT Core, the Lambda that splits gateway batches into single readings. What is new is one DynamoDB table with two sparse indexes, a handful of Lambdas, a change stream wired through an EventBridge Pipe into SQS, and the event tables on the relational side.

Three things to notice on the map, each of which becomes a decision below. The state table is the contract: the poller, the slot Lambda and the reconciler all write it; consumers only ever read it by probe id; so the producer can change from Phase 1 to Phase 2 without the readers noticing. Nothing downstream of the table is triggered by a Lambda: alerting starts from the committed write, through the table's own change stream. And the two sparse indexes are what keep the sweeper flat: they contain only silent probes and only unacknowledged deviations, so a minute-by-minute scan reads almost nothing regardless of fleet size.

Decision 1 — The Gate Is a Moving Average, and Nothing Else

Compliance is evaluated on the moving average of the last hour of readings, compared with the control's limits, every few minutes. Average inside the limits: green. Average outside: red. That is the whole gate.

The alternative was the intuitive one: compare the raw value and require it to stay out of range for a persistence window before going red. It gives earlier warning and cleaner recovery. It also needs a fourth traffic-light state ("under verification"), a persistence counter, and hysteresis to stop the light from flapping — three mechanisms doing what the average does alone. A defrost cycle spikes for a few minutes; an average over an hour barely moves. The average is the anti-transient mechanism.

We accepted the trade-offs out loud. No early warning: the transition is green-to-red with no orange. And slow recovery: after a repair the light stays red until the average comes back, up to most of an hour after the operator has fixed the problem. The second one needed a line of UI copy ("the value returns gradually"), not a state.

A later infrastructure draft tried to sneak two rejected mechanisms back in: a pending state requiring two consecutive bad evaluations, and a minimum number of samples before evaluating at all. Both were removed. A consecutive-breach counter redoes the average's job and delays detection; a minimum sample count reintroduces a blind hour at start-up that the next rule already covers.

Evaluation is decoupled from recording

Evaluating every few minutes is cheap only if it does not call the management system every few minutes. So evaluation (inside the IoT service, against a working copy of the limits) is a different act from recording (a write to the management system, only on state changes and at the scheduled check). Evaluating every minute was considered — it is the granularity the data has — but it multiplies round-trips for a few minutes' gain on a slow thermal signal.

Start "at regime"

After a probe is associated or re-activated, monitoring starts neither immediately nor after a fixed timer, but at the first reading inside the limits. The same gesture covers two opposite situations the system cannot tell apart: an empty fridge cooling down (readings high but harmless) and a technical re-association of a full fridge already cold. A fixed grace period protects the first and opens a blind hour on the second. "First reading inside limits" handles both: on the empty fridge it waits; on the full one it finishes instantly. From then on the average runs on whatever partial window exists, rather than waiting for a full hour.

Decision 2 — Three Levels of State: Phase, Gate, Stage

The confusion in the old system came from one status field answering three different questions. The new model keeps three:

Three rules follow from the split, and each was argued over:

• Green permits closing; it does not cause it. A deviation closes only when the average is back and the operator has confirmed a corrective action. Temperature OK plus documentary proof of the intervention is the most defensible criterion in an inspection.
• Closing and escalating have different conditions. Closing requires both; escalation stops as soon as either holds. Average back but nobody signed: stays open, does not escalate (the physical problem is gone). Signed but average still red: stays open, does not escalate (someone has it — escalation is about coverage, not authority). We wrote the second case wrong at first, and the ADR won. The accepted risk is on record: an operator who stamps "fixed" without fixing stops the escalation.
• A non-conformity is never automatic. Detection is automatic and immutable ("red from 23:50, escalated at 01:00, not handled" stays in history). The conclusion — declaring an NC — is a human act, attributed by the operator code typed at that moment. Not clicking cannot erase a deviation; only a person can call it a non-conformity.

Decision 3 — One State Row per Probe, Keyed by the Hardware Serial

Evaluation state has to live somewhere between polls. The decision was one row per probe in DynamoDB, partitioned by the probe's hardware serial. Not per control, not in worker memory, not in a cache. Here is the row, with field names paraphrased:

{
  "probe_serial": "ac233fabcd12",       // partition key — hardware id, canonical form

  // what this probe is watching (copied from the management system, remapped on publication)
  "control_ref":    "…",                // the control definition that carries the limits
  "equipment_ref":  "…",                // the fridge
  "quantity":       "TEMPERATURE",      // which channel of the payload to read
  "limit_min": 2.0, "limit_max": 8.0, "unit": "C",
  "config_version": 4,                  // working copy of the limits; bumped by the reconciler

  // the window — Phase 2 only. Phase 1 recomputes it from raw readings instead.
  "slots": {                            // absolute 5-minute buckets, ~12 kept
    "1770000000": { "v": 4.1, "ts": 1770000287 },
    "1770000300": { "v": 4.3, "ts": 1770000581 }
  },

  // the three levels of state
  "phase":     "AT_REGIME",             // WAITING_REGIME | AT_REGIME | SUSPENDED
  "gate":      "COMPLIANT",             // COMPLIANT | NON_COMPLIANT | DATA_ABSENT
  "red_cause": null,                    // OUT_OF_LIMITS | PROBE_SILENT
  "event_ref": null,                    // the open deviation event on the relational side

  // keys of the two sparse indexes
  "shard":     37,                      // hash(probe_serial) % N, stable for life
  "last_seen": 1770000300               // sort key of the stale-sweep index; REMOVED when suspended
  // "escalation_due": 1770003900       // sort key of the escalation index; present ONLY while a
                                        // deviation is open and unacknowledged, REMOVED on ack
  // no TTL attribute: the row dies by reconciliation, never by time
}

Why this shape, group by group:

The row is small on purpose. With one measurement per slot it stays under a single write unit, and DynamoDB bills an update on the whole row size, not on the delta. The two indexes are keys-only projections, so every state write costs one base write plus two index writes and nothing more.

Why the serial and not the control

The control is the subject of compliance: it carries the limits, it is what goes green or red. Keying by it looks natural. It is wrong here, for a reason buried in the management system: publishing a new version of the HACCP manual recreates the touched entities with new ids. Rename a room and its equipment gets new ids; touch a control and its whole procedure is recreated. Keying the state row by the control would orphan every row on every publication — window lost, every control back to waiting-for-regime, an hour of blindness across the fleet.

The hardware serial is the only identity a publication does not touch. So the control and equipment ids are attributes on the row, remapped by the publication flow, and the state — phase, gate, open event — survives a re-publication untouched.

Why not memory, why not a cache

State in worker memory needs consistent hashing from probe to pod, sticky routing, rebalancing on every scale event, and a rebuild of every window on restart — a simultaneous read burst across the whole fleet. With durable state the workers hold nothing: any pod serves any probe, scaling is adding pods, restart has no repopulation procedure. A cache as the source of truth was excluded because it loses data on failover, which brings the repopulation procedure straight back.

Why the average is never stored

Whoever reads computes the average from the window of samples. The window length stays a read parameter: the same row serves an hour today and another duration tomorrow with no migration. The cost is that every reader must compute it the same way, so the computation lives behind one interface with two source implementations — raw readings in the first phase, pre-bucketed slots in the second. The gate, the stage and the escalation do not know where the average came from, which is what lets the transport change underneath them later.

Two details that would have silently falsified compliance data

Absolute slot timestamps, not a circular index. With a fixed one-hour window of five-minute slots, an array of twelve positions indexed by (minute / 5) % 12 is the obvious optimisation. It is a bug. The gateway ships in batches, so a 09:00 reading can arrive at 10:05. With a circular index it lands in the 10:00 slot and falsifies the average — no exception, no metric, just a wrong compliance value. With an absolute key it lands in its historical slot, outside the window, and is correctly ignored.

Last value wins, not sum and count. Delivery is at-least-once twice over: the MQTT layer retransmits until acknowledged, and the stream consumer replays a batch on error. A sum/count accumulator counts the duplicate and moves the average that decides an alert. A conditional write — "set this slot only if absent or older than my timestamp" — is idempotent, tolerates out-of-order delivery, and needs no lock.

Decision 4 — The Absence of Data Has to Be Hunted

An evaluator cannot, by construction, detect a silent probe. A probe that sends nothing produces no writes and is never evaluated; and a poller running every few minutes is too slow for an "offline" alarm that must be immediate. Absence has to be sought actively, at a finer cadence than evaluation.

Three ways were considered. A per-item expiry with a delete event is elegant and needs no sweeper, but the store guarantees expiry only "typically within 48 hours" — unacceptable for an alarm. One schedule per control is a pattern we already used, but at fleet scale it is millions of schedules against a quota of about that size, plus a create and a delete per control per cycle. The third is what we built:

The sparse index has a consequence that became its own decision. The original design put a sliding expiry on the state row, renewed by every write. That expiry moves forward while the probe transmits and stops when the probe goes quiet. A probe silent long enough expired its own row — and the sweeper can only find silent probes among rows that exist. The alarm ceased exactly in the case it existed to catch: surveillance ending by statute of limitations, with no error and no metric. The state row has no expiry. A row dies because the assignment is gone (see reconciliation, below), an explicit decision with a trace — not because time passed.

Same principle for suspension. A suspended row receives no readings by definition, so whoever suspends must remove the last-seen attribute, and whoever re-activates restores it. Otherwise the row keeps an old value, matches the cutoff on every sweep forever, and the sweeper's cost scales with the number of suspended fridges instead of open incidents.

Decision 5 — Edge-Triggered Notification Without Loss or Duplication

Notify on the change of state, never "still red" every poll. That needs a mechanism that fires exactly once per transition and survives a crash halfway through. The naive order — read state, notify, write state — breaks: a crash between notify and write leaves the old state, and the next evaluation notifies again.

Two properties fall out. First, the transition is a conditional write that names the expected prior state. Two evaluators that both see red both attempt it; the store serialises writes on the row; the first passes and the second's condition is false. Exactly one winner, with no coordination. Second, the notification does not start from the evaluator. The committed change appears on the table's change stream, a filter keeps only records where the gate changed, and a worker turns them into push and email. It is a transactional outbox where the stream is the outbox.

One more subtlety, which was the actual defect in an earlier draft: the notification is attached to the opening of the stage, not the transition of the gate. If the light flaps — red, green, red — the stage stays open and absorbs it. Hysteresis was rejected for the gate precisely because the stage already does this job, provided the notification hangs off the right event.

Decision 6 — Escalation Is a Sweep, Acknowledgement Is a Push

If nobody handles a deviation within the policy window, the next level of contacts is notified. The first draft scheduled a one-shot timer per event, justified by "exact timing instead of sweep granularity". That argument lost: on a one-hour policy threshold, a one-minute sweep is off by under 2%, and the threshold is a policy number, not a physical deadline. Meanwhile the one-shot cost three real things: schedule churn per event, an extra failure surface, and above all a distributed cancellation problem — acknowledgement happens in the management system, which would then have to call the IoT service to cancel a timer.

Instead, a deadline attribute is written when the gate opens a deviation and removed when acknowledgement arrives. A second sparse index on the same shard attribute therefore contains, by construction, only open and unacknowledged deviations — a handful of rows across the fleet, zero in a healthy one — and the sweep already running for silent probes reads it too.

Acknowledgement is a push, not a poll, and arithmetic decides it. Polling — the IoT service asking "is this handled yet?" on every sweep — is tens of thousands of calls a day for a few dozen open deviations, nearly all answering "no change". A push is one call per acknowledgement and one per escalation that actually fires. Three properties follow. Cancellation stops being distributed: it is an idempotent attribute removal. The error direction is safe: a lost acknowledgement leaves the deadline in place and fires an escalation on a handled event — a visible, harmless false positive, never a missed one. And it scales flat.

Decision 7 — Two Comparisons, Two Roles

The limits are compared in two places, and the difference is declared rather than suffered:

• The alerting gate, in the IoT service, against a working copy of the limits refreshed periodically from the management system. It decides only when to wake someone up.
• The record outcome, in the management system, against the live limits, at the moment the scheduled recording is written. It is the only outcome that lands in the HACCP history an inspector reads.

This makes the staleness of the copy harmless by construction: a copy a few minutes old can produce one alert too many or too late, never a wrong record. Alerts are recoverable; records are not. A single decider in the IoT service would move authority over the limits out of the system that owns them and make the HACCP record depend on a possibly stale copy — the risk exactly where it cannot be recovered. A single decider in the management system, queried on every evaluation, brings back the round-trip that the polling cadence exists to avoid. A push on limit changes is acknowledged as the better long-term answer and deferred; the two-role split is what makes deferring it safe.

The price is that the two rules must match to the letter: inclusive thresholds, and the same rounding applied to the average before comparing. And the IoT service now hands the management system the average of the window, not the point reading — which is also what removes the branch that opened "under monitoring" on a single sample.

Decision 8 — State Rows Are Reconciled, Not Commanded

The assignment — probe, equipment, control, suspension — lives in the management system, because it owns the manual and the manual is what you are inspected against. The IoT service does not receive commands to create state rows. It reconciles. Periodically it walks the set of active assignments with their limits, creates missing rows in waiting-for-regime, updates the copied fields in place, and removes rows no longer in the set. Assignment events also trigger an immediate pass for latency, but the periodic walk is the self-healing floor, not the primary channel.

The alternative — create on event only — is less code and lower latency. But a lost event (a deploy, a failed request, a queue in error) leaves a probe unmonitored forever, and nobody finds out, because there is no row from which to notice. That is the worst failure mode in the domain. Reconciliation gives three things for free that a command path would build one by one: bootstrap of a new environment, recovery after a lost event, and propagation of the id remap on publication.

Reconciling a fleet-sized set needs care in the walk itself. It is keyset pagination, not offset: with offsets, a row removed between two pages shifts everything after it, and the walk skips a probe that is still active — the reconciler would then delete a live row. The response carries a completeness marker compared against the rows collected over the whole walk; a count that changes mid-walk invalidates the walk; and a contraction of the set beyond a threshold is treated as an alarm, not a truth to apply. Deleting on the basis of an incomplete answer is the one thing the reconciler must never do.

Decision 9 — Cutover in Shadow, Then One Switch per Site

The new path went to production first in shadow: running completely and writing its own state, but neither writing recordings nor notifying. A single flag governs both effects, because a shadow that notifies sends alarms from a non-authoritative system, which is the fastest way to lose trust in a product.

Cutover then happens per site with one boolean. Off, the scheduled recording carries the point value and the old chain notifies; on, the same recording carries the hourly average and the new path notifies. Per site and not per fridge on purpose: with the finer switch, two adjacent fridges in the same kitchen would behave differently and neither the user nor support could explain why. Direct cutover without shadow was rejected because the first divergence between the two models would be discovered on a customer's HACCP record, and a rollback does not unwrite a recording. Double-writing was rejected because it dirties the history with records from a system under test.

The exit criterion from shadow is explicitly not "zero differences": the two models disagree by design. The criterion is that every difference is explained by the model change and none by a bug.

The Cost Shape That Decided the Design

The fleet is small today and the declared target is orders of magnitude larger. No functional decision took a position on that, yet it changes which design is correct. When we priced the pipeline at target scale, the dominant line was not the window, the evaluation, or the sweeps. It was the ingestion rule that writes every raw reading straight into the key-value store — tens of times the cost of carrying the same events through a stream. Retiring that write is the goal of the second phase.

Two other readings of that exercise shaped the plan. Compute is not the lever: all the functions together were a rounding error next to the store's write costs, so moving them onto a cluster would trade a small saving for lease tables and shard rebalancing. And the state row fits in a single write unit only because it carries one measurement; adding a second one would cross the unit boundary and double the write line, because the store bills on total row size, not on the delta. That is the only cost effect of keeping humidity out of scope: the model already carries it end to end and discards it at the last step.

The Adoption Path

Why not go straight to Phase 2? Because the first release would then depend on new transport before the compliance engine — the user-visible value and the part with the most domain rules — is in production and verified. The thing that makes this a phase plan and not a rewrite is the storage contract: consumers only ever read the state row by serial, and the average sits behind one interface with two source implementations.

Phase 2 has an ordering constraint that is easy to get backwards. The sweeper must exist before raw-telemetry retention is shortened. Today a probe that has been switched off returns a stale reading that gets recorded as compliant — wrong, but visible. Shorten retention first and that reading no longer exists; the value arrives empty; the management system skips the write without saying anything. Inverting the order turns a wrong value into a fridge with no surveillance and no alarm.

What Changed, Side by Side

Key Takeaways

1. Write down what production actually does before planning phases. The continuous evaluation existed in documents and nowhere in code.
2. One anti-transient mechanism, not three. The moving average made the intermediate state, the persistence counter and the hysteresis all redundant. Accept the trade-offs out loud.
3. Separate the sampled from the eventful. A gate is re-evaluated every poll; a deviation is one event with a life cycle. Fusing them was the root of the old bug.
4. Key state on the identity that survives. Business ids get recreated; the hardware serial does not. Everything else is an attribute that can be remapped.
5. Read the code before trusting the decision record. "The association persists by itself" was false. The correction became a new ADR, not a footnote.
6. Absence of data must be hunted, and expiry is not a detector. A sparse index and a one-minute sweep cost nearly nothing at any fleet size; item expiry guarantees 48 hours.
7. The state change is the lock; the stream is the outbox. A conditional write and a change stream give exactly-once transitions and at-least-once delivery with no coordination service.
8. Prefer the failure direction you can see. A lost acknowledgement causes a redundant escalation, never a missed one. A missing row would be silent; an orphan row gets reported.
9. Declare which comparison is authoritative. The alert can be stale; the record cannot. Two roles make a stale copy harmless.
10. Reconcile, don't command. A lost event must never mean a fridge nobody watches. Keyset walks and a completeness marker make the reconciler safe to run.
11. Shadow first, then a switch at the granularity people can explain. Per site, not per fridge.
12. Find the cost that decides the design. It was one ingestion rule. Compute was noise.