Building an Alarm Strategy
Why most notification setups fail, and how to design one people still read after six months.
Most Infohub notification setups fail, and they fail the same way: too many messages, so people stop reading them, and the one that mattered arrives in a folder nobody opens. The technical configuration is straightforward — this article is about the part that decides whether it works.
The governing principle#
Every notification must have a person who will do something specific when they receive it.
If nobody would act, it is not a notification. It might be a dashboard indicator, a line in a report, or nothing at all — but sending it costs the attention that real alarms depend on.
Applying that one test removes most of what gets configured in a first enthusiastic pass.
Alarm fatigue is the failure mode#
It develops predictably.
A plant configures notifications for everything measurable, because it is easy and it feels thorough. Volume is high from day one. Within a week recipients are skimming; within a month they have a mail rule filing Infohub messages into a folder; within a quarter nobody reads them. The system is still running, still sending, and delivering no value.
The insidious part is that nothing looks broken. Messages are being generated and delivered successfully. Only the outcome has failed.
You cannot fix fatigue by adding urgency. Once recipients are filtering, marking messages high-priority, adding "URGENT" to subjects, or moving to SMS does not restore attention — it briefly borrows it and then trains people to ignore the new channel too. The only fix is reducing volume to the point where every message is worth reading.
Design for volume from the start#
Set a budget before configuring anything. Something like:
| Channel | Reasonable volume |
|---|---|
| SMS or push | A handful per week, per person |
| Chat channel | A few per shift |
| Email, individual | A few per day |
| Email, scheduled reports | One per day or per week |
Then configure within it. If the design implies fifty emails a day, the design is wrong regardless of how correct each individual notification is.
Tiers rather than one flat level#
Not everything deserves the same channel.
Tier 1 — act now. A condition where minutes matter and someone must intervene. SMS or push, to the person on duty. Should be a very short list — a handful of conditions per plant.
Tier 2 — act this shift. Something needs attention but not immediately. A chat channel the team already has open, so the group sees it and someone can claim it.
Tier 3 — awareness. Something worth knowing. An email, or better, a dashboard indicator with no message at all.
Tier 4 — the record. Scheduled summaries. Email, on a period boundary.
Most conditions belong in Tier 3 or 4, and most of Tier 3 is better served by colour on a dashboard than by a message. That is the reduction that makes the whole thing work.
Use the right trigger#
Matching the trigger to the intent avoids a lot of noise. See The Message Engine.
| You want | Trigger |
|---|---|
| To know a value went out of range | Measure Limit Exceeded |
| A shift or daily summary | Time Period Ended |
| Targets at the start of a shift | Time Period Started |
| To know how a batch or stoppage turned out | Event Period Ended |
Event Period Ended is the one that reduces volume most, and it is under-used. "Tell me when a batch finishes and how it went" is one message per batch — a naturally paced, genuinely informative stream. Built as limits on measures instead, the same information arrives as a scatter of threshold breaches at arbitrary moments.
Limits must be set where action begins#
A notification on Measure Limit Exceeded is only as good as the limit.
A limit inside the normal operating range fires constantly and is the single largest source of notification volume. Chart the measure over a few months and set the threshold outside where it normally runs.
A limit at the specification boundary fires too late to prevent anything. A warning band before the boundary gives someone the chance to act.
Consider whether the condition needs to persist. A momentary spike crossing a threshold and immediately returning is usually not worth waking anyone for. Where the underlying data is event-driven, a hold time on the event trigger filters this at source.
Name a recipient, not a group#
"The maintenance team" is not a recipient. When a message goes to everyone, it is owned by nobody, and the observed behaviour is that everyone assumes someone else is handling it.
Better patterns:
- A role-based address with a defined on-duty holder, so the responsibility is unambiguous even as people change.
- A chat channel with a claiming convention — the first person to respond says so, and the thread records who has it.
- An individual for Tier 1 conditions, with a defined escalation if they do not respond.
Also: send to roles rather than named individuals wherever possible, because distribution lists of named people decay. Within a year, some recipients have left and some who should receive it do not.
Escalation#
For Tier 1 conditions, decide in advance what happens if nobody acts. An alarm nobody answers is the situation the strategy exists to prevent.
Escalation needs a defined path — who is contacted next, after how long — and it needs to be agreed with the people on it rather than configured unilaterally. It is an organisational agreement that Infohub implements, not the other way round.
Use redundant channels for these conditions. A single channel has a single point of failure: an expired webhook, an SMTP relay down, a phone with no signal.
Review it#
An alarm strategy is not a configuration task that completes. Schedule a review — quarterly is realistic — and look at:
Volume per recipient. Is anyone receiving more than the budget?
Which notifications fire most. The top few are candidates for a limit change or removal. A notification firing daily is telling you about a chronic condition, which is a process problem to fix rather than a message to keep sending.
Which have never fired. Either the threshold is wrong, or the condition genuinely does not occur and the notification can go.
Whether anyone acts. Ask the recipients directly. The answers are usually informative and sometimes uncomfortable.
Whether the recipients are still right. Roles change.
The most valuable review question. Ask recipients: "which of these do you actually read?" Whatever they name, keep. Everything else is a candidate for deletion. This one question typically halves the volume and improves the outcome, because the messages that survive get read.
Getting started#
For a plant with no notifications configured, a sound first pass:
- One scheduled shift summary on Time Period Ended, to the shift team's channel. Useful, low volume, and it establishes the habit of reading Infohub output.
- Two or three genuine Tier 1 alarms, thresholds set from real operating data, to the person on duty.
- Everything else on the dashboard, in colour, with no message.
- Wait a month. Then ask what people wish they had been told, and add that.
This inverts the usual approach, which is to configure everything and prune later. Pruning rarely happens, and by the time anyone tries, the recipients have already stopped reading.
Diagnosing a failing strategy#
| Symptom | Cause |
|---|---|
| Recipients have mail rules filing Infohub messages | Volume — reduce it, do not escalate channels |
| An important condition was missed | It was in the noise, or nobody owned it |
| Nobody responded to an alarm | No named recipient, or no escalation path |
| One notification dominates the volume | Limit inside the normal range, or a chronic process problem |
| Notifications nobody can explain | Configured by someone who has left, for a reason nobody recorded |
| SMS bill is unexpectedly high | A misconfigured limit firing repeatedly |
| Alerts stopped entirely | Deployment or credentials — see Notification Services |