Backfilling and Recalculation
What backfill actually does, when it is the right tool, and the behaviours that surprise people — evaluation before the date, manual calendars, and events.
Backfill is the most-used and least-documented operation in Infohub. It is the right tool for some jobs, the wrong tool for others, and it has behaviours that look like bugs until you know the model underneath.
What backfill does#
Setting a backfill date on a measure (or an event scheme) tells the engine: re-derive everything from this date forward. Retrieved measures re-query their source for the range; aggregations and calculations recompute; the changes then propagate up the dependency chain exactly as a live change would.
Backfill is stored per measure–time-scheme (and per event scheme), alongside the refresh configuration. It is a request, serviced by the engine on its normal passes — not an instantaneous action.
When it is the right tool#
- The source has data Infohub missed — an outage, a late-loaded historian, a repointed tag
- Configuration was fixed — an unguarded division, a wrong aggregation type, a corrected expression — and history must be reworked under the new rules
- An event scheme must rebuild events from source history
- A consumer must resend a range (backfill the measures; the consumer resends what changed)
When it is the wrong tool#
- The measure is stuck. Backfill often appears to fix a stuck measure because it forces a fresh evaluation — but if the underlying cause remains (a missing dependency, a retry loop, an unassigned engine), it sticks again. Diagnose first: Diagnosing Stuck or Faulted Measures.
- Manual data is in the range. Recalculation regenerates derived values; it must never be pointed at manually entered measures with the expectation that they regenerate — they cannot.
- Routinely, on a schedule. A site that needs scheduled backfills has a timing problem (source latency vs refresh offset) that should be fixed at the refresh configuration, not papered over.
The behaviours that surprise people#
Calculations can evaluate before your backfill date. A calculation's window can legitimately reach earlier than the backfill date — a rolling window [-7, 0], a previous-period reference [-1], or a dependency whose own chain extends back. The engine evaluates what the expression demands. If the earlier range has no data, guard the expression rather than fighting the backfill date.
Backfill against a manual calendar fails on missing periods. The error time period containing Utc '…' does not exist means the backfill date falls before the first period of a manual time scheme. The engine cannot invent periods for a calendar it does not generate. Move the backfill date inside the calendar's range, or extend the calendar first.
A backfill request can appear to "reset". For event schemes especially: you set the backfill forward, the engine picks it up, evaluates, and resumes from its own processing position — which can be earlier than your date if that is where its dependencies stand. The scheme processes from where its inputs allow, not from where you would like.
Duplicate-key noise during heavy backfill. Bulk inserts during a large backfill can collide with rows that already exist and retry; the work completes but does double duty. Prefer bounded ranges over "backfill everything since 2019".
Quality in windows. A moving-window calculation recomputed by backfill takes the quality of its inputs. If the leading days of the window predate good data, those outputs carry degraded quality — correct, if surprising on a chart that colours by quality.
Licence expiry does not block backfill; it blocks processing. If nothing processes at all, the licence has lapsed — a backfill request will sit unserviced. See Licence Keys in Practice.
Doing a large backfill well#
- Bound the range. A year of minutely data across a large model is millions of evaluations, competing with live collection.
- Do it off-peak, and expect current data to lag while it runs.
- Warn downstream. Every consumer resends the restated range; a metered destination or a central replication target will feel it.
- Verify a sample at the bottom of the chain first — if the retrieved layer is wrong, everything above it recomputed wrong values efficiently.
- Watch progress in the Monitor, not by refreshing a chart. See Monitoring Infohub and Its Health.
Backfill vs recalculate vs refresh#
| You want | Use |
|---|---|
| Rework history from the source | Backfill on the retrieved measure |
| Rework derived history after a config fix | Backfill/recalculate from the fixed measure up |
| Catch late-arriving recent data automatically | Refresh offset / refresh period — standing configuration |
| Re-detect events from history | Backfill on the event scheme |
| Push history to a consumer | Backfill the measures; integration resends changes |
The standing-configuration point matters: refresh settings (RefreshOffset, RefreshPeriod) are how the engine re-examines the recent past every pass. Backfill is the one-off reach into deeper history. Sites that confuse the two either backfill daily (wrong tool) or set a refresh offset of months (expensive every single pass).