Running the Data Engine

What it is, and is not#

The Data Engine is not a Windows service. It is a service object defined in the model and hosted by the Flow Bootstrap service on a Platform.

So there is nothing called "Flow Data Engine" in the Services list. To start or stop an engine you deploy or undeploy it; to have it run at all, Flow Bootstrap must be running on the machine whose host name matches its Platform. See Infohub Architecture at a Glance.

What happens each pass#

On each pass the engine, broadly:

1. Works out which measures need values for which periods.
2. Asks the relevant data sources for the raw data.
3. Stores retrieved values.
4. Aggregates upward through the interval chains that depend on what changed.
5. Evaluates calculations whose dependencies changed.
6. Records how far it has progressed.

Two things follow from that shape.

Work is dependency-driven, not scheduled per measure. You do not schedule a measure. You declare its dependencies, and the engine derives what needs recomputing when something changes.

Corrections propagate. A late-arriving or corrected value causes everything above it to recalculate for the affected periods. This is why backfilling works, and why a large backfill is expensive.

Work is partitioned across engines#

An engine owns metrics, time schemes, and event schemes. That is how a large model is divided: run more than one engine and assign different parts of the model to each.

Reasons to do it:

• Isolate a slow source. A historian that answers slowly holds up whatever engine is waiting on it. Giving it its own engine keeps the rest of the model current.
• Spread across machines. Engines on different Platforms run on different servers.
• Separate priorities. Operational hourly measures on one engine, heavy monthly reporting on another.

Most single-site models run one engine and never need more. Reach for partitioning when you can point at a specific contention problem, not speculatively — several engines are more to operate and monitor.

Engine properties#

Engines carry a property bag. The ones that matter operationally:

Poll trades latency against load. A short poll makes values appear sooner and means the engine wakes more often to find nothing to do. For hourly reporting, a poll measured in minutes is ample.

MaxConcurrentMeasures is the main throughput control. Raising it increases parallelism up to the point where the source or the database becomes the bottleneck, after which it makes things worse. See Engine Performance and Tuning.

TimeZoneID deserves care: it determines how period boundaries land in local terms. Getting it wrong shifts every period by the offset, which shows up as daily totals that disagree with the plant's own by a few hours' worth of production.

Diagnostic is for investigation, not steady state. It produces a great deal of output; turn it on to diagnose something and off again afterwards.

Starting and stopping#

An engine starts when it is deployed and Flow Bootstrap on its Platform is running. It stops when undeployed.

Undeploy rather than delete to stop an engine temporarily — a source system outage, a maintenance window. Configuration is preserved and redeploying restores it exactly.

Note that deployment restarts the engine, so there is a brief gap in processing. Deploy substantial changes in a maintenance window.

Confirming it is working#

In order, cheapest first:

1. Is Flow Bootstrap running on the Platform host?
2. Does the Platform's host name match that machine? A mismatch means nothing runs, silently.
3. Is the engine deployed, and showing Deployed rather than Different or Faulted?
4. Are new values appearing for a measure you expect? Look at the newest interval, not today's total.
5. What does the Flow event log say?

Steps 2 and 3 account for the large majority of cases where an engine appears to be doing nothing.

Catch-up and initial load#

When an engine starts against a model with history to build — a new instance, a backfilled scheme, a long outage — it has to work through the backlog.

Expect this to take time proportional to the work, and expect current data to be slower to appear while it happens. Two practical notes:

Do not conclude the engine is broken because it is behind. Check whether values are advancing at all. An engine making progress needs time; an engine making none needs investigation.

Bound large backfills. Backfilling a year at minutely resolution across a large model is a very large amount of work. Do it in ranges, and outside peak hours.

Time and the freeze point#

Two settings shape which periods the engine will touch.

The freeze point on a time scheme marks everything before it as closed. The engine will not rework frozen periods, which is what stops it re-examining years of history on every pass — and what causes late-arriving values before the freeze point to be discarded. If old periods stop updating and the configuration looks right, check the freeze point.

An event scheme's refresh offset determines how far back it re-examines source data, which is how late-arriving events are captured. Too small and late data is never seen. See Designing an Event Scheme.

Routine operation#

Things worth doing regularly rather than in response to a problem:

• Check deployment status. Different means a saved change is not in effect; Faulted means something is failing. Both are quiet.
• Watch the Flow event log for recurring errors. A source failing every pass is worth fixing even if the numbers look adequate.
• Confirm the newest period is current for a sample of measures across the model, not just one.
• Watch database growth. See Database Maintenance.

The Config CLI has a --deployment-status recipe that makes the first of these a scheduled check rather than a habit.

Diagnosing#

See also#

• Monitoring Infohub and Its Health
• Engine Performance and Tuning
• The Maintenance Window
• Backfilling and Recalculation