Late Data Handling

Storage and the .late file#

When a data point arrives with a timestamp that falls inside a completed hourly block, the Historian cannot write it directly into that block - the block may already be in use by a reader. Instead, the Historian buffers the late point in a temporary .late file alongside the corresponding .data block. During the next maintenance cycle (approximately every minute), the Historian merges all pending .late files back into their .data blocks and deletes the .late files. After merging, the data is available for queries.

You can monitor late data arrival using the Dataset.Writes.Late system tag, which reports the rate of late data ingestion in points per second.

Late Data Tolerance#

The Late Data Tolerance setting defines a clock-skew window in milliseconds. Its purpose is to distinguish genuine late data (a backfill) from clock-skew noise (a device whose clock is slightly off and repeatedly sends timestamps a few milliseconds in the past).

Example: With Late Data Tolerance set to 100 ms, a point arriving 50 ms before the most recent stored timestamp is discarded as clock noise. A point arriving 5 seconds before the most recent stored timestamp is accepted as a genuine backfill point.

Late Data Age#

The Late Data Age setting bounds the window in days. A late point beyond the tolerance window is stored only while it is no more than lda days behind the newest stored point:

Importing history older than 30 days (a migration from another historian, a long backfill) requires raising lda first - for example to 400 to import roughly a year of history - and it can be lowered again once the import completes.

How to configure#

In the Historian browser (Dataset Editor)#

1. Open the Historian at https://<historian>:4512.
2. Choose Datasets in the sidebar.
3. Select the edit pencil at the right of the dataset's row.
4. Enter the tolerance in Late Data Tolerance (ms) and the window in Late Data Age (days). The two purge limits sit above them.
5. Select Save.

Expand a dataset's row on the Datasets page and its current Late Data Tolerance and Late Data Age are shown beside its purge limits, with its late-write rate among the live status cards below.

Via the REST API#

Set ldt (Late Data Tolerance, milliseconds) and/or lda (Late Data Age, days) when creating or updating a dataset. The update body is a partial update: only the fields you send are changed, so adjusting the tolerance cannot disturb the late-data window.

POST /api/datasets/{datasetId}
Content-Type: application/json

{
  "ldt": 100
}

POST /api/datasets/{datasetId}
Content-Type: application/json

{
  "lda": 400
}

Both fields are returned in GET /api/datasets and GET /api/datasets/{datasetId} responses, and every data write's response body reports how many points were accepted and how many were dropped per reason (see the Historian REST API).

Choosing a value#

Troubleshooting#