connector plugin documentation

Overview#

The HTTP/REST connector reads time-series data from any HTTP or REST API that returns JSON. Use this plugin when your data source exposes an HTTP endpoint that serves paginated, timestamped records and you can express the response shape using JSONPath expressions. The connector handles pagination, authentication (including OAuth2), request chaining (for lookups before the main call), and fan-out (one request per element of a list). It also supports real-time data via polling or Server-Sent Events (SSE).

Type: IETF.Http

Important: Binding paths#

This plugin advertises binding paths verbatim with no instance-id prefix. Unlike every other connector plugin, the BindingPath from each endpoint configuration is advertised to Atlas exactly as written, with no instance identifier prepended.

For example, if your endpoint configuration has:

{
  "BindingPath": "weather/station-a/temperature"
}

Atlas will see exactly weather/station-a/temperature, not my-http-instance/weather/station-a/temperature.

Similarly, template paths are advertised as templates:

{
  "BindingPath": "weather/[station]/temperature"
}

Atlas sees the template weather/[station]/temperature. When Atlas queries weather/denver/temperature, the connector extracts station=denver.

Placeholders are square brackets everywhere in this plugin, in binding paths, request paths, headers and bodies alike: [from], [to], [params.name], [vars.name], [steps.name.field], [chain.name.field] and [fanout.item]. Curly braces are not template syntax. A token written with braces is sent as written, so the request goes out with the braces still in the URL. Anything that merely looks bracketed and does not start with a letter, an underscore or $, such as an IPv6 host [::1] or a JSONPath [0] or [*], is left alone.

If you assume the standard instance-id-prefixed rule, your binding paths will match nothing.

Configuration#

All configuration lives in appsettings.json under a Plugins array entry. There is no file; the type hierarchy is:

• HttpConnectorConfig (top-level)
• AuthConfig, RetryConfig, SubscriptionConfig (nested objects)
• StepConfig, EndpointConfig (lists)
• ChainStepConfig, ColumnMapping, PaginationConfig, SseConfig (nested in endpoints/steps)

Top-level connector options#

Authentication (Auth block)#

The Auth block is optional and shortcuts the most common authentication patterns. When configured, it runs before any user-defined Steps and injects headers or query parameters automatically. For OAuth2 types, a hidden step named __auth is synthesized and its token is available as [steps.__auth.token].

Auth RefreshInterval defaults to 55 minutes for all OAuth2 types. HTTP 401 responses trigger an immediate refresh followed by one automatic retry of the original request.

Retry (Retry block)#

Subscription (Subscription block)#

Instance-level defaults for polling cadence. Per-endpoint configuration overrides these.

Instance steps (Steps array)#

Steps run once on startup and their results are cached for the instance lifetime. Use them for auth token fetches, static lookups, or any value that should be fetched once and reused.

Extracted values are referenced in any template as [steps.NAME.FIELD].

Endpoints (Endpoints array)#

Each endpoint maps to one or more Atlas binding paths and defines how to call the API and map the response.

Columns#

Each column maps a JSONPath expression to a named typed column in the output.

Pagination (Pagination block)#

Automatically loops through paginated API responses and merges all pages into a single rowset.

Chain steps (Chain array within an endpoint)#

Per-request chain steps run on every GetRange call, in order, before the main endpoint call. Use them when a lookup value depends on request parameters.

SSE configuration (Sse block within an endpoint)#

Server-Sent-Events settings for real-time subscriptions. When configured, the connector opens StreamPath and maps each event's data: JSON payload to one Update row. SSE events often use a different JSON shape from the range response (one point per event vs a batch), so this block carries its own TimestampPath and Columns. Any field left null falls back to the endpoint's history mapping.

Connection setup#

1. Choose your target API. Identify the HTTP/REST endpoint that serves your time-series data.

2. Create a connector instance in appsettings.json. Add an entry to the Plugins array with "Type": "IETF.Http" and a unique "Id".

3. Set the BaseUrl. This is prepended to all relative paths in endpoints and steps.

4. Configure authentication (if required by the target API):

   • No auth: leave Auth unset or set Auth.Type to "None".
   • Static Bearer token: set Auth.Type to "Bearer" and pass the token via Auth.Token (or env var Plugins__N__Auth__Token=...).
   • Basic auth: set Auth.Type to "Basic", Auth.Username, and Auth.Password.
   • API key in header: set Auth.Type to "ApiKey", Auth.Header to the header name (e.g. "X-Api-Key"), and Auth.Value to the key.
   • API key in query param: set Auth.Type to "ApiKey", Auth.QueryParam to the param name, and Auth.Value to the key.
   • OAuth2 Client Credentials: set Auth.Type to "OAuth2ClientCredentials", Auth.TokenUrl, Auth.ClientId, Auth.ClientSecret, and optional Auth.Scope. The connector fetches and refreshes tokens automatically.
   • OAuth2 Refresh Token: set Auth.Type to "OAuth2RefreshToken", Auth.TokenUrl, Auth.ClientId, Auth.ClientSecret, and Auth.RefreshToken. The refresh token must be obtained once externally.
   • OAuth2 Password Grant: set Auth.Type to "OAuth2Password", Auth.TokenUrl, Auth.ClientId, Auth.ClientSecret, Auth.Username, Auth.Password, and optional Auth.Scope.
   • Custom multi-step auth (e.g. nonce + HMAC): use Steps with DependsOn to express the sequence, then reference the final token in DefaultHeaders.

5. Add startup steps if needed (optional). Use Steps to fetch auth tokens, resolve static IDs, or load reference tables. Steps run once on startup and are cached. Reference extracted values as [steps.NAME.FIELD].

6. Define endpoints. For each API endpoint:

   • Set BindingPath to the path Atlas will see (without instance-id prefix).
   • Set Method (GET, POST, etc.).
   • Set Path to the relative URL (supports templates).
   • Set TimestampPath to the JSONPath selecting timestamps.
   • Add Columns for each value column you want to extract.
   • Optionally add Chain steps for per-request lookups.
   • Optionally add Pagination if the API returns pages.
   • Optionally add Sse for real-time subscriptions via Server-Sent-Events.

7. Test the binding. Query one of your binding paths from Atlas and check that data flows. If you bind nothing, verify that your BindingPath exactly matches what Atlas expects (remember: no instance-id prefix).

Binding paths#

This plugin advertises binding paths exactly as configured, with no instance-id prefix.

Static paths#

If your endpoint has no parameters:

{
  "BindingPath": "weather/station-a/temperature"
}

Atlas sees exactly weather/station-a/temperature. Use this path to query.

Template paths#

Binding paths can contain [param] placeholders:

{
  "BindingPath": "weather/[station]/temperature",
  "Path": "/stations/[params.station]/readings?from=[from]&to=[to]",
  ...
}

Atlas sees the template weather/[station]/temperature and matches any concrete path of the same shape:

• weather/denver/temperature -> extracts station=denver, calls /stations/denver/readings?from=...&to=...
• weather/seattle/temperature -> extracts station=seattle, calls /stations/seattle/readings?from=...&to=...

The extracted parameter is available as [params.station] throughout the request.

Multiple endpoints, different paths#

You can define multiple endpoints with overlapping shapes:

"Endpoints": [
  {
    "BindingPath": "weather/[station]/temperature",
    "Path": "/stations/[params.station]/readings?from=[from]&to=[to]",
    ...
  },
  {
    "BindingPath": "weather/[station]/pressure",
    "Path": "/stations/[params.station]/pressure?from=[from]&to=[to]",
    ...
  }
]

Atlas sees two templates and routes weather/denver/temperature to the first and weather/denver/pressure to the second.

Polling vs. SSE#

The connector supports two real-time modes for each endpoint:

Polling (default)#

If you don't configure Sse, subscriptions use polling:

• Cadence: re-query every PollInterval (default 10 seconds, or Subscription.PollInterval).
• Change detection: the connector tracks the maximum Timestamp from the last emission (watermark), re-reads a window slightly before that timestamp (to account for out-of-order points), filters rows already seen, and emits only new rows as Update events.
• Snapshot first: each subscription gets a snapshot from the range endpoint before polling starts, then SnapshotComplete, then Update rows as new points arrive.

Use polling when the API exposes a range endpoint (most common).

Server-Sent-Events (SSE)#

If you configure Sse with a StreamPath, subscriptions stream instead:

{
  "BindingPath": "alerts/all",
  "Path": "/alerts/history?from=[from]&to=[to]",
  "Sse": {
    "StreamPath": "/alerts/stream"
  }
}

The connector:

1. Emits an initial snapshot from Path (the history endpoint).
2. Emits SnapshotComplete.
3. Opens StreamPath and consumes data: JSON events, mapping each to one Update row.
4. Reconnects with 2-second backoff if the stream drops while the subscription is active.

SSE event shapes often differ from the history response (one point per event vs a batch). If they differ, configure the event mapping in the Sse block:

{
  "Sse": {
    "StreamPath": "/alerts/stream",
    "TimestampPath": "$.ts",
    "Columns": [
      { "Name": "Status", "Path": "$.status", "Type": "string" }
    ]
  }
}

Any field left null in the Sse block falls back to the endpoint's history mapping.

Disabling real-time#

To make an endpoint pull-only (subscriptions return NotSupported):

{
  "Realtime": "None"
}

Response shape handling#

The connector maps JSON responses to rows and columns using JSONPath.

Example response#

Suppose your API returns:

{
  "rows": [
    { "ts": "2024-05-22T10:00:00Z", "value": 42.5, "quality": "good" },
    { "ts": "2024-05-22T10:05:00Z", "value": 43.1, "quality": "good" }
  ]
}

And your endpoint configuration is:

{
  "TimestampPath": "$.rows[*].ts",
  "TimestampFormat": "Iso8601",
  "Columns": [
    { "Name": "Value", "Path": "$.rows[*].value", "Type": "double" },
    { "Name": "Quality", "Path": "$.rows[*].quality", "Type": "string" }
  ]
}

The connector:

1. Extracts timestamps via -> ["2024-05-22T10:00:00Z", "2024-05-22T10:05:00Z"]
2. Parses each using ISO 8601 format
3. Extracts values via $.rows[*].value -> [42.5, 43.1]
4. Extracts quality strings via $.rows[*].quality -> ["good", "good"]

Result: a 2-row rowset with columns Timestamp (auto-added), Value (double), and Quality (string).

JSONPath syntax#

The connector uses JPath (simplified JSONPath): $ is the response root, .field accesses an object property, [n] accesses an array element by index, [*] projects over all array elements.

Examples:

• $.data[*].value -> all value fields in the data array
• -> the ts field of the first row
• $.readings[*].temperature.celsius -> the celsius field nested in temperature for each reading

The number of elements extracted by TimestampPath must match the number extracted by each Columns[*].Path. If they don't match, rows are padded with nulls to the longer length.

Unit conversion#

Use Scale and Offset to transform numeric columns:

{
  "Name": "TempC",
  "Path": "$.rows[*].temp_k",
  "Type": "double",
  "Scale": 1,
  "Offset": -273.15
}

Formula: output = (parsed_value * Scale) + Offset.

Examples:

• Kelvin to Celsius: Scale: 1, Offset: -273.15
• kPa to Pa: Scale: 1000, Offset: 0
• mph to m/s: Scale: 0.44704, Offset: 0

Authentication#

All authentication patterns are configured via the Auth block or custom steps.

No authentication#

{
  "Auth": { "Type": "None" }
}

Or omit the Auth block entirely.

Static Bearer token#

{
  "Auth": {
    "Type": "Bearer",
    "Token": "eyJhbGci..."
  }
}

To avoid storing secrets in config files, pass via environment variable:

Plugins__0__Auth__Token=eyJhbGci...

Basic authentication#

{
  "Auth": {
    "Type": "Basic",
    "Username": "myuser",
    "Password": "mypass"
  }
}

The connector base64-encodes username:password and injects Authorization: Basic <base64>.

API key in header#

{
  "Auth": {
    "Type": "ApiKey",
    "Header": "X-Api-Key",
    "Value": "my-key-12345"
  }
}

API key in query parameter#

{
  "Auth": {
    "Type": "ApiKey",
    "QueryParam": "apikey",
    "Value": "my-key-12345"
  }
}

OAuth2 Client Credentials#

For service-to-service authentication (no user interaction):

{
  "Auth": {
    "Type": "OAuth2ClientCredentials",
    "TokenUrl": "https://auth.example.com/oauth/token",
    "ClientId": "my-client-id",
    "ClientSecret": "my-secret",
    "Scope": "data:read",
    "RefreshInterval": "00:55:00"
  }
}

The connector automatically:

1. Fetches a token on startup via the client credentials grant.
2. Injects it as Authorization: Bearer {token} on all requests.
3. Refreshes the token before expiry (55 minutes by default).
4. Refreshes immediately if any request returns 401.

The token is available as [steps.__auth.token] in templates if you need it in custom headers.

OAuth2 Refresh Token#

For cases where you have a long-lived refresh token (obtained once via a browser or Postman):

{
  "Auth": {
    "Type": "OAuth2RefreshToken",
    "TokenUrl": "https://auth.example.com/oauth/token",
    "ClientId": "my-client-id",
    "ClientSecret": "my-secret",
    "RefreshToken": "eyJhbGci...",
    "RefreshInterval": "00:55:00"
  }
}

The connector exchanges the refresh token for access tokens and handles renewal automatically.

OAuth2 Password Grant#

For service accounts with a username and password:

{
  "Auth": {
    "Type": "OAuth2Password",
    "TokenUrl": "https://auth.example.com/oauth/token",
    "ClientId": "my-client-id",
    "ClientSecret": "my-secret",
    "Username": "service-user@example.com",
    "Password": "service-password",
    "Scope": "data:read",
    "RefreshInterval": "08:00:00"
  }
}

Custom multi-step authentication#

For APIs that require a nonce, HMAC challenge, or intermediate lookup, use steps with DependsOn to chain them:

{
  "Steps": [
    {
      "Name": "get-nonce",
      "Method": "GET",
      "Path": "/auth/nonce",
      "Extract": { "nonce": "$.nonce" }
    },
    {
      "Name": "exchange-token",
      "Method": "POST",
      "Path": "/auth/token",
      "Body": { 
        "nonce": "[steps.get-nonce.nonce]",
        "client_id": "[vars.clientId]"
      },
      "Extract": { "token": "$.access_token" },
      "DependsOn": ["get-nonce"],
      "RefreshInterval": "00:55:00"
    }
  ],
  "DefaultHeaders": {
    "Authorization": "Bearer [steps.exchange-token.token]"
  }
}

Timestamp formats#

The TimestampFormat controls both how [from] and [to] bounds are formatted in outgoing requests and how timestamps in API responses are parsed.

Example for an API that uses Unix seconds:

{
  "TimestampFormat": "UnixSeconds",
  "Path": "/data?start=[from]&end=[to]",
  "TimestampPath": "$.rows[*].epoch"
}

When Atlas requests the range [2024-05-22 10:00:00 UTC, 2024-05-22 11:00:00 UTC], the connector formats it as start=1716372000&end=1716375600 and sends that in the URL.

Connection status#

The instance's status in Connect says whether it can reach the API, separately from whether it started. Connect checks about every 20 seconds, by running again any startup step that has not yet succeeded, the token fetch among them, within a bounded budget, and the check never writes anything to the API. A blank or malformed base URL, an OAuth2 grant with no token URL, or a server certificate the connector will not trust reads Connection Fault at once, with the reason; no token request is ever sent without a token URL, so credentials never go anywhere else. An API that cannot be reached reads Connecting… for a minute and then Connection Fault. A startup step that fails does not stop the instance: it keeps running, its status says which step failed and why, and the step is retried until it succeeds. See Plugin Instance States, Connection Status and Retries for the status model and the retry ladders.

Troubleshooting#

Issue: Binding paths match nothing#

Symptom: You configure an endpoint with BindingPath: "my-api/temperature", but when you try to bind that path in Atlas, nothing matches.

Cause: This is the binding-path exception. Unlike other connectors, HTTP advertises BindingPath verbatim with no instance-id prefix. If you assume the usual rule and try to bind my-http-instance/my-api/temperature, it won't match my-api/temperature.

Fix: Bind exactly the path as written in BindingPath, with no prefix.

Issue: Columns are empty (no values extracted)#

Symptom: Data arrives but all columns show nulls.

Cause: JSONPath mismatch. The Path expression in a column definition didn't match any elements in the response, or the array length didn't match the timestamp array.

Fix: Verify your JSONPath expression against the actual response. Use a JSONPath tester (online or in your API client) to confirm the path selects the correct array. Ensure the number of elements matches TimestampPath.

Issue: Timestamps are wrong or fail to parse#

Symptom: Timestamp parsing error in logs, or timestamps appear in the wrong year.

Cause: Format mismatch between TimestampFormat and the actual format in the API response, or date parsing ambiguity.

Fix: Check the response and match the format exactly. If the response contains ISO 8601 strings like 2024-05-22T10:00:00Z, use Iso8601. If it contains Unix epoch seconds, use UnixSeconds. If you need a custom format, set TimestampFormat to Custom and provide TimestampCustomFormat with the format string.

Issue: 401 Unauthorized#

Symptom: All requests return HTTP 401.

Cause: Authentication not configured, misconfigured, or token expired.

Fix: Verify your Auth block matches the target API's requirements. For OAuth2, verify TokenUrl, ClientId, and ClientSecret; an OAuth2 grant with no TokenUrl reads Connection Fault with the reason "OAuth2 needs a token URL" and sends nothing. If using a refresh token or password grant, ensure the credentials are correct. Check that the token is not expired. The connector automatically refreshes on 401, but if the refresh itself fails, check the logs for details.

Issue: Pagination runs forever or hits the MaxPages limit#

Symptom: Pagination loops without terminating, or you hit the MaxPages safety cap (default 100).

Cause: The API pagination is not correctly configured, or NextCursorPath (for cursor pagination) is not extracting a cursor, or the API never signals end-of-results.

Fix: Check that your pagination type (OffsetLimit, PageNumber, Cursor) matches the API's pagination scheme. For cursor pagination, verify NextCursorPath correctly extracts the cursor from the response. If the API stops returning new rows before hitting MaxPages, that's normal and not an error. If it truly loops, increase MaxPages (at risk of long queries) or fix the API integration.

Issue: Realtime subscriptions don't emit Update rows#

Symptom: You see the snapshot and SnapshotComplete, but no Update rows arrive even though data is changing.

Cause: Polling is configured but timestamps are duplicated, out-of-order, or the query window is too narrow.

Fix: Check that your API returns unique, monotonically increasing timestamps. If timestamps are duplicated or out-of-order, the watermark-based change detection will skip them. Increase Subscription.AllowedLatency to expand the re-read window and account for out-of-order arrival. Check that the poll interval (Subscription.PollInterval or per-endpoint PollInterval) is frequent enough for your use case.

Issue: SSE stream connects but never emits events#

Symptom: The connector opens the SSE stream (no error), emits the snapshot, then no Update rows arrive.

Cause: The stream is open but the API is not sending events, or the event shape doesn't match the configured mapping.

Fix: Verify the SSE endpoint is actually pushing events (test it manually with curl or a client). Check that StreamPath is correct and any template variables (like [params.sensor]) are being substituted. Verify that the event's TimestampPath and Columns mapping match the actual event JSON shape. If the event shape differs from the history response, configure the Sse block with its own mapping.

Issue: Scale and Offset not applied#

Symptom: Numeric values are not transformed by Scale and Offset.

Cause: You configured a non-numeric column type (e.g. Type: "string"), or the column type is not double or float (only these types support Scale/Offset).

Fix: Ensure the column Type is double or float. Scale and Offset are ignored for other types.

Notes#

• No Quality column: This plugin does not emit a Quality column. Unlike the Historian, OPC UA, i3X, and some other data sources, HTTP has no standard way to signal data quality. If you need quality data, embed it in the JSON response and extract it as a regular column.

• No config hot-reload: Endpoint definitions are static. If you change them in appsettings.json, restart the instance.

• Array injection in POST bodies: If a POST body value is a single template expression (e.g. "[steps.list-tags.ids]") and the referenced value is a JSON array, the array is injected as proper JSON, not as a string. This is used for batch calls where an entire array should be part of the request body.