Connecting Data Sources

Overview#

Data reaches an Atlas domain through the Timebase Connect. Connect is a separate service that sits between your data sources and Atlas. It connects to Atlas over a long-lived network connection, tells Atlas which data points it can serve, and then pushes values in as they change.

This article takes you from "I have a data source" to "values are arriving in my instances". It covers the plugins that ship today, where the configuration files live, how to point Connect at Atlas, how to add a source, and how to check that data is actually flowing.

Two terms used throughout:

• Source - one configured data source, for example one Historian server or one SQL database. The Connect interface calls these Sources; the configuration file and the API call them connectors. They are the same thing.
• Binding path - the slash-delimited text that identifies a single data point, for example historian-juice-factory/The Juice Factory/131-TT-01.PV. You type this into a property's Bound strategy in the Instance Map. See Property Sourcing Strategies for the Atlas side of that.

Data sources you can connect#

Twelve plugins ship with the Connector. Each is named Timebase.Connect.{Company}.{Solution}, and the Type column below is the exact string you select in the interface or type into the configuration file.

Default ports#

Atlas listens on 4551 and 4552. The Connect hosts its own interface on 4561 and 4562.

Where the configuration lives#

Connect reads three configuration files. On a normal Windows install they are under C:\ProgramData\Flow Software\Timebase\Connector\.

In Docker the directories are set by environment variables on the image: Settings=/settings, Config=/config, Plugins=/plugins, Logs=/logs and Certificates=/certificates.

Step 1: Point Connect at Atlas#

1. Open atlas.config in the Config folder.

2. Set the address. It is a host, a port and a TLS flag rather than a URL:

   jsonCopy

   {
     "atlases": {
       "atl_7c3e18": {
         "name": "Primary",
         "alias": "primary",
         "description": "",
         "config": {
           "host": "<YourAtlasServerHere>",
           "port": 4552,
           "useTls": true,
           "reconnectDelaySeconds": 5
         }
       }
     }
   }
   

3. The key is a surrogate, not a name. atl_7c3e18 is minted by the host and never changes. name is a label you can edit freely; alias is the handle a source uses to name this target (see Multiple Atlas targets). Renaming a target rewrites one line and breaks nothing.

4. Save the file and restart the Connect service.

You can also do this from the Connect interface at https://<YourConnectorHostHere>:4562 on the Targets page, which writes the same file for you. Each target row shows its endpoint, whether it is Connected, which sources advertise to it and whose identity Connect signs in with.

The address per deployment:

Atlas listens for HTTPS on 4552 and plain HTTP on 4551. Change port and useTls together: TLS on 4551 cannot connect.

Leaving the address out gives you localhost on 4552 over TLS, which is the same as the shipped file. That is deliberate: absence and the shipped default mean the same thing, so a missing address fails against a plausible address rather than a stale one.

HubUrl is no longer something you set. Connect composes it from the three values above and reports it back on its status API and in its log, so you will still see it in a connection message. Writing a HubUrl into atlas.config configures nothing.

In Docker, the container name is the hostname. Using localhost inside the Connect container points at Connect itself, not at Atlas.

When Atlas is served by a different Pulse#

Each target carries its own optional auth block, and you only need it when that Atlas trusts a different Pulse from the one this Connect uses:

"config": {
  "host": "<YourAtlasServerHere>",
  "port": 4552,
  "useTls": true,
  "auth": {
    "enabled": true,
    "clientId": "Connect",
    "clientSecret": "<the secret from THAT Pulse>",
    "idP": { "host": "<that Atlas's Pulse>", "port": 4542, "useTls": true }
  }
}

Left off - which is the shipped state - Connect authenticates to the target exactly as it authenticates to everything else, using its own Pulse:Auth identity. That covers every deployment where one Pulse serves the whole installation, so most sites never touch this.

Switch it on and Connect presents the client named here instead, obtaining its token from the Pulse named in idP. That is the only way to reach an Atlas belonging to a different Pulse: a token from your own Pulse is signed by the wrong issuer, and Atlas's hub refuses it.

This idP is not the one that was removed. The old component-wide Auth:IdP was a second copy of the Pulse address and is gone. This one is per target, and its whole purpose is to name a different Pulse from the one in Pulse:Host. If they would be the same, leave the block off.

Step 2: Give Connect its Pulse credentials#

Complete this step before you look for a network problem. Atlas gates its connector hub, so a connector without a usable credential is refused with 401 Unauthorized on every attempt and retries. The retry log reads the same as an unreachable host, so check the credential first when a connector will not register.

Both products ship with "Auth": { "Enabled": true }, so the work is registering Connect in Pulse and telling it what to send.

1. In Pulse, create a client registration for Connect. Note its secret.

2. Add Atlas's client id (Atlas by default) to that client's audiences. Without it, Connect receives a valid token that Atlas rejects, which produces the same 401 and is the harder half of this step to diagnose.

3. Open Connect's settings.config and fill in Pulse:Auth:ClientId and Pulse:Auth:ClientSecret from that registration.

4. Confirm Pulse:Host points at your Pulse server. The shipped values are host localhost, port 4542, TLS on. In Docker the first start composes the host as pulse into settings.config, and a Pulse__Host environment variable in the Compose file outranks the file.

   jsonCopy

   "Pulse": {
     "Host": "<YourPulseServerHere>",
     "Port": 4542,
     "UseTls": true,
     "Auth": {
       "Enabled": true,
       "ClientId": "Connect",
       "ClientSecret": "<YourClientSecretHere>"
     }
   }
   

5. Restart Connect.

For a local trial without Pulse#

Set Pulse:Auth:Enabled to false in Atlas's settings.config and in Connect's. The two settings are read independently - each product reads its own - so they have to be changed together.

Step 3: Add a data source#

Connect has its own interface with a schema-driven editor and a connection test. Use it in preference to hand-editing JSON.

1. Open https://<YourConnectorHostHere>:4562 in a browser.
2. Select Sources in the sidebar. The page lists every source grouped by plugin type, with its alias, its status and the targets it advertises to, and the header counts how many are running.
3. Select Add Source and choose the plugin from the list (see Data sources you can connect). The list scrolls. A New source drawer opens with three tabs: General, Settings and Targets.
4. On General, give the source a Name. Pick something short and stable, for example historian-main or erp: Connect derives the source's alias from it, and the alias is the first segment of every binding path this source publishes.
5. On Settings, fill in the connection fields for the plugin. The form is generated from the plugin's own schema and starts on its defaults. Each plugin's fields are described in Plugin settings below, and the topics, datasets or queries the source should publish are entries on the same tab.
6. Select Test. This probes the connection to the source and reports success or the exact error. It does not read any data.
7. Select Create. The source is created live, stopped. Select Start on its row to switch it on. No restart is needed.

Nothing exists until you select Create. The whole record, name, description and every plugin setting, is written once. Abandoning the drawer leaves nothing behind.

The name matters more than most names do. You cannot set the alias on the create form, because Connect derives it from the name you give here, and once derived it is what Atlas binding paths are written against. Changing it later is possible in the source's Edit drawer, but it stops every hand-authored path that named the old spelling from resolving, so Connect asks you to confirm. Get it right at creation and the question never comes up.

Id and Enabled are not on the create form. The id is minted for you, and a new source starts out disabled. Both are on the General tab of the Edit drawer once the source exists, and the row's Start button flips the same switch.

Once created, the source has a page of its own: select its name on the Sources page. The page shows its status, the targets it advertises to with the time of the last handshake, the routes it advertises, and the topics it publishes. Stop and Start switch it off and on, Logs opens a log tail scoped to this source, and Edit opens the same three-tab drawer, now with Alias, Id and Enabled on the General tab.

The same thing in connector.config#

Every source you add is stored as one record in the connectors map of connector.config, keyed by a minted id. The file has a version envelope at the top that must be preserved:

{
  "version": 1,
  "schemas": {},
  "connectors": {
    "con_a1c4f7": {
      "alias": "historian-main",
      "name": "historian-main",
      "type": "Timebase.Historian",
      "enabled": true,
      "config": {
        "Host": "<YourHistorianServerHere>",
        "Port": 4512,
        "UseTls": true,
        "ClientName": "Timebase.Connect.Historian",
        "Datasets": {
          "The Juice Factory/FillerTotals": {
            "Dataset": "The Juice Factory",
            "Filter": "uom[1]:\"ft³\" Fields[\"Instrument Code\"]:\"FQ\""
          }
        },
        "DefaultLookback": "01:00:00"
      }
    }
  }
}

Two levels, and the casing differs between them. The record's own fields are lowerCamel and belong to Connect. Everything under config belongs to the plugin and keeps the plugin's own casing, which is PascalCase for every plugin that ships.

Overriding one setting from the environment names the key and the nesting, for example connectors__con_a1c4f7__config__ConnectionString. Look the id up in this file. It is rename-proof, which a position in an array never was.

A file still holding a Connectors[] array predates this shape. Delete it and let Connect write a fresh one, or convert it by hand.

Plugin settings#

Only the fields you are most likely to change are listed. Every plugin also exposes its full field set in the Sources editor with inline descriptions.

Timebase Historian#

Reads tags from a Timebase Historian and republishes them.

Datasets is a flat dictionary. Each key is a topic name you choose. Each value has two fields:

"Datasets": {
  "The Juice Factory/FillerTemps": {
    "Dataset": "The Juice Factory",
    "Filter": "uom[1]:\"°F\" Fields[\"Instrument Code\"]:\"TT\""
  }
}

Slashes in the key are cosmetic. They give the binding path a hierarchical look, nothing more. The Dataset value is what the Historian is actually queried with.

Authenticating to a secured Historian

The source's Auth.Mode says whose identity Connect presents to the Historian, the same way every link between two Timebase components does:

A Custom client block names everything itself; the identity provider has no default host or port:

"Auth": {
  "Mode": "Custom",
  "ClientId": "<YourClientIDHere>",
  "ClientSecret": "<YourClientSecretHere>",
  "IdP": { "Host": "<YourPulseServerHere>", "Port": 4542, "UseTls": true }
}

Connect fetches a token from that Pulse and attaches it to every Historian request. A Custom block missing its client, secret or identity provider host reads Connection Fault at once, with the reason naming what is missing; so does This host on a Connect whose own Pulse sign-in is switched off. A source saved on an earlier Beta build with "Enabled": true is converted on load: to Custom, or to Host when the client it named was Connect's own.

SQL databases#

Four plugins share one identical settings shape: Microsoft.SqlServer, PGDG.PostgreSql, Oracle.MySql and Oracle.Database. Only the connection string format and one SQL rule differ between them.

"con_3e91b2": {
  "alias": "erp",
  "name": "erp",
  "type": "Microsoft.SqlServer",
  "enabled": true,
  "config": {
    "ConnectionString": "Server=<YourSqlServerHere>;Database=ERP;Trusted_Connection=True;TrustServerCertificate=True;",
    "CommandTimeout": "00:00:30",
    "Subscription": { "PollInterval": "00:00:30" },
    "Queries": {
      "Customers": {
        "Sql": "SELECT * FROM Customers"
      },
      "WorkOrders/[CustomerId]": {
        "Sql": "SELECT id, created, status FROM WorkOrders WHERE CustomerId = @CustomerId",
        "Parameters": { "CustomerId": "int" }
      },
      "Setpoints": {
        "Sql": "SELECT TargetTempC AS [Value], ModifiedAt AS [Timestamp] FROM FillerSetpoints",
        "Subscription": { "WatermarkColumn": "Timestamp" }
      }
    }
  }
}

Query fields:

Rules your SQL must follow. Your statement is wrapped as WITH __src AS (<your SQL>) SELECT * FROM __src ... so that Atlas can push a time filter into the database. That wrapping means:

• On every database: no trailing semicolon, and the statement must not start with WITH.
• On SQL Server only: no top-level ORDER BY, no OPTION (...), no FOR XML or FOR JSON, and no EXEC.
• On PostgreSQL, MySQL and Oracle only: no CALL.

Live polling. SQL sources do not receive change notifications; they re-run the query on a timer.

OPC UA#

Browses an OPC UA server's address space for Variable nodes and republishes them. History comes from HistoryRead, live values from monitored-item subscriptions.

There are also five certificate-store paths (ApplicationSubjectName, ApplicationStorePath, TrustedIssuerStorePath, TrustedPeerStorePath, RejectedStorePath) for a secured connection.

"con_5b20d9": {
  "alias": "opcua-plant",
  "name": "opcua-plant",
  "type": "OPCFoundation.OpcUa",
  "enabled": true,
  "config": {
    "Url": "opc.tcp://<YourOpcUaServerHere>:4840",
    "SecurityPolicy": "None",
    "SecurityMode": "None",
    "AutoAcceptUntrusted": true,
    "BrowseDepth": 8,
    "DefaultLookback": "01:00:00"
  }
}

i3X (CESMII)#

Reads an i3X v1.0 server over REST and its event-stream channel. A Timebase Historian exposes an i3X endpoint, so this is one way to connect Historian to Historian across a boundary.

Expansion matters when the server publishes composite objects:

HTTP / REST#

Reads JSON from an HTTP API and maps it into rows using JSONPath expressions. Live data comes from re-polling the endpoint or from a Server-Sent Events stream.

Instance-level fields:

Endpoint fields:

"con_9d41ae": {
  "alias": "http-api",
  "name": "http-api",
  "type": "IETF.Http",
  "enabled": true,
  "config": {
    "BaseUrl": "https://<YourApiHostHere>",
    "Subscription": { "PollInterval": "00:00:10" },
    "Endpoints": [
      {
        "BindingPath": "http-api/readings",
        "Path": "api/readings?from=[from]&to=[to]",
        "TimestampPath": "$.data[*].ts",
        "TimestampFormat": "Iso8601",
        "Columns": [
          { "Name": "Temperature", "Path": "$.data[*].temperature", "Type": "double" },
          { "Name": "Pressure",    "Path": "$.data[*].pressure",    "Type": "double" }
        ]
      }
    ]
  }
}

AVEVA Historian#

Reads an AVEVA Historian (formerly Wonderware Historian) through its Runtime SQL Server database. Authentication is SQL Server authentication only.

There is no raw connection-string field. Connect assembles it from the fields above.

AVEVA PI System#

Reads an AVEVA PI System over PI Web API.

Canary Historian#

Reads a Canary Labs Historian over its Views gRPC service.

InfoHub#

Reads Flow Software InfoHub through its v1 REST API: Tag history, Measure values per calendar or event period, and event occurrences. Read-only and polled. See Flow Software InfoHub Connector Plugin.

References are identified by id, never by name, because InfoHub allows any node to be renamed at any time.

Reference#

A built-in synthetic source with no external dependency. Use it to prove that configuration, advertisement and binding work before you connect anything real. It publishes a fixed set of paths regardless of its settings:

fake/sine/double
fake/text/string
fake/onoff/bool
fake/counter/long
fake/temp/float
fake/index/int

This source serves history rather than live values. A property bound to a fake/... path reads no value in the live cell - it reads as null - and returns generated values for a range query.

Use it to prove the plumbing: Connect starts, advertises its paths, Atlas matches your bindingPath against them, the domain compiles, and a historical read returns rows. To check the live path end to end, use a source that serves live values.

Binding paths#

A binding path is the text Atlas uses to route a request to a single data point. When you set a property to the Bound strategy in an Instance Map, the bindingPath you enter must exactly match a path Connect advertises.

Three rules:

• A binding path is slash-delimited segments, and the first segment is the source's alias. The minted con_ id is accepted in the same position, but the alias is what hand-authored paths use, and what stays readable in a git history.
• It is not a URI. Never write a scheme:// prefix such as historian://, opc:// or sql://. A path written that way is accepted, compiles without complaint, and then produces no value for as long as it stays in the file.
• Paths are case-sensitive. historian-main/Plant Data/Tank01.Level will not match historian-main/plant data/tank01.level.

Segments may contain spaces and dots. The Juice Factory and 131-TT-01.PV are each a single segment.

For the Timebase Historian, AVEVA Historian, AVEVA PI and Canary per-point paths, <TagName> is everything after the dataset or filter, so a tag whose name contains / (such as an Ignition tag path) is written exactly as named: historian-main/Plant Data/site/area/Process/humidity. Do not escape the slashes; %2F and square brackets are not decoded.

Version: tag names that contain / bind from the first Timebase release after 2.0.0, which needs both Atlas and Connect from that release. In 2.0.0 the tag part of the path is a single segment, so a path to such a tag is never routed: the domain compiles and the property stays empty, with nothing logged.

Path shapes per plugin#

"Datasets": {
  "The Juice Factory/FillerTemps": {
    "Dataset": "The Juice Factory",
    "Filter": "Fields[\"Instrument Code\"]:\"TT\""
  }
}

Get the exact path, do not type it from memory#

A binding path is matched against the connector's advertised paths at start-up rather than when you save, so copy it rather than typing it.

To get the exact string:

1. Open Connect's Sources page and select the source.
2. Read the advertised paths and templates from its page, under Advertised routes. Open the datasource row to list them.
3. Copy the path character for character into the Instance Map's Binding path field.

Why a bound property shows the wrong number#

Two cases produce a value that is present but not the one you meant:

• A compound property bound to a source whose column names do not match. Atlas matches the source's columns to the property's fields by name, and drops any name it does not recognise. A source emitting Val for a field named Value writes nothing at all.
• A plain number property bound to a multi-column result. For a non-compound property, Atlas reads the first column by position, so binding a plain double to the standard [Timestamp, Value, Quality] result lands the Timestamp in it, which reads as a large number. Model time-series properties with a compound type such as atlas-common:TVQ.

See Property Sourcing Strategies for how to model the property so this cannot happen.

How live data reaches Atlas#

Once Connect is connected, it does not wait to be asked. It pushes.

1. Connect connects to Atlas and identifies itself.
2. It advertises every binding path and path template its enabled sources can serve.
3. Atlas compares those paths against the domains it has compiled.
4. For every path a compiled domain actually binds, Atlas opens a subscription.
5. Connect pushes values on those subscriptions as they arrive, and Atlas writes each one straight into the property.

It is one Atlas process end to end. There is no separate cache service and no separate engine service to check.

How each source produces its live values depends on the plugin: OPC UA, i3X, PI System and Canary receive change notifications from the source; Historian, SQL, AVEVA Historian, InfoHub and HTTP re-query on a timer. The PollInterval field on the polled plugins controls how often.

Historical data#

When a query or a chart asks for a window of time in the past, Atlas asks Connect for that window at read time, every time. The source system remains the system of record.

Three consequences worth knowing:

• A property holds its current value. History lives in the source, and Atlas reads it there on demand.
• A connector that was offline loses no history. Because history is read live from the source, a gap in the live stream fills itself the moment a chart asks for that window - so there is no backfill step to run.
• A source with no queryable history answers only for the present. The chart is empty for the past however long the connection has been up.

Data quality#

Seven of the shipped plugins send a quality code alongside every value: Timebase Historian, AVEVA Historian, AVEVA PI System, Canary Historian, i3X, OPC UA and InfoHub.

For that quality to land in Atlas, the property must be modelled as a compound whose type has a field named exactly Quality. The standard type is atlas-common:TVQ, which has Timestamp, Value and Quality. The field names must match exactly, including case.

The property is atlas-common:Quality. It is a whole number, not the words Good or Bad. It is the raw OPC quality code, and only the top two bits carry the status:

In a query, read the field through the compound:

MATCH (p:Pump) RETURN p.Label, p.Temperature.Value, p.Temperature.Quality

A new binding reads as no value#

This is worth knowing when you verify a new connection. A bound property holds no value until the first one arrives, so a new binding reads as null, not as zero:

A new binding, an offline tag and a connector that is down all read the same way, because nothing has arrived. A real zero is a value: a source that delivers 0.0 reads as 0.0, and a Bad quality of 0 is a reading, not an absence. A source that sends a null cell also leaves the property with no value.

Test the property, or the field. In Cypher a compound property that has never received a sample is null as a whole:

MATCH (p:Pump) WHERE p.Temperature IS NULL RETURN p.Label

Once any field holds a value the property is a compound, so check the field you care about, for example WHERE p.Temperature.Value IS NULL. The pull read of a property's values returns "value": null with a quality of GoodNoData, and the Instances page shows a dash. When you are verifying a new source, look for those first.

Verify the connection#

On Connect#

Open https://<YourConnectorHostHere>:4562 and look at the Sources page. The header shows how many sources are running out of how many configured.

The link to Atlas is on the Targets page. Each target's Status reads Connected once Connect has connected and registered, Connecting… while a first attempt or a retry is in progress, and Connection Fault with the reason when Atlas refused it. A source's own page repeats the targets it advertises to, with the Hub URL and the time of the Last handshake, and lists its Advertised routes: each datasource with its paths and templates.

Connect has no unauthenticated status endpoint. The calls behind its pages sit under /app and expect a browser session; a script that wants the same facts reads them from Atlas, as described below.

Each source's own status#

The Targets page describes Connect's link to Atlas. Each source also carries a status of its own on the Sources page: Running when it can reach its system, Connecting… during the first attempt or an outage being waited out, Connection Fault when it cannot reach its system or was refused, with the reason beside it, and Config Fault when it could not start at all. A source whose system is down keeps its subscriptions through the outage and recovers on its own. The statuses, the one-minute grace and the retry rhythm are described in Plugin Instance States, Connection Status and Retries, and each plugin's own article says what its connection check does.

In Atlas#

Atlas keeps the list of connectors that are registered with it, and a script can read it:

GET https://<YourAtlasServerHere>:4552/api/v1/connectors

The call needs a bearer token from Pulse, like everything under /api/v1. It returns one entry per connector with connectorId, displayName, datasourceCount and registeredAt. If your connector is in the list, it is connected and completed a handshake. Presence in the list is the online signal; there is no separate status field, and a connector whose connection drops is removed at once.

The list counts datasources and does not name binding paths. Use the source's page in Connect, under Advertised routes, for those. The Atlas interface itself has no connectors page; its Logs page is where a connector's registration and subscriptions show up, as described in Connector Diagnostics.

Hot reload#

Edits to connector.config take effect without a restart. The Connector detects the change, reconciles the running sources against the file, and re-advertises its paths to Atlas. The Atlas connection is not dropped.

Dropping a new plugin folder into the plugins directory also triggers a reload. The plugins directory is set by the top-level Plugins value in appsettings.json, or by the Plugins environment variable in Docker (/plugins on the shipped image). Its default is Plugins, with a capital P, which matters on Linux.

Four timing settings live in a PluginManager section and rarely need changing:

Multiple Atlas targets#

A single connector can be configured with several Atlas targets, each an entry in atlas.config. A source then names the targets it advertises to, by alias (matched without regard to case) or by the atl_ key:

"con_3e91b2": {
  "alias": "erp-system",
  "name": "example-erp-system",
  "type": "Microsoft.SqlServer",
  "enabled": true,
  "atlasIds": [ "primary" ],
  "config": { "ConnectionString": "..." }
}

Leave atlasIds out, or leave it empty, and the source advertises to every configured target. In the interface this is the Targets tab of the source's Edit drawer: one switch per target, and with every switch off the source advertises to all of them. The Sources page shows the result in its Targets column, as the target names or All Targets.

Multiple Connectors#

You can connect any number of Connectors to one Atlas. Each connects independently and no coordination is needed between them.

Two rules govern collisions:

• Two Connectors with the same id overwrite each other. The last one to connect wins. The id comes from the host's own state file rather than settings.config, so give each deployment site or environment its own name on the Instance Info page, opened from the bottom of the sidebar. The id a connector advertises is still reported as connectorId on the wire; only where its value comes from has changed.

• Only one connector serves a binding path. If two advertise the same path, the last to connect takes it and Atlas logs the overlap. The other connector stays connected and stops serving that path, so values are neither merged nor delivered twice. Give each source a distinct Id so its paths are unambiguous.

Troubleshooting#

Related articles#

• How Atlas Aligns and Times Values - what Atlas does with a connector's watermark, and why AllowedLatency on a source affects the rows Atlas emits
• SQL Server Connector Plugin - the watermark and re-snapshot subscription modes in full, shared by all four SQL plugins
• Binding Concepts - how a binding path resolves to a source and a topic