Kafka, Kinesis, and Event Hub

Streams are not databases#

Worth stating before the detail, because it shapes every decision below.

A stream is an ordered, retained-for-a-while sequence of messages. It is not queryable, it is not indexed, and it does not hold your history indefinitely. Something downstream consumes the stream and puts the data somewhere durable.

That means publishing to a stream is only half an integration. If nobody has built the consuming side and the durable store behind it, Infohub will publish faithfully into a buffer that expires. Confirm the whole path exists before deploying.

Apache Kafka#

The most fully-featured of the three, and the most configurable.

Points that matter when publishing:

Topic and partitioning. Kafka preserves order within a partition, not across a topic. If a consumer needs a measure's values in order, all of that measure's messages must land on one partition — which usually means keying on measure identity. Publishing with no key round-robins across partitions and consumers may see values out of order.

Retention. A broker setting, not an Infohub one. Confirm it, because it determines how long a downstream outage can last before data is lost permanently.

Restatements arrive as new messages. A corrected value is a new message later in the stream, not an update to the earlier one. The downstream consumer must handle a later message superseding an earlier one for the same measure and period — which is straightforward if it upserts into a store, and a real problem if it appends.

AWS Kinesis Data Streams#

A smaller property set, reflecting a simpler service.

Shards and ordering. Kinesis orders within a shard, so the same partitioning consideration applies as for Kafka: key by measure identity if order matters.

Capacity. Kinesis is provisioned in shards or on-demand, and exceeding capacity throttles writes. Estimate the message rate before deploying, and remember that a backfill produces a burst far above steady state.

Credentials. An IAM identity with least privilege — permission to write to that stream and nothing else. IAM credentials also rotate, and Infohub will not know when they do.

Region. The Platform hosting the consumer needs outbound access to the correct regional endpoint. A cross-region write adds latency and cost.

Azure Event Hub#

Partitioning. Same consideration. A partition key based on measure identity preserves per-measure order.

Throughput units. Capacity is provisioned, and exceeding it throttles. Again, plan for backfill bursts.

Authentication. A connection credential that has an expiry and a rotation policy. Record that Infohub depends on it, because when it is rotated Infohub stops publishing and nothing in Infohub knows why.

Cost and volume#

All three platforms meter by volume, and the arithmetic is easy to get wrong in the optimistic direction.

Measures × periods per day × days is the steady state. A thousand measures at hourly intervals is 24,000 messages a day. That is unremarkable. A backfill of one year for those measures is 8.7 million messages in a burst, which may exceed provisioned capacity, throttle, and cost noticeably.

Before deploying, and again before any large backfill:

• Estimate the steady-state rate and confirm the provisioned capacity covers it.
• Warn the platform's owner ahead of a backfill.
• Use the namespace filter to publish only what is consumed.
• Publish at the interval the downstream needs, not the finest interval you have.

The remaining consumers#

Three more consumers exist that do not need a page of their own.

CSV#

Writes values to CSV files. Useful for a downstream process that collects files, for a periodic extract someone works on in a spreadsheet, or as a simple archive.

The practical notes are the same as any file output: the Integration Engine runs as the Bootstrap service account, so that account needs write access to the destination; use a UNC path rather than a mapped drive letter; and agree a file naming and rotation convention so the folder does not become unmanageable.

Canary#

Writes into a Canary historian. Infohub also reads Canary, so both directions exist and it is worth being clear which you want.

Writing to a historian is a legitimate pattern: Infohub's calculated, contextualised values become tags in the historian alongside the raw ones, available to anything that already reads it. It is a good route for making Infohub's outputs available to existing SCADA screens and reports without those systems needing to know about Infohub.

Endpoint Consumer#

The general-purpose outbound consumer: it posts to a URL or writes to a file, with a configurable body.

This is the fallback for any system with an HTTP endpoint and no dedicated consumer — a webhook, an internal API, a third-party service. Body Text is a template, so you shape the payload to whatever the destination expects.

Treat it as you would the Web Service data source in reverse: capture what the destination expects, build the body to match, and test with one measure before scaling up. Be conservative about volume, because an endpoint that is fine receiving a message a minute may not be fine receiving a backfill.

Diagnosing#

See also#

• What a Data Consumer Is
• MQTT and Sparkplug B Consumers
• Consumer Reference
• Multi-Site Architecture