Industrial telemetry pipeline

Context

Industrial controllers expose hundreds of process values that change many times a second. Operations and analytics teams need that data on-site and in the cloud, aligned in time, stored as a time series and searchable.

Problem

Controller data was already being recorded, but only inside the plant's existing systems and in a shape built for operations, not analysis. Training models and running inference needed the same data moved out continuously, aligned in time and structured.

Role

I designed and built the pipeline end to end: the edge acquisition service, the ingestion API, the stream workers, storage and search.

Architecture

1. Edge acquisition. A service beside each controller reads its full data block about 50 times a second and decodes around 200 tags. Tag maps live in MongoDB, with a file fallback, so a new site or tag set is configuration rather than code.
2. Time alignment. Each second of readings is resampled onto a fixed 20ms grid. Every row carries the latest value of every tag, so downstream consumers always receive 50 complete rows per second.
3. Dual delivery. Each one-second batch goes to an on-site and a cloud endpoint. Each destination has its own queue and retry loop with exponential backoff, so an outage at one never stalls the other.
4. Ingestion API. A FastAPI backend validates incoming batches and publishes them to Redis Streams.
5. Stream workers. Multiprocessing workers in consumer groups sanitise payloads, remove out-of-order duplicates and write in batches to TimescaleDB.
6. Search. Key parameters are indexed in Elasticsearch for full-text and analytical queries across the device history.
7. Self-healing and observability. The edge service restarts itself after 30 seconds without a successful controller read or delivery. Grafana, Loki and OpenTelemetry dashboards drive alerting.

Key decisions

A fixed time grid instead of raw read timestamps. Controller reads drift by a few milliseconds under load. Resampling to a 20ms grid gives every consumer the same row cadence and makes joins across tags and sites trivial. The cost is that a row can repeat a value when a read is late.

Independent delivery per destination instead of a single fan-out. One shared send loop would couple the on-site and cloud paths, so a slow cloud link would back up local delivery. Separate queues and retry budgets keep each path's failures contained.

Bounded buffers that shed the oldest data instead of unbounded queues. An edge box cannot grow memory without limit during a long outage. The edge queue keeps the most recent few minutes and drops older readings once full, and a batch that exhausts its retries is logged and released. Recent data stays live; the trade-off is a gap during extended outages rather than a crash.

Results

• Controller data that previously stayed inside the plant's systems now lands structured in TimescaleDB, ready for model training and inference.
• Around 10,000 tag values per second from each controller (about 200 tags read 50 times a second), aligned to a 20ms grid.
• Full-text and analytical search over the device history through Elasticsearch.
• The edge service recovers on its own within 30 seconds of a stalled read or delivery.

What I'd change

Delivery still depends heavily on the site's internet connection. When bandwidth drops, batches back up at the edge, and a long slowdown means the oldest readings are shed. I would add store-and-forward at the edge: persist each batch to a small disk-backed queue before sending, compress batches with delta encoding since most tag values change slowly, and drain the backlog in larger chunks once the link recovers. A slow connection would then delay data rather than lose it.