Sensor Data Platform: Multi-Source Ingestion at 10M+ Points a Day

Some details are generalized where a client needs them to be.

My role was to define the target architecture and build the adapter pattern that let new field sources land without changing the rest of the platform.

Problem

The platform pulls from a lot of different field hardware: vibration sensors, energy meters, groundwater monitors, OPC-UA controllers, and vendor devices that ship their own payload formats. Each source speaks a slightly different language. Some publish over MQTT, some expose OPC-UA, some arrive through TCP or HTTP bridges, and each one carries its own idea of what a measurement looks like.

The original problem was not only volume. It was shape. If every device integration leaks its own schema into storage, APIs, dashboards, and alerting, every new source becomes a platform change. That does not scale. We needed ingestion services that could absorb protocol-specific mess at the edge and present one stable contract to everything downstream.

Approach

So we treated ingestion as a set of small services around one contract: adapters at the edge, a normalized sensor model in the middle, and TimescaleDB as the durable time-series store.

One shared model sits between the messy protocol edge and everything downstream. New hardware is one new adapter service; storage and consumers stay stable.

• Thin adapter services, one shared model. Each field protocol (MQTT, OPC-UA, vendor TCP/HTTP) lands in a small service whose only job is to translate into a common observation model, following the OGC SensorThings shape. Add a new device type and you write one adapter; the storage and consumers never know the difference.
• TimescaleDB instead of a dedicated time-series engine. The platform already ran PostgreSQL everywhere. Hypertables, compression, and continuous aggregates gave us the time-series performance we needed without shipping a second database to every deployment. Fewer moving parts beat chasing peak throughput we were not going to use.
• Batched, idempotent writes. Adapters micro-batch their inserts, big enough to amortize the per-write cost, small enough to keep freshness in seconds. Writes are idempotent on (datastream, timestamp), so if something falls over and replays, it just lands correctly. No careful cleanup required.
• Scale by edge pressure, not by rewriting the core. No adapter owns the whole platform. High-volume sources can run more workers or larger batches, while quiet sources stay small. The normalized model is the boundary that keeps that scaling local.

Implementation

Each tenant gets the same ingestion shape: selected adapter services, broker, database, and API consumers around the normalized model. The adapter boundary made deployment composition straightforward. A tenant that needs MQTT and OPC-UA gets those adapters. Another tenant with a vendor bridge gets that adapter without changing the rest of the stack.

The important implementation detail was keeping protocol handling out of the core. Adapters own connection behavior, retries, payload quirks, timestamp cleanup, and source-specific validation. The shared layer owns identity, normalized observations, idempotent writes, and storage. That split kept failure domains small: a broken vendor payload could fail one adapter without forcing every consumer to understand that vendor.

Scaling followed the same boundary. High-volume adapters could increase workers and batch sizes independently, and inserts stayed idempotent on the normalized datastream/timestamp key. That made replay safe: if a gateway reconnected and sent old data again, the platform converged instead of duplicating history.

Outcome

• 10M+ observations a day, sustained across production tenants.
• New field sources land as adapter services, not as changes across storage, APIs, and consumers.
• High-volume sources scale independently, through adapter workers and batching, while the normalized model keeps downstream behavior stable.
• Real-time monitoring became practical because tilt, vibration, energy, and process signals could be compared through one shared observation shape.

What I’d do differently: put the normalized model in on day one. Retrofitting it while the system was live was the most expensive part of the project. Writing the adapters was straightforward. Migrating history out of older schemas, under load, was not.