Real-Time Industrial Data Platform

Context and problem

What made it difficult

• Sustained high event throughput with peaks well above steady-state load
• Industrial data sources that cannot be paused or reshaped to suit the pipeline
• Near-real-time expectations for operational visibility, not overnight batch reporting
• Enterprise security and access requirements on every layer of the environment
• Cost that has to stay defensible against the operational value delivered

Role and responsibilities

• Led requirements gathering and technical workshops with client stakeholders
• Designed the cloud and data architecture
• Implemented the ingestion, processing, and analytics layers
• Owned delivery planning and coordination across client and internal teams
• Translated technical tradeoffs for non-engineering decision-makers

Generalized architecture

Key decisions and tradeoffs

• Managed ingestion over self-managed brokers

  At tens of thousands of events per second, dropped data during a peak was a greater concern than a few milliseconds of latency. Managed Azure ingestion services absorbed burst load without requiring a team on call to scale a broker cluster. The tradeoff was less control over the transport internals.

• Processing on the way in, not on the way out

  Validating and shaping events at ingest keeps bad data out of the analytics layer, where it is far more expensive to detect and undo. It costs more compute per event, and it means schema changes have to be handled carefully at the boundary.

• Choosing the store by query pattern, not by default

  Time-series-optimized storage answers operational questions over recent windows far more cheaply than a general-purpose relational store; relational storage still won where the questions were transactional. Splitting them meant accepting more than one store to operate.

• Generalized, reusable environment patterns

  Client environments differ, but the shape of the problem repeats. Standardizing the environment pattern made delivery predictable across clients. It required upfront design work that a single deployment would not have justified.

Implementation

• Ran technical workshops to establish data sources, volumes, latency expectations, and the decisions the data needed to support
• Built the ingestion and processing pipeline and validated it against realistic sustained and peak loads
• Modelled operational storage around the query patterns identified in discovery
• Delivered near-real-time dashboards and iterated on them with the operators using them
• Planned rollout and handover so client teams could run the environment day to day

Results

• Systems processing 50,000+ events per second in production
• Operational data made visible to client teams in near real time rather than after the fact
• Environments supported representing more than $3M in contract value

Lessons and next iteration

• The workshop is the highest-leverage hour of the project. Volume and latency numbers agreed in a room beat the same numbers discovered during load testing.
• Throughput headroom is cheaper than an incident. Designing for peak rather than average avoided the class of failure that is hardest to explain to a client.
• Next iteration: push more validation and schema contracts to the source side, so bad data is caught before it costs pipeline compute.

Technologies

Azure IoT Hub · Azure Event Hubs · Microsoft Fabric · Azure Data Explorer · Time Series Insights · Azure Functions · Azure SQL · Power BI · Python · SQL · KQL

Relevant technologies across this body of work. Not every technology listed was part of a single identical deployment.