Message Queues

Edge modules are able to communicate using message queues, and all modules supporting the Nexus SDK uses this, including the DataDistributionModule.

The queues decouple the enqueuing of messages from processing, which means that any module can be restarted without affecting other modules. This provides reliability, by ensuring that all messages are processed at least once. The queue absorbs spikes in message generation, prevents processing saturation, and ensures a retry / backoff strategy in the event of errors.

The Nexus platform allows for a "time to live" setting that sets the maximum time any measurement can exist in the queue. The default setting is 2 hours. The reason for having this setting is to avoid the queue to grow unlimited, exceeding either memory or disk space. Expired measurements on the queue are dropped.

Measurements

All measurements produced on the edge device, by any module, must be sent to the DataDistributionModule measurement input queue. The name of the measurement input queue is always:

"/modules/DataDistributionModule/inputs/distinput"

Streaming logs

Normally, all logs are sent to cloud periodically using cold path upload directly from the modules that generate the log. However, it is also possible to stream the logs in real-time from any module that supports the Nexus SDK. The way that works is that the module sends the logs to a specific input queue on the DataDistributionModule, where logs are routed directly to cloud as hot path data.

The name of the log input queue is always:

"/modules/DataDistributionModule/inputs/streaminglogs"

All logs are always sent to cloud using cold path batched file uploads, even if logs are also being streamed in real time.