The MQTT protocol: advantages for industrial networks

The first of a two-part article explaining the fundamentals of the MQTT protocol, and how it can be adapted to industrial applications.

Since 2015, MQTT has consistently ranked as the most popular Internet of Things (IoT)-specific messaging protocol in the Eclipse Foundation’s annual IoT Developer Survey. An open-source OASIS/ISO standard, MQTT is used in many consumer applications, like mobile chat, home automation and car-sharing. It is supported by major cloud computing and IoT platforms for enterprise applications in smart energy, health and banking services as well. And more recently, MQTT has gained traction within the manufacturing and processing industries.

However, there are several obstacles to bringing the power of MQTT to bear in an industrial environment. MQTT’s innate flexibility is also a potential drawback, requiring stronger guarantees of interoperability and state management to meet the needs of a diverse industrial network. Likewise, the integration of disparate device protocols cannot be addressed purely through MQTT, given its current level of support in field devices and the long lifespan of legacy systems. And while MQTT goes a long way to addressing fundamental cybersecurity issues, MQTT by itself isn’t sufficient to create a secure industrial IoT (IIoT) infrastructure.

For system integrators, developers, engineers and managers wondering what MQTT can do for them, this two-part article explains the fundamentals of the protocol, demonstrates how the Sparkplug B specification adapts MQTT to industrial applications and shows how to establish and scale MQTT networks using industrial edge computing.

Fundamentals of MQTT v3.1.1

MQTT (formerly MQ Telemetry Transport) was designed for industrial networks. In the 1990s, ConocoPhillips (now Phillips 66) needed a way to improve telemetry reporting over its costly, low-bandwidth dial-up and satellite SCADA infrastructure. IBM partnered with system integrator Arcom Control Systems (now Cirrus Link Solutions) to develop a minimalist communication protocol that gracefully handled intermittent network outages and high latency among distributed devices over TCP/IP.

With efficiency and stability as principal goals, IBM and Arcom made a few critical decisions in designing the MQTT protocol, including the use of a publish-subscribe architecture, a message payload with very low overhead and mechanisms for dealing with client disconnections.

Publish-subscribe communication

Traditional communication models use a poll-response mechanism to interrogate field devices and maintain state awareness. Data is refreshed when a master device or application issues a request for new data, causing field devices to respond with the requested data. Queries are often triggered cyclically based on timing parameters.

IBM abandoned this model in favour of a brokered publish-subscribe model that is central to the strength of MQTT. In that scheme, a central server acts as a broker, managing data delivery for the entire network.

Figure 1: In the MQTT architecture, publishers and subscribers from across the organisation connect to a common data broker.

Rather than being prompted by command, MQTT-enabled field devices and gateways publish data to the broker only when they detect a change in a monitored value, a behaviour known as report-by-exception. Other clients, including software and other field devices, can register with the broker as subscribers to any data published on the network.

Since some devices may not need to publish very often, state awareness is maintained by periodically sending a small keep-alive packet to the broker, in addition to other mechanisms that we’ll discuss later.

Flexible topic paths

Although each client identifies itself to the broker with a unique client ID, with the broker as the one and only endpoint, there is no device addressing scheme. Instead, clients identify published data using individual, hierarchical topic paths represented as plain text strings.

For example, CellA/Oven/Temperature could represent the internal temperature of an oven in a particular work cell. A client could subscribe to that topic by using the exact path as a topic filter or could enter a more general filter to capture related data as well.

For instance, there might be other devices in that same cell that also publish temperature data as CellA/Pump/Temperature and CellA/Fluid/Temperature. A subscriber could use the single-level wildcard (+) character to register a filter with the broker that would match all three: CellA/+/Temperature.

Or maybe that oven in CellA also publishes pressure and humidity data. The subscriber could register those topic filters independently as CellA/Oven/Pressure and CellA/Oven/Humidity or could capture all subtopics using the multi-level wildcard (#) character: CellA/Oven/#.

Figure 2: MQTT’s flexible topic filters provide a simple subscription mechanism.

The broker stores these topic filters as part of each client’s network session and routes matching updates from publishers to any and all matching subscribers: one-to-one or one-to-many. This creates a system that is something like Twitter for machines: a free flow of anonymous, interest-based, event-driven, bidirectional communication.

Message payload

MQTT is an application layer protocol like HTTP, meaning that it defines the interface methods that applications and devices use to talk to each other over a network, and both are commonly used as messaging transports. In fact, MQTT has become a contender with HTTP for the most widely used protocol in IoT applications. But unlike HTTP, MQTT was designed for machine-to-machine communication.

While HTTP is famously heavyweight, with a long list of message headers used to describe and respond to resources, MQTT is data-agnostic, with a streamlined on-the-wire footprint that can be processed efficiently by devices with limited power and processing capabilities. It uses a simple byte array payload with a fixed 2-byte header and variable-length header fields (up to a few additional bytes) to indicate packet length or control codes. A packet can be up to 256 MB in size and can transport anything from process variables to a picture of your favourite pet.

Fault tolerance mechanisms

Reliable messaging is critical for industrial communications because of the need for accurate system state awareness and timely control response. Understanding this, MQTT’s designers included features to gracefully handle unintended client disconnections and ensure data delivery to subscribers.

Messages can be transmitted or received with one of three quality of service levels (QoS 0,1, or 2). Higher QoS levels carry more overhead but provide a stronger guarantee of delivery.

• QoS 0 publishes a message once with no required acknowledgment.
• QoS 1 publishes a message repeatedly until it is acknowledged.
• QoS 2 publishes a message repeatedly but guarantees that it is received only once.
   

Published topics can be flagged as retained messages. This flag instructs the MQTT broker to store the most recent update to that topic. When a new client subscribes to that topic, it will receive this message immediately, rather than waiting for the next publication.

A client can also opt to connect to an MQTT broker with a persistent session. When the client disconnects, intentionally or otherwise, the broker keeps these network sessions open and stores the client’s subscription filters along with any matching QoS 1 or 2 messages that arrive while it is disconnected. When the client reconnects, messages are issued in the order they were received.

Finally, each client can also designate a special message as a last will and testament (LWT), with its own topic path, QoS and retention settings. This message is registered with the broker upon connecting and is distributed to subscribers should the client connection be interrupted unexpectedly. Each client’s keep-alive timer value determines when the broker considers this connection lost.

Reduced bandwidth

Combined with its streamlined payload, the communications model used by MQTT results in an 80–90% reduction in overall bandwidth consumption compared to poll-response protocols, according to Cirrus Link. This efficiency creates significant room for existing networks to grow — up to millions of connections — with MQTT.

Decoupled data

It is the decoupled nature of MQTT data exchange that unlocks the scalability of industrial networks. Without the need for point-to-point connections or direct addressing, MQTT networks can grow and share data flexibly.

Any client that wants access to published data — a maintenance database, ERP or SCADA system, for instance — can simply point to the common MQTT broker and subscribe to any desired topics, without needing the details of the publishing source. Network traffic between publishers and the broker is unaffected by the number of subscribers receiving updates, and subscribers do not require reconfiguration if a field device type or IP address changes.

Figure 3: Brokered traffic dramatically reduces the number of network connections.

Data integrity and security

The architecture also has important implications for data integrity and security. Since the MQTT broker doesn’t house or modify data, but only distributes it, each publisher is the single source of truth for its respective topics, reducing the potential for discrepancies and data siloing.

MQTT’s fault-tolerance features complement the flexible nature of the publish-subscribe model by building in guarantees that ensure that clients are advised of changes in data quality and equipment state without constant polling.

The broker alone manages user authentication, data access rights and message delivery, simplifying network management and allowing each client to remain anonymous to other network members. And since MQTT connections are device-originating (outgoing), only the broker is required to have an open firewall port. Field devices can be completely locked down while still permitting bidirectional communication.

Figure 4: MQTT’s device-originating connections simplify cybersecurity.

MQTT also supports optional username and password fields, but in an effort to keep the specification as simple as possible, it primarily relies on security mechanisms in other layers. The most common method is to make use of the transport layer security (TLS) mechanisms already built into the TCP/IP stack (port 8883 is registered for MQTT TLS). In combination with certificates of trust to authenticate the identity of connected endpoints, secure site-to-site MQTT communication is feasible even over public networks.

Faster, better with Sparkplug B

Because of its flexibility, MQTT has been adopted for many IoT solutions. It imposes few naming conventions and accommodates any payload of an appropriate size with equal efficiency. Developers can customise MQTT’s basic structure (topic paths) and behaviours (retained messages, persistent sessions and so on) for their MQTT client as best suits the application, with no extraneous design elements to account for. However, this same flexibility can also be a limitation, particularly for the kind of large, diverse networks characteristic of industrial installations.

With no uniform naming standard, no common data format and no contextual information on published topics, the details of each publisher’s data — which topic paths to subscribe to, how to decode or interpret the payload, the appropriate QoS level and how to recognise a publisher’s LWT — must be known ahead of time in order to be discovered and used by subscribers. Since each device is free to use different conventions, configuring a large network can require significant effort akin to mapping conventional tag data between applications. These factors slow the pace of development and may even inhibit interoperability, adding to the cost of integration and undermining the goals of IIoT.

Likewise, while MQTT includes features to monitor client connection status and data quality, there’s no guarantee that a vendor will utilise them in a field device or software client.

These weaknesses — and other observations about MQTT’s adoption in the marketplace — inspired the development of the open-source Sparkplug MQTT Topic Namespace and Payload Definition, under the leadership of MQTT co-inventor Arlen Nipper and Cirrus Link Solutions. The current version, Sparkplug B (SpB), expands on the basic MQTT architecture to address common industrial use cases and adds a handful of important implementation details to MQTT clients that conform to the specification.

Components of an MQTT/Sparkplug B network

Sparkplug adds to and clarifies the roles of basic MQTT clients, enabling new features and more explicit messaging that are the basis for other enhancements in the specification.

The specification distinguishes between two types of MQTT clients:

• MQTT/Sparkplug B Edge of Network (EoN) Nodes: These clients provide physical and/or logical gateway functions to enable MQTT/Sparkplug B communications for legacy devices and sensors. EoN nodes also include smart devices and sensors capable of publishing their own Sparkplug B data, process variables or metrics directly to an MQTT broker.
• MQTT/Sparkplug B Application Nodes: These are software clients, optionally including one primary application responsible for sending commands and receiving historical data. An MQTT/SpB application node may also be a gateway to legacy software systems.
   

Sparkplug nodes use predefined message types to distinguish between internal data and data originating from connected devices. Nodes are also responsible for reporting on the state of their connected devices, if present.

Any MQTT 3.1.1-compliant broker will support Sparkplug B payloads, and one or more can be used as needed for redundancy, high availability or scalability.

In Part 2

In Part 2 of this article we will further describe the features of the Sparkplug B specification and how it makes MQTT more suitable for industrial use. We will also look at how MQTT with Sparkplug B can be practically implemented with the application of edge computing.

Top image credit: ©stock.adobe.com/au/Michael Traitov