Energy infrastructure demands mission-critical networking

Standard networking methods are acceptable for most SCADA systems, but mission-critical power transmission and distribution system messaging and real-time control demands a zero-loss solution.

As far back as the early 1990s, the electrical power industry was looking for ways to standardise the increasing numbers of ‘intelligent electrical devices’ used in substation equipment. Up to that point, much of the interconnection was hardwired, and intelligent communication was commonly achieved with serial protocols. These efforts eventually led to a new IEC 61850 standard titled ‘Communication networks and systems for power utility automation’. The standard was needed to ensure real-time substation automation applications could be implemented using digital communication networks.

This standard was initially released in the early 2000s, and it defined communications methods and services to promote high interoperability among all makes of hardware. More recently, this standard has been updated to what is commonly known as IEC 61850 Edition 2 to add extensions and improvements, and to correct various shortcomings.

Key portions of IEC 61850 identify data structures, commands and conformance testing requirements. Without rigorous testing methodology and certification, it is impossible to ensure various products will interoperate properly and meet the constraints defined within the standard. Therefore, it is imperative that any devices to be integrated within a network are in compliance with the IEC 61850 standard, which means each device must undergo conformance testing by a suitable agency and receive a certificate.

One update in IEC 61850 Edition 2 from the previous version was the addition of maximum allowable recovery time requirements for various communication events (some of these values are actually zero time), which can only be achieved by means of higher availability for communications links. Fortunately, the rise in prominence of industrial Ethernet has provided various redundancy solutions meeting these needs. As Electric Light & Power (ELP.com) states, “With Ethernet advancements, communications is no longer a limiting factor inside or outside substations.” Furthermore, “the engineering definition and structure defined in the standard will simplify engineering and integration for the technical team”.¹

IEC 61850 is able to fulfil the link redundancy requirements in part by referencing IEC 62439-3, which is titled ‘Industrial communication networks - High availability automation networks - Part 3: Parallel Redundancy Protocol (PRP) and High-availability Seamless Redundancy (HSR)’. It identifies methods, protocols and topologies for achieving Ethernet network redundancy.

Benefits of hardware-based network redundancy

Clearly, the communication redundancy requirements indicated by the IEC standards set the direction for the power utility automation industry. Specific approaches employing hardware-based network redundancy solutions in compliance with IEC standards offer a number of benefits:

• Compliance with IEC 61850 Edition 2 to provide total communication redundancy
• Promotion of interoperability
• Mixed topologies possible
• The ability to connect with non-redundant networks
• No impact on CPU loading
• Simplified network upgrades

Perhaps the most compelling reason to select hardware-based redundancy is that all elements will unmistakably be certified as compliant with a common standard, leading to a total solution. Any other hybrid approach using non-compliant devices or a mix of hardware and software calls into question whether it is truly compliant with the standards. T&D World (TDWorld.com) reminds us that historically, proprietary equipment from various manufacturers would not interoperate, leading to implementations that were “piecemeal and fragmented. One might say they were mutually incompatible and kept that way with an assortment of nonstandard messaging protocols. It was the technological equivalent to the Tower of Babel.”²

In direct contrast to the old way of doing things, HSR and PRP are specified to be interoperable with each other and among devices complying with these standards. Generally speaking, it is most straightforward to implement HSR as a ring and PRP as a parallel star. However, there are times when the architecture may require more complex or mixed topologies, such as multiple coupled rings, or connection to single-port devices, or interconnection of HSR and PRP networks.

Hardware-based components are available to handle these situations in a standard manner. In fact, specific devices called redundancy boxes (RedBoxes) are available to allow any one-port device or non-redundant network to connect seamlessly into a redundant HSR or PRP network.

Looking at performance, hardware-based network redundancy devices are purpose-built for this role. Therefore, they handle all redundancy functions on board and are completely transparent to any external devices. Not only does this simplify implementation, it means that there is no additional CPU loading for any other devices. In the case of industrial automation computers, embedding all the redundant networking functionality on the network adapter ensures the computer can operate optimally and not be hindered in any way by networking issues.

Power automation network redundancy requirements

What exactly does it mean for hardware to offer the right redundancy capabilities for switchgear and substation networking? The IEC standards spell out the performance requirements and indicates some ways of achieving them:

• Must be IEC 61850 certified
• Must achieve mandated recovery times
• Must achieve zero data loss — every command makes it through
• Ring topology possible
• Redundant star topology possible

To start with, hardware vendors must submit their products to testing agencies for evaluation against the IEC 61850 standard in a form of simulated service to achieve certification. Without this certification, the product should not be considered for substation automation applications. Typically, devices are tested individually to confirm basic functionality and also in conjunction with other related devices to confirm interoperability. Networking traffic will also be evaluated.

Table 1 shows the fundamental requirements of the IEC 61850 standard as mandated recovery times for various system communication events.

Table 1: IEC 61850 recovery times — applications and communications (Source: IEC).

Note that some of the recovery times are zero, which is also known as ‘bumpless’, ‘zero data loss’ or ‘zero packet loss’. In practice, the typical method of achieving zero data loss in a networked system is to introduce redundancy such that the communication packet is sent along two different paths so that it will always make it through, even if there is any single failure.

Common Ethernet protocols are neither deterministic nor do they guarantee that a packet will ever actually make it to the destination. There are some other more advanced protocols to address recovery time and redundancy, such as Rapid Spanning Tree Protocol (RSTP), but they are nowhere near fast enough for the most demanding IEC 61850 requirements.

High performance Ethernet

So how it is that common Ethernet media can be considered and leveraged for substation automation? As noted earlier, IEC 61850 in turn refers to IEC 62439-3 to introduce high-performance redundancy protocols acceptable for meeting the recovery time requirements. HSR and PRP are specifically defined methods of achieving suitable redundancy, capable of meeting the zero recovery time requirement since their architecture ensures no packet is lost. They share some similarities, but also have their own pros and cons.

HSR network uses a ring topology and requires no dedicated switches. Instead, each intelligent device has at least two ports and acts as a switch so each data packet frame received on a given port is retransmitted (forwarded) out the other port. The basic concept is that if the ring is healthy, each destination node should receive two identical frames from a source node, with minimal time delay between the two. Normally the second frame is discarded, but if it is never received it indicates trouble on one of the paths. Even in the case of one break in the ring, operation continues normally.

Figure 1: A HSR ring network.

The PRP approach is a parallel star. Every network path consists of two connections, effectively creating two networks completely in parallel. Every frame is sent from each source to each destination down both paths. Again, the destination device normally receives both frames and discards the second, but any time only one frame is received it indicates trouble on the other path. As with HSR, operation continues normally even in the event of any one failure.

Figure 2: A PRP network.

As communication technology continues to focus on Ethernet, and as the electrical power industry complies with IEC 61850, applications will naturally standardise to implementing HSR and PRP. HSR has some limitations on node quantity and bandwidth, and requires specialised node hardware, but avoids the need for additional switches. PRP allows more common network methods but doubles the number of switches and field cables.

Protection, Automation & Control World (PACW.com) concludes “that a system based only on fibre cables can replace a conventional system. The availability of faster CPUs and multiple communications ports enables this approach as an alternate for a conventional system. In the near future protocol-based systems will bypass the conventional way of using miles of copper cables, not only in speed, but also in reliability”.³

Each installation must be evaluated to identify if one or both solutions are optimal. The following application examples illustrate the strengths of each topology.

Application example: switchgear with protective relays

The present generation of switchgear is more likely than not to be specified with advanced intelligent devices for metering, control and protection. Legacy switchgear is commonly upgraded with smart devices, so the entire installation is intelligent and compliant.

Figure 3: Example of switchgear on a HSR network.

In the example of Figure 3, the various smart devices — such as protection relays, control units and metering units — are installed within the switchgear and interconnected with standard Ethernet fibre-optic media in an HSR ring configuration. The fibre-optic connections are high speed, electrically isolated and immune to electrical noise.

In a traditional configuration, many of these devices would need hardwired interconnections to achieve their protective functions. However, with a HSR ring established, it is possible to perform these peer-to-peer connections using the IEC 61850 Generic Object Oriented Substation Event (GOOSE) control model. GOOSE messaging is extremely high speed, and communications will continue reliably even in the event of any one cable or device failure.

Note that there are two locations in Figure 3 where industrial computers are indicated. In this case, the equipment consists of a power automation computer with a gigabit Ethernet adapter card, with both components certified as IEC 61850 compliant.

The computer on the bottom right is inserted in the HSR ring and acts as an end device application platform, just as any other relay or intelligent device. It could be used to provide local data gathering or more advanced control functionality. The computer at the top of the ring is acting as a RedBox or gateway for the WAN workstations to access devices on the HSR ring, so that a SCADA system could have connectivity with the HSR ring.

HSR rings are very suitable for installation within switchgear and present the most economical redundancy method, since additional dedicated switches are not required.

Application example: substation automation

For substation automation, the reliable operation of protection relays is crucial. In the example of Figure 4, multiple protection relays are in turn connected to measurement and control units. To achieve complete reliability, a PRP architecture with redundant optical fibres is implemented.

Figure 4: Example of substation network redundancy with PRP.

The top level device on this monitoring and control network once again is a power automation computer with a dual-port gigabit Ethernet adapter card. The dual ports on this card provide connections to the two completely independent but parallel networks. Even in the event of any failure on a network path — whether it is cabling, the switch or control power — the communication will continue normally through the other path.

As is evident in Figure 4, there are some additional costs involved with a second switch and dual fibre runs along every path. However, this architecture allows each network to operate at full performance with no slowdowns imposed by the redundancy scheme.

Conclusion

Critical energy infrastructure, such as electrical substations and switchgear used for power transmission and distribution, serves a demanding role in society. It is crucial that automation methods applied to these systems offer the highest level of performance.

In particular, many of the intelligent electrical devices used in energy infrastructure systems offer advanced communications capabilities over and above their basic protective features. Furthermore, industrial Ethernet has made great advances in all types of automation. However, more is involved than simply patching switchgear devices into an Ethernet switch.

The relevant IEC standards require that the most critical communication events experience zero data loss, and define two network topologies that can readily achieve this requirement. HSR rings are economical to implement, while PRP parallel networks require more switches and media but offer better performance. When implemented properly, even high-speed protection and control can be performed over Ethernet, which reduces installation complexity. Hardware-based products are available to provide an easily implemented solution for many different topologies, without negatively impacting intelligent device performance in any way. When correctly specified and implemented, these hardware-based network redundancy products will provide years of trouble-free service.

References

1. Self H, Guglielmo A 2013, IEC 61850 - Why all the Hype?, ELP.com, <http://www.elp.com/articles/powergrid_international/print/volume-18/issue-01/features/iec-61850-why-hype.html>
2. Wolf G 2016, The Smart Substation, TDWorld.com, <http://www.tdworld.com/grid-opt-smart-grid/smart-substation>
3. Becker F, Nohe S, Echeverria A 2015, Designing Non-Deterministic PAC Systems to Meet Deterministic Requirements, PACW.org, <https://www.pacw.org/issue/june_2015_issue/deterministic_system/designing_nondeterministic_pac_systems_to_meet_deterministic_requirements/complete_article/1.html>

Image credit: ©stock.adobe.com/Phuchit Aunmuang