OT/IT network convergence: experience-based best practices

The idea of network convergence sets the stage for interactions between IT and OT network personnel with very different training, experiences and cultures.

For years, office information technology (IT) networks and plant floor operational technology (OT) networks were wholly separate. On top of that, IT and OT personnel often had little to do with one another. With the advent of Industrial Ethernet replacing fieldbus protocols on the plant floor, they now share a common network, creating valuable opportunities to combine resources and collaborate on goals for overall organisational success.

However, this network convergence also sets the stage for interactions — some might say showdowns — between IT and OT network personnel with very different training, experiences and cultures. The extent to which these necessary collaborations become adversarial or collaborative is dependent on the approach taken by the organisations and individuals involved.

There is a great deal of misunderstanding about what convergence is and what it entails. The chances of success in this environment are low due to the steep learning curve and the opportunity for costly missteps when combining different perspectives.

Fortunately, these challenges have become less necessary to endure. As more and more organisations converge their networks, there is a growing body of resources and best practices being published.

What ‘one network’ means

With Ethernet now commonly running on both the IT side and the OT plant floor, isolated networks are no longer advisable. Converged networks give us the ability to selectively share data. Thus, we are seeing the emergence of what has become known as the convergence of IT and OT — or the creation of a single network.

The properly converged OT/IT network is not one big flat network, but one network strategically protected, so only appropriate data flows. Selective sharing controls device connectivity and data access to ensure only authorised information and resources are accessed. Specific data might flow one way, from plant to office or office to plant; back and forth both ways; or not at all. This selective sharing is a key to an effective and secure network.

The benefits of a converged network

Economies of scale

Moving to Ethernet in the OT environment is both practical and cost-effective. Since Ethernet is prevalent and standards-based, it can be found in consumer appliances, IIoT devices and ruggedised industrial devices. By leveraging the availability of Ethernet products and associated standards, you can now choose the best solutions from different manufacturers and they should communicate with each other with little effort.

Interoperability

The flexibility of implementing Ethernet on a converged network provides exponential benefits to the individuals in both OT and IT. Historically, industrial devices communicated through fieldbus protocols. However, implementing a fieldbus protocol, such as Profibus, limits device options to only those which speak Profibus. Alternatively, Ethernet supports multiple protocols.

Information

All machines are gathering data. However, data without context is useless. With the speed and immediacy of Ethernet communications, operators can, for the first time, collect highly detailed, real-time production data that can be strategically deployed to make smarter, cheaper, more efficient business decisions. By converging your network, IT and OT can leverage the skillsets of both teams to interpret and analyse the information.

The value of data

The primary interest of many manufacturers is often data capture and analysis due to its powerful and quick rewards. They can raise the bar on production goals, then gather the appropriate data and determine how to get there.

Value is locked in OT production data nearly everywhere. For example, a company that produces consumer goods lacked insight on the speed or functionality of their machines. When machine issues occurred, operators had difficulty communicating with maintenance staff. To combat this, an OT network was built that allowed their existing HMI to connect to a communication server and contact the appropriate maintenance personnel.

Through that data, they are able to monitor machines more effectively, measure response times and use real-time production data to proactively contact the appropriate person when a machine reached certain milestones.

Many companies like these are also finding that having production and sourcing information down to each individual component is extremely valuable. They can use this information to track and trace issues with specific units and ensure that such issues are minimised. Further, many industrial companies are finding that collecting data and storing it is valuable, even if you don’t have the right questions to ask yet. Manufacturers might want to investigate something later, and having production data to analyse from previous months and years is very valuable in the pursuit of such knowledge.

In the pre-Ethernet days, if this type of information was collected at all, it might be hand captured on clipboards and all but lost. Even if it was later looked at, it was subject to illegibility and many other types of human error. Using Ethernet to capture and analyse information makes it potentially useful intelligence as opposed to pen scribbles.

How to design a converged network

It is important to avoid quick fixes and short-sighted solutions, such as connecting existing OT and IT networks. A converged network should not be formed from two existing networks. The methodology “just plug them in” seldom works.

Network audit

The first step in designing a new network is identifying what is on the network currently. This process gives insight into what devices are where, and what each is currently talking to. This is also a good time to develop accurate documentation as to the network structure.

Odds are you may be in for some surprises. Things tend to be added over the years without concern for the holistic nature of the network. This is your opportunity to start with a clean, streamlined, efficient slate.

Assessment

Once you’ve inventoried everything, your next step is to assess the status of your current network. At this snapshot in time, what is the quality of your network? You will identify the purpose of each device and decide what should be talking to what. Then you can create the optimal data flows for each case. It’s a very individual and technical discussion for the organisation, and strategic planning should be done.

As a few general examples, production data might flow up to analysis software that may reside in the enterprise where it may be selectively reported to salespeople and non-technical managers. Other OT-generated data, such as real-time status reports or maintenance schedules, would likely stay in OT. By the same manner, IT data, such as personnel records and salary data, should not be accessible by the plant floor.

Structuring OT and IT

The inventory/audit will help you keep all OT machine functions out of the IT world and vice versa, ensuring that nothing is inappropriately tied to the wrong network, so the proper security protections, resources and connections can be applied. The often cited Purdue Architecture Model is a good, simplified illustration of a basic network architecture.

There are certainly some grey areas. Remember, it’s not where the device is located, it’s what it does. For example, there might be a device used to access email on the plant floor and these would be connected to the IT network, not the OT network. Purposes should NOT be mixed; mixing capacity opens up serious vulnerabilities.

Consider a DMZ

In between the OT and IT domains is what is known as the DMZ. This shared territory is where both worlds come together and what is shared with whom is determined. Physically, this area is a collection of servers and PCs, with information flowing up from OT and down from IT, directionally protected by firewalls. Here it is appropriately processed and then directed back to the predetermined location. The information flowing in and out is carefully controlled — selectively shared one way or both ways as appropriate.

One important function of the DMZ is to keep a wide buffer zone between the outside world accessed by IT and the bread-and-butter world of OT. Threats from the business side need to be isolated from the OT world and can be accomplished through compartmentalisation such as ISA99/IEC62443. This protects the manufacturing side from being impacted by IT threats and allows production to continue. Further, the DMZ helps ensure that production equipment would not be subject to IT necessities, such as virus scans or firmware updates.

Don’t make security an afterthought

A plan needs to exist and be integrated as to how you will share data. Begin with determining security needs that should be built into your network. The US National Institute of Standards and Technology (NIST) has made recommendations on cybersecurity for reference.

Don’t wait for the perfect solution to solve every scenario. As part of this plan, document what simple actions you can take to increase your security and implement them immediately.

A new organisational agreement

Even in an organisation where OT and IT people work well together, inevitably, it will come up: Who is in charge in situation X? Does IT or OT have the final word on equipment and operations in the DMZ? Who specifies network-wide Ethernet equipment?

When the converged organisation is built, the purpose is to share information and support both the OT mission and the IT mission. Decisions need to be thoughtfully made to ensure there is not a ‘winner’ and a ‘loser’ and subsequent disgruntlement. A better way may be to create a new dotted line organisation, frontloading universal buy-in from both IT and OT, at all levels.

In most organisations, this starts with immediate and demonstrated support from the top. It’s good practice to see leaders from both the business and production teams join together and express their support for all OT/IT convergence activities. It is vitally important that IT and OT collaborate and communicate, establishing clear responsibilities. Whether that is two individuals serving as representatives, a committee or a newly created role such as an automation and Data Exchange Engineer.

The Automation and Data Exchange Engineer

It helps to have a new individual in your team, a professional who understands first-hand the functions and priorities of both the IT and the OT worlds and is capable of communicating with and relating to both departments. Let’s call this individual the Automation and Data Exchange (ADX) Engineer. It is imperative that this person is cross-trained substantially in both OT and IT practices with their background of what discipline they came from originally being less important.

Led by the ADX Engineer, there should be governance responsibilities for all things related to the converged network, and answering directly to upper management. One of their early duties might be to develop proper procedures for management and operation of the converged network. They can create a standard operating procedures (SOP) guide for everyone to be aware of the new road ahead. The valid concerns of both IT and OT disciplines will be accounted for, with potential SOPs including directives such as “Patches will always be tested in an isolated sandbox before being applied to any OT equipment” or “Internet-connected devices shall not be placed directly on the OT network.”

The committee or ADX Engineer should also lead all convergence establishment and maintenance activities. If it is a multi-location organisation, they can start with a pilot project at a smaller location and take key learnings on to additional locations. After assessing the extent of the convergence challenge at each location, they can also decide, case by case, whether internal resources possess the expertise — and the extra time — to tackle each project. They can work together to identify and select a turnkey third-party expert, identify local resources to handle the job, or some combination of both as the team sees fit.

Often, an outside third party is beneficial as they can provide insight from a different perspective, share best practices and provide instant, on-demand manpower.

Encouraging cooperation

In the drive for successful OT/IT convergence, sometimes there are situations where one group or the other, resisting change, sticks their head in the sand and refuses to cooperate, causing very difficult roadblocks. Sometimes one group or the other calls in outside help and, literally, says “don’t let the (other department) know that you’re here”. Fortunately, this is not the norm; most organisations are made up of professionals who will work together for the common good and it is assumed that your organisation will not experience anything like this. But, theoretically, what if it does?

The visible involvement of C-Level executives will help in this regard. If it’s holding up progress, they will hash it out.

Another effective strategy is to involve a third party, at least at first. It’s often amazing, humbly speaking, how an idea repeatedly expressed by an insider is ignored, but that same idea expressed by an outside expert is considered genius. That’s reality and it’s helpful to understand. Of course, a well-chosen OT/IT consultant who has “been there, done that” provides both technological and psychological mediator-type assistance and will deliver much more than inside people ever could, drawing upon the experience of driving convergence in other organisations and helping to flatten the learning curve.

It is important that the consultant understands, has experience in and speaks the language of both IT and OT. They should be without loyalty to one side and have knowledge of both so they are not seen as “the IT consultant” or “the OT consultant” but as the “Convergence Consultant.”

Conclusion

The march towards the convergence of OT and IT functions on a single Ethernet network is inevitable for companies that wish to maximise the benefits of Ethernet connectivity while also optimising the efficiency of the network. This will not come without challenges and growing pains, which vary from costly, multi-year processes, to being accomplished by a smooth, mutually beneficial effort.

Image: ©stock.adobe.com/au/vegefox.com