by Ravi Baghavatula and Morris Novello, Nebbiolo Tech, and members of the OpenFog Consortium.
This blog originally appeared in Embedded Computing Design.
The Industrial Internet of Things (IIoT) is heralding a new wave of modernization, across many industries, with customers and internal stakeholders demanding advances in productivity, management, security, and flexibility across various verticals. However, IIoT deployments continue to face considerable head-winds in terms of (largely) manually managed infrastructure, that are mostly un-secure and setup as silos. Fog Computing offers an innovative solution to address these challenges by providing secure access to Operational Technology (OT) infrastructure within the framework of Information Technology (IT) toolsets. Figure 1 shows the layout of a typical factory, with services and workloads being more IT centric at the top layers (Factory Datacenter, say) and progressively becoming OT centric as they move towards the lower layers (Factory Machine, say). Software defined resource allocation and management is gaining traction within the Fog Computing paradigm, as it empowers plant operators to be more ‘adaptive’ to their future needs. From a networking perspective, this translates into implementing Virtual Network Functions (VNFs) across the plant floor using Software Defined Networking (SDN).
Figure 2 presents one view of a typical SDN solution and includes the following components.
- SDN Applications
- The SDN Northbound Interface (NBI) which allows the SDN application(s) to talk to the SDN Controller
- The SDN Controller
- The SDN Control Data Plane Interface (CDPI) which allows the SDN Controller to talk to the SDN Datapath
- The SDN Datapath
- Management and administrative functions which are responsible for policy enforcement and performance monitoring of the entire solution
In an IT-centric environment, each of the above components are realized using a plethora of open-source (and a few closed) solutions. More prominent amongst these include OpenStack, VMware NSX, Cisco Digital Network Architecture, etc.
However, an IIoT environment introduces several constraints on the SDN eco-system that necessitate a re-design of a few SDN components.
- Harsh operating environments, which lead towards mostly fan less compute systems. These fan less designs severely curtail the amount of available compute, primarily due to the restrictions on their available thermal headroom. While new processor designs keep pushing the envelope on a performance-per-watt basis, it’s reasonable to assume that (for the foreseeable future) these headless systems would lag (sometimes quite significantly) behind their cousins from a typical Data Center environment.
- Unlike in a data center environment, compute in a typical factory floor is mostly sparse and usually not universally reachable (see Figure 1). This introduces issues in scale-up and scale-out of SDN components.
- Colocation of SDN components with customer’s business logic, on to the feeble amount of available compute, necessitates a re-think on how these SDN components are implemented (and they inter-connect to one another).
- The lack of a cohesive (and effective) perimeter (in terms of firewalls, BUM rate policers etc.) in a typical factory floor (especially bottom up) implies that these SDN components need to pay special attention to their availability and resiliency. Such considerations make, compute-wise, an already bad situation even worse.
Consider a simple example of a Virtual Network Function (VNF) based solution which involves:
- a learning bridge
- multiple virtual-machines (or containers) connected to the above learning bridge
- one or more machines or sensors connected to the above learning bridge firewall to restrict the flow of data between the above communication end-points
- some means for an operator to manage the firewall
Table 1 provides a summary of the manner in which the VNF solution maps to SDN components, and how these are implemented (kernel-space vs user-space).
Table 2 shows the manner in which various traffic types are handled by the SDN components using (a) Linux tools (using Linux bridge and Linux firewall) and (b) OVS.
Since the entire solution is implemented in software, it is imperative to enumerate the design considerations in selecting either options with respect to an IIoT deployment.
- The communication between the controller and the CDPI is software switched (either collocated onto the same compute, or across the plant network) and hence consumes CPU cycles
- The controller itself consumes CPU cycles
- The OVS solution involves punt (i.e., exception and slow path protocol) traffic being handled by the CDPI agent and the controller (both of which are implemented in user-space. This is in contrast with the Linux tools solution, wherein this traffic is handled entirely within the in-kernel forwarding plane. As such, the OVS solution incurs the overhead of context switches (between kernel and user spaces) on a per-punt-packet basis. These context switch overheads can get quite aggravating in the presence of network disturbances (say, flood of traffic due to a faulty end point, bursts of expected traffic, network re-organization resulting in STP recalculations etc.)
In general, SDN deployments within an IIoT environment need to account for the following broad considerations
- CPU core counts matter. VNFs would need to scale-up locally in terms of CPU cores. Further, the larger the number of available CPU cores, the easier it is to ‘pin’ VNFs to specific cores and therefore provide greater performance and resilience. In general, this improves the availability of not just the VNF but also that of the customer business logic.
- Single threaded performance matters. Due to scale-out difficulties in an IIoT environment, it is imperative that a VNF has the compute headroom to handle bursts of network workloads. Since typical VNFs are single threaded, higher single threaded performance would result in better performance of the overall SDN solution.
- Thermal envelope matters. Since the IIoT environment typically employs fan less designs, an effective SDN solution mandates that the CPU core and single threaded performance be available at a lower thermal envelope.
- VNF Offload matters. While there is merit in considering fast path offloads like DPDK and IOVisor, these offloads are still tuned for IT centric workloads. For example, Intel’s DPDK necessitates a few CPU cores to be reserved for DPDK processing. This doesn’t fit in an IIoT environment as compute in these environments is core count challenged. The need is to offload not just a general fast path, but to offload the entire VNF. This does not imply IIoT environments need to embrace dedicated routers and switches. Rather, the offloads need to include reconfigurable FPGAs. Each VNF IP could be optimized for use in an IIoT environment and flashed on a per-use basis onto an available FPGA for use on the factory floor. This not only offloads the VNF functionality, but also the slow path (and other control plane) aspects of the VNF thereby relieving the feeble IIoT compute of handling networking duties.
- Software stack matters. An IT centric SDN software stack lacks the optimizations necessary to operate in an IIoT environment. Such environments require the use of a purpose-built software stack where each individual component has been fine-tuned for its use in these demanding settings. For example, OVS would incur context switching costs (between kernel space fast path and user space slow path) in comparison to Linux bridging (wherein both fast path and slow path reside within the kernel).
Summary
Fog Computing holds the key to the next big leap in industrial automation and is being embraced by multiple industrial verticals as the means to modernizing not just the equipment but also the processes and manageability toolsets involved in large scale automation infrastructures. NFV is an important aspect of Fog Computing, with SDN fast becoming the de-facto means of implementing NFVs in an IIoT environment.
While IT-centric concepts of SDN may be mapped onto an IIoT environment, an industrial network designer needs to account for a very disparate set of design considerations for realizing the true benefits of Virtualized Network Functions (VNFs) in such an environment.
References
- Fog Networking: An Overview on Research Opportunities, Mung Chiang
- Fog Computing Overview Video
- OpenFog Consortium Website
- Fog Computing and Its Role in the Internet of Things, Flavio Bonomi, Rodolfo Milito, Jiang Zhu, Sateesh Addepalli