Distributed Realtime Simulation

A simulation model may be split up and run on multiple hardware nodes in parallel. The advantages of this approach are:

Increased number of I/Os
Shorter simulation time steps

However, running different parts of a model on different hardware nodes also poses some difficulties:

The nodes may need to exchange information in every simulation step.
The I/Os of the different nodes must be synchronized to guarantee consistent electrical states.
The simulation startup must be synchronized to guarantee consistent model states.

This chapter describes how distributed realtime simulation can be faciliated using multiple RT Boxes with minimal additional modifications to the simulation model.

Model Setup for Distributed Realtime Simulation

Identifying Suitable Subsystems

As a first step the simulation model needs to be divided into suitable subsystems for parallel simulation. Obvious candidates for independent subsystems are isolated electrical circuits.

Another common solution is to separate circuit models at a capacitor or inductor that can be replaced by a current source in one part of the model and a voltage source in the other part. Both sources are controlled by respective measurements in the other part of the model. This approach can often be applied to DC-link capacitors:

../_images/splitup_before.svg — Fig. 4 Model with DC-link before split-up

../_images/splitup_after.svg — Fig. 5 Model with DC-link after split-up

Note that in an offline simulation a Delay block is necessary in at least one direction of communication to prevent algebraic loops.

Exchanging signals between two models

For physical distribution of the model onto different RT Boxes the code for each part of the model must be generated separately. Therefore we need to create a subsystem for each RT Box and place the corresponding part of the model inside it.

The signals between the subsystems are now replaced by SFP blocks. The SFP ports establish a high-speed bidirectional serial communication link between two RT Boxes with a bandwith of 6.25 Gbps. To send signals from a model over a SFP port use the SFP Out. To receive signals from an SFP port and use it in a model use the SFP In.

../_images/splitup_subsystems.svg — Fig. 6 Model with communication over SFP

The SFP communication introduces a latency of two simulation timesteps (i.e. model B processes the information from model A two timesteps after it was processed by Model A). For offline simulations, these delays are modelled internally in the SFP blocks, so no external delay block is necessary anymore.

In this example, SFP port A is used on the first box running the rectifier and SFP port B is used on the second box running the inverter. This assumes that a SFP cable connects port A on the first box with port B on the second box.

Signal relaying

Signals can be exchanged between any two boxes that have a direct physical SFP connection. To exchange signals between boxes that are connected only via other boxes, signal relaying must be manually provided on the boxes in between:

../_images/splitup_relay.svg — Fig. 7 Signal relaying

In the case shown above the latency for a signal from Model A to Model C (and vice versa) is four simulation steps.

Time and Startup Synchronization

In addition to simulation data exchange, the SFP connection allows synchronizing the simulation time steps and the simulation startup between multiple RT Boxes. Synchronization requires assigning “primary” and “secondary” roles to SFP ports. For time step synchronization the clock signal from the primary box is distributed to a number of secondary boxes. All boxes with time step synchronization enabled must use the same simulation step time.

For startup synchronization the primary box first waits for all registered secondary boxes to become available (and optionally synchronized with the primary clock). Then the primary box sends a start signal to all secondary boxes simultaneously and also starts its own simulation model.

The assignment of primary and secondary roles is done in the Interconnect settings of the RT Box specific coder options. Please see section Interconnect for further details.

Note that one RT Box can connect to a primary box on one SFP port and act as a primary box for further RT Boxes connected to the other SFP ports. This results in a tree-like structure where the clock signal is distributed from the tree root to the subsequent levels. However, you should consider that clock jitter increases with each level that is added to the tree. Configuration settings for four RT-Boxes arranged in a tree with 3 levels are shown in the figure below.

../_images/sync_tree.svg — Fig. 8 Startup and time synchronization in a 3 level tree

Synchronized versus Unsynchronized Simulation

As stated above, the latency of a signal sent from one box to another box is two simulation steps when the two boxes use synchronized simulation time steps. This is the best solution for distributed simulations where the calculation time for each simulation step is roughly the same on all boxes.

Another common scenario is to use additional RT Boxes for I/O extension only. To minimize I/O latency it may be advisable to not use simulation time step synchronization. The RT Boxes running as I/O extender should run very simple models that only connect incoming SFP blocks to peripheral blocks and vice versa. These models can typically be run with a step size of just \(1\, \mu{\textrm s}\). In this case, the SFP latency is reduced to the step size of the fast I/O extenders plus the time needed for SFP data transmission (typically well below \(1\, \mu{\textrm s}\)).

Another scenario where time step synchronization is not recommended is when splitting the model into a fast part (e.g. for the electrical system) and a slow part (e.g. the mechanical system). Both parts can then run on two RT Boxes with different time steps.