View on GitHub

CoSimIO

Small standalone tool for interprocess communication in CoSimulation contexts for coupling and exchanging data between different solvers or other software-tools.

Communication

Main Page of Documentation

Table of Contents


Exchanging data between two codes using interprocess communication (IPC) is the main task of the CoSimIO. Different methods with specific advantags / disadvantages exist, see below. The default way of communicating is by using sockets, see here.

When initially establishing a connection between two codes, a handshake of both partners is performed. During this handshake some basic information is exchanged between the partners, including the check if the versions of CoSimIO are compatible. The handshake is always done via files as it is the most robust way of communication.

For the partners to connect without problems, it is recommended to start the connection process close to each other and not with too much time delay.

One connection partner acts as the primary connection, the other one is the secondary. This is only required for some technical details, they both have the exact same function and functionalities. Examples of those details:

By default it is automatically determined from the names of the partners which is the primary and which is the secondary. In some cases it can be useful to specify this explicitly, this can be done with the setting is_primary_connection. Either both partners or none can specify this setting. Note that specifying both partners as either primary or secondary will lead to a deadlock.

It is important to mention that leftover files (e.g. when one side crashes) can interfere with following executions, as these leftover files can be mistakenly read. A remedy (but not a solution) is to use a dedicated folder where the files are written (this is the default behavior and can be changed with the setting use_folder_for_communication, see below). The folder is created (or deleted and recreated if it exists from a previous execution) at the beginning of the execution by the primary connection, the files for the data exchange are then written inside this folder. At the end of the execution it is automatically deleted. This way it is less likely that leftover files from previous executions interfere.
Note however that also in this case it is possible that a leftover folder can cause issues. Two safe methods exist to avoid this, but they require more input / actions from the user:

The implementation of the different methods for Communication can be found here.

Input for all communication methods:

The following settings are available for all methods of communication:

name type required default description
my_name string x - necessary for establishing a connection
connect_to string x - name of partner to connect to
communication_format string - file select format of communication, see below
is_primary_connection bool - determined from other input whether this is the primary connection, if not specified it is determined automatically from the names of the partners
working_directory string - current working directory path to the working directory
use_folder_for_communication bool - true whether the files used for communication are written in a dedicated folder. Deadlocks from leftover files from previous executions are less likely to happen as they can be cleanup up.
always_use_serializer bool - false use the Serializer also when it is not necessary, e.g. for basic types such as Im-/ExportData. This is ~ 10x slower but more stable, especially when combined with ascii-serialization
serializer_trace_type string - no_trace mode for the Serializer: no_trace (fastest method, binary format, without any debugging checks), ascii (ascii format, without any debugging checks), trace_error (ascii format, checks are enabled), trace_all (ascii format, checks are enabled and printed, hence very verbose!)
echo_level int - 0 decides how much output is printed
print_timing bool - false whether timing information should be printed

File-based communication

As the name indicates, this method uses files for communicating data. It is robust and useful for debugging. It is the preferred method for implementing the CoSimIO in a new code, as it is intuitive to follow the flow of data through the files. In order to prevent race conditions when accessing the files, two mechanisms are available (which can be selected with use_aux_file_for_file_availability):

The implementation of the FileCommunication can be found here.

Specific Input:

Set communication_format to file.

name type required default description
use_aux_file_for_file_availability bool - Windows: true; Unix: false select whether files are made available by use of an auxiliary file or via rename.
use_file_serializer bool - true Using the FileSerializer (which directly uses a file stream to read/write data) over the StreamSerializer (which first to reads/writes to a stringstream before writing everything to the file at once)

Socket-based communication

The data is communicated through network sockets by using the TCP communication protocol (using IPv4). No data is written to the filesystem, this makes it more efficient than the file-based communication. Important: By default the loopback interface / localhost is used, which cannot communicate across compute nodes. Hence for using this form of communication on a cluster or other system with multiple compute nodes, it is required to specify which network should be used (e.g. infiniband), see below how to specify the input. Otherwise it will hang! On most linux systems the ifconfig command can be used to check the available networks. Otherwise contact your system administrator.

The ASIO library is used as a high level interface for the sockets.

The implementation of the SocketCommunication can be found here.

Specific Input:

Set communication_format to socket.

name type required default description
ip_address string - “127.0.0.1” specify the ip address used to establish the connection
network_name string - - the name of the network can be specified alternatively to specifying the ip address. This is used to determine the ip address. Will print the available networks if a wrong name is specified.

The following logic is used for selecting the ip-address

  1. If the user has specified ip_address, then this one is used directly
  2. If instead the network_name is specified, then this is used to determine the ip-address to be used
  3. If neither is specified, then the localhost ip-address is used (127.0.0.1)

Unix domain socket-based communication

This form of communication is experimental

This type of communication uses unix domain sockets for the data exchange. It is similar to the tcp socket communication, but instead of passing the data through the network, it uses the kernel memory. This makes it faster, but at the same time it only works locally on one compute node.

The ASIO library is used as a high level interface for the sockets.

This form of communication is currently only available under Unix, the Windows implementation is work in progress.

The implementation of the LocalSocketCommunication can be found here.

Important: This form of communication does not support distributed memory machines!

Specific Input:

Set communication_format to local_socket.

name type required default description
currently_nothing        

Pipe-based communication

This form of communication is experimental

A pipe is a data channel to perform interprocess communication between two processes. No data is written to the filesystem, it is directly exchanged through the kernel memory. This makes it more efficient than the file-based communication, but at the same time it only works locally on one compute node.

The (default) buffer size is specific for each operating system, see e.g. here for information for Linux. If the data to be exchanged is larger than the buffer size, then it is exchanged in chuncks. Hence a larger buffer size results in less data exchanges.

This form of communication is currently only available under Unix, the Windows implementation is work in progress.

The implementation of the PipeCommunication can be found here.

Important: This form of communication does not support distributed memory machines!

Specific Input:

Set communication_format to pipe.

name type required default description
buffer_size int - Linux: 65536 (64 KB); others: 8192 (8 KB) buffer size of pipe, differs between OSs.

MPI-based communication

This form of communication is experimental

MPI is usually used to communicate between different ranks within an executable/one MPI-communicator. MPI 2.0 added functionalities with which the communication can be done also between independent communicators (i.e. if two executables were started separately with MPI as shown below). This can be done similarly to the socket based communication through opening ports and accepting connection (on the primary/server side) and connecting to the opened port (on the secondary/client side). After the connection is established, communication is done with the standard MPI calls like MPI_Send and MPI_Recv. This is oftentimes the fastest way of exchanging data in a distributed memory environment.

The disadvantage of this form of communication is that the features required for establishing communication across communicators are not robustly available for all MPI implementations. Experience shows that it is problematic with OpenMPI but works well with IntelMPI. Furthermore it might be required to use the same compilers and MPI implementation for successfully connecting.

As this form of communication is less robust and relies on MPI features that might not be available on older systems, it is required to manually enable it at compile time with CO_SIM_IO_BUILD_MPI_COMMUNICATION.

This form of communication is based on MPI and is hence only available if a connection is established with CoSimIO::ConnectMPI.

The two executables are expected to be started with separate MPI calls:

mpiexec -np 4 ./execubtable_1  &  mpiexec -np 4 ./execubtable_2

OpenMPI works only with very recent versions (4.1) and requires additionally to start an ompi-server:

ompi-server -r server.txt
mpiexec --ompi-server file:server.txt -np 4 ./execubtable_1  &  mpiexec --ompi-server file:server.txt -np 4 ./execubtable_2

The implementation of the MPIInterCommunication can be found here.

Specific Input:

Set communication_format to mpi_inter.

name type required default description
currently_nothing