~sbinet/talks

255786244fa7af564b3b608e010f47b19ce27521 — Sebastien Binet 3 years ago 9c2e8a8 chep-2018-paper
update
1 files changed, 77 insertions(+), 69 deletions(-)

M 2018/2018-07-09-chep-alice-fer/paper/chep-2018.tex
M 2018/2018-07-09-chep-alice-fer/paper/chep-2018.tex => 2018/2018-07-09-chep-alice-fer/paper/chep-2018.tex +77 -69
@@ 5,6 5,7 @@
%
\documentclass{webofc}
\usepackage[varg]{txfonts}   % Web of Conferences font
\usepackage[left]{lineno}

%% \usepackage{colortbl}
%% \usepackage[english]{babel}


@@ 20,6 21,7 @@
%
%
\begin{document}
\linenumbers
%
\title{Exploring polyglot software frameworks in ALICE with \texttt{FairMQ} and \texttt{fer}}
%


@@ 27,24 29,25 @@
%
%%%\subtitle{Do you have a subtitle?\\ If so, write it here}

\author{\firstname{Laurent} \lastname{Aphecetche}\inst{1}\fnsep\thanks{\email{laurent.aphecetche@subatech.in2p3.fr}} \and
        \firstname{S\'ebastien} \lastname{Binet}\inst{2}\fnsep\thanks{\email{binet@cern.ch}} \\
\author{
	%%\firstname{Laurent} \lastname{Aphecetche}\inst{1}\fnsep\thanks{\email{laurent.aphecetche@subatech.in2p3.fr}} \and
        \firstname{S\'ebastien} \lastname{Binet}\inst{1}\fnsep\thanks{\email{binet@cern.ch}} 
		on behalf of the ALICE collaboration
        % etc.
}

\institute{CNRS/IN2P3, Subatech
\and
\institute{
	%CNRS/IN2P3, Subatech
%\and
           CNRS/IN2P3, LPC, F-63000 Clermont-Ferrand, France
          }

\abstract{%
In order to meet the challenges of the Run-3 data rates and volumes, the ALICE collaboration is merging the online and offline infrastructures into a common framework: ALICE-O2.
O2 is based on FairRoot and FairMQ, a message-based, multi-threaded and multi-process control framework.
In order to meet the challenges of the Run 3 data rates and volumes, the ALICE collaboration is merging the online and offline infrastructures into a common framework: ALICE-$O^2$.
$O^2$ is based on FairRoot and FairMQ, a message-based, multi-threaded and multi-process control framework.
In FairMQ, processes (possibly on different machines) exchange data via message queues either through 0MQ or nanomsg.
FairMQ allows to build sophisticated topologies of processes: router/dealer, request/reply, publish/subscribe, client/server, etc...
In turn, this enables developers to write their reconstruction or analysis process in whatever language they choose or deem appropriate for the task at hand, as long as that programming language can send and receive data through these message queues.

This paper introduces \texttt{fer}, a Go-based toolkit that interoperates with the C++ toolkit FairMQ, to explore the realm of polyglot distributed frameworks.
}
%
\maketitle


@@ 52,24 55,26 @@ In turn, this enables developers to write their reconstruction or analysis proce
\section{Introduction}
\label{intro}

In order to meet the challenges of the Run-3 data rates and volumes, the ALICE collaboration is merging the online and offline infrastructures into a common framework: ALICE-O2~\cite{ref-aliceo2}.
O2 is based on FairRoot~\cite{ref-fairroot} and FairMQ~\cite{ref-fairmq}, a message-based, multi-threaded and multi-process control framework.
In FairMQ, processes (possibly on different machines) exchange data via message queues either through \texttt{ZeroMQ}~\cite{ref-zeromq} or \texttt{nanomsg}~\cite{ref-nanomsg}.
FairMQ allows to build sophisticated topologies of processes: router/dealer, request/reply, publish/subscribe, client/server, etc\ldots
The ALICE collaboration is merging the online and offline infrastructures into a common framework: ALICE-$O^2$~\cite{ref-aliceo2}.
This work is needed to meet the challenges of the Run 3 data rates and volumes.
$O^2$ is a message-based, multi-threaded and multi-process control framework based on two libraries: FairRoot~\cite{ref-fairroot} and FairMQ~\cite{ref-fairmq}.
Sophisticated processing pipelines can be modeled with FairMQ: router/dealer, request/reply, publish/subscribe, client/server, etc.
The nodes of these pipelines are processes (possibly on different machines) exchanging data via message queues either through \texttt{ZeroMQ}~\cite{ref-zeromq} or \texttt{nanomsg}~\cite{ref-nanomsg}.

In turn, this enables developers to write their reconstruction or analysis process in whatever language they choose or deem appropriate for the task at hand, as long as that programming language can send and receive data through these message queues.
From the standpoint of the FairMQ toolkit, the programming language used to implement a given process in the pipeline is irrelevant, as long as data is correctly propagated through the message queues. 
This enables developers to write their reconstruction or analysis process in whatever language they choose or deem appropriate for the task at hand.

This paper presents \texttt{fer}, a Go-based library compatible and interoperable with FairMQ.
We will start with a brief introduction of the builtin features that make Go a solid choice when dealing with I/O and concurrency.
We will then describe the principal components of \texttt{fer} and how they interact with C++ FairMQ.
Finally, we will report on the performances (CPU, VMem) of \texttt{fer} and conclude with the main figures of merit of \texttt{fer}, in the context of deployment in a distributed computing setup.
This paper presents \texttt{fer}, a Go-based~\cite{ref-golang} library compatible and interoperable with FairMQ.
This paper starts with a brief introduction of the builtin features that make Go a solid choice when dealing with I/O and concurrency.
The principal components of \texttt{fer} and how they interact with C++ FairMQ will then be described.
Finally, Sec.~\ref{sec-fer} will report on the performance (CPU, VMem) of \texttt{fer} and conclude with the main figures of merit of \texttt{fer}, in the context of deployment in a distributed computing setup.

\section{FairMQ concepts}
\label{sec-fairmq-concepts}

FairMQ is a distributed processing toolkit, written in \texttt{C++}, with pluggable transports (\texttt{ZeroMQ}, \texttt{nanomsg}, \texttt{shmem}, InfiniBand.)
FairMQ is a distributed processing toolkit, written in \texttt{C++}, with pluggable transports (\texttt{ZeroMQ}, \texttt{nanomsg}, \texttt{shmem}, InfiniBand).
In FairMQ, the bulk of the processing is performed by devices: UNIX processes that are connected with other devices through message queues.
Devices can be connected in various ways, through a wide variety of topologies: router/dealer, request/reply, publish/subscribe, client/server, etc\ldots.
Devices can be connected in various ways, through a wide variety of topologies: router/dealer, request/reply, publish/subscribe, client/server, etc.
The medium used for the connection between devices can be chosen among the following options: \texttt{tcp}, \texttt{udp}, \texttt{ipc}, \texttt{inproc}, shared-memory.
Figure~\ref{fig-topology} shows a typical FairMQ processing pipeline, fanning out data read off a source (\texttt{sampler}) to a pair of processors and then fanning in the data to a single output sink.



@@ 80,8 85,8 @@ Figure~\ref{fig-topology} shows a typical FairMQ processing pipeline, fanning ou
\label{fig-topology}
\end{figure}

FairMQ applications are currently configured via a \texttt{JSON} configuration file, declaring the devices, the ports they are listening on or sending data to, the type and topology of the message queues, etc\ldots
An example of such a configuration file is shown in figure~\ref{fig-cfg-json}.
FairMQ applications are currently configured via a \texttt{JSON} configuration file, declaring the devices, the ports they are listening on or sending data to, the type and topology of the message queues, etc.
An example of such a configuration file is shown in Fig.~\ref{fig-cfg-json}.

\begin{figure}[h]
\centering


@@ 114,10 119,10 @@ An example of such a configuration file is shown in figure~\ref{fig-cfg-json}.
\label{fig-cfg-json}       % Give a unique label
\end{figure}

Topologies in FairMQ can be described in JSON, like in figure~\ref{fig-cfg-json}, but other formats and mechanisms are available: XML or via the Dynamic Deployment System~\cite{ref-dds} (DDS.)
Devices are completely agnostic with regard to the technology used to modify, deploy and push the final configuration.
Once the user has written the correct configuration, declaring inputs and outputs of the whole pipeline, the \texttt{C++} device is launched, passed that configuration and essentially runs an infinite for-loop, waiting for data to be received, sending refined data down the pipeline, or waiting for external commands to control the finite state machine of a FairMQ application.
An abridged declaration of the \texttt{C++} base class for devices is shown in figure~\ref{fig-fairmq-cxx-device}.
Topologies in FairMQ can be described in JSON, like in Fig.~\ref{fig-cfg-json}, but other formats and mechanisms are available: XML or via the Dynamic Deployment System (DDS)~\cite{ref-dds}.
Devices are completely agnostic with regard to the technology used to modify, deploy, and push the final configuration.
Once the user has written the correct configuration, declaring inputs and outputs of the whole pipeline, the \texttt{C++} device is launched, is given that configuration and essentially runs an infinite \texttt{for} loop, waiting for data to be received, sending refined data down the pipeline, or waiting for external commands to control the finite state machine of a FairMQ application.
An abridged declaration of the \texttt{C++} base class for devices is shown in Fig.~\ref{fig-fairmq-cxx-device}.

\begin{figure}[h]
\centering


@@ 145,35 150,35 @@ protected:
\end{figure}

Users are supposed to at least override the \texttt{FairMQDevice::Run()} method in their derived classes.
This method is where data is sent to the downstream devices (using the \texttt{FairMQDevice::Send(\ldots)} method) or received from upstream devices (using the \texttt{FairMQDevice::Receive(\ldots)} method.)
As a device is instantiated within its own UNIX process, users are usually free to leverage any kind of library or programming paradigm inside the boundaries of their own memory address space.
This method is where data is sent to the downstream devices (using the \texttt{FairMQDevice::Send(\ldots)} method) or received from upstream devices (using the \texttt{FairMQDevice::Receive(\ldots)} method).
As a device is usually instantiated within its own UNIX process, users are usually free to leverage any kind of library or programming paradigm inside the boundaries of their own memory address space.

With FairMQ's microservice-like architecture, processing pipelines can horizontally scale much more easily when data taking or data processing demand it.
Indeed, it is just a matter of adding more processing resources to the pool and connect them via message queues to accomodate a change in the processing load.
Abstracting away the minute details of how devices are connected and distributed (all in the same address space, on different machines, etc\ldots) allows to very easily isolate a faulty device and mitigate its inpact on others, \emph{e.g.} preventing it from corrupting memory of other devices.
This also allows to use processes' boundaries to segregate components that are multithreaded friendly from others that are not.
Abstracting away the minute details of how devices are connected and distributed (all in the same address space, on different machines, etc.) allows users to very easily isolate a faulty device and mitigate its impact on others, \emph{e.g.} preventing it from corrupting memory of other devices.
This also allows to use processes' boundaries to segregate components that are multithread friendly from others that are not.

On one hand, requiring that data is communicated between devices through these message queues, healthly constrains the set of \texttt{C++} features one can use to implement the data model.
Indeed, the data being exchanged will tend to resemble plain old data (POD.)
On the other hand, relying on off the shelf libraries such as \texttt{ZeroMQ} or \texttt{nanomsg} to implement the actual data transport between devices, enables to implement devices in any language that has support for \texttt{ZeroMQ} or \texttt{nanomsg}.
A FairMQ pipeline could theoretically be composed of $n$ devices, each device implemented in a different programming language, presumably one that fits best for the task at hand.
On one hand, requiring that data is communicated between devices through these message queues healthly constrains the set of \texttt{C++} features one can use to implement the data model.
Indeed, the data being exchanged will tend to resemble plain old data (POD).
On the other hand, relying on off the shelf libraries such as \texttt{ZeroMQ} or \texttt{nanomsg} to implement the actual data transport between devices, enables the implementation of devices in any language that has support for \texttt{ZeroMQ} or \texttt{nanomsg}.
A FairMQ pipeline could theoretically be composed of $n$ devices, each device implemented in a different programming language, presumably one that fits best the task at hand.

\texttt{fer} is investigating the feasability of a \texttt{Go}~\cite{ref-golang} based toolkit, interoperable with FairMQ.
In the following, the feasability of a \texttt{Go}~\cite{ref-golang} based toolkit, interoperable with FairMQ, \texttt{fer}, is investigated.

\section{Elements of Go}
\label{sec-intro-go}
Go is an open source general programming language with built-in support for concurrency programming.
It has been released in 2009 under the BSD-3 license.
Go has a static type system with first-class functions, closures and interfaces.
Go has a static type system with first-class functions, closures, and interfaces.
The general syntax of Go is close to that of C and C++.
Even if Go programs are statically typed, code verbosity is limited thanks to Go's type inference system.
Go's concurrency primitives derive from Hoare's Communicating Sequential Processes~\cite{ref-csp} (CSP): \emph{goroutines} and \emph{channels}.
Go's concurrency primitives derive from Hoare's Communicating Sequential Processes (CSP)~\cite{ref-csp}: \emph{goroutines} and \emph{channels}.
Channels are typed conduits that connect goroutines together.
Goroutines are very lightweight green threads: it is possible to schedule thousands of them on regular hardware.
This is achieved thanks to the Go runtime multiplexing goroutines on OS threads and the ability to grow and shrink each goroutine's stack as needed.
The last element for easy concurrency programming is the garbage collector that takes care of freeing memory from the correct thread, when it is not needed anymore.

Go is available on all major platforms (Linux, Windows, macOS, Android, iOS, \ldots) and for many architectures (\texttt{amd64}, \texttt{arm64}, \texttt{i386}, \texttt{s390x}, \texttt{mips64}, \ldots)
Go is available on all major platforms (Linux, Windows, macOS, Android, iOS, etc.) and for many architectures (\texttt{amd64}, \texttt{arm64}, \texttt{i386}, \texttt{s390x}, \texttt{mips64}, etc.).
The Go toolchain can very easily cross-compile from one OS/Arch pair to any other pair, by just modifying two environment variables.
The Go toolchain generates completely statically compiled binaries.
This enables a very efficient deployment model where binaries produced on a developer laptop can be quickly provisioned on a bare metal production cluster.


@@ 195,13 200,13 @@ Indeed, installing Go packages is usually just a matter of entering in the shell

A single command that works on all platforms and operating systems supported by the Go toolchain.

\section{\texttt{fer}}
\section{Design and components of \texttt{fer}}
\label{sec-fer}

As introduced in section~\ref{sec-fairmq-concepts}, implementing a toolkit compatible with FairMQ requires to be able to configure FairMQ-like devices, using \emph{e.g.} the JSON configuration file format; to create topologies of devices according to the configuration and connected via a transport library (\emph{e.g.} \texttt{ZeroMQ} or \texttt{nanomsg}); and finally to execute the main routines of the devices.
As introduced in Sec.~\ref{sec-fairmq-concepts}, implementing a toolkit compatible with FairMQ requires to be able to configure FairMQ-like devices, using \emph{e.g.} the JSON configuration file format; to create topologies of devices according to the configuration and connected via a transport library (\emph{e.g.} \texttt{ZeroMQ} or \texttt{nanomsg}); and finally to execute the main routines of the devices.

\texttt{fer}~\cite{ref-fer} is a toolkit implemented in Go that realizes the requirements above.
Similar to its \texttt{C++} sibling, \texttt{fer} exposes the concept of a device by way of the \texttt{Device} interface, as shown in figure~\ref{fig-fer-device}.
The requirements above are met by \texttt{fer}~\cite{ref-fer}, a toolkit implemented in Go.
Similar to its \texttt{C++} sibling, \texttt{fer} exposes the concept of a device by way of the \texttt{Device} interface, as shown in Fig.~\ref{fig-fer-device}.

\begin{figure}[h]
\centering


@@ 224,10 229,10 @@ type DevReseter    interface { Reset(ctl Controler) error }
\label{fig-fer-device}
\end{figure}

A type implements the \texttt{fer.Device} interface as soon as it has a method named \texttt{Run()} that takes a \texttt{Controler} interface and returns an \texttt{error}.
The \texttt{Controler} interface is shown in figure~\ref{fig-fer-controler}.
A type implements the \texttt{Device} interface as soon as it has a method named \texttt{Run()} that takes a \texttt{Controler} interface and returns an \texttt{error}.
The \texttt{Controler} interface is shown in Fig.~\ref{fig-fer-controler}.
It is used to retrieve named channels bound to incoming or outgoing data, through the \texttt{Chan()} method.
An additional \texttt{Done()} method exposes transitions inthe \texttt{fer} final state machine (\texttt{RUN}, \texttt{PAUSE}, \texttt{STOP}, \ldots)
An additional \texttt{Done()} method exposes transitions in the \texttt{fer} final state machine (\texttt{RUN}, \texttt{PAUSE}, \texttt{STOP}, etc.)

\begin{figure}[h]
	\centering


@@ 249,7 254,7 @@ type Msg struct {
// Cmd describes commands to be sent to a device, via a channel.
type Cmd byte
	\end{minted}
	\caption{Declaration of the \texttt{Logger} and \texttt{Controler} interfaces, in the \texttt{fer} package.
	\caption{Declaration of the \texttt{Logger} and \texttt{Controler} interfaces in the \texttt{fer} package.
	\texttt{Controler}s expose communication channels that can exchange data messages, \texttt{Msg}, that embed the actual payload or an error if any.
}
	\label{fig-fer-controler}


@@ 259,12 264,12 @@ The input configuration is parsed by the \texttt{fer/config} package and passed 

\subsection{Implementing a \texttt{fer} device}
This section shows how implementing a \texttt{fer} device and integrating it inside a \texttt{C++} pipeline can be done.
Figure~\ref{fig-fer-processor} shows how one can implement the \texttt{processor} from figure~\ref{fig-topology}.
Figure~\ref{fig-fer-processor} shows how one can implement the \texttt{processor} from Fig.~\ref{fig-topology}.
The \texttt{processor} retrieves two channels, the \texttt{"data1"} input channel and the \texttt{"data2"} output channel.
The \texttt{Run()} method contains an infinite for-loop.
At each iteration, the device waits for data coming in from one of the two channels.
When the input channel is ready -- a message has been delivered to the device -- the associated payload is modified by the device (just appending some data) and then sent downstream \emph{via} the output channel \texttt{odatac}.
Concurrently, if any command is sent on the \texttt{Done()} channel, the for-loop exits and the Go runtime can reclaim resources used by the associated goroutine.
The \texttt{Run()} method contains an infinite \texttt{for} loop.
At each iteration, the device waits for data coming in from either the \texttt{idatac} or the \texttt{Done()} channel.
When the input channel is ready -- a message has been delivered to the device -- the associated payload is modified by the device (just appending some data) and then sent downstream through the output channel \texttt{odatac}.
Concurrently, if any command is sent on the \texttt{Done()} channel, the \texttt{for} loop exits and the Go runtime can reclaim resources used by the associated goroutine.

\begin{figure}[h]
	\centering


@@ 313,8 318,8 @@ func (dev *processor) Run(ctl fer.Controler) error {
\label{fig-fer-processor}
\end{figure}

Once a type implements the \texttt{fer.Device} interface, a value can be created and passed to the \texttt{fer.Main} function that takes care of the minute details of scheduling devices, establishing socket connections and forwarding external commands.
This simplifies the creation of the main entry point for the whole program, as shown in figure~\ref{fig-fer-main}.
Once a type implements the \texttt{fer.Device} interface, a value of that type can be created and passed to the \texttt{fer.Main} function that takes care of the minute details of scheduling devices, establishing socket connections and forwarding external commands.
This simplifies the creation of the main entry point for the whole program, as shown in Fig.~\ref{fig-fer-main}.

\begin{figure}[h]
	\centering


@@ 332,19 337,21 @@ func main() {
	\label{fig-fer-main}
\end{figure}

Compiling and running this processor is done like so:
Compiling and running this processor is done via:
\begin{minted}{bash}
%> go get ./my-device
%> $GOPATH/bin/my-device --id processor --mq-config ./path-to/config.json
\end{minted}

At the beginning of the development of \texttt{fer}, a pure-Go implementation of the \texttt{nanomsg} package was readily available but not for \texttt{ZeroMQ}.
\subsection{Go implementation of \texttt{ZeroMQ}}

At the beginning of the development of \texttt{fer}, a pure Go implementation of the \texttt{nanomsg} package was readily available but not for \texttt{ZeroMQ}.
Only a so-called \texttt{cgo} based package existed, one that linked to the C/C++ \texttt{ZeroMQ} library and wrapped the C/C++ calls with Go functions and types.
Calling Go functions that call C/C++ and back is not only costly -- around ten times a normal function call, it also breaks the easy compilation, installation and deployment promises that Go users are accustomed to.
To correct this deficiency, we developed a pure-Go package that implements the required subset of the \texttt{ZeroMQ} protocol and sockets to provide interoperability with the C++ implementation: \texttt{go-zeromq/zmq4}~\cite{ref-go-zmq}.
Calling Go functions that call C/C++ and back is not only costly -- around ten times a normal function call -- it also breaks the easy compilation, installation, and deployment promises that Go users are accustomed to.
To correct this deficiency, a pure Go package that implements the required subset of the \texttt{ZeroMQ} protocol and sockets to provide interoperability with the C++ implementation was developed: \texttt{go-zeromq/zmq4}~\cite{ref-go-zmq}.

As shown in figure~\ref{fig-tof-tcp}, the mean value for the time of flight of a single \texttt{uint64} token exchanged between a source device, a processor device and a sink device, goes from $\sim 70\mu s$ for the \texttt{czmq} C++ version (called from Go) down to $\sim 45\mu s$ for the pure-Go \texttt{nanomsg} and \texttt{zeromq} transports.
The maximum resident set size (\texttt{MaxRSS} from \texttt{/proc/self/statm}) for all configurations was around 22Mb.
As shown in Fig.~\ref{fig-tof-tcp}, the mean value for the time of flight of a single \texttt{uint64} token exchanged between a source device, a processor device and a sink device, goes from $\sim 70~\mu$s for the \texttt{czmq} C++ version (called from Go) down to $\sim 45~\mu$s for the pure Go \texttt{nanomsg} and \texttt{zeromq} transports.
The maximum resident set size (\texttt{MaxRSS} from \texttt{/proc/self/statm}) for all configurations was around 22~Mb.

\begin{figure}[h]
\centering


@@ 357,25 364,26 @@ The maximum resident set size (\texttt{MaxRSS} from \texttt{/proc/self/statm}) f
\end{figure}

\section{Conclusions}
\label{sec-conclusions}

The AliceO2 framework based on the message passing infrastructure from FairMQ enables to build applications that resemble microservices architectures.
This type of architectures allows to scale horizontally quite easily, by just adding new processor devices and fanning out data to these new processors, in \emph{e.g.} a round-robin fashion.
The ALICE-$O^2$ framework based on the message passing infrastructure from FairMQ enables one to build applications that resemble microservices architectures.
These types of architectures allow systems to scale horizontally quite easily, by just adding new processor devices and fanning out data to these new processors, in \emph{e.g.} a round-robin fashion.
FairMQ, and microservices in general, is language agnostic: developers can thus implement their device in whatever language they deem adequate for the task at hand.
We introduced \texttt{fer}, a Go-based toolkit that interoperates with the C++ toolkit FairMQ, to explore the realm of polyglot distributed frameworks.
\texttt{fer} provides a fast, "one-command" installation procedure that works on all platforms and operating systems supported by the Go toolchain.
This paper introduced \texttt{fer}, a Go-based toolkit that interoperates with the C++ toolkit FairMQ, to explore the realm of polyglot distributed frameworks.
The \texttt{fer} toolkit provides a fast, "one-command" installation procedure that works on all platforms and operating systems supported by the Go toolchain.
It also provides Grid-friendly deployments thanks to its single, statically compiled, binary.
As such it can be a great testbed environment to explore new avenues.


\begin{thebibliography}{}

\bibitem{ref-aliceo2} M. Al-Turany \emph{et al.}, Journal of Physics: Conference Series, \textbf{Volume 664}, (2015)
\bibitem{ref-fairroot} M. Al-Turany \emph{et al.}, Journal of Physics: Conference Series, \textbf{Volume 396}, (2012)
\bibitem{ref-fairmq} \texttt{https://github.com/FairRootGroup/FairMQ}
\bibitem{ref-zeromq} \texttt{http://zeromq.org/}
\bibitem{ref-nanomsg} \texttt{https://nanomsg.org/}
\bibitem{ref-dds} DDS: The Dynamic Deployment System, \texttt{doi:10.15120/GR-2015-1-INFRASTRUCTURE-IT-04}
\bibitem{ref-golang} The \texttt{Go} programming language, \texttt{https://golang.org}
\bibitem{ref-aliceo2} M. Al-Turany \emph{et al.} ALFA: The new ALICE-FAIR software framework, Journal of Physics: Conference Series, \textbf{Volume 664}, 072001 (2015)
\bibitem{ref-fairroot} M. Al-Turany \emph{et al.} The FairRoot framework, Journal of Physics: Conference Series, \textbf{Volume 396}, 022001 (2012)
\bibitem{ref-fairmq} \texttt{https://github.com/FairRootGroup/FairMQ}, visited: August 2018
\bibitem{ref-zeromq} \texttt{http://zeromq.org/} (\texttt{v4.2.5}, visited: August 2018)
\bibitem{ref-nanomsg} \texttt{https://nanomsg.org/} (\texttt{v1.1.5}, visited: August 2018)
\bibitem{ref-dds} A. Manafov and A. Lebedev, DDS: The Dynamic Deployment System, GSI Report 2015-1, p 371 (2015) \texttt{doi:10.15120/GR-2015-1-INFRASTRUCTURE-IT-04}
\bibitem{ref-golang} The \texttt{Go} programming language, \texttt{https://golang.org} (\texttt{v1.11}, visited: August 2018)
\bibitem{ref-csp} C. A. R. Hoare, Communications of the ACM \textbf{Volume 21} Issue 8, pp 666-677 (1978)
\bibitem{ref-fer} \texttt{https://github.com/sbinet-alice/fer}, \texttt{doi:10.5281/zenodo.1470586}
\bibitem{ref-go-zmq} \texttt{https://github.com/go-zeromq/zmq4}, \texttt{doi:10.5281/zenodo.1470580}