94 lines
10 KiB
TeX
94 lines
10 KiB
TeX
|
|
\section{MWRE Architecture}\label{sec:workflow_environment}
|
|
|
|
|
|
\begin{wrapfigure}{R}{0.5\textwidth}
|
|
\centering
|
|
\includegraphics[width=4cm,keepaspectratio=true]{./mwre/figures/Povray.pdf}
|
|
\caption{The POV-Ray workflow.}
|
|
\label{fig:ex-povray}
|
|
\end{wrapfigure}
|
|
|
|
Based on the requirements outlined in the introduction, we designed a \emph{Manycore Workflow Runtime Engine (MWRE)} tailored to heterogeneous shared-memory manycore systems. Our main design principle was to provide a set of feature similar to the ones found in the current DCI workflow engines, while addressing the special characteristics and requirements of manycore systems. In this section we discuss the basic architecture of our workflow system and give short descriptions of the components.
|
|
|
|
We use the Persistence of Vision Raytracer (POV-Ray) workflow as a running example to illustrate the workflow representation, translation and enactment.
|
|
POV-Ray~\cite{Plachetka98-PVPR} is a free tool for creating three-dimensional graphics, which is known to be a time-consuming process used not only by hobbyists and artists, but also in biochemistry research, medicine, architecture, and mathematical visualization. We modelled a POV-Ray rendering scenario as a workflow depicted in Figure~\ref{fig:ex-povray}, where the description of a movie can be separated in several scenes, each scene being composed of several frames that can be rendered (e.g. in \texttt{.png} format) in a \texttt{parallel for} loop. The \texttt{Scene} activity before the \texttt{Render} returns the scene file for a given scene name.
|
|
Finally, all frames are merged into a \texttt{.mpg} movie using a \texttt{Convert} activity (e.g. running a \texttt{png2yuv} followed by an \texttt{ffmpeg} conversion).
|
|
|
|
Figure~\ref{fig:overall-architecture} presents the overall architecture of the MWRE consisting of three main parts: \circled{1} the workflow specification, \circled{2} a source-to-source compiler, and \circled{3} the workflow runtime environment.
|
|
The main difference between the MWRE's architecture and the traditional DCI workflow engines is the source-to-source compiler that generates a native C++ program from the workflow specification. Furthermore, MWRE does the mapping of abstract activity types to concrete activity implementations at compile-time, instead of runtime as done by most DCI engines.
|
|
|
|
\begin{figure}[t]
|
|
\centering
|
|
\includegraphics[keepaspectratio=true, width=\textwidth]{./mwre/figures/overall-architecture.pdf}
|
|
\caption{MWRE architecture.}
|
|
\label{fig:overall-architecture}
|
|
\end{figure}
|
|
|
|
|
|
\subsection{Workflow specification}
|
|
The input to MWRE is a workflow specification (\circled{1}) encoded in the Interoperable Workflow Intermediate Representation (IWIR)~\cite{Plankensteiner2013}. IWIR is an intermediate workflow specification language designed in the European SHIWA project\footnote{\url{http://www.shiwa-workflow.eu/}} to enable interoperability of different workflow systems. As most workflow environments require workflows to be written in their respective workflow language to be able to execute them, exchanging workflows usually means that the workflow specification needs to be translated from one workflow language to another. IWIR eases this translation process by providing a common intermediate representation and thus reducing the number of required translators from $2\cdot n^2$ to $2\cdot n$. IWIR is currently supported by four workflow systems: ASKALON~\cite{Fahringer2007}, MOTEUR~\cite{glatard2008flexible}, Triana~\cite{taylor2003triana} and WS-PGRADE~\cite{kacsuk2011p}.
|
|
|
|
Using IWIR to specify MWRE workflows has several advantages allowing reusing existing tool chains with no need to implement existing functionality.
|
|
First, because it is designed for interoperability, it captures all concepts and constructs found in most workflow languages, and can therefore be seen as a superset of the major workflow languages. Second, it is well-defined and easy to parse. Third, it allows automatic translation and reuse of existing workflows for experiments across different systems without further modifications.
|
|
Fourth, it allows integration of new languages and new platforms with O(1) complexity through IWIR front-end or back-end support. And fifth, the implementation of a MWRE workflows is not tied to a specific development environment, but domain scientists can use their favourite workflow development tools from any workflow environment that supports IWIR translation.
|
|
|
|
|
|
\lstset{ %
|
|
backgroundcolor=\color{white}, % choose the background color; you must add \usepackage{color} or \usepackage{xcolor}
|
|
basicstyle=\scriptsize\ttfamily, % the size of the fonts that are used for the code
|
|
breakatwhitespace=true, % sets if automatic breaks should only happen at whitespace
|
|
breaklines=true, % sets automatic line breaking
|
|
captionpos=b, % sets the caption-position to bottom
|
|
commentstyle=\color{black}, % comment style
|
|
deletekeywords={...}, % if you want to delete keywords from the given language
|
|
escapeinside={(*@}{@*)}, % if you want to add LaTeX within your code
|
|
extendedchars=true, % lets you use non-ASCII characters; for 8-bits encodings only, does not work with UTF-8
|
|
frame=none, % adds a frame around the code
|
|
keepspaces=false, % keeps spaces in text, useful for keeping indentation of code (possibly needs columns=flexible)
|
|
keywordstyle=\color{black}, % keyword style
|
|
otherkeywords={*,...}, % if you want to add more keywords to the set
|
|
numbers=left, % where to put the line-numbers; possible values are (none, left, right)
|
|
numbersep=2pt, % how far the line-numbers are from the code
|
|
numberstyle=\tiny\color{gray}, % the style that is used for the line-numbers
|
|
rulecolor=\color{black}, % if not set, the frame-color may be changed on line-breaks within not-black text (e.g. comments (green here))
|
|
showspaces=false, % show spaces everywhere adding particular underscores; it overrides 'showstringspaces'
|
|
showstringspaces=false, % underline spaces within strings only
|
|
showtabs=false, % show tabs within strings adding particular underscores
|
|
stepnumber=1, % the step between two line-numbers. If it's 1, each line will be numbered
|
|
stringstyle=\color{black}, % string literal style
|
|
tabsize=2, % sets default tabsize to 2 spaces
|
|
columns=fullflexible,
|
|
}
|
|
|
|
\lstdefinelanguage{XML2}{
|
|
language=XML,
|
|
commentstyle=\color{gray}\upshape,
|
|
morestring=[b][\color{black}\itshape\mdseries]",
|
|
morestring=[s][\bfseries]{>}{<},
|
|
morecomment=[s]{<?}{?>},
|
|
stringstyle=\color{black},
|
|
identifierstyle=\color{blue}\bfseries,
|
|
keywordstyle=\color{black}\bfseries, % keyword style
|
|
tagstyle=\color{black}\bfseries,
|
|
}
|
|
|
|
\begin{listing}
|
|
\resizebox*{!}{\textheight}{
|
|
\lstinputlisting[language=XML2]{mwre/povray.iwir}}
|
|
\caption{IWIR specification of the POV-Ray workflow.}
|
|
\label{lst:povray_iwir}
|
|
\end{listing}
|
|
|
|
Listing~\ref{lst:povray_iwir} shows the IWIR specification of our POV-Ray workflow. The \texttt{toplevel} composite activity represents the start of the workflow and is defined in lines~\ref{povray:iwir:toplevel:start}~--~\ref{povray:iwir:toplevel:end}. The input ports of the \texttt{toplevel} activity (one file and two integers) are defined in lines~\ref{povray:iwir:toplevel:inports:start}~--~\ref{povray:iwir:toplevel:inports:end} and also represent the workflow input ports. A real-world POV-Ray workflow has more input parameters, but we omitted them for brevity reasons. The output ports of the \texttt{toplevel} activity (one file) defined in lines~\ref{povray:iwir:toplevel:outports:start}~--~\ref{povray:iwir:toplevel:outports:end} again represent the workflow output ports too. Lines~\ref{povray:iwir:pfor:start}~--~\ref{povray:iwir:pfor:end} represent the \texttt{parallel for} \texttt{ParallelFor} loop activity containing the atomic activities \texttt{SceneFile} (lines~\ref{povray:iwir:SceneFile:start}~--~\ref{povray:iwir:SceneFile:end}) and \texttt{Render} (lines~\ref{povray:iwir:Render:start}~--~\ref{povray:iwir:Render:end}). The loop activity is then followed by the atomic activity \texttt{Convert}, represented in lines~\ref{povray:iwir:Convert:start}~--~\ref{povray:iwir:Convert:end}.
|
|
|
|
\subsection{Source-to-source compiler}
|
|
The IWIR workflow specification is then given to the source-to-source compiler (\circled{2}), which translates it into a C++ \emph{workflow program} using the API provided by MWRE. We believe that compiling the workflow application into a native executable program is the way to achieve the ``best'' performance on shared-memory manycore systems while minimizing enactment latencies and other middle-ware overheads. The compiler is also responsible for mapping the abstract workflow activity types to concrete activity implementations. Performing this task at compile-time, instead of runtime as done by most DCI workflow systems, saves additional overhead and eases configuration of the workflow program. For this purpose, the compiler has access to a repository that contains activity implementations in form of pre-compiled libraries, binaries, GPU kernels (i.e. \texttt{CUDA}, \texttt{OpenCL}), source-code snippets, and shell-scripts. The compiler also allows a $1$-to-$n$ mapping of activity types to implementations. Meta-data stored for each mapping allows the scheduler to efficiently select the most suitable implementation at runtime.
|
|
Using a source-to-source compiler also allows applying transformations to the workflow structure to optimize the performance of the generated workflow program, similar to the compiler optimisations for shared memory systems.
|
|
We describe the source-to-source compiler in Section~\ref{sec:Source-to-Source-Compiler} in detail.
|
|
|
|
|
|
\subsection{Runtime environment}
|
|
The runtime environment of MWRE (\circled{3}) invokes and executes the workflow program created by the source-to-source compiler. It is implemented as a C++ library considering several performance concerns, such as overhead, memory footprint, and resource utilization.
|
|
We describe MWRE workflow engine in Section~\ref{sec:MWRE-EnactmentModel} in detail.
|