48 lines
5.0 KiB
TeX
48 lines
5.0 KiB
TeX
|
|
\section{Workflow Model}
|
|
|
|
The design and implementation of scientific workflows consists of two parts: an \emph{abstract part} and a \emph{concrete part}. We give a short overview of these two parts in the following subsections.
|
|
|
|
\begin{figure}{ht}
|
|
\centering
|
|
\includegraphics[width=.5\textwidth]{./background/figures/workflow-abstractpart.pdf}
|
|
\caption{An abstract part of a scientific workflow.}
|
|
\label{fig:dag}
|
|
\end{figure}
|
|
|
|
\subsection{Workflow design: the abstract part}
|
|
The abstract part (see Figure~\ref{fig:dag}) of a scientific workflow specification comprises a hardware and middleware agnostic (and therefore easily portable) description of the workflow structure, the activities involved (identified by a unique name and a type), and the data and control flow dependencies between the activities. The individual activities are treated as black-boxes such that only the input and the output signatures are known.
|
|
|
|
There are usually two different types of workflow activities:
|
|
\begin{enumerate}
|
|
\item \emph{Atomic activities} are basic indivisible computations units such as a legacy codes;
|
|
\item \emph{Composite activities} combine several fine granular workflow activities, including atomic and other composite ones, to form coarse granular activities and impose a control flow on the contained inner activities.
|
|
\end{enumerate}
|
|
Typical composite activities are sequential and parallel loops, conditional activities, and sub-workflows.
|
|
|
|
|
|
\subsection{Workflow implementation: the concrete part}
|
|
The concrete part of a workflow contains the hardware and middleware-dependent implementations of the atomic activities and their accompanying meta-information. This part is often highly specific to each individual workflow system and the underlying DCI. It usually contains information about the available activity implementations, locations where they are installed, how they can be executed, and any other further information indented to help the workflow engine in selecting the most appropriate activity implementation.
|
|
|
|
|
|
\subsection{Workflow enactment}
|
|
A workflow engine executes a workflow instance (operation usually called \emph{workflow enactment}) by traversing the DAG representing the workflow structure, determining the state of the individual activities, transferring data from finished activities to their successors in the dependency graph, unrolling composite activities and replacing them with the resulting subgraph, and delegating the actual execution of atomic activities to the scheduling and execution subsystems.
|
|
We call the resulting DAG where composite activities have been replaced with their contained subgraphs enriched with additional state information a \emph{Workflow Execution Plan} (WEP).
|
|
|
|
We distinguish between two types of workflow enactment modes:
|
|
\begin{itemize}
|
|
\item In \emph{early enactment mode} the engine re-evaluates the WEP as soon as there are activity state changes, and tries to complete it as early as possible. This mode usually comes with a much higher overhead, but results in a more complete WEP comprising more information, which allows the scheduler to better plan the workflow execution on the underlying resources;
|
|
\item In \emph{late enactment mode} (also called \emph{lazy evaluation mode}) the engine only re-evaluates and completes the WEP when it is absolutely necessary for the workflow enactment, and only partially completes it as far as it is required for continuing the enactment.
|
|
\end{itemize}
|
|
|
|
\subsection{Workflow scheduling}
|
|
Scheduling describes the process of mapping atomic activities to available computing resources where they are executed. A concrete mapping of activities to computing resources is called \emph{workflow schedule}. The scheduler usually tries to optimize the schedule by maximising or minimizing a given utility function, typically the overall execution time.
|
|
|
|
To be able to produce an optimal mapping the scheduler need to know the resource requirements of an individual activity in advance. We assume in this paper that the scheduler has accurate knowledge of the resources required by an individual activity, but in practice obtaining this knowledge is still a steep challenge.
|
|
|
|
Existing scheduling algorithms can be broadly divided into two categories~\cite{Wieczorek2009}:
|
|
\begin{itemize}
|
|
\item \emph{Just-in-time scheduling algorithms} only consider the next activities to be scheduled when deciding on a mapping and ignore the rest of the WEP. They are usually linear in complexity with the number of activities (i.e. $O(N)$) and come with a low overhead, but due to limited information the generated schedule may not be that good;
|
|
\item \emph{Full-ahead scheduling algorithms} use the entire WEP when deciding on a mapping and try to find an optimal schedule in the search space. They usually come with a much higher overhead, but consider more workflow information and therefore, produce in general better results. Generating a full-ahead schedule is an NP-hard problem~\cite{ullman1975np} and therefore, most existing full-ahead scheduling algorithms are approximate heuristic algorithms~\cite{heft02}.
|
|
\end{itemize}
|