144 lines
19 KiB
TeX
144 lines
19 KiB
TeX
\section{Experiments}\label{sec:transformationexperiments}
|
|
|
|
We conducted all experiments on a multi-core system with four Intel Xeon E7-4870 10-core CPUs at \SI{2.40}{\giga\hertz} with \SI{128}{\giga\byte} of RAM. We used the GNU \texttt{gcc} compiler version 4.9.1 to compile the generated workflow programs with the \texttt{-O3} optimization flag.
|
|
|
|
We used a combination of real-world workflows to show that compiler transformations are practically applicable,
|
|
supplemented by a number of synthetic workflows to better highlight their potential positive effects. We manually chose the applied the transformations, and plan to research an automatic optimizer based on the experience gained from these experiments in future work.
|
|
For each experiment, we recorded the workflow enactment time in both early and late evaluation modes, and their memory utilization. The enactment time refers to the time spent by the engine for evaluating the WEP and resolving the dependencies and does not include the time spent in executing atomic activities. We used the resident size reported by the operating system to measure the engine's memory consumption. Since there is no difference in memory consumption in early and late evaluation modes, we report a single joint result for both. We also report a single value for the number of WEP nodes for early and late evaluation modes for the same reasons.
|
|
We used the HEFT~\cite{heft02} scheduling algorithm in all experiments unless otherwise stated.
|
|
|
|
|
|
\subsection{Engine Overhead Reduction}
|
|
|
|
\begin{figure*}[t]
|
|
\centering
|
|
\subfloat[Original workflow.]{\includegraphics[height=4cm]{./workflowtransformations/figures/Raincloud-experiments1.pdf}\label{fig:raincloud-structure1}}\qquad
|
|
\subfloat[After loop unswitching.]{\includegraphics[height=4cm]{./workflowtransformations/figures/Raincloud-experiments2.pdf}\label{fig:raincloud-structure2}}\qquad
|
|
\subfloat[After loop unswitching and loop unrolling.]{\includegraphics[height=4cm]{./workflowtransformations/figures/Raincloud-experiments3.pdf}\label{fig:raincloud-structure3}}
|
|
\caption{RainCloud workflow.}
|
|
\label{fig:raincloud-structure}
|
|
\end{figure*}
|
|
|
|
We show in this section the effects of transformations on reducing engine overhead for two real-world workflows.
|
|
|
|
\subsubsection{RainCloud} is a meteorological workflow for weather simulation in mountainous regions that uses a simple numerical linear model of orographic precipitations~\cite{Schuller2013} (see Figure~\ref{fig:raincloud-structure1}). The \texttt{prepareLM} activity partitions the input data in several chunks, independently processed by a \texttt{parallel for each} loop. The number of chunks and the degree of parallelism is controlled by an input port called \texttt{NGroups}. The workflow features an optional post-processing activity of the results called \texttt{PPS}, and an optional post-processing activity of all results called \texttt{PPF}.
|
|
We first applied loop unswitching that moves the \texttt{if} activity out of the \texttt{parallel foreach} loop (see Figure~\ref{fig:raincloud-structure2}), followed by a ten-fold loop unrolling, as shown in Figure~\ref{fig:raincloud-structure3} (we only unrolled the loop twice in the figure for brevity reasons).
|
|
To enable loop unrolling, we exchanged \texttt{parallel foreach} with a \texttt{parallel for} loop to process more data chunks per iteration in the transformed workflow. We also introduced three additional helper activities: (1) \texttt{H1} that calculates the loop counter boundaries of the new \texttt{parallel for} loop, (2) \texttt{H2} that distributes the data chunks among the unrolled activities, and (3) \texttt{H3} that collects the results from the unrolled activities and passes them to the output port of the \texttt{parallel for} loop.
|
|
%We compared in our experiments the original version of the Raincloud workflow with a version that only has Loop Unswitching applied, and with a version that has Loop Unswitching and Loop Unrolling applied.
|
|
|
|
\begin{figure*}[t]
|
|
\centering
|
|
\subfloat[Early evaluation enactment.]{\includegraphics[width=.33\textwidth]{./workflowtransformations/figures/raincloud-EnactmentTimeEarly.pdf}\label{fig:raincloud-results-enactment-early}}
|
|
\subfloat[Late evaluation enactment.]{\includegraphics[width=.33\textwidth]{./workflowtransformations/figures/raincloud-EnactmentTimeLate.pdf}\label{fig:raincloud-results-enactment-late}}
|
|
\subfloat[WEP length.]{\includegraphics[width=.33\textwidth]{./workflowtransformations/figures/raincloud-NodeCountLate.pdf}\label{fig:raincloud-results-nodecount}}\\
|
|
\subfloat[Early evaluation WEP visitation.]{\includegraphics[width=.33\textwidth]{./workflowtransformations/figures/raincloud-VisitationCountEarly.pdf}\label{fig:raincloud-results-visitations-early}}
|
|
\subfloat[Late evaluation WEP visitation.]{\includegraphics[width=.33\textwidth]{./workflowtransformations/figures/raincloud-VisitationCountLate.pdf}\label{fig:raincloud-results-visitations-late}}
|
|
\subfloat[Memory usage.]{\includegraphics[width=.33\textwidth]{./workflowtransformations/figures/raincloud-MemoryLate.pdf}\label{fig:raincloud-results-memory}}
|
|
\caption{RainCloud workflow experimental results.}
|
|
\label{fig:raincloud-results-enactment}
|
|
\end{figure*}
|
|
|
|
Figure~\ref{fig:raincloud-results-enactment-early} shows that loop unswitching enables an up to three times better enactment, while loop unrolling an up to 88 times better enactment as the original version. The better performance is caused on one hand by a reduced WEP length, defined as the length of its critical path (see Figure~\ref{fig:raincloud-results-nodecount}), and a consequentially lower WEP node visitation (see Figure~\ref{fig:raincloud-results-visitations-early}). On the other hand, loop unrolling reduces the loop iterations by an order of magnitude, which massively reduces the loop handling overhead.
|
|
|
|
Figure~\ref{fig:raincloud-results-enactment-late} shows a better scalability of the enactment time with the problem size, although the performance improvements are lower than in early evaluation mode. Applying loop unswitching shows an up to 1.4 times better enactment, while applying both loop unswitching and loop unrolling increases it to a twice better enactment time. The lower overhead is again caused by a reduced WEP length, a consequentially lower WEP node visitation, and a reduced loop handling overhead. Late evaluation, however, eliminates a lager number of unnecessary WEP node visits reducing the optimization potential compared to the early evaluation mode. Figure~\ref{fig:raincloud-results-visitations-late} shows that, in contrast to the early evaluation mode, the workflow after loop unswitching and loop unrolling features more WEP node visits in late evaluation mode compared to loop unswitching only. The helper activities needed by loop unrolling cause more WEP node visits in late evaluation mode than eliminated by the reduced number of loop iterations.
|
|
|
|
% memory usage of the original and the transformed versions of the RainCloud workflow.
|
|
Loop unswitching reduces the number of WEP nodes, which improves the memory usage of the engine by up to 29\% (see Figure~\ref{fig:raincloud-results-memory}).
|
|
Loop unrolling reduces loop iteration management overhead, which additionally improves the memory usage up to 4\% for a total reduction of up to 33\%.
|
|
|
|
\begin{figure*}[t]
|
|
\centering
|
|
\subfloat[Original workflow.]{\includegraphics[height=6.1cm]{./workflowtransformations/figures/meteoag01.pdf}\label{fig:meteoag-structure1}}\qquad
|
|
\subfloat[After loop unswitching.]{\includegraphics[height=6.1cm]{./workflowtransformations/figures/meteoag02.pdf}\label{fig:meteoag-structure2}}\qquad
|
|
\subfloat[After loop unswitching and loop unrolling.]{\includegraphics[height=6.1cm]{./workflowtransformations/figures/meteoag03.pdf}\label{fig:meteoag-structure3}}
|
|
\caption{MeteoAG workflow.}
|
|
\label{fig:meteoag-structure}
|
|
\end{figure*}
|
|
|
|
\subsubsection{MeteoAG} \cite{cotton2003rams} is another workflow designed for meteorological simulations based on the RAMS numerical atmospheric model (see Figure~\ref{fig:meteoag-structure1}). The simulations produce spatial and temporal fields of heavy precipitation cases over the western part of Austria to resolve most alpine watersheds and thunderstorms. The \texttt{simulation\_init} activity splits the input data into several simulation cases, independently processed by the \texttt{parallel for each} loop. For each simulation case, another nested parallel loop is executed with different parameter values. The workflow contains two optional \texttt{rams\_hist} and \texttt{revu\_dump} activities within an \texttt{if} activity. As with RainCloud, we first applied loop unswitching to move the \texttt{if} activity out of the \texttt{parallel foreach} loop (see Figure~\ref{fig:meteoag-structure2}), followed by a ten-fold loop unrolling shown in Figure~\ref{fig:meteoag-structure3} (unrolled twice in the figure for brevity reasons). Again, the helper activity \texttt{H1} calculates the loop boundaries for the new \texttt{parallel for} loop, \texttt{H2} distributes the input data among the unrolled loop iterations, and \texttt{H3} collects the results. We fixed in our experiments the inner loop iteration number to 20 and varied the outer loop iteration count.
|
|
|
|
\begin{figure*}[t]
|
|
\centering
|
|
\subfloat[Speedup.]{\includegraphics[width=.33\textwidth]{./workflowtransformations/figures/meteoag-speedup.pdf}\label{fig:meteoag-speedup}}
|
|
\subfloat[Early evaluation enactment.]{\includegraphics[width=.33\textwidth]{./workflowtransformations/figures/meteoag-enactment-early.pdf}\label{fig:meteoag-enactment-early}}
|
|
\subfloat[Late evaluation enactment.]{\includegraphics[width=.33\textwidth]{./workflowtransformations/figures/meteoag-enactment-late.pdf}\label{fig:meteoag-enactment-late}}
|
|
\caption{MeteoAG workflow experimental results.}
|
|
\label{fig:meteoag-results}
|
|
\end{figure*}
|
|
|
|
Figure~\ref{fig:meteoag-speedup} shows the speedup of the transformed workflows of up to 1.32.
|
|
The MeteoAG workflow is much more complex than RainCloud with atomic activities having a much larger impact on enactment performance than the composite activities, as indicated by the late enactment mode having the same performance as early enactment mode (see Figures~\ref{fig:meteoag-enactment-early} and \ref{fig:meteoag-enactment-late}).
|
|
For this reason, the measured enactment times lie well within the respective error margins and, therefore, the results only hint at a better performance achieved by the transformations.
|
|
The performance improvements achieved by loop unswitching and loop unrolling are largely hidden by the remaining unoptimized workflow structure.
|
|
More specialized transformations suitable for the unoptimized workflow parts may obtain significant
|
|
performance improvements, which we plan to research
|
|
in future work.
|
|
|
|
\subsection{Atomic Activity Reduction}
|
|
We show in this section the results of applying loop-invariant code hoisting and common activity elimination. We implemented the workflow from Figure~\ref{fig:transformation-loopInvariantHoisting}, and for activity elimination the workflow from Figure~\ref{fig:transformation-CommonSubexpressionElimination}, surrounded by a \texttt{parallel for} loop. For both workflows, we implemented activities with an execution time of \SI{1}{\second} each that, in contrast to DCIs, is a common order of magnitude for manycore parallel systems.
|
|
Loop-invariant code hoisting halves the number of activities in the loop body of our example, which produces a theoretical speedup of up to 2. Figure~\ref{fig:raincloud-results-loophoisting-makespan} shows a performance speedup of up to 2.6, surpassing the theoretical estimate because the original loop loop overhead significantly contributes to the workflow makespan, as the loop body consists of two short-running activities. %Loop hoisting cuts the number of activities in half, thereby also reducing loop overhead.
|
|
Common activity elimination reduces the number of atomic activities in our example workflow from five to four, which gives us a theoretical speedup of up to 1.25. Figure~\ref{fig:raincloud-results-elimination-makespan} shows a performance improvement of up to 1.7 times, again surpassing our theoretical estimate. The reason is that the total number of activities in the transformed workflow is a multiple of the number of cores, in contrast to the original workflow where the average activity time spent in the waiting queue was twice as high.
|
|
|
|
\begin{figure}[t]
|
|
\centering
|
|
\subfloat[Loop hoisting makespan.]{\includegraphics[width=.8\columnwidth]{./workflowtransformations/figures/loophoisting-Makespan.pdf}\label{fig:raincloud-results-loophoisting-makespan}}\\
|
|
\subfloat[Common activity elimination makespan.]{\includegraphics[width=.8\columnwidth]{./workflowtransformations/figures/elimination-Makespan.pdf}\label{fig:raincloud-results-elimination-makespan}}
|
|
\caption{Loop hoisting and common activity elimination results.}
|
|
\label{fig:raincloud-results-elimination-loophoisting}
|
|
\end{figure}
|
|
|
|
|
|
\subsection{Schedule Improvement}
|
|
We show in this section the effects of transformations to improving scheduling of the Sparselu workflow, adapted from the BOTS benchmark suite~\cite{duran2009barcelona} for LU factorization of sparse matrices. The original version of the program consists of three parallel inner loops (see Figure~\ref{fig:Sparselu-experiments1}) within a sequential outer loop. However, the first and the second inner loops have the same loop range and the contained activities \texttt{fwd} and \texttt{bvid} operate on separate parts of the sparse matrix and have no dependencies on each other. Therefore, we can apply loop fusion to combine the two inner loops (see Figure~\ref{fig:Sparselu-experiments2}) in a new
|
|
inner loop containing twice as many parallel activities. This gives the scheduler more opportunities to utilize the available computing resources.
|
|
%We simulated in this experiment the execution of the activities to be able to assign them arbitrary execution times matching the predicted execution times used by HEFT.
|
|
The iteration count of the inner loops determines the number of activities to be executed in parallel. Figure~\ref{fig:Sparselu-Heft-Speedup} shows that
|
|
%the speed-up of the transformed workflow version.
|
|
in all scenarios where the number of inner loop iterations is not a multiple of the number of cores, the transformed workflow version shows a significant speedup up to 36\%. Even when the inner loop iteration count is a multiple of the number of cores, the transformed version still shows a slightly better performance when the outer loop counter is large enough. The higher number of activities to be simultaneously scheduled allows a better mapping.
|
|
|
|
\begin{figure}[t]
|
|
\centering
|
|
\subfloat[Original workflow.]{\includegraphics[height=4.2cm]{./workflowtransformations/figures/Sparselu-experiments1.pdf}\label{fig:Sparselu-experiments1}}\qquad
|
|
\subfloat[After loop fusion.]{\includegraphics[height=4.2cm]{./workflowtransformations/figures/Sparselu-experiments2.pdf}\label{fig:Sparselu-experiments2}}
|
|
\caption{Sparselu workflow.}
|
|
\label{fig:sparselu-structure}
|
|
\end{figure}
|
|
|
|
\begin{figure}[t]
|
|
\centering
|
|
\subfloat[Early evaluation mode.]{\includegraphics[width=.8\columnwidth]{./workflowtransformations/figures/Sparselu-Heft-Speedup-early.pdf}\label{fig:Sparselu-Heft-Speedup-early}}\\
|
|
\subfloat[Late evaluation mode.]{\includegraphics[width=.8\columnwidth]{./workflowtransformations/figures/Sparselu-Heft-Speedup-late.pdf}\label{fig:Sparselu-Heft-Speedup-late}}
|
|
\caption{Speedup of Sparselu caused by scheduling improvements.}
|
|
\label{fig:Sparselu-Heft-Speedup}
|
|
\end{figure}
|
|
|
|
|
|
|
|
\subsection{Parallel Throughput Improvement}
|
|
|
|
\begin{figure}[t]
|
|
\centering
|
|
\subfloat[Original workflow.]{\includegraphics[height=4.5cm]{./workflowtransformations/figures/blast-experiment01.pdf}\label{fig:blast-experiment01}}\hspace{2mm}
|
|
\subfloat[After reduction recognition.]{\includegraphics[height=4.5cm]{./workflowtransformations/figures/blast-experiment02.pdf}\label{fig:blast-experiment02}}
|
|
\caption{Blast workflow.}
|
|
\label{fig:blast-structure}
|
|
\end{figure}
|
|
|
|
We first show the results of applying reduction recognition to the Basic Local Alignment Search Tool Blast~\cite{altschul1990basic} workflow (see Figure~\ref{fig:blast-structure}). Blast is a bioinformatics application that finds regions of local similarity between sequences by comparing nucleotide or protein sequences to sequence databases, and calculating the statistical significance of matches. The workflow consists of a parallel \texttt{foreach} loop that conducts the parallel search operation, and a few subsequent activities that collect the intermediate results and produce the final result. Each iteration of the \texttt{foreach} loop produces three files (called \texttt{output}, \texttt{error} and \texttt{total}), collected by the \texttt{cat\_output}, \texttt{cat\_error} and \texttt{cat\_total} activities and merged into a single file each. The merge operations are associative and, therefore, appropriate for the reduction recognition transformation.
|
|
|
|
Figure~\ref{fig:blast-experiment02} shows the transformed workflow featuring an additional \texttt{for} loop, which distributes the results of the \texttt{foreach} loop among several parallel merge activities \texttt{cat\_*\_p}.
|
|
The helper activity \texttt{H1} splits the original input of the \texttt{foreach} loop.
|
|
The results of the parallel merge operations are then collected and merged by the final merge operations \texttt{cat\_*\_f}. To exclusively measure the time for the merge operation, we replaced in our experiments all activities not involved in the merge operation with empty implementations. The execution time of the merge operations is around \SI{1}{\milli\second} per file. We configured the outer loop to have 40 parallel iterations to fully utilize all available cores, and varied the iteration count of the inner loop.
|
|
|
|
Figure~\ref{fig:results-reduction-makespan} shows that reduction recognition can significantly improve the performance of the merge operation up to 3.8 times. However, this is still far from the theoretical estimate of 13.3 times because the parallel merge operations have a very short execution time and therefore, a comparably larger execution overhead. However, we expect a higher speedup for longer running reduction operations.
|
|
|
|
For software pipelining, we implemented the workflow from Figure~\ref{fig:transformation-SoftwarePipelining} with activities' execution time of \SI{1}{\second} each. Figure~\ref{fig:results-pipelining-makespan} shows that executing the three previously sequential activities in parallel improves the performance by up to 2.8 times, close to the theoretical estimate of 3.
|
|
|
|
\begin{figure}[t]
|
|
\centering
|
|
\subfloat[Reduction recognition makespan.]{\includegraphics[width=.8\columnwidth]{./workflowtransformations/figures/blast-reduction.pdf}\label{fig:results-reduction-makespan}}\\
|
|
\subfloat[Software pipelining makespan.]{\includegraphics[width=.8\columnwidth]{./workflowtransformations/figures/pipelining-Makespan.pdf}\label{fig:results-pipelining-makespan}}
|
|
\caption{Software pipelining and reduction recognition results.}
|
|
\label{fig:results-pipelining-reduction}
|
|
\end{figure}
|