include/doxygen_pages.doxygen

Go to the documentation of this file.

/**
 * \page restrictions Restrictions of the Cell-Architecture
 *
 * \par
 * \section res_arch CBE Architecture
 *
 * There are some restriction caused by the CBE architecture. Programming a SPU can be
 * compared to programming a microcontroller:
 * \li \b Small \b local-store \b of \b SPUs: Only 256 KByte of local-store are available
 *      on a SPU which is moreover shared by data and instructions. Because of this a economic
 *      stewardship of available resources is needed.
 * \li \b No \b underlying \b OS: The operation system is running on the PPU, but not on
 *      the SPUs. Although the OS also manages the SPUs, there are some things which
 *      are missing or have to be done in a very performance decreasing and complex way.
 *      E.g. there is no support for several threads on one SPU (unless you program it yourself)
 *      and most of the system-calls are delegated to the PPU.
 * \li \b Limited \b I/O-Capabilities: File-operations are only possible on the PPU. So if you
 *      want to store data of an SPU into a file, you have to send it to the PPU who then stores
 *      the data into a file. Main-memory is managed by the PPU and the SPUs can only read
 *      from or write into the main-memory. Allocation of main-memory has to be done by the PPU.
 *
 *
 * \par
 * \section res_alignment Memory-Alignment
 *
 * All Buffers used in send- or receive-functions have to be aligned at a 16-byte boundary
 * (for better performance a 128-byte alignment is recommended). This restriction cannot be
 * circumvented.
 *
 *
 * \par
 * \section res_size Transfer-Size
 *
 * DMA-calls only transfer data in chunks of 16 bytes. Send-receive operations and
 * DMA-get-operations provided by this library hide this restriction by the use of intermediate
 * buffers. But you should keep this restriction in mind because to gain maximum performance
 * the size of data to be transfered should be a multiple of 16 bytes.
 *
 *
 *
 **/


/**
 * \page violations Violations of the MPI-Standard
 *
 * \par
 * \section viol_prefix Prefix of functions:
 *
 * The prefix of functions and data-structures is not 'MPI_' as stated by the MPI-standard.
 * Instead it is 'MOC_' to allow the usage of another MPI-Library (e.g. to connect several
 * playstations) and to stress that this implementation is not fully conform with the
 * MPI-standard.
 *
 *
 * \par
 * \section viol_startup Application Startup
 *
 * MPI-applications (Version 1.3) are started using mpiexec/mpirun and normally only one
 * binary is executed on all MPI-nodes. Because PPU- and SPU-instructions aren't compatible
 * at least two executables (and two compilers) are needed. Also SPU-threads have to be initialized
 * and started by the PPU. Because of this the SPU-binaries are embedded into the PPU-binary
 * and started by this PPU-binary during runtime.
 * In this library this concept is also used. So you have to program at least one PPU- and one
 * SPU-program, embed the SPU-program into the PPU-executable and add the SPU-program-handle to
 * the MOC-environment (see \ref MOC_Add_spu_thread) during runtime before calling MOC_Init.
 *
 *
 * \par
 * \section viol_queue Order of received messages:
 *
 * Because of the internal design of this implementation the order of
 * incoming messages cannot be preserved in all cases. Currently the
 * correct order is only preserved from messages from the same source,
 * with the same message-tag and same communicator.
 *
 *
 * \par
 * \section viol_tag Number of usable message-tag
 *
 * According to the MPI-standard there should be at least 32767 tags available. But to save
 * memory we use only one byte to encode the message-tag. So there are only 256 usable
 * message-tags. Message-tags greater than 249 are reserved for internal purposes.
 *
 *
 * \par
 * \section viol_nr_pending Number of simultaneous pending send-receive operations
 *
 * According to the MPI-standard there should be hardly any limitations in the number of
 * possible simultaneous transfers. It is stated that only in pathological cases the maximum
 * of possible outstanding transfers should be reached. Because the resources of an SPU are
 * very limited our implementation has a tight limit of possible simultaneous transfers.
 * Although the upper-limit can be changed using \ref MOC_Set_queue_size, it is limited by
 * the 256 KByte of local-storage of a SPU.
 * There is also a limit of the number of simultaneous DMA-calls, because for each DMA-operation
 * an exclusive DMA-tag is used and there are only 32 DMA-tag (on the PPU only 15) available.
 *
 * \par
 * \section viol_init MPI_Init:
 *
 * The parameters of the init-function for the SPU does not match the parameters
 * defined by the MPI-standard (see \ref moc_init_spu "MOC_Init (SPU)")
 *
 *
 **/


/**
 * \page performance Performance Measurements
 *
 * The measurements were taken on a Playstation 3 running Yellow Dog Linux release 6.0
 * using Cell SDK 3.1. To measure send-receive operations ping-pongs were used and to
 * measure DMA-calls the time till the used DMA-tag is marked to have no pending calls
 * is used. The stated latencies in the pictures above are one-way latencies.
 *
 * \section perf_send_recv Send-Receive performance
 *
 * The following two figures show latency and bandwidth achieved by our implementation.
 * The theoretical maximum bandwidth is 25 GByte/s and synchronous send between SPUs is
 * at a close range to this limit if the data-size is large enough. There is also a big
 * difference between SPU-SPU and SPU-PPU operations. The reason for this is the fact that
 * the PPU cannot handle DMA-operations itself but only remote-control the SPU's
 * Memory Flow Controller. And as you will see later the main-memory performance of the PPU
 * is much lower than that of the SPUs. Because we use ping-pongs there is no way to
 * determine differences in sending from or to the PPU, but other measurements indicate
 * such differences.
 *
 * \image html latency-kb.png
 *
 * \image html bandwidth-kb.png
 *
 * The following figure shows the latency of send-receive operations at a lower scale than
 * the first figure. If you look at the latency of synchronous send operations then you can
 * see that there is a big start-up latency and only a very small data-size dependend latency.
 * So there's only a very small time-difference between sending 10 byte and 1 Kbyte, but there
 * is naturally a difference in throughput.
 *
 * \image html latency-b.png
 *
 * \section perf_moc_vs_dma Send-Recv vs. DMA performance
 *
 * The following two figures demonstrate the difference in performance of send-recv operations
 * and DMA-calls. The performance of DMA-calls reaches it's peak much earlier than send-recv
 * and the peak-value is higher than that of send-recv even though the difference is not much.
 * So DMA is a lot faster than send-recv especially for small messages, but for big messages
 * the difference is much lower. It seems that it is always better to use DMA, but keep in mind
 * that in practical applications calculation time easily exceeds communication time.
 *
 * \image html dma-vs-moc-lat.png
 *
 * \image html dma-vs-moc-bw.png
 *
 * \section perf_spu_vs_ppu SPU- vs PPU-initiated transfer
 *
 * The following figure shows the difference between SPU- and PPU-initiated DMA-calls.
 * As you can see the highest performance can be achieved by using SPU-SPU DMA-calls. There
 * is also hardly any difference between put and get operations. DMA-operations between SPU and
 * main-memory achieves lower performance and there is a difference between put and get
 * operations. PPU to SPU DMA calls have the lowest performance with small transfer sizes, but
 * if the transfer-size is big enough they are faster than SPU-main-memory transfers.
 * The lowest performance has a memcopy on the PPU, which also explains the bad performance of
 * buffered send between SPU and PPU.
 *
 * \image html spu-ppu-dma-bw-kb.png
 *
 * \section perf_mem DMA vs. memcpy
 *
 * The official Cell documentation states that it is also possible to use
 * memcpy to transfer data from the PPU to a SPU. However one should use
 * DMA, because it is always faster.
 * If you look at the following figure, this is not always true. If the
 * data-size is small enough memcpy is faster.
 *
 * \image html ppu-transfer-bw-b.png
 *
 **/



/**
 * \page conclusion Summary & Conclusions
 *
 * \details MPI wasn't designed for the Cell Broadband Engine. A common MPI-node has one or more
 * normal CPUs and several GBytes of RAM. The Cell Broadband Engine features a normal CPU
 * (the PPU) and several special processing units with only a very limited amount
 * of memory. The SPUs support only one thread per SPU and a complex handling of system-calls
 * is needed. Because of this there are several restrictions in implementing MPI on the Cell
 * Broadband Engine.
 *
 * Nevertheless MPI on Cell makes sense. It greatly simlifies programming on the Cell Broadband
 * Engine, features a public API which is well known and widely used and message passing operations
 * are generally easier to use than DMA (DMA needs memory-addresses and explicit synchronization,
 * whereas MPI only needs the ranks of participating processes and features implicit
 * synchronization).
 *
 * But message parsing operations cannot replace DMA. Both methods of communication have their
 * advantages and disadvantages and they complement one another. So this library additionally
 * features some easy to use DMA operations, if someone wants to gain maximum performance. Using
 * DMA alltogether with send-recv especially speeds up application with a lot of communication
 * going on between SPUs and PPU.
 **/

---