/** * \page restrictions Restrictions of the Cell-Architecture * * \par * \section res_arch CBE Architecture * * There are some restriction caused by the CBE architecture. Programming a SPU can be * compared to programming a microcontroller: * \li \b Small \b local-store \b of \b SPUs: Only 256 KByte of local-store are available * on a SPU which is moreover shared by data and instructions. Because of this a economic * stewardship of available resources is needed. * \li \b No \b underlying \b OS: The operation system is running on the PPU, but not on * the SPUs. Although the OS also manages the SPUs, there are some things which * are missing or have to be done in a very performance decreasing and complex way. * E.g. there is no support for several threads on one SPU (unless you program it yourself) * and most of the system-calls are delegated to the PPU. * \li \b Limited \b I/O-Capabilities: File-operations are only possible on the PPU. So if you * want to store data of an SPU into a file, you have to send it to the PPU who then stores * the data into a file. Main-memory is managed by the PPU and the SPUs can only read * from or write into the main-memory. Allocation of main-memory has to be done by the PPU. * * * \par * \section res_alignment Memory-Alignment * * All Buffers used in send- or receive-functions have to be aligned at a 16-byte boundary * (for better performance a 128-byte alignment is recommended). This restriction cannot be * circumvented. * * * \par * \section res_size Transfer-Size * * DMA-calls only transfer data in chunks of 16 bytes. Send-receive operations and * DMA-get-operations provided by this library hide this restriction by the use of intermediate * buffers. But you should keep this restriction in mind because to gain maximum performance * the size of data to be transfered should be a multiple of 16 bytes. * * * **/ /** * \page violations Violations of the MPI-Standard * * \par * \section viol_prefix Prefix of functions: * * The prefix of functions and data-structures is not 'MPI_' as stated by the MPI-standard. * Instead it is 'MOC_' to allow the usage of another MPI-Library (e.g. to connect several * playstations) and to stress that this implementation is not fully conform with the * MPI-standard. * * * \par * \section viol_startup Application Startup * * MPI-applications (Version 1.3) are started using mpiexec/mpirun and normally only one * binary is executed on all MPI-nodes. Because PPU- and SPU-instructions aren't compatible * at least two executables (and two compilers) are needed. Also SPU-threads have to be initialized * and started by the PPU. Because of this the SPU-binaries are embedded into the PPU-binary * and started by this PPU-binary during runtime. * In this library this concept is also used. So you have to program at least one PPU- and one * SPU-program, embed the SPU-program into the PPU-executable and add the SPU-program-handle to * the MOC-environment (see \ref MOC_Add_spu_thread) during runtime before calling MOC_Init. * * * \par * \section viol_queue Order of received messages: * * Because of the internal design of this implementation the order of * incoming messages cannot be preserved in all cases. Currently the * correct order is only preserved from messages from the same source, * with the same message-tag and same communicator. * * * \par * \section viol_tag Number of usable message-tag * * According to the MPI-standard there should be at least 32767 tags available. But to save * memory we use only one byte to encode the message-tag. So there are only 256 usable * message-tags. Message-tags greater than 249 are reserved for internal purposes. * * * \par * \section viol_nr_pending Number of simultaneous pending send-receive operations * * According to the MPI-standard there should be hardly any limitations in the number of * possible simultaneous transfers. It is stated that only in pathological cases the maximum * of possible outstanding transfers should be reached. Because the resources of an SPU are * very limited our implementation has a tight limit of possible simultaneous transfers. * Although the upper-limit can be changed using \ref MOC_Set_queue_size, it is limited by * the 256 KByte of local-storage of a SPU. * There is also a limit of the number of simultaneous DMA-calls, because for each DMA-operation * an exclusive DMA-tag is used and there are only 32 DMA-tag (on the PPU only 15) available. * * \par * \section viol_init MPI_Init: * * The parameters of the init-function for the SPU does not match the parameters * defined by the MPI-standard (see \ref moc_init_spu "MOC_Init (SPU)") * * **/ /** * \page performance Performance Measurements * * The measurements were taken on a Playstation 3 running Yellow Dog Linux release 6.0 * using Cell SDK 3.1. To measure send-receive operations ping-pongs were used and to * measure DMA-calls the time till the used DMA-tag is marked to have no pending calls * is used. The stated latencies in the pictures above are one-way latencies. * * \section perf_send_recv Send-Receive performance * * The following two figures show latency and bandwidth achieved by our implementation. * The theoretical maximum bandwidth is 25 GByte/s and synchronous send between SPUs is * at a close range to this limit if the data-size is large enough. There is also a big * difference between SPU-SPU and SPU-PPU operations. The reason for this is the fact that * the PPU cannot handle DMA-operations itself but only remote-control the SPU's * Memory Flow Controller. And as you will see later the main-memory performance of the PPU * is much lower than that of the SPUs. Because we use ping-pongs there is no way to * determine differences in sending from or to the PPU, but other measurements indicate * such differences. * * \image html latency-kb.png * * \image html bandwidth-kb.png * * The following figure shows the latency of send-receive operations at a lower scale than * the first figure. If you look at the latency of synchronous send operations then you can * see that there is a big start-up latency and only a very small data-size dependend latency. * So there's only a very small time-difference between sending 10 byte and 1 Kbyte, but there * is naturally a difference in throughput. * * \image html latency-b.png * * \section perf_moc_vs_dma Send-Recv vs. DMA performance * * The following two figures demonstrate the difference in performance of send-recv operations * and DMA-calls. The performance of DMA-calls reaches it's peak much earlier than send-recv * and the peak-value is higher than that of send-recv even though the difference is not much. * So DMA is a lot faster than send-recv especially for small messages, but for big messages * the difference is much lower. It seems that it is always better to use DMA, but keep in mind * that in practical applications calculation time easily exceeds communication time. * * \image html dma-vs-moc-lat.png * * \image html dma-vs-moc-bw.png * * \section perf_spu_vs_ppu SPU- vs PPU-initiated transfer * * The following figure shows the difference between SPU- and PPU-initiated DMA-calls. * As you can see the highest performance can be achieved by using SPU-SPU DMA-calls. There * is also hardly any difference between put and get operations. DMA-operations between SPU and * main-memory achieves lower performance and there is a difference between put and get * operations. PPU to SPU DMA calls have the lowest performance with small transfer sizes, but * if the transfer-size is big enough they are faster than SPU-main-memory transfers. * The lowest performance has a memcopy on the PPU, which also explains the bad performance of * buffered send between SPU and PPU. * * \image html spu-ppu-dma-bw-kb.png * * \section perf_mem DMA vs. memcpy * * The official Cell documentation states that it is also possible to use * memcpy to transfer data from the PPU to a SPU. However one should use * DMA, because it is always faster. * If you look at the following figure, this is not always true. If the * data-size is small enough memcpy is faster. * * \image html ppu-transfer-bw-b.png * **/ /** * \page conclusion Summary & Conclusions * * \details MPI wasn't designed for the Cell Broadband Engine. A common MPI-node has one or more * normal CPUs and several GBytes of RAM. The Cell Broadband Engine features a normal CPU * (the PPU) and several special processing units with only a very limited amount * of memory. The SPUs support only one thread per SPU and a complex handling of system-calls * is needed. Because of this there are several restrictions in implementing MPI on the Cell * Broadband Engine. * * Nevertheless MPI on Cell makes sense. It greatly simlifies programming on the Cell Broadband * Engine, features a public API which is well known and widely used and message passing operations * are generally easier to use than DMA (DMA needs memory-addresses and explicit synchronization, * whereas MPI only needs the ranks of participating processes and features implicit * synchronization). * * But message parsing operations cannot replace DMA. Both methods of communication have their * advantages and disadvantages and they complement one another. So this library additionally * features some easy to use DMA operations, if someone wants to gain maximum performance. Using * DMA alltogether with send-recv especially speeds up application with a lot of communication * going on between SPUs and PPU. **/