205 lines
9.5 KiB
Plaintext
Executable File
205 lines
9.5 KiB
Plaintext
Executable File
|
|
|
|
/**
|
|
* \page restrictions Restrictions of the Cell-Architecture
|
|
*
|
|
* \par
|
|
* \section res_arch CBE Architecture
|
|
*
|
|
* There are some restriction caused by the CBE architecture. Programming a SPU can be
|
|
* compared to programming a microcontroller:
|
|
* \li \b Small \b local-store \b of \b SPUs: Only 256 KByte of local-store are available
|
|
* on a SPU which is moreover shared by data and instructions. Because of this a economic
|
|
* stewardship of available resources is needed.
|
|
* \li \b No \b underlying \b OS: The operation system is running on the PPU, but not on
|
|
* the SPUs. Although the OS also manages the SPUs, there are some things which
|
|
* are missing or have to be done in a very performance decreasing and complex way.
|
|
* E.g. there is no support for several threads on one SPU (unless you program it yourself)
|
|
* and most of the system-calls are delegated to the PPU.
|
|
* \li \b Limited \b I/O-Capabilities: File-operations are only possible on the PPU. So if you
|
|
* want to store data of an SPU into a file, you have to send it to the PPU who then stores
|
|
* the data into a file. Main-memory is managed by the PPU and the SPUs can only read
|
|
* from or write into the main-memory. Allocation of main-memory has to be done by the PPU.
|
|
*
|
|
*
|
|
* \par
|
|
* \section res_alignment Memory-Alignment
|
|
*
|
|
* All Buffers used in send- or receive-functions have to be aligned at a 16-byte boundary
|
|
* (for better performance a 128-byte alignment is recommended). This restriction cannot be
|
|
* circumvented.
|
|
*
|
|
*
|
|
* \par
|
|
* \section res_size Transfer-Size
|
|
*
|
|
* DMA-calls only transfer data in chunks of 16 bytes. Send-receive operations and
|
|
* DMA-get-operations provided by this library hide this restriction by the use of intermediate
|
|
* buffers. But you should keep this restriction in mind because to gain maximum performance
|
|
* the size of data to be transfered should be a multiple of 16 bytes.
|
|
*
|
|
*
|
|
*
|
|
**/
|
|
|
|
|
|
/**
|
|
* \page violations Violations of the MPI-Standard
|
|
*
|
|
* \par
|
|
* \section viol_prefix Prefix of functions:
|
|
*
|
|
* The prefix of functions and data-structures is not 'MPI_' as stated by the MPI-standard.
|
|
* Instead it is 'MOC_' to allow the usage of another MPI-Library (e.g. to connect several
|
|
* playstations) and to stress that this implementation is not fully conform with the
|
|
* MPI-standard.
|
|
*
|
|
*
|
|
* \par
|
|
* \section viol_startup Application Startup
|
|
*
|
|
* MPI-applications (Version 1.3) are started using mpiexec/mpirun and normally only one
|
|
* binary is executed on all MPI-nodes. Because PPU- and SPU-instructions aren't compatible
|
|
* at least two executables (and two compilers) are needed. Also SPU-threads have to be initialized
|
|
* and started by the PPU. Because of this the SPU-binaries are embedded into the PPU-binary
|
|
* and started by this PPU-binary during runtime.
|
|
* In this library this concept is also used. So you have to program at least one PPU- and one
|
|
* SPU-program, embed the SPU-program into the PPU-executable and add the SPU-program-handle to
|
|
* the MOC-environment (see \ref MOC_Add_spu_thread) during runtime before calling MOC_Init.
|
|
*
|
|
*
|
|
* \par
|
|
* \section viol_queue Order of received messages:
|
|
*
|
|
* Because of the internal design of this implementation the order of
|
|
* incoming messages cannot be preserved in all cases. Currently the
|
|
* correct order is only preserved from messages from the same source,
|
|
* with the same message-tag and same communicator.
|
|
*
|
|
*
|
|
* \par
|
|
* \section viol_tag Number of usable message-tag
|
|
*
|
|
* According to the MPI-standard there should be at least 32767 tags available. But to save
|
|
* memory we use only one byte to encode the message-tag. So there are only 256 usable
|
|
* message-tags. Message-tags greater than 249 are reserved for internal purposes.
|
|
*
|
|
*
|
|
* \par
|
|
* \section viol_nr_pending Number of simultaneous pending send-receive operations
|
|
*
|
|
* According to the MPI-standard there should be hardly any limitations in the number of
|
|
* possible simultaneous transfers. It is stated that only in pathological cases the maximum
|
|
* of possible outstanding transfers should be reached. Because the resources of an SPU are
|
|
* very limited our implementation has a tight limit of possible simultaneous transfers.
|
|
* Although the upper-limit can be changed using \ref MOC_Set_queue_size, it is limited by
|
|
* the 256 KByte of local-storage of a SPU.
|
|
* There is also a limit of the number of simultaneous DMA-calls, because for each DMA-operation
|
|
* an exclusive DMA-tag is used and there are only 32 DMA-tag (on the PPU only 15) available.
|
|
*
|
|
* \par
|
|
* \section viol_init MPI_Init:
|
|
*
|
|
* The parameters of the init-function for the SPU does not match the parameters
|
|
* defined by the MPI-standard (see \ref moc_init_spu "MOC_Init (SPU)")
|
|
*
|
|
*
|
|
**/
|
|
|
|
|
|
/**
|
|
* \page performance Performance Measurements
|
|
*
|
|
* The measurements were taken on a Playstation 3 running Yellow Dog Linux release 6.0
|
|
* using Cell SDK 3.1. To measure send-receive operations ping-pongs were used and to
|
|
* measure DMA-calls the time till the used DMA-tag is marked to have no pending calls
|
|
* is used. The stated latencies in the pictures above are one-way latencies.
|
|
*
|
|
* \section perf_send_recv Send-Receive performance
|
|
*
|
|
* The following two figures show latency and bandwidth achieved by our implementation.
|
|
* The theoretical maximum bandwidth is 25 GByte/s and synchronous send between SPUs is
|
|
* at a close range to this limit if the data-size is large enough. There is also a big
|
|
* difference between SPU-SPU and SPU-PPU operations. The reason for this is the fact that
|
|
* the PPU cannot handle DMA-operations itself but only remote-control the SPU's
|
|
* Memory Flow Controller. And as you will see later the main-memory performance of the PPU
|
|
* is much lower than that of the SPUs. Because we use ping-pongs there is no way to
|
|
* determine differences in sending from or to the PPU, but other measurements indicate
|
|
* such differences.
|
|
*
|
|
* \image html latency-kb.png
|
|
*
|
|
* \image html bandwidth-kb.png
|
|
*
|
|
* The following figure shows the latency of send-receive operations at a lower scale than
|
|
* the first figure. If you look at the latency of synchronous send operations then you can
|
|
* see that there is a big start-up latency and only a very small data-size dependend latency.
|
|
* So there's only a very small time-difference between sending 10 byte and 1 Kbyte, but there
|
|
* is naturally a difference in throughput.
|
|
*
|
|
* \image html latency-b.png
|
|
*
|
|
* \section perf_moc_vs_dma Send-Recv vs. DMA performance
|
|
*
|
|
* The following two figures demonstrate the difference in performance of send-recv operations
|
|
* and DMA-calls. The performance of DMA-calls reaches it's peak much earlier than send-recv
|
|
* and the peak-value is higher than that of send-recv even though the difference is not much.
|
|
* So DMA is a lot faster than send-recv especially for small messages, but for big messages
|
|
* the difference is much lower. It seems that it is always better to use DMA, but keep in mind
|
|
* that in practical applications calculation time easily exceeds communication time.
|
|
*
|
|
* \image html dma-vs-moc-lat.png
|
|
*
|
|
* \image html dma-vs-moc-bw.png
|
|
*
|
|
* \section perf_spu_vs_ppu SPU- vs PPU-initiated transfer
|
|
*
|
|
* The following figure shows the difference between SPU- and PPU-initiated DMA-calls.
|
|
* As you can see the highest performance can be achieved by using SPU-SPU DMA-calls. There
|
|
* is also hardly any difference between put and get operations. DMA-operations between SPU and
|
|
* main-memory achieves lower performance and there is a difference between put and get
|
|
* operations. PPU to SPU DMA calls have the lowest performance with small transfer sizes, but
|
|
* if the transfer-size is big enough they are faster than SPU-main-memory transfers.
|
|
* The lowest performance has a memcopy on the PPU, which also explains the bad performance of
|
|
* buffered send between SPU and PPU.
|
|
*
|
|
* \image html spu-ppu-dma-bw-kb.png
|
|
*
|
|
* \section perf_mem DMA vs. memcpy
|
|
*
|
|
* The official Cell documentation states that it is also possible to use
|
|
* memcpy to transfer data from the PPU to a SPU. However one should use
|
|
* DMA, because it is always faster.
|
|
* If you look at the following figure, this is not always true. If the
|
|
* data-size is small enough memcpy is faster.
|
|
*
|
|
* \image html ppu-transfer-bw-b.png
|
|
*
|
|
**/
|
|
|
|
|
|
|
|
/**
|
|
* \page conclusion Summary & Conclusions
|
|
*
|
|
* \details MPI wasn't designed for the Cell Broadband Engine. A common MPI-node has one or more
|
|
* normal CPUs and several GBytes of RAM. The Cell Broadband Engine features a normal CPU
|
|
* (the PPU) and several special processing units with only a very limited amount
|
|
* of memory. The SPUs support only one thread per SPU and a complex handling of system-calls
|
|
* is needed. Because of this there are several restrictions in implementing MPI on the Cell
|
|
* Broadband Engine.
|
|
*
|
|
* Nevertheless MPI on Cell makes sense. It greatly simlifies programming on the Cell Broadband
|
|
* Engine, features a public API which is well known and widely used and message passing operations
|
|
* are generally easier to use than DMA (DMA needs memory-addresses and explicit synchronization,
|
|
* whereas MPI only needs the ranks of participating processes and features implicit
|
|
* synchronization).
|
|
*
|
|
* But message parsing operations cannot replace DMA. Both methods of communication have their
|
|
* advantages and disadvantages and they complement one another. So this library additionally
|
|
* features some easy to use DMA operations, if someone wants to gain maximum performance. Using
|
|
* DMA alltogether with send-recv especially speeds up application with a lot of communication
|
|
* going on between SPUs and PPU.
|
|
**/
|
|
|