Uni/MPIonCell
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Initial commit.

Browse Source
This commit is contained in:
matzman
2017-10-26 19:57:24 +02:00
commit 9ce8b5485d
1090 changed files with 47036 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
include/doxygen_main.doxygen
Executable
+244
View File
@@ -0,0 +1,244 @@
#ifndef MOC_DOXYGEN_MAIN_H_
#define MOC_DOXYGEN_MAIN_H_
/**
* \mainpage MPI On Cell
*
* The idea of this project is to implement a message passing
* library for work on the Cell BE. The main idea behind such
* an implementation would be to help the programmer by adding
* an layer of abstraction between the very low level libraries
* by IBM and give him the possibility to work with an interface
* he is more familiar with. To achieve this we decided to try
* to keep very closely to the standard provided by the Message
* Passing Interface (MPI), so that programmers knowing how to
* write MPI programmers can easily work with our library.
* Because of the very special nature of the Cell, it might not
* be possible to keep 100% true to the standard, but we try to
* introduce changes only when absolutely necessary and of great
* benefit, because of some specialties of the Cell's hardware.
*
*
* \par Table of Content
* \par
* \ref end_user
* \par
* \subpage restrictions
* \par
* \subpage violations
* \par
* \subpage performance
* \par
* \subpage conclusion
* \if full
* \par
* \par
* \ref internal_main
* \endif
**/
/**
* \defgroup end_user End-User Documentation
*
* This section describes the public API provided by the MOC-library.
*
* If some things remain unclear, consult the official MPI-standard (Version 1.3).
*
* This library uses Direct Memory Access (DMA) provided by the Cell Broadband Engine.
* So the use of other DMA operations than provided of this library is discouraged,
* because it may interfere with this library. One exceptions are signals.
* We don't use signals at all at the moment, so one can savely use signal.
**/
/**
* \defgroup environment_main Environmental Management
* \ingroup end_user
*
* This chapter discusses routines for getting and, where appropriate, setting various parameters
* that relate to the MPI implementation and the execution environment (such as error handling,
* startup&shutdown and Time).
**/
/**
* \defgroup environment_startup Startup & Shutdown
* \ingroup environment_main
*
* Functions to start and terminate the MOC-execution environment.
*
* To be able to communicate with other processes every MOC-process needs to setup his
* MOC-Environment before any MOC-function can be called (there are some exceptions though).
*
* The PPU-process has a special role, because he is the one who starts the SPU-processes.
* SPU-processes can be added to the MOC-environment by calling \ref MOC_Add_spu_thread with
* the appropriate SPU-thread-handle (This implementation only supports the embedded binary
* concept where the SPU-binary is embedded into the PPU-binary and accessed via a handle).
* The order of calling determines the rank of the SPU-processes. To get the number of total
* and available SPUs the functions \ref MOC_Get_avail_spu_count and \ref MOC_Get_max_spu_count
* can be used.
*
* The buffer-size and the size of the message-queues can be set before calling \ref MOC_Init
* on the PPU using \ref MOC_Set_buffer_size and \ref MOC_Set_queue_size.
*
* MOC_Init only returns after all processes have called this function and have set up their
* environment.
*
* If the MOC-environment isn't needed anymore, \ref MOC_Finalize should be called by all
* MOC-processes. MOC_Finalize only returns after all MOC-processes have called this functions
* and have shutdown their environments.
**/
/**
* \defgroup environment_information Implementation Information
* \ingroup environment_main
**/
/**
* \defgroup environment_error Error-Handling
* \ingroup environment_main
**/
/**
* \defgroup environment_timer Timers and Synchronization
* \ingroup environment_main
**/
/**
* \defgroup p2p_main Point-to-Point Communication
* \ingroup end_user
*
* Sending and receiving of messages by processes is the basic MPI communication mechanism.
* The basic point-to-point communication operations are send and receive.
*
* \ref MOC_Send uses either buffered or synchronous send depending on the message-size
* (Short messages are transmitted using buffered send and long messages are transmitted using
* synchronous send).
*
* \ref MOC_Recv receives all messages irrespective wether buffered or synchronous
* communication is used.
*
* \note Because there are separate message-queues for each remote process and no distinct
* global ordering of messages, there may be some situation where a message from a
* process may never be received. This may occur when MOC_Recv, MOC_Probe and MOC_IProbe
* are used with MOC_ANY_SOURCE, because then there is the possibility that new messages
* are received before a specific message from a process with a higher rank than the other
* senders. This may also occur with lower possibility when using MOC_ANY_TAG between
* messages from the same sender but with other tags.
*
**/
/**
* \defgroup p2p_status Return Status
* \ingroup p2p_main
**/
/**
* \defgroup p2p_datatype Datatypes
* \ingroup p2p_main
**/
/**
* \defgroup p2p_synch Synchronous Communication
* \ingroup p2p_main
*
* Direct blocking sending of messages without any intermediate buffer.
*
* The blocking synchronous send-operation returns only when the receiver has completely
* received the message.
**/
/**
* \defgroup p2p_buffered Buffered Communication
* \ingroup p2p_main
*
* Blocking sending of messages with the use of an intermediate buffer.
*
* The blocking buffered send-operation returns if the send-buffer can be safely reused
* (after the data has been copied into the intermediate buffer).
**/
/**
* \defgroup p2p_sendrecv Send-Receive
* \ingroup p2p_main
*
* The send-receive operations combine in one call the sending of a message
* to one destination and the receiving of another message, from another process.
*
* A message sent by a send-receive operation can be received by a regular receive
* operation or probed by a probe operation; a send-receive operation can receive
* a message sent by a regular send operation.
**/
/**
* \defgroup p2p_imm Non-Blocking Communication
* \ingroup p2p_main
*
* Non-blocking sending and receiving messages.
*
* Because of restrictions of the Cell Broadband Engine non-blocking communication can
* only be provided with a lot of limitations. In fact the only operation where
* non-blocking communication make sense is synchronous send.
*
* \ref MOC_ISend uses blocking buffered send or non-blocking synchronous send according to
* the message size.
*
* \note Every non-blocking communication operation has to be finished using \ref MOC_Test
* or \ref MOC_Wait, otherwise resources are no freed and some data are not copied correctly.
**/
/**
* \defgroup broadcast Collective Communication
* \ingroup end_user
*
* Communication that involves a group of processes.
*
* A collective operation is normally executed by having all processes in the group call
* the communication routine, with matching arguments. Since there are no means for creating
* groups currently implemented, the only available groups are MOC_COMM_WORLD and
* MOC_COMM_SELF.
**/
/**
* \defgroup dma_access Direct Memory Access
* \ingroup end_user
*
* Functions which use DMA to transfer Data.
*
* The operations defined in this section provide simple means to use Direct Memory
* Access (DMA).
*
* The MPI-standard assumes a distributed memory architecture, where
* active participation of the sending and the receiving processes is required. But
* the Cell Broadband Engine is a distributed shared memory architecture which provides
* a common address space and DMA-Operations enabling processes to write data directly
* into the memory of an remote process without active participation of the receiving
* process.
* DMA is faster than Send-Receive but more complicated to handle, and so there are some
* functions provided to enable the advantages of DMA and to hide as much complexity as
* possible. So the user doesn't have to care about the upper-limit of an DMA-transfer
* and in get-operations the data-size doesn't have to be a multiple of 16-byte. However
* the data-address should be at least 16-byte aligned and in put-operations the data-size
* has to be a multiple of 16-byte.
*
* There are two kind of addresses. Local-store addresses are used on the SPUs and are only
* valid in context of a specific SPU. Effective addresses define a common address-space and
* are used on the PPU. Functions to translate between those address types are provided.
*
* DMA-operations can be blocking or non-blocking. The status of non-blocking DMA-calls can
* be requested by using \ref MOC_Test and \ref MOC_Wait.
*
* \note Every non-blocking DMA operation has to be finished using \ref MOC_Test
* or \ref MOC_Wait, otherwise resources are no freed and some data are not copied correctly.
**/
/**
* \defgroup comm_main Groups, Context and Communicators
* \ingroup end_user
*
* The MOC communicator specifies a group of processes inside which
* a communication occurs.
* Initially there is a single communicator, MPI_COMM_WORLD, which
* contains all of the processes involved in the computation.
**/
/**
* \defgroup comm_mgt Communicator Management
* \ingroup comm_main
**/
#endif /*MOC_DOXYGEN_MAIN_H_*/
include/doxygen_pages.doxygen
Executable
+204
View File
@@ -0,0 +1,204 @@
/**
* \page restrictions Restrictions of the Cell-Architecture
*
* \par
* \section res_arch CBE Architecture
*
* There are some restriction caused by the CBE architecture. Programming a SPU can be
* compared to programming a microcontroller:
* \li \b Small \b local-store \b of \b SPUs: Only 256 KByte of local-store are available
* on a SPU which is moreover shared by data and instructions. Because of this a economic
* stewardship of available resources is needed.
* \li \b No \b underlying \b OS: The operation system is running on the PPU, but not on
* the SPUs. Although the OS also manages the SPUs, there are some things which
* are missing or have to be done in a very performance decreasing and complex way.
* E.g. there is no support for several threads on one SPU (unless you program it yourself)
* and most of the system-calls are delegated to the PPU.
* \li \b Limited \b I/O-Capabilities: File-operations are only possible on the PPU. So if you
* want to store data of an SPU into a file, you have to send it to the PPU who then stores
* the data into a file. Main-memory is managed by the PPU and the SPUs can only read
* from or write into the main-memory. Allocation of main-memory has to be done by the PPU.
*
*
* \par
* \section res_alignment Memory-Alignment
*
* All Buffers used in send- or receive-functions have to be aligned at a 16-byte boundary
* (for better performance a 128-byte alignment is recommended). This restriction cannot be
* circumvented.
*
*
* \par
* \section res_size Transfer-Size
*
* DMA-calls only transfer data in chunks of 16 bytes. Send-receive operations and
* DMA-get-operations provided by this library hide this restriction by the use of intermediate
* buffers. But you should keep this restriction in mind because to gain maximum performance
* the size of data to be transfered should be a multiple of 16 bytes.
*
*
*
**/
/**
* \page violations Violations of the MPI-Standard
*
* \par
* \section viol_prefix Prefix of functions:
*
* The prefix of functions and data-structures is not 'MPI_' as stated by the MPI-standard.
* Instead it is 'MOC_' to allow the usage of another MPI-Library (e.g. to connect several
* playstations) and to stress that this implementation is not fully conform with the
* MPI-standard.
*
*
* \par
* \section viol_startup Application Startup
*
* MPI-applications (Version 1.3) are started using mpiexec/mpirun and normally only one
* binary is executed on all MPI-nodes. Because PPU- and SPU-instructions aren't compatible
* at least two executables (and two compilers) are needed. Also SPU-threads have to be initialized
* and started by the PPU. Because of this the SPU-binaries are embedded into the PPU-binary
* and started by this PPU-binary during runtime.
* In this library this concept is also used. So you have to program at least one PPU- and one
* SPU-program, embed the SPU-program into the PPU-executable and add the SPU-program-handle to
* the MOC-environment (see \ref MOC_Add_spu_thread) during runtime before calling MOC_Init.
*
*
* \par
* \section viol_queue Order of received messages:
*
* Because of the internal design of this implementation the order of
* incoming messages cannot be preserved in all cases. Currently the
* correct order is only preserved from messages from the same source,
* with the same message-tag and same communicator.
*
*
* \par
* \section viol_tag Number of usable message-tag
*
* According to the MPI-standard there should be at least 32767 tags available. But to save
* memory we use only one byte to encode the message-tag. So there are only 256 usable
* message-tags. Message-tags greater than 249 are reserved for internal purposes.
*
*
* \par
* \section viol_nr_pending Number of simultaneous pending send-receive operations
*
* According to the MPI-standard there should be hardly any limitations in the number of
* possible simultaneous transfers. It is stated that only in pathological cases the maximum
* of possible outstanding transfers should be reached. Because the resources of an SPU are
* very limited our implementation has a tight limit of possible simultaneous transfers.
* Although the upper-limit can be changed using \ref MOC_Set_queue_size, it is limited by
* the 256 KByte of local-storage of a SPU.
* There is also a limit of the number of simultaneous DMA-calls, because for each DMA-operation
* an exclusive DMA-tag is used and there are only 32 DMA-tag (on the PPU only 15) available.
*
* \par
* \section viol_init MPI_Init:
*
* The parameters of the init-function for the SPU does not match the parameters
* defined by the MPI-standard (see \ref moc_init_spu "MOC_Init (SPU)")
*
*
**/
/**
* \page performance Performance Measurements
*
* The measurements were taken on a Playstation 3 running Yellow Dog Linux release 6.0
* using Cell SDK 3.1. To measure send-receive operations ping-pongs were used and to
* measure DMA-calls the time till the used DMA-tag is marked to have no pending calls
* is used. The stated latencies in the pictures above are one-way latencies.
*
* \section perf_send_recv Send-Receive performance
*
* The following two figures show latency and bandwidth achieved by our implementation.
* The theoretical maximum bandwidth is 25 GByte/s and synchronous send between SPUs is
* at a close range to this limit if the data-size is large enough. There is also a big
* difference between SPU-SPU and SPU-PPU operations. The reason for this is the fact that
* the PPU cannot handle DMA-operations itself but only remote-control the SPU's
* Memory Flow Controller. And as you will see later the main-memory performance of the PPU
* is much lower than that of the SPUs. Because we use ping-pongs there is no way to
* determine differences in sending from or to the PPU, but other measurements indicate
* such differences.
*
* \image html latency-kb.png
*
* \image html bandwidth-kb.png
*
* The following figure shows the latency of send-receive operations at a lower scale than
* the first figure. If you look at the latency of synchronous send operations then you can
* see that there is a big start-up latency and only a very small data-size dependend latency.
* So there's only a very small time-difference between sending 10 byte and 1 Kbyte, but there
* is naturally a difference in throughput.
*
* \image html latency-b.png
*
* \section perf_moc_vs_dma Send-Recv vs. DMA performance
*
* The following two figures demonstrate the difference in performance of send-recv operations
* and DMA-calls. The performance of DMA-calls reaches it's peak much earlier than send-recv
* and the peak-value is higher than that of send-recv even though the difference is not much.
* So DMA is a lot faster than send-recv especially for small messages, but for big messages
* the difference is much lower. It seems that it is always better to use DMA, but keep in mind
* that in practical applications calculation time easily exceeds communication time.
*
* \image html dma-vs-moc-lat.png
*
* \image html dma-vs-moc-bw.png
*
* \section perf_spu_vs_ppu SPU- vs PPU-initiated transfer
*
* The following figure shows the difference between SPU- and PPU-initiated DMA-calls.
* As you can see the highest performance can be achieved by using SPU-SPU DMA-calls. There
* is also hardly any difference between put and get operations. DMA-operations between SPU and
* main-memory achieves lower performance and there is a difference between put and get
* operations. PPU to SPU DMA calls have the lowest performance with small transfer sizes, but
* if the transfer-size is big enough they are faster than SPU-main-memory transfers.
* The lowest performance has a memcopy on the PPU, which also explains the bad performance of
* buffered send between SPU and PPU.
*
* \image html spu-ppu-dma-bw-kb.png
*
* \section perf_mem DMA vs. memcpy
*
* The official Cell documentation states that it is also possible to use
* memcpy to transfer data from the PPU to a SPU. However one should use
* DMA, because it is always faster.
* If you look at the following figure, this is not always true. If the
* data-size is small enough memcpy is faster.
*
* \image html ppu-transfer-bw-b.png
*
**/
/**
* \page conclusion Summary & Conclusions
*
* \details MPI wasn't designed for the Cell Broadband Engine. A common MPI-node has one or more
* normal CPUs and several GBytes of RAM. The Cell Broadband Engine features a normal CPU
* (the PPU) and several special processing units with only a very limited amount
* of memory. The SPUs support only one thread per SPU and a complex handling of system-calls
* is needed. Because of this there are several restrictions in implementing MPI on the Cell
* Broadband Engine.
*
* Nevertheless MPI on Cell makes sense. It greatly simlifies programming on the Cell Broadband
* Engine, features a public API which is well known and widely used and message passing operations
* are generally easier to use than DMA (DMA needs memory-addresses and explicit synchronization,
* whereas MPI only needs the ranks of participating processes and features implicit
* synchronization).
*
* But message parsing operations cannot replace DMA. Both methods of communication have their
* advantages and disadvantages and they complement one another. So this library additionally
* features some easy to use DMA operations, if someone wants to gain maximum performance. Using
* DMA alltogether with send-recv especially speeds up application with a lot of communication
* going on between SPUs and PPU.
**/
include/internal/doxygen_internal_main.doxygen
Executable
+218
View File
@@ -0,0 +1,218 @@
#ifndef MOC_INTERNAL_DOXYGEN_MAIN_H_
#define MOC_INTERNAL_DOXYGEN_MAIN_H_
/**
* \defgroup internal_main Developer Documentation
*
* This section deals with the internal design and functionality. For correct
* understanding knowledge about the functionality of the Cell Broadband Engine
* is needed. So you should read the official CBE documentation before you
* continue.
*
* \note Because we had only access to a playstation, we cannot guarantee correct
* execution on another CBE-machine than a playstation 3.
**/
/**
* \defgroup internal_core Implementation-Core
* \ingroup internal_main
**/
/**
* \defgroup internal_am Address-Macros
* \ingroup internal_core
*
* Predefined Macros for calculating Addresses of Message-Queue and Status-Array.
**/
/**
* \defgroup internal_tm Test-Macros
* \ingroup internal_core
*
* Predefined Macros for testing the validity of parameters.
**/
/**
* \defgroup internal_env Environmental Management
* \ingroup internal_main
*
* Internals about Startup & Shutdown and Error-Handling
**/
/**
* \defgroup internal_init Startup & Shutdown
* \ingroup internal_env
*
* Internals about the startup of the execution-environment and
* SPU-Thread-Management.
*
* \section iinit_start Startup
*
* The PPU-Process creates for every SPU-thread an own thread using the
* pthread-library (see \ref internal_spu_thread). The rank of each prozess,
* the overall process count, the sizes of message-queue and buffer-space
* and the effective base address of the buffer is send to each SPU-thread via
* mailbox-messages. The SPU-threads then send the memory-addresses of their
* message-queues and status-arrays back to the PPU-process which writes this
* data into the prozess-list. After all SPU-Threads have reported back to the
* PPU-process the effective address of the PPU's process-list ist distributed
* among the SPU-processes which then copy the process-list via DMA in their
* local storages.
*
* \section iinit_shut Shutdown
*
* \subsection iinit_shut_ppu PPU
*
* The PPU-process waits till all SPU-threads have signalled that they are also
* executing the finalize-function. After that a signal is send back to each
* SPU-Thread that they can now stop execution. Then local ressources are freed
* and the finalize-function returns.
*
* \subsection iinit_shut_spu SPU
*
* A mailbox-message is send to the PPU to signal that the SPU-process is now
* executing the finalize-function. After that the SPU-process is waiting till
* a mailbox-message from the PPU-process is received. Then local ressources are
* freed and the finalize-function returns.
*
**/
/**
* \defgroup internal_spu_thread SPU-Thread Management
* \ingroup internal_init
*
* This section deals about the creation and management of SPU-threads.
**/
/**
* \defgroup internal_error Error-Handling
* \ingroup internal_env
**/
/**
* \defgroup internal_timer Timers and Synchronization
* \ingroup internal_env
*
* Internals about Time.
*
* On the PPU "gettimeofday()" from time.h is used, because it is the only function
* which seems to work ("clock()" from time.h always returns 0).
*
* On the SPU there are two implementations. One uses also "gettimeofday()" and the
* other uses the spu-timer-library. The reason for this lies in the fact, that the
* spu-timer-library was marked as prototype in Cell SDK 3.0 and the future development
* of this library was uncertain. Cell SDK 3.1 has integrated the spu-timer-library in
* the main-library and so the gettimeofday-implementation is now obsolete.
*
* \note The timebase-constant which is needed to calculate time is hardcoded, because
* there was no according function for retrieving this value in Cell SDK 3.0 and we had
* only access to a playstation 3. Now Cell SDK 3.1 provides a function for retrieving
* the timebase value, but we didn't use it to provide backwards-compatibility. So if
* you plan to use MOC on another platform than playstation 3 keep in mind to check if
* the timebase-value is correct.
*
**/
/**
* \defgroup internal_comm Groups, Context and Communicators
* \ingroup internal_main
*
* At the moment user-defined groups and communicators are not implemented and maybe never
* will be.
* At the moment user-defined groups and communicators are not implemented and maybe never
* will be. Nevertheless user-defined groups and communicators have been taken into
* consideration while making the software-design (e.g. reserved field for communicator
* in message-envelope). So it should be not too hard to implement groups and communicators.
* However a lot of things concerning communicators are hardcoded (e.g. parameter checks).
*
* To save memory only one byte is used to encode the communicator. So if one
* day user-defined communicators are implemented there are only 256 possible
* communicators.
**/
/**
* \defgroup internal_p2p_main Point-to-Point Communication
* \ingroup internal_main
*
* Internals about Point-to-Point Communication.
**/
/**
* \defgroup internal_p2p_dma DMA-Helper Functions
* \ingroup internal_p2p_main
*
* Functions wrapping DMA-Transfers.
*
* The Cell Broadband Engine uses Direct Memory Access (DMA) for communication. Therefore
* data of a MOC-message is transfered using DMA operations. Because programming DMA is
* rather complex, some functions are provided to hide this complexity. One DMA-call can
* only tranfer a maximum of 16KByte of data and the data-size has to be a multiple of
* 16 byte. The provided functions split the data into multiple DMA-calls if needed. If
* the data-size requires it, an intermediate buffer is used into which a separate
* DMA-Call transfers 16 byte containing the piece of data not fitting into the 16-byte
* scheme. Then only the needed bytes are copied into the receive-buffer. But the latter
* only works for a get-operation.
*
* The internal DMA-functions are also made publicly available via wrapping-function
* (see chapter \ref dma_access) which provide parameter checks.
**/
/**
* \defgroup internal_p2p_blocking Blocking Communication
* \ingroup internal_p2p_main
*
* Internals about blocking Point-to-Point Communication.
**/
/**
* \defgroup internal_p2p_synch Synchronous Communication
* \ingroup internal_p2p_blocking
*
* Internals about blocking synchronous Communication.
*
* \image html synchronous_communication_1.png
**/
/**
* \defgroup internal_p2p_buffered Buffered Communication
* \ingroup internal_p2p_blocking
*
* Internals about blocking buffered Communication.
*
* \image html buffered_communication_1.png
**/
/**
* \defgroup internal_p2p_bmalloc Buffer-Management
* \ingroup internal_p2p_buffered
*
* This section shows how the buffer used for buffered communication is managed.
**/
/**
* \defgroup internal_p2p_imm Non-Blocking Communication
* \ingroup internal_p2p_main
*
* Internals about non-blocking communication.
*
* The SPUs are very limited in usage, e.g. only one thread per SPU is supported,
* and so it is hard to make communication non-blocking. In fact only synchronous
* send and receive provide possibilities for non-blocking communication. Because
* we think that the advantages of a non-blocking receive aren't worth the effort,
* only non-blocking send is implemented at the moment. The picture below illustrates
* a non-blocking synchronous communication call.
*
* \image html isyncronous_communication_1.png
**/
/**
* \defgroup internal_collective Collective Communication
* \ingroup internal_main
*
* Internals about Collective Communication
*
* Currently there are only two operations implemented.
*
* \section MOC_Bcast
*
* The root node sends normal messages using the reserved tag MOC_BCAST_TAG, which
* is received by all other nodes.
*
* \section MOC_Barrier
*
* All but the process with the highest rank send a special-message (special-flag set)
* to the latter process. This process then sends another special-message back to
* the others. After that the function returns on all calling processes.
*/
#endif /*MOC_DOXYGEN_MAIN_H_*/
include/internal/imoc.h
Executable
+201
View File
@@ -0,0 +1,201 @@
#ifndef IMOC_H_
#define IMOC_H_
#include <stdint.h>
#include "../moc.h"
#include "imoc_core.h"
/**
* \addtogroup internal_p2p_imm
**/
//@{
/**
* @name Constants for MOC_Request.request_type
*/
//@{
/**
* MOC_Request is associated with a DMA-Communication.
**/
#define MOC_REQ_TYPE_DMA 0
/**
* MOC_Request is associated with aISsend-Communication.
**/
#define MOC_REQ_TYPE_ISSEND 1
//@}
//@}
/**
* \addtogroup internal_p2p_main
**/
//@{
/**
* @name Internal constants
*/
//@{
/**
* Defines a treshold, when to use memcpy and when to use DMA in a get-operation
* on the PPU.
**/
#define MEMCPY_GET_TRESHOLD 156
/**
* Defines a treshold, when to use memcpy and when to use DMA in a put-operation
* on the PPU.
**/
#define MEMCPY_PUT_TRESHOLD 2210
//@}
/**
* @name Internal shared functions
*/
//@{
/**
* Get the next valid message-envelope from "source" with specified "tag" and "comm".
*
* @param source \b in rank of sending node
* @param tag \b in message-tag
* @param comm \b in message-communicator
* @param isSpecial \b in true to return special-message
* @param status \b inout MOC_status-instance to add some information (e.g. source by MOC_ANY_SOURCE)
* @returns pointer to the appropriate message-queue-entry.
**/
moc_message_envelope_t* IMOC_get_entry(int source, int tag, MOC_Comm comm, int isSpecial, MOC_Status *status);
/**
* Get next free entry in message-queue (a local copy of the message-queue has to be used).
* To ensure correct ordering message-tag and -comm and -type has to be specified.
*
* @param pqueue \b in pointer to local copy of remote message-queue
* @param tag \b in message-tag
* @param comm \b in message-communicator
* @param isSpecial \b in true if message should be a special message
* @param index \b out index of free entry
* @return 0 when success otherwise error-code
**/
int IMOC_getFreeEntry ( moc_message_envelope_t* pqueue, uint8_t tag, MOC_Comm comm, int isSpecial, int* index );
/**
* Get free message-ID.
*
* \note Message-ID is index of message-status-array
*
* @param mid \b out free message-ID
* @retval MOC_SUCCESS No error; MPI routine completed successfully.
* @retval MOC_ERR_MESSAGE_EXHAUSTED No further messages possible.
**/
int IMOC_getFreeMID ( int *mid );
/**
* Wait till message with message-ID mid has been received.
*
* Waits till message-status-flag 'outstanding' is 0.
*
* @param mid \b in message-ID
* @return 0 when success otherwise error-code
**/
int IMOC_waitForMessageCompletion ( uint8_t mid );
/**
* Tests if message with message-ID mid has been received.
*
* Tests if message-status-flag 'outstanding' is 0.
*
* @param mid \b in message-ID
* @param status \b out status of message
* @retval MOC_SUCCESS No error; MPI routine completed successfully.
**/
int IMOC_testForMessageCompletion ( uint8_t mid, uint8_t *status );
//@}
//@}
/**
* \addtogroup internal_error
*
* This section deals with error-messages.
*
* An array of type \ref error_strings_struct stores error-messages and
* their corresponding error-codes. \ref MOC_Error_string searches this
* list for a given error-code and returns the error-message.
**/
//@{
/**
* Error-Strings structure.
*
* stores the error-string to a given error-code
**/
typedef struct error_strings_struct
{
int error_code; /**< error-code */
char * text; /**< error-text */
} error_strings_t; /**< typedef for \ref error_strings_struct */
//@}
/**
* \addtogroup internal_p2p_bmalloc
*
* Information about already allocated buffer-space is stored in the
* status-array. But this information has no order which forces us
* everytime we want to allocate buffer-space to create an ordered list
* of already allocated buffer-space. To avoid this we have additionally
* an ordered linked-list. This list orders the indexes (== message-IDs)
* of the status-array-entries in ascending order of the memory-addresses.
* This enables fast searching for free buffer-space.
*
* Allocated buffer-space is cleared by the receiving process by setting
* the appropriate flag in the status-array-entry. This is done by a
* DMA-Transfer and so no collaboration of the sending-process is needed.
* During the next allocation of buffer-space nodes with disabled buffer-flags
* are freed.
**/
//@{
/**
* Node of linked list holding information about allocated buffer-space.
**/
typedef struct moc_internal_buffer_node_struct
{
struct moc_internal_buffer_node_struct *next_node; /**< Pointer to
next entry **/
uint8_t mid; /**< Message-ID of message using buffer-space.
Information about the used buffer-space is stored in the
appropriate message-status-entry (mid is index of entry). **/
} moc_internal_buffer_node_t;
/**
* Allocate buffer-space.
*
* @param size \b in size of buffer-space
* @param mid \b in corresponding message-ID
* @return effective address of allocated buffer-space (16-byte aligned)
**/
uint64_t IMOC_bmalloc ( uint32_t size, uint8_t mid );
//@}
#endif /*IMOC_H_*/
include/internal/imoc_core.h
Executable
+173
View File
@@ -0,0 +1,173 @@
#ifndef MOC_INTERNAL_CORE_
#define MOC_INTERNAL_CORE_
#include <stdint.h>
/**
* \addtogroup internal_core
*
* This section deals with the data-structures and the basic concepts which
* form the core of this implementation.
*
* \section icore_struct Data-Structures
*
* Basically there are four data-structures. \ref moc_message_envelope_struct
* holds meta-information about a message and \ref moc_message_status_struct holds
* information about used buffer-space and provide means for a receiving proccess
* to send some information back to the sending process.
*
* \ref moc_process_struct stores information every node needs to know about the
* other nodes to successfully communicate with them.
*
* The forth data-structure (actually there are two) \ref moc_node_handle_ppu_struct
* respectively \ref moc_node_handle_spu_struct holds information a node know about
* itself and the execution environment.
*
* \section core_conc Concepts
*
* Every node has (N-1) queues for incoming messages, where N is the overall number
* of node. Each queue is exclusively used by a specific remote node to write
* meta-data about the message to be send into it (queue-entries are of type \ref
* moc_message_envelope_struct). Because there are several independend queues and no
* global time is available, an exact order of all incoming messages cannot be guaranteed.
* Corrent ordering can only be guaranteed for messages from the same source-node.
* The queues are implemented in form of a big array which is divided in (N-1) equally
* sized pieces.
*
* Furthermore every node has an array of size (N-1)*m where m is the size of a
* message-queue (entries are of type \ref moc_message_status_struct).
* This array is used for buffer-management and to report the status of a message
* back to the sender.
* Every message currently under progress has a message-id which is unique in respect
* to the sending-node. This message-id is equal to the index of the corresponding
* entry of the status-array.
*
* In order to communicate with other nodes every node also keeps a list of all nodes
* in the execution environment (entries are of type \ref moc_process_struct).
*
* Pointers to the own arrays and some further information a node needs to know (e.g.
* own rank) is stored in the globally accessible variable \ref node_handle.
**/
//@{
/**
* @name PPU & SPU
**/
//@{
/**
* Default size of the message-queues.
**/
#define MOC_QUEUE_SIZE 4
/**
* Default size of the buffers used for buffered communication.
**/
#define MOC_BUFFER_SIZE 2048
/**
* Message-envelope structure.
*
* Is used to store meta-data about the message in the message-queues of the receiving process.
**/
typedef struct moc_message_envelope_struct
{
uint8_t id; /**< Message-ID. Also index of message-status-array*/
uint8_t tag; /**< Message-tag*/
uint8_t communicator; /**< Communicator of message*/
union
{
struct
{
unsigned int valid : 1; /**< Is the message valid? */
unsigned int signal_back : 1; /**< Should the status be reported back to the
sending process */
unsigned int special : 1; /**< Is this message a special message? */
unsigned int collaborative : 1; /**< Is this message part of a collaborative
communication? */
unsigned int channel : 2; /**< Which channel is used to transport the message? */
unsigned int in_mm : 1; /**< Are the data in Main-Memory? */
unsigned int keep_valid : 1; /**< Should the message-status-entry be invalidated? */
} __attribute__((__packed__)) bits; /**< flag-struct */
uint8_t value; /**< union-value */
} flags; /**< flag-union */
unsigned int size : 24; /**< Message-size in byte*/
unsigned int datatype : 8; /**< Data-type of message-entries */
uint64_t message_address; /**< Effective address of message-data*/
} __attribute__((__packed__)) moc_message_envelope_t;
/**
* Constant for moc_message_envelope_t.flags.channel.
*
* Message is transported via a DMA-Call.
**/
#define IMOC_MESSAGE_CHANNEL_DMA 0
/**
* Constant for moc_message_envelope_t.flags.channel.
*
* Message is embedded in message-envelope.
**/
#define IMOC_MESSAGE_CHANNEL_ENV 1
/**
* Message-status structure.
*
* Is used to store information about the message-status and the buffer used in the
* message-status-array.
**/
typedef struct moc_message_status_struct
{
uint32_t buffer_offset; /**< allocated buffer-offset from buffer-base */
uint32_t message_size; /**< size of message in byte */
uint32_t error_value; /**< optional error-code of message */
union
{
struct
{
unsigned int valid : 1; /**< Is this entry valid? */
unsigned int outstanding : 1; /**< Message is not yet received? */
unsigned int buffer_used : 1; /**< Is Bufferspace used? */
unsigned int reserved : 5; /**< reserved */
} __attribute__((__packed__)) bits; /**< flag-struct */
uint8_t value; /**< union-value */
} flags; /**< flag-union */
uint8_t reserved[3]; /**< reserved */
} __attribute__((__packed__)) moc_message_status_t;
/**
* Constant for moc_process_struct.type.
**/
#define MOC_PPU_PROCESS 0
/**
* Constant for moc_process_struct.type.
**/
#define MOC_SPU_PROCESS 1
/**
* Process-information structure.
*
* Stores information about a MOC-Process. This information is used for communication
* between the processes.
*/
typedef struct moc_process_struct
{
uint8_t type; /**< Process-type (SPU or PPU) */
uint8_t reserved[7]; /**< reserved */
uint64_t inbox_queue_base; /**< effective base-address of message-queues */
uint64_t stat_array_base; /**< effective base-address of message-status-array */
uint64_t ps_base; /**< effective base-address of problem-state (Signals, mailboxes, ...)*/
} __attribute__((__packed__)) moc_process_t;
//@}
//@}
#endif /*MOC_INTERNAL_CORE_*/
include/internal/imoc_macros.h
Executable
+141
View File
@@ -0,0 +1,141 @@
#ifndef MOC_INTERNAL_ADDRESS_MACROS_H_
#define MOC_INTERNAL_ADDRESS_MACROS_H_
/**
* \addtogroup internal_am
**/
//@{
/**
* @name PPU & SPU
**/
//@{
/**
* Effective base-address of own remote queue at node 'rank'.
*
* Address is of type uint64_t
**/
#define IMOC_EA_REMOTE_QUEUE(proc) ( (proc > node_handle.rank) ? node_handle.moc_process_list[proc].inbox_queue_base+sizeof(moc_message_envelope_t)*node_handle.queue_size*node_handle.rank : node_handle.moc_process_list[proc].inbox_queue_base+sizeof(moc_message_envelope_t)*node_handle.queue_size*(node_handle.rank-1) )
/**
* Effective Base-Address of local queue responsible for node 'rank'.
*
* Address is of type moc_message_envelope_t*
**/
#define IMOC_LOCAL_QUEUE(proc) ( (proc < node_handle.rank) ? node_handle.message_queue_base+node_handle.queue_size*(proc) : node_handle.message_queue_base+node_handle.queue_size*(proc-1) )
/**
* Effective base-address of own remote queue-entry at node 'rank'.
*
* Address is of type uint64_t
**/
#define IMOC_EA_REMOTE_QUEUE_ENTRY(proc, index) ( (proc > node_handle.rank) ? node_handle.moc_process_list[proc].inbox_queue_base+sizeof(moc_message_envelope_t)*node_handle.queue_size*node_handle.rank+sizeof(moc_message_envelope_t)*(index) : node_handle.moc_process_list[proc].inbox_queue_base+sizeof(moc_message_envelope_t)*node_handle.queue_size*(node_handle.rank-1)+sizeof(moc_message_envelope_t)*(index) )
/**
* Effective Base-Address of local queue-entry responsible for node 'rank'.
*
* Address is of type moc_message_envelope_t*
**/
#define IMOC_LOCAL_QUEUE_ENTRY(proc, index) ( (proc < node_handle.rank) ? node_handle.message_queue_base+node_handle.queue_size*(proc)+(index) : node_handle.message_queue_base+node_handle.queue_size*(proc-1)+(index) )
/**
* Effective Address of remote status-array entry at node 'rank'.
*
* Address is of type uint64_t
**/
#define IMOC_EA_REMOTE_STATUS_ENTRY(id, rank) ( node_handle.moc_process_list[rank].stat_array_base+sizeof(moc_message_status_t)*(id) )
//@}
//@}
/**
* \addtogroup internal_tm
**/
//@{
/**
* Tests if invalid communicator in a send-operation.
* Calls return with appropriate error-code itself.
**/
#define IMOC_TEST_COMM_SEND(comm) if ( (comm) != MOC_COMM_WORLD && (comm) != MOC_COMM_SELF ) \
{ return MOC_ERR_COMM; }
/**
* Tests if invalid communicator in a recv-operation.
* Calls return with appropriate error-code itself.
**/
#define IMOC_TEST_COMM_RECV(comm) if ( (comm) != MOC_COMM_WORLD && (comm) != MOC_COMM_SELF ) \
{ return MOC_ERR_COMM; }
/**
* Tests if invalid tag in a send-operation.
* Calls return with appropriate error-code itself.
**/
#define IMOC_TEST_TAG_SEND(tag) if ( (tag) > 255 || (tag) < 0 ) \
{ return MOC_ERR_TAG; }
/**
* Tests if invalid tag in a recv-operation.
* Calls return with appropriate error-code itself.
**/
#define IMOC_TEST_TAG_RECV(tag) if ( (tag)!=MOC_ANY_TAG && ((tag) > 255 || (tag) < 0) ) \
{ return MOC_ERR_TAG; }
/**
* Tests if invalid rank.
* Calls return with appropriate error-code itself.
**/
#define IMOC_TEST_RANK(rank) if ( (rank) < 0 || (rank) >= node_handle.comm_size ) \
{ return MOC_ERR_RANK; }
/**
* Tests if invalid rank in a recv-operation.
* Calls return with appropriate error-code itself.
**/
#define IMOC_TEST_RANK_RECV(rank) if ( rank != MOC_ANY_SOURCE && ((rank) < 0 || (rank) >= node_handle.comm_size) ) \
{ return MOC_ERR_RANK; }
/**
* Tests if invalid byte-count in a recv-operation.
* Calls return with appropriate error-code itself.
**/
#define IMOC_TEST_BYTECOUNT(count) if ( (count) < 0 || (count) > MAX_MOC_MESSAGE_SIZE ) \
{ return MOC_ERR_COUNT; }
/**
* Tests if invalid buffer-pointer.
* Calls return with appropriate error-code itself.
**/
#define IMOC_TEST_BUFFER(buf) if ( (buf) == 0 ) \
{ return MOC_ERR_BUFFER; }
/**
* Tests if rank is MOC_PROC_NULL and returns with MOC_SUCCESS itself.
**/
#define IMOC_TEST_PROC_NULL(rank) if ((rank) == MOC_PROC_NULL ) \
{ return MOC_SUCCESS; }
/**
* Tests if rank is own rank and returns with MOC_SUCCESS itself.
**/
#define IMOC_TEST_PROC_SELF(proc) if ( (proc) == node_handle.rank ) \
{ return MOC_SUCCESS; }
/**
* Tests if address is 16-byte aligned.
* Calls return with appropriate error-code itself.
**/
#define IMOC_TEST_ALIGNMENT(address) if ( ((unsigned long)(address)) % 16 != 0 ) \
{ return MOC_ERR_BAD_ALIGNMENT; }
//@}
#endif /*MOC_INTERNAL_ADDRESS_MACROS_H_*/
include/internal/ppu/imoc_core_ppu.h
Executable
+132
View File
@@ -0,0 +1,132 @@
#ifndef MOC_INTERNAL_CORE_PPU_H_
#define MOC_INTERNAL_CORE_PPU_H_
#include <stdint.h>
#include <libspe2.h>
#include <pthread.h>
#include "../imoc_core.h"
#include "../../ppu/moc.h"
/**
* \addtogroup internal_core
**/
//@{
/**
* @name PPU only
**/
//@{
/**
* Node-handle structure.
*
* \note PPU only
*
* Information each node knowns about himself and the world.
**/
typedef struct moc_node_handle_ppu_struct
{
uint8_t rank; /**< rank of node */
uint8_t comm_size; /**< number of overall processes */
uint8_t queue_size; /**< Size of a message-queue */
volatile moc_message_envelope_t* message_queue_base; /**< base-address of own message-queues */
volatile moc_message_status_t* status_array_base; /**< base-address of own message-status-array */
void* buffer_base; /**< base-address of own buffer */
uint32_t buffer_size; /**< size of own buffer */
volatile uint32_t* signal_area; /**< address of own signal-area */
volatile moc_process_t* moc_process_list; /**< list of all other MOC-processes
(index is rank of process) */
} moc_node_handle_ppu_t; /**< typedef of \ref moc_node_handle_ppu_struct */
/**
* Information each node knowns about himself and the world.
*
* \note PPU only
*
**/
moc_node_handle_ppu_t node_handle;
//@}
//@}
/**
* \addtogroup internal_spu_thread SPU-Thread Management
*
* \note PPU only
**/
//@{
/**
* SPU-Thread structure.
*
* \note PPU only
*
* Stores information about the SPU-Threads the managing PPU-Process
* needs to know.
**/
typedef struct moc_spu_thread_struct
{
uint8_t rank; /**< rank of SPU-thread */
pthread_t pthread; /**< PThread-handle <*/
spe_context_ptr_t speid; /**< SPE-Context */
spe_stop_info_t stop_info; /**< Error-Information */
uint32_t spe_entry_adress; /**< Spe-entry-address */
void *argp; /**< effective address of user-defined information
in main-memory */
void *envp; /**< Pointer to environment information */
void* buffer_base; /**< base-address of the SPU-Thread's buffer */
uint32_t buffer_size; /**< size of the SPU-Thread's buffer */
uint64_t ls_base; /**< Effective Basis-Address of SPU's local-storage */
uint32_t queue_base_in_ls; /**< Address of message-queue in SPU's local-storage */
uint32_t status_base_in_ls; /**< Address of status-array in SPU's local-storage */
} moc_spu_thread_t; /**< typedef of \ref moc_spu_thread_struct */
/**
* actual number of SPU-Threads.
*
* \note PPU only
**/
uint8_t moc_spu_thread_count;
/**
* actual number of PPU-Threads.
*
* \note PPU only
**/
uint8_t moc_ppu_thread_count;
/**
* init buffer size
**/
uint32_t moc_spu_buffer_size;
/**
* init message-queue size
**/
uint8_t moc_spu_queue_size;
/**
* array of SPU-Threads.
*
* \note PPU only
**/
moc_spu_thread_t moc_spu_thread_list[MOC_SPU_THREAD_MAX];
/**
* PThread-function for creating SPU-Threads
**/
void* moc_spe_pthread_func(void *arg);
//@}
#endif /*MOC_INTERNAL_CORE_PPU_H_*/
include/internal/ppu/imoc_dma_ppu.h
Executable
+148
View File
@@ -0,0 +1,148 @@
#ifndef MOC_INTERNAL_DATA_TRANSFER_PPU_H_
#define MOC_INTERNAL_DATA_TRANSFER_PPU_H_
#include <stdint.h>
#include "../../moc.h"
/**
* \addtogroup internal_p2p_dma
**/
//@{
/**
* @name PPU-only
**/
//@{
/**
* Internal wrapper-function for a blocking DMA-get operation.
* Data can be of any size, constraints of low-level DMA-functions
* are dealt by this function.
* However data-addresses have to be at least 16-byte aligned.
*
* This function actually always reads a multiple of 16 from the source,
* but copies only size bytes of it to target. This means however that up
* to 15 bytes are read from an invalid location. As it is never written to
* them this should always be safe.
*
* On the ppu the system for tag reservation is not present, so a hardcoded tag is used.
* It has also to be noted that the dma calls are actually initiated on the spu by the ppu.
*
* @param target \b in local-address of buffer
* @param source \b in ea-address of data
* @param size \b in size of data in byte
* @param dest \b in id of source spe
* @return 0 when success otherwise error-code
**/
int IMOC_get_data(void* target, uint64_t source, uint32_t size, int dest);
/**
* Internal wrapper-function for a blocking DMA-put operation.
* Data-size should be a multiple of 16 because of low-level constraints, but
* there is no upper limit of transfer-size.
* Data-addresses have to be at least 16-byte aligned.
*
* This function actually always reads a multiple of 16 from the source, and copies
* it to target. This means however that up to 15 bytes are written to and invalid
* location, so if one wants to use it with invalid sizes precautions have to be taken
* in order to make it safe.
*
* On the ppu the system for tag reservation is not present, so a hardcoded tag is used.
* It has also to be noted that the dma calls are actually initiated on the spu by the ppu.
*
* @param source \b in local-address of data
* @param target \b in ea-address of buffer
* @param size \b in size of data in byte
* @param dest \b in id of source spe
* @return 0 when success otherwise error-code
**/
int IMOC_put_data(void* source, uint64_t target, uint32_t size, int dest);
/**
* Internal wrapper-function for a non-blocking DMA-get operation.
* Data can be of any size, constraints of low-level DMA-functions
* are dealt by this function.
* However data-addresses have to be at least 16-byte aligned.
*
* This function actually always reads a multiple of 16 from the source,
* but copies only size bytes of it to target. This means however that up
* to 15 bytes are read from an invalid location. As it is never written to
* them this should always be safe.
*
* On the ppu the system for tag reservation is not present, so a hardcoded tag is used.
* It has also to be noted that the dma calls are actually initiated on the spu by the ppu.
*
* @param target \b in local-address of buffer
* @param source \b in ea-address of data
* @param size \b in size of data in byte
* @param dest \b in id of source spe
* @param request \b out request-handle
* @return 0 when success otherwise error-code
**/
int IMOC_iget_data(void* target, uint64_t source, uint32_t size, uint32_t dest, imoc_dma_request* request);
/**
* Internal wrapper-function for a non-blocking DMA-put operation.
* Data-size should be a multiple of 16 because of low-level constraints, but
* there is no upper limit of transfer-size.
* Data-addresses have to be at least 16-byte aligned.
*
* This function actually always reads a multiple of 16 from the source, and copies
* it to target. This means however that up to 15 bytes are written to and invalid
* location, so if one wants to use it with invalid sizes precautions have to be taken
* in order to make it safe.
*
* On the ppu the system for tag reservation is not present, so a hardcoded tag is used.
* It has also to be noted that the dma calls are actually initiated on the spu by the ppu.
*
* @param source \b in local-address of data
* @param target \b in ea-address of buffer
* @param size \b in size of data in byte
* @param dest \b in id of source spe
* @param request \b out request-handle
* @return 0 when success otherwise error-code
**/
int IMOC_iput_data(void* source, uint64_t target, uint32_t size, uint32_t dest, imoc_dma_request* request);
/**
* Waits till a IMOC_iget_data or IMOC_iput_data is completed.
*
* @param request \b in request-handle
* @return 0 when success otherwise error-code
**/
int IMOC_wait_dma(imoc_dma_request* request);
/**
* Tests if a IMOC_iget_data or IMOC_iput_data is completed.
*
* @param request \b in request-handle
* @param flag \b out true if operation is completed
* @return 0 when success otherwise error-code
**/
int IMOC_test_dma(imoc_dma_request* request, int *flag);
/**
* Reserves a tag on the ppu, for communication with a given spu.
*
* @param speid \b id of spu to communicate with
* @return tag (0-14) when success otherwise error code
**/
int moc_tag_reserve(unsigned int speid);
/**
* Releases a reserved tag.
*
* @param speid \b id of spu for which the tag was reserved
* @param tag \b tag to be released
* @return 0 when success otherwise error-code
**/
int moc_tag_release(unsigned int speid, int tag);
//@}
//@}
#endif /*MOC_INTERNAL_DATA_TRANSFER_PPU_H_*/
include/internal/ppu/imoc_macros_ppu.h
Executable
+48
View File
@@ -0,0 +1,48 @@
#ifndef MOC_INTERNAL_ADDRESS_MACROS_PPU_H_
#define MOC_INTERNAL_ADDRESS_MACROS_PPU_H_
#include "../imoc_macros.h"
/**
* \addtogroup internal_am
**/
//@{
/**
* @name PPU-only
**/
//@{
/**
* LS-Address of own remote queue at node 'rank'.
*
* Address is of type uint32_t.
**/
#define IMOC_LS_REMOTE_QUEUE(proc) ( (proc > node_handle.rank) ? moc_spu_thread_list[proc-moc_ppu_thread_count].queue_base_in_ls+sizeof(moc_message_envelope_t)*node_handle.queue_size*node_handle.rank : moc_spu_thread_list[proc-moc_ppu_thread_count].queue_base_in_ls+sizeof(moc_message_envelope_t)*node_handle.queue_size*(node_handle.rank-1) )
/**
* LS-Address of own remote queue-entry at node 'rank'.
*
* Address is of type uint32_t.
**/
#define IMOC_LS_REMOTE_QUEUE_ENTRY(proc, index) ( (proc > node_handle.rank) ? moc_spu_thread_list[proc-moc_ppu_thread_count].queue_base_in_ls+sizeof(moc_message_envelope_t)*node_handle.queue_size*node_handle.rank+sizeof(moc_message_envelope_t)*(index) : moc_spu_thread_list[proc-moc_ppu_thread_count].queue_base_in_ls+sizeof(moc_message_envelope_t)*node_handle.queue_size*(node_handle.rank-1)+sizeof(moc_message_envelope_t)*(index) )
/**
* Effective Address of remote status-array entry at node 'rank'.
*
* Address is of type uint32_t
**/
#define IMOC_LS_REMOTE_STATUS_ENTRY(id, rank) ( moc_spu_thread_list[rank-moc_ppu_thread_count].status_base_in_ls+sizeof(moc_message_status_t)*(id) )
//@}
//@}
#endif /*MOC_INTERNAL_ADDRESS_MACROS_PPU_H_*/
include/internal/spu/imoc_core_spu.h
Executable
+53
View File
@@ -0,0 +1,53 @@
#ifndef MOC_INTERNAL_CORE_SPU_H_
#define MOC_INTERNAL_CORE_SPU_H_
#include "../imoc_core.h"
/**
* \addtogroup internal_core
**/
//@{
/**
* @name SPU only
**/
//@{
/**
* Node-handle structure.
*
* \note SPU only
*
* Information each node knowns about himself and the world.
**/
typedef struct moc_node_handle_spu_struct
{
uint8_t rank; /**< rank of node */
uint8_t comm_size; /**< number of overall processes */
uint8_t queue_size; /**< Size of a message-queue */
uint64_t ls_base; /**< Effective address of own local-store */
volatile moc_message_envelope_t* message_queue_base; /**< base-address of own message-queues */
volatile moc_message_status_t* status_array_base; /**< base-address of own message-status-array */
uint64_t buffer_base; /**< base-address of own buffer */
uint32_t buffer_size; /**< size of own buffer */
volatile moc_process_t* moc_process_list; /**< list of all other MOC-processes
(index is rank of process) */
} moc_node_handle_spu_t; /**< typedef of \ref moc_node_handle_spu_struct */
/**
* Information each node knowns about himself and the world.
*
* \note SPU only
**/
moc_node_handle_spu_t node_handle;
//@}
//@}
#endif /*MOC_INTERNAL_CORE_SPU_H_*/
include/internal/spu/imoc_dma_spu.h
Executable
+115
View File
@@ -0,0 +1,115 @@
#ifndef MOC_INTERNAL_GET_DATA_H_
#define MOC_INTERNAL_GET_DATA_H_
#include <stdint.h>
#include "../../moc.h"
/**
* \addtogroup internal_p2p_dma
**/
//@{
/**
* @name SPU-only
**/
//@{
/**
* Internal wrapper-function for a blocking DMA-get operation.
* Data can be of any size, constraints of low-level DMA-functions
* are dealt by this function.
* However data-addresses have to be at least 16-byte aligned.
*
* This function actually always reads a multiple of 16 from the source,
* but copies only size bytes of it to target. This means however that up
* to 15 bytes are read from an invalid location. As it is never written to
* them this should always be safe.
*
* @param target \b in ls-address of buffer
* @param source \b in ea-address of data
* @param size \b in size of data in byte
* @return 0 when success otherwise error-code
**/
int IMOC_get_data(void* target, uint64_t source, uint32_t size);
/**
* Internal wrapper-function for a blocking DMA-put operation.
* Data-size should be a multiple of 16 because of low-level constraints, but
* there is no upper limit of transfer-size.
* Data-addresses have to be at least 16-byte aligned.
*
* This function actually always reads a multiple of 16 from the source, and copies
* it to target. This means however that up to 15 bytes are written to and invalid
* location, so if one wants to use it with invalid sizes precautions have to be taken
* in order to make it safe.
*
* @param source \b in ls-address of data
* @param target \b in ea-address of buffer
* @param size \b in size of data in byte
* @return 0 when success otherwise error-code
**/
int IMOC_put_data(void* source, uint64_t target, uint32_t size);
/**
* Internal wrapper-function for a non-blocking DMA-get operation.
* Data can be of any size, constraints of low-level DMA-functions
* are dealt by this function.
* However data-addresses have to be at least 16-byte aligned.
*
* This function actually always reads a multiple of 16 from the source,
* but copies only size bytes of it to target. This means however that up
* to 15 bytes are read from an invalid location. As it is never written to
* them this should always be safe.
*
* @param target \b in ls-address of buffer
* @param source \b in ea-address of data
* @param size \b in size of data in byte
* @param request \b out request-handle
* @return 0 when success otherwise error-code
**/
int IMOC_iget_data(void* target, uint64_t source, uint32_t size, imoc_dma_request* request);
/**
* Internal wrapper-function for a blocking DMA-put operation.
* Data-size should be a multiple of 16 because of low-level constraints, but
* there is no upper limit of transfer-size.
* Data-addresses have to be at least 16-byte aligned.
*
* This function actually always reads a multiple of 16 from the source, and copies
* it to target. This means however that up to 15 bytes are written to and invalid
* location, so if one wants to use it with invalid sizes precautions have to be taken
* in order to make it safe.
*
* @param source \b in ls-address of data
* @param target \b in ea-address of buffer
* @param size \b in size of data in byte
* @param request \b out request-handle
* @return 0 when success otherwise error-code
**/
int IMOC_iput_data(void* source, uint64_t target, uint32_t size, imoc_dma_request* request);
/**
* Waits till a IMOC_iget_data or IMOC_iput_data is completed.
*
* @param request \b in request-handle
* @return 0 when success otherwise error-code
**/
int IMOC_wait_dma(imoc_dma_request* request);
/**
* Tests if a IMOC_iget_data or IMOC_iput_data is completed.
*
* @param request \b in request-handle
* @param flag \b out true if operation is completed
* @return 0 when success otherwise error-code
**/
int IMOC_test_dma(imoc_dma_request* request, int *flag);
//@}
//@}
#endif /*MOC_INTERNAL_GET_DATA_H_*/
include/moc.h
Executable
+680
View File
@@ -0,0 +1,680 @@
#ifndef MOC_H_
#define MOC_H_
#include <stdint.h>
#include "moc_datatype.h"
#include "moc_error_codes.h"
/**
* Indicates, in a receive, to accept a message with any tag value.
**/
#define MOC_ANY_TAG 1000
/**
* Reserved Tag for Broadcast-Messages (MOC_Bcast).
**/
#define MOC_BCAST_TAG 250
/**
* Specifies a rank that can be used to send or receive from no-one.
**/
#define MOC_PROC_NULL 1000
/**
* Indicates, in a receive, to accept a message from any source.
**/
#define MOC_ANY_SOURCE 1001
/**
* Maximum byte-count which can be transferred by one DMA-transfer.
**/
#define MAX_DMA_CALL_SIZE 16384
/**
* Maximum byte-count of a MOC-message. It's only the theoretical size, because
* practically the SPU's local-stores can only hold much fewer bytes.
**/
#define MAX_MOC_MESSAGE_SIZE 16777215
/**
* \internal
* \addtogroup internal_p2p_imm
**/
//@{
/**
* \internal
* @name Possible structures for MOC_Request.request_data.
**/
//@{
/**
* \internal
* Struct with information about a non-blocking DMA-Communication.
**/
typedef struct imoc_dma_request_struct
{
char *buffer;
void *target;
uint32_t size;
uint32_t remainder;
uint32_t dest;
uint32_t tag_id;
} imoc_dma_request;
/**
* \internal
* Struct with information about a non-blocking ISsend-Communication.
**/
typedef struct imoc_issend_request_struct
{
uint8_t mid;
uint8_t isFinished;
uint8_t source;
uint8_t tag;
uint32_t message_size;
} imoc_issend_request;
//@}
//@}
/**
* \addtogroup environment_error
*
* This section describes how errors are handled.
*
* Every function should return an integer if possible. This integer should be
* MOC_SUCCESS if the operation teminated successfully otherwise an error-code
* telling the reason of the error is returned.
**/
//@{
/**
* Maximal length of error-text.
**/
#define MOC_MAX_ERROR_STRING 255
/**
* Returns an error-text for a given error code.
*
* The error-text is already terminated by '\\0'.
*
* \param errorcode \b in Error code returned by an MPI routine
* \param string \b out Text that corresponds to the errorcode
* \param resultlen \b out Length of string
* \retval MOC_SUCCESS No error; MPI routine completed successfully.
* \retval MOC_ERR_ARG Invalid argument. Most likely an error-code out of range.
**/
int MOC_Error_string ( int errorcode, char *string, int *resultlen );
/**
* Converts an error code into an error class.
*
* \param errorcode \b in Error code returned by an MPI routine
* \param errorclass \b out Error class associated with errorcode
* \retval MOC_SUCCESS No error; MPI routine completed successfully.
* \retval MOC_ERR_COMM Invalid argument. Most likely an error-code out of range.
**/
int MOC_Error_class ( int errorcode, int *errorclass );
//@}
/**
* \addtogroup environment_information
*
* Functions to return some information about the execution environment and
* the MOC-implementation.
**/
//@{
/**
* Major version-number of this MOC-Implementation
**/
#define MOC_VERSION 0
/**
* Minor version-number of this MOC-Implementation
**/
#define MOC_SUBVERSION 9
/**
* The maximum length of the string returned by MOC_Get_processor_name.
**/
#define MOC_MAX_PROCESSOR_NAME 4
/**
* Gets the name of the processor.
*
* \note This function returns "SPU" if the calling process is running on a SPU otherwise
* it's running on the "PPU".
*
* \param name \b out Specifies the processor-type the calling process is runnning on
* \param resultlen \b out Length (in characters) of the name
* \retval MOC_SUCCESS No error; MPI routine completed successfully.
**/
int MOC_Get_processor_name ( char *name, int *resultlen);
/**
* Returns the major and minor version of this MOC-implementation.
*
* \note Can be called before MOC_Init and after MOC_Finalize.
*
* @param version \b out Major version of this MOC-implementation
* @param subversion \b out Minor version of this MOC-implementation
* @retval MOC_SUCCESS Function returned successfully
**/
int MOC_Get_version(int *version, int *subversion);
//@}
/**
* \addtogroup environment_startup
**/
//@{
/**
* @name PPU & SPU
**/
//@{
/**
* Terminates MPI execution environment.
*
* \note All processes must call this routine before exiting.
*
* \retval MOC_SUCCESS No error; MPI routine completed successfully.
**/
int MOC_Finalize (void);
//@}
//@}
/**
* \addtogroup p2p_status
*
* Receiving operations return a status-object holding information about
* the message.
*
* There are three Fields of the status-Struct which are predefined by the MPI-Standard:
* MOC_SOURCE gives the message-source, MOC_TAG gives the message-tag and MOC_ERROR gives
* an error-code (not implemented by MOC at the moment). To get the number of element the
* function \ref MOC_Get_count should be used.
**/
//@{
/**
* Structure holding information about a received message.
**/
typedef struct MOC_Status_struct
{
uint8_t MOC_SOURCE; /**< message-source */
uint8_t MOC_TAG; /**< message-tag */
int MOC_ERROR; /**< error-code */
int message_size; /**< message-size in byte (internal-variable) */
uint8_t mid; /**< message-ID (internal-variable) */
} MOC_Status; /**< */
/**
* Gets the number of received elements which is encoded in MOC_Status.
*
* @param status \b in status-object
* @param datatype \b in Type of received elements
* @param count \b out Number of received elements
* @retval MOC_SUCCESS No error; MPI routine completed successfully.
* @retval ... error-code
**/
int MOC_Get_count(MOC_Status *status, MOC_Datatype datatype, int *count);
//@}
/**
* \addtogroup comm_main
*
* \details
* Currently there is no support for creating and managing user-defined groups and
* communicators and probably never will be. The local storage of a SPU is very small
* and also the maximal number of nodes is limited to a small number and so there is
* doubt if the benefits will be greater than the overhead introduced by supporting
* user-defined groups and communicators.
*
* Only the predefined communicators \ref MOC_COMM_WORLD, \ref MOC_COMM_SELF and
* \ref MOC_COMM_NULL are available.
*
* The only operations which are currently implemented are \ref MOC_Comm_size and \ref
* MOC_Comm_rank.
**/
/**
* \addtogroup comm_mgt
*
* Funktions for managing communicators. Because of reasons already stated (see \ref comm_main)
* there is no support for user-defined communicators. So this section is limited to functions
* which return the size of a specific communicator or the rank of the calling process.
**/
//@{
/**
* Communicator-Handle.
*
* Masks the actual type which is used to encode the communicator.
* Char is used and therefore the maximum number of possible communicators is limited to 256.
* But because there are three predefined communicators (MPI_COMM_NULL, MPI_COMM_WORLD and
* MPI_COMM_SELF) only 253 user-defined communicators are possible.
**/
typedef int MOC_Comm;
/**
* Specifies a null communicator.
**/
#define MOC_COMM_NULL 0
/**
* Specifies a constant that contains all of the processes.
**/
#define MOC_COMM_WORLD 1
/**
* Specifies a constant that contains only the calling process.
**/
#define MOC_COMM_SELF 2
/**
* Determines the rank of the calling process in the communicator.
*
* \param comm \b in commicator-handle
* \param rank \b out rank of the calling process in group of comm
* \retval MOC_SUCCESS No error; MPI routine completed successfully.
* \retval MOC_ERR_COMM Invalid communicator. A common error is to use a null communicator in a call.
**/
int MOC_Comm_rank ( MOC_Comm comm, int *rank );
/**
* Determines the size of the group associated with a communictor.
*
* \param comm \b in commicator-handle
* \param size \b out number of processes in the group of comm
* \retval MOC_SUCCESS No error; MPI routine completed successfully.
* \retval MOC_ERR_COMM Invalid communicator. A common error is to use a null communicator in a call.
**/
int MOC_Comm_size ( MOC_Comm comm, int *size );
//@}
/**
* \addtogroup environment_timer
*
* Functions to measure time.
*
* On the PPU gettimeofday from time.h is used. On the SPU either also gettimeofday or the
* spu-timer-library is used depending on the configuration during compilation of the MOC-library.
* When gettimeofday is used on the SPU the time is global in the MOC-environment with the
* disadvantage that the SPU-threads get their times from the PPU using DMA causing inaccuracy and
* delays.
**/
//@{
/**
* Returns a floating-point number of seconds, representing elapsed wall-clock
* time since some time in the past.
*
* @return wall-clock time in seconds.
**/
double MOC_Wtime(void);
/**
* Returns the resolution of Moc_Wtime in seconds.
*
* @return resolution in seconds.
**/
double MOC_Wtick(void);
//@}
/**
* \addtogroup p2p_main
**/
//@{
/**
* Indicates, at which message-size \ref MOC_Send switches to synchronous send.
*/
#define MOC_SEND_TRESHOLD 4096
/**
* Performs a blocking standard mode send operation.
*
* \note This routine may block until the message is received.
*
* @param buf \b in Pointer to buffer where data is stored
* @param count \b in Number of elements to be send (not equal to actual byte-value)
* @param datatype \b in Type of elements to be send
* @param dest \b in Rank of receiving process
* @param tag \b in Tag of message
* @param comm \b in communicator of message
* @retval MOC_SUCCESS No error; MPI routine completed successfully.
* @retval ... error-code
**/
int MOC_Send(void* buf, int count, MOC_Datatype datatype, int dest, int tag, MOC_Comm comm);
/**
* Performs a blocking standard mode receive operation.
*
* \note This routine blocks until the message is received.
*
* @param buf \b inout Buffer into which data should be stored
* @param count \b in Number of elements in receive buffer (not equal to actual byte-value)
* @param datatype \b in Datatype of each receive buffer element
* @param source \b in Rank of source
* @param tag \b in Tag of message
* @param comm \b in Communicator of message
* @param status \b inout Object holding status-information (has to be allocated by user)
* @retval MOC_SUCCESS No error; MPI routine completed successfully.
* @retval ... error-code
**/
int MOC_Recv(void* buf, int count, MOC_Datatype datatype, int source, int tag,
MOC_Comm comm, MOC_Status *status);
/**
* Waits till a message is received and returns information without actually
* receiving the message.
*
* @param source \b in source of message
* @param tag \b in message-tag
* @param comm \b in communicator
* @param status \b out status-object holding information about message
* @retval MOC_SUCCESS No error; MPI routine completed successfully.
* @retval ... error-code
**/
int MOC_Probe(int source, int tag, MOC_Comm comm, MOC_Status *status);
/**
* Tests if a message is available and returns information without actually
* receiving the message.
*
* @param source \b in source of message
* @param tag \b in message-tag
* @param comm \b in communicator
* @param flag \b out boolean value telling if a message is available
* @param status \b out status-object holding information about message
* @retval MOC_SUCCESS No error; MPI routine completed successfully.
* @retval ... error-code
**/
int MOC_Iprobe(int source, int tag, MOC_Comm comm, int *flag, MOC_Status *status);
//@}
/**
* \addtogroup p2p_synch
**/
//@{
/**
* Performs a blocking synchronous mode send operation.
*
* \note This routine blocks until the message is received.
*
* @param buf \b in Pointer to buffer where data is stored
* @param count \b in Number of elements to be send (not equal to actual byte-value)
* @param datatype \b in Type of elements to be send
* @param dest \b in Rank of receiving process
* @param tag \b in Tag of message
* @param comm \b in communicator of message
* @retval MOC_SUCCESS No error; MPI routine completed successfully.
* @retval ... error-code
**/
int MOC_Ssend(void* buf, int count, MOC_Datatype datatype, int dest, int tag, MOC_Comm comm);
//@}
/**
* \addtogroup p2p_buffered
**/
//@{
/**
* Performs a blocking buffered mode send operation.
*
* \note This routine blocks until the send-buffer is free.
*
* @param buf \b in Pointer to buffer where data is stored
* @param count \b in Number of elements to be send (not equal to actual byte-value)
* @param datatype \b in Type of elements to be send
* @param dest \b in Rank of receiving process
* @param tag \b in Tag of message
* @param comm \b in communicator of message
* @retval MOC_SUCCESS No error; MPI routine completed successfully.
* @retval ... error-code
**/
int MOC_Bsend(void* buf, int count, MOC_Datatype datatype, int dest, int tag, MOC_Comm comm);
//@}
/**
* \addtogroup p2p_sendrecv
**/
//@{
/**
* Execute a blocking send and receive operation. Both send and receive use the same
* communicator, but possibly different tags. The send buffer and receive buffers may
* be disjoint, and may have different lengths and datatypes.
*
* @param sendbuf \b in Pointer to buffer where data is stored
* @param sendcount \b in Number of elements to be send (not equal to actual byte-value)
* @param sendtype \b in Type of elements to be send
* @param dest \b in Rank of receiving process
* @param sendtag \b in Tag of message to be send
*
* @param recvbuf \b inout Buffer into which data should be stored
* @param recvcount \b in Number of elements in receive buffer (not equal to actual byte-value)
* @param recvtype \b in Datatype of each receive buffer element
* @param source \b in Rank of source
* @param recvtag \b in Tag of message to be received
*
* @param comm \b in Communicator of message
* @param status \b inout Object holding status-information (has to be allocated by user)
*
* @retval MOC_SUCCESS No error; MPI routine completed successfully.
* @retval ... error-code
**/
int MOC_Sendrecv(void *sendbuf, int sendcount, MOC_Datatype sendtype, int dest,
int sendtag, void *recvbuf, int recvcount, MOC_Datatype recvtype,
int source, int recvtag, MOC_Comm comm, MOC_Status *status);
/**
* Execute a blocking send and receive. The same buffer is used both for the send
* and for the receive, so that the message sent is replaced by the message
* received.
*
* @param buf \b inout Pointer to buffer
* @param count \b in Number of elements (not equal to actual byte-value)
* @param datatype \b in Type of elements
*
* @param dest \b in Rank of receiving process
* @param sendtag \b in Tag of message to be send
*
* @param source \b in Rank of source
* @param recvtag \b in Tag of message to be received
*
* @param comm \b in Communicator of message
* @param status \b inout Object holding status-information (has to be allocated by user)
*
* @retval MOC_SUCCESS No error; MPI routine completed successfully.
* @retval ... error-code
**/
int MOC_Sendrecv_replace(void* buf, int count, MOC_Datatype datatype, int dest,
int sendtag, int source, int recvtag, MOC_Comm comm,
MOC_Status *status);
//@}
/**
* \addtogroup p2p_imm
**/
//@{
/**
* Communication-Request (Handle)
**/
typedef struct moc_request_struct
{
int request_type;
union
{
imoc_dma_request dma_data;
imoc_issend_request issend_request;
} request_data ;
} MOC_Request /**< Communication-Request (Handle) */;
/**
* Performs a non-blocking standard mode send operation.
*
* \note This routine returns as soon as possible.
*
* @param buf \b in Pointer to buffer where data is stored
* @param count \b in Number of elements to be send (not equal to actual byte-value)
* @param datatype \b in Type of elements to be send
* @param dest \b in Rank of receiving process
* @param tag \b in Tag of message
* @param comm \b in communicator of message
* @param request \b out communication request (handle)
* @retval MOC_SUCCESS No error; MPI routine completed successfully.
* @retval ... error-code
**/
int MOC_ISend(void* buf, int count, MOC_Datatype datatype, int dest, int tag, MOC_Comm comm,
MOC_Request *request);
/**
* Performs a non-blocking synchronous mode send operation.
*
* \note This routine returns as soon as possible (After the message-envelope has
* been written into the receiver's message-queue).
*
* @param buf \b in Pointer to buffer where data is stored
* @param count \b in Number of elements to be send (not equal to actual byte-value)
* @param datatype \b in Type of elements to be send
* @param dest \b in Rank of receiving process
* @param tag \b in Tag of message
* @param comm \b in communicator of message
* @param request \b out communication request (handle)
* @retval MOC_SUCCESS No error; MPI routine completed successfully.
* @retval ... error-code
**/
int MOC_ISsend(void* buf, int count, MOC_Datatype datatype, int dest, int tag, MOC_Comm comm,
MOC_Request *request);
/**
* Tests, if a non-blocking communication has been completed.
*
* @param request \b in Communication-Request (Handle)
* @param flag \b out true, if communication-request has been completed
* @param status \b out Object holding status-information
* @retval MOC_SUCCESS No error; MPI routine completed successfully.
* @retval ... error-code
**/
int MOC_Test(MOC_Request *request, int *flag, MOC_Status *status);
/**
* Wait till a non-blocking communication has been completed.
*
* @param request \b in Communication-Request (Handle)
* @param status \b out Object holding status-information
* @retval MOC_SUCCESS No error; MPI routine completed successfully.
* @retval ... error-code
**/
int MOC_Wait(MOC_Request *request, MOC_Status *status);
//@}
/**
* \addtogroup broadcast
**/
//@{
/**
* Broadcasts a message from the process with rank "root" to all other processes of the group.
*
* \note The broadcast-message can also be received by a MOC_Recv using MOC_BCAST_TAG.
*
* @param buffer \b inout starting address of buffer
* @param count \b in number of elements to be send or received
* @param datatype \b in Type of elements to be send or received
* @param root \b in rank of the process who sends the broadcast-message
* @param comm \b in communicator of message
**/
int MOC_Bcast(void* buffer, int count, MOC_Datatype datatype, int root, MOC_Comm comm );
/**
* Blocks the caller until all processes in the communicator have called it; that is, the
* call returns at any process only after all members of the communicator have entered the
* call.
*
* @param comm \b in communicator of message
* @retval MOC_SUCCESS No error; MPI routine completed successfully.
* @retval ... error-code
**/
int MOC_Barrier(MOC_Comm comm );
//@}
#endif /*MOC_H_*/
include/moc_datatype.h
Executable
+149
View File
@@ -0,0 +1,149 @@
#ifndef MOC_DATATYPE_H_
#define MOC_DATATYPE_H_
#include <stdint.h>
/**
* \addtogroup p2p_datatype
*
* Datatypes supported by MOC.
*
* There are the normal datatypes defined by C (char, short, int, ...) and special
* vector-datatypes used by the Cell Broadband Engine. For more information see
* "C/C++ Language Extensions for Cell Broadband Engine Architecture - Version 2.5"
* chapter 1.1.
**/
//@{
/**
* @name PPU & SPU
**/
//@{
/**
* Typedef defining MOC_Datatype.
*
* One byte is used to encode the Data-Type. The first four msb hold the
* size in byte / 2 ( if ((datatype & 0xF0)>>4) == 0 then size = 1 else
* size = ((datatype & 0xF0)>>4)*2 ) and the four lsb hold an id. The
* combination of id and size is used to uniquely identify
* a datatype (or rather a group of datatypes belonging together).
**/
typedef uint8_t MOC_Datatype;
/**
* MOC_Byte is a special datatype. It can match any datatype.
* It has a size of 1 byte.
**/
#define MOC_BYTE ((MOC_Datatype)0+0)
/**
* C-Datatype char.
* It has a size of 1 byte.
**/
#define MOC_CHAR ((MOC_Datatype)0+1)
/**
* C-Datatype unsigned char.
* It has a size of 1 byte.
**/
#define MOC_UNSIGNED_CHAR ((MOC_Datatype)0+2)
/**
* C-Datatype short.
* It has a size of 2 byte.
**/
#define MOC_SHORT ((MOC_Datatype)16+0)
/**
* C-Datatype unsigned short.
* It has a size of 2 byte.
**/
#define MOC_UNSIGNED_SHORT ((MOC_Datatype)16+1)
/**
* C-Datatype int.
* It has a size of 4 byte.
**/
#define MOC_INT ((MOC_Datatype)32+0)
/**
* C-Datatype unsigned int.
* It has a size of 4 byte.
**/
#define MOC_UNSIGNED ((MOC_Datatype)32+1)
/**
* C-Datatype long long.
* It has a size of 8 byte.
**/
#define MOC_LONG_LONG ((MOC_Datatype)64+0)
/**
* C-Datatype unsigned long long.
* It has a size of 8 byte.
**/
#define MOC_UNSIGNED_LONG_LONG ((MOC_Datatype)64+1)
/**
* C-Datatype float.
* It has a size of 4 byte.
**/
#define MOC_FLOAT ((MOC_Datatype)32+2)
/**
* C-Datatype double.
* It has a size of 8 byte.
**/
#define MOC_DOUBLE ((MOC_Datatype)64+5)
/**
* Cell-specific Datatype vector unsigned char (vec_uchar16).
* It has a size of 16 byte.
**/
#define MOC_VEC_UCHAR16 ((MOC_Datatype)128+0)
/**
* Cell-specific Datatype vector unsigned short (vec_ushort8).
* It has a size of 16 byte.
**/
#define MOC_VEC_USHORT8 ((MOC_Datatype)128+1)
/**
* Cell-specific Datatype vector unsigned int (vec_uint4).
* It has a size of 16 byte.
**/
#define MOC_VEC_UINT4 ((MOC_Datatype)128+2)
/**
* Cell-specific Datatype vector signed char (vec_char16).
* It has a size of 16 byte.
**/
#define MOC_VEC_CHAR16 ((MOC_Datatype)128+4)
/**
* Cell-specific Datatype vector signed short (vec_short8).
* It has a size of 16 byte.
**/
#define MOC_VEC_SHORT8 ((MOC_Datatype)128+3)
/**
* Cell-specific Datatype vector signed int (vec_int4).
* It has a size of 16 byte.
**/
#define MOC_VEC_INT4 ((MOC_Datatype)128+5)
/**
* Cell-specific Datatype vector float (vec_float4).
* It has a size of 16 byte.
**/
#define MOC_VEC_FLOAT4 ((MOC_Datatype)128+6)
//@}
//@}
#endif /*MOC_DATATYPE_H_*/
include/moc_error_codes.h
Executable
+72
View File
@@ -0,0 +1,72 @@
#ifndef MOC_ERROR_H_
#define MOC_ERROR_H_
/**
* \addtogroup environment_error
*
* This section describes how errors are handled.
*
* Every function should return an integer if possible. This integer should be
* MOC_SUCCESS if the operation teminated successfully otherwise an error-code
* telling the reason of the error is returned.
*
* For more information see Chapter 7.2 of the MPI 1.3 Standard.
**/
//@{
/**
* @name MPI_ERROR-CLASSES
*
* Error-classes defined by the MPI-standard. Every error-code returned by a
* MOC-function should belong to an error-class or be an error-class itself.
* This enables implementation-independend error-handling.
*
* Value-range: 0 = MOC_SUCCESS < MOC_ERR_... <= MOC_ERR_LASTCODE
**/
//@{
#define MOC_SUCCESS 0 /**< No error */
#define MOC_ERR_BUFFER 1000 /**< Invalid buffer pointer */
#define MOC_ERR_COUNT 2000 /**< Invalid count argument */
#define MOC_ERR_TYPE 3000 /**< Invalid datatype argument */
#define MOC_ERR_TAG 4000 /**< Invalid tag argument */
#define MOC_ERR_COMM 5000 /**< Invalid communicator */
#define MOC_ERR_RANK 6000 /**< Invalid rank */
#define MOC_ERR_REQUEST 7000 /**< Invalid request (handle) */
#define MOC_ERR_ROOT 8000 /**< Invalid root */
#define MOC_ERR_GROUP 9000 /**< Invalid group */
#define MOC_ERR_OP 10000 /**< Invalid operation */
#define MOC_ERR_TOPOLOGY 11000 /**< Invalid topology */
#define MOC_ERR_DIMS 12000 /**< Invalid dimension argument */
#define MOC_ERR_ARG 13000 /**< Invalid argument of some other kind */
#define MOC_ERR_UNKNOWN 14000 /**< Unknown error */
#define MOC_ERR_TRUNCATE 15000 /**< Message truncated on receive */
#define MOC_ERR_OTHER 16000 /**< Known error not in this list */
#define MOC_ERR_INTERN 17000 /**< Internal MOC (implementation) error */
#define MOC_ERR_IN_STATUS 18000 /**< Error code is in status */
#define MOC_ERR_PENDING 19000 /**< Pending request */
#define MOC_ERR_LASTCODE 20000 /**< Last error code */
//@}
/**
* @name MOC-Error-Codes
*
* Implementation dependent Error-Codes.
**/
//@{
#define MOC_ERR_NOT_IMPLEMENTED 17001 /**< Function not implemented */
#define MOC_ERR_BUFFER_OVERFLOW 17002 /**< Internal Buffer Overflow */
#define MOC_ERR_NO_MEMORY 17003 /**< Insufficient Memory */
#define MOC_ERR_NO_BUFFER 17004 /**< Insufficient Buffer-Space */
#define MOC_ERR_MESSAGE_EXHAUSTED 17005 /**< No further messages possible */
#define MOC_ERR_BAD_ALIGNMENT 16001 /**< Base-Adress of Buffer isn't 16-byte aligned */
#define MOC_ERR_BAD_SIZE 16002 /**< Bad Size (probably no multiple of 16) */
//@}
//@}
#endif /*MOC_ERROR_H_*/
include/ppu/moc.h
Executable
+129
View File
@@ -0,0 +1,129 @@
#ifndef MOC_H_PPU
#define MOC_H_PPU
#include "moc_dma_ppu.h"
#include "moc_datatype_ppu.h"
#include "../moc.h"
#include <stdint.h>
#include <libspe2.h>
/**
* Maximum number of possible SPU-threads.
**/
#define MOC_SPU_THREAD_MAX 16
/**
* \addtogroup environment_startup
**/
//@{
/**
* @name PPU-only
**/
//@{
/**
* Returns the number of available SPUs.
*
* \attention PPU-only
*
* \note Must be called before \ref MOC_Init
*
* \note This function should return the count of free SPUs. But this
* doesn't really works. To get the available SPU-count the
* function spe_cpu_info_get() from libspe2 with
* SPE_COUNT_USABLE_SPES-flag is used. However this function can only
* detect if a SPU-thread is used by the calling function. There seems
* to be no means to detect SPU-thread used by other applications.
*
*
* \param count \b out number of available SPUs
* \retval MOC_SUCCESS No error; MPI routine completed successfully.
* \retval ... error-code
**/
int MOC_Get_avail_spu_count(int* count);
/**
* Returns the number of SPUs.
*
* \attention PPU-only
*
* \note Must be called before \ref MOC_Init
*
* \param count \b out SPU-count
* \retval MOC_SUCCESS No error; MPI routine completed successfully.
* \retval ... error-code
**/
int MOC_Get_max_spu_count(int* count);
/**
* Adds a SPU-Thread.
*
* \attention PPU-only
*
* \note Must be called before calling \ref MOC_Init
*
* \note The order of executions defines the order of the ranks of the SPU-Threads
*
* \param phandle \b in Handle of the SPE-Program
* \param argp \b in Pointer to some user-defined information in the main-memory
* \retval MOC_SUCCESS No error; MPI routine completed successfully.
* \retval MOC_ERR_BUFFER_OVERFLOW A Buffer-Overflow occured. Mosst likely you tried to add more
* SPU-threads than defined by \ref MOC_SPU_THREAD_MAX.
* \retval MOC_ERR_INTERN Internal Error
**/
int MOC_Add_spu_thread ( spe_program_handle_t* phandle, void* argp );
/**
* Sets the size of the buffer used for buffered communication.
*
* \attention PPU-only
*
* \note Must be called before \ref MOC_Init
*
* \param size buffer-size
* \return 0 if success, otherwise error-code
**/
int MOC_Set_buffer_size ( uint32_t size );
/**
* Sets the size of the message-queues.
*
* \attention PPU-only
*
* \note Must be called before \ref MOC_Init
*
* \param size queue-size
* \return 0 if success, otherwise error-code
**/
int MOC_Set_queue_size ( uint8_t size );
/**
* Initialize the MPI execution environment.
*
* \attention PPU-only function
*
* \note All processes must call this routine before executing any MOC-routine.
*
* \param argc \b in Pointer to the number of arguments
* \param argv \b in Pointer to the argument vector
* \retval MOC_SUCCESS No error; MPI routine completed successfully.
* \retval MOC_ERR_NO_MEMORY Insufficient Memory.
* \retval MOC_ERR_INTERN Internal Error
**/
int MOC_Init ( int *argc, char ***argv );
//@}
//@}
#endif /*MOC_H_*/
include/ppu/moc_datatype_ppu.h
Executable
+65
View File
@@ -0,0 +1,65 @@
#ifndef MOC_DATATYPE_PPU_H_
#define MOC_DATATYPE_PPU_H_
#include "../moc_datatype.h"
/**
* \addtogroup p2p_datatype
**/
//@{
/**
* @name PPU-only
**/
//@{
/**
* C-Datatype long.
* It has a size of 8 byte.
* \attention On the SPU long has a size of 4 byte. So if
* you are trying to send data of type long from the PPU to a
* SPU or vice versa there will be an error.
**/
#define MOC_LONG ((MOC_Datatype)64+2)
/**
* C-Datatype unsigned long.
* It has a size of 8 byte.
* \attention On the SPU unsigned long has a size of 4 byte. So if
* you are trying to send data of type long from the PPU to a
* SPU or vice versa there will be an error.
**/
#define MOC_UNSIGNED_LONG ((MOC_Datatype)64+3)
/**
* Cell-specific Datatype vector bool char (vec_bchar16).
* It has a size of 16 byte.
* \note PPU-only. Is mapped to the SPU-Datatype vector unsigned char.
**/
#define MOC_VEC_BCHAR16 ((MOC_Datatype)128+0)
/**
* Cell-specific Datatype vector bool short (vec_bshort8).
* It has a size of 16 byte.
* \note PPU-only. Is mapped to the SPU-Datatype vector unsigned short.
**/
#define MOC_VEC_BSHORT8 ((MOC_Datatype)128+1)
/**
* Cell-specific Datatype vector pixel (vec_pixel8).
* It has a size of 16 byte.
* \note PPU-only. Is mapped to the SPU-Datatype vector unsigned short.
**/
#define MOC_VEC_PIXEL8 ((MOC_Datatype)128+1)
/**
* Cell-specific Datatype vector bool int (vec_bint4).
* It has a size of 16 byte.
* \note PPU-only. Is mapped to the SPU-Datatype vector unsigned int.
**/
#define MOC_VEC_BINT4 ((MOC_Datatype)128+2)
//@}
//@}
#endif /*MOC_DATATYPE_PPU_H_*/
include/ppu/moc_dma_ppu.h
Executable
+119
View File
@@ -0,0 +1,119 @@
#ifndef MOC_DATA_TRANSFER_PPU_H_
#define MOC_DATA_TRANSFER_PPU_H_
#include <stdint.h>
#include "../moc.h"
#include "moc_datatype_ppu.h"
/**
* \addtogroup dma_access
**/
//@{
/**
* @name PPU-only
**/
//@{
/**
* Wrapper-function for a blocking DMA-get operation.
* Data can be of any size, constraints of low-level DMA-functions
* are dealt by this function.
* However data-addresses have to be at least 16-byte aligned.
*
* @param target \b out pointer to buffer
* @param source \b in local-store address of data
* @param count \b in size of data in byte
* @param datatype \b in Type of Elements
* @param dest \b in rank of source spe
* @retval MOC_SUCCESS No error; MPI routine completed successfully.
* @retval ... error-code
**/
int MOC_Get_data(void* target, uint32_t source, int count,
MOC_Datatype datatype, int dest);
/**
* Wrapper-function for a blocking DMA-put operation.
* Data-size should be a multiple of 16 because of low-level constraints, but
* there is no upper limit of transfer-size.
* Data-addresses have to be at least 16-byte aligned.
*
* @param source \b in pointer to data
* @param target \b in local-store address of buffer
* @param count \b in size of data in byte
* @param datatype \b in Type of Elements
* @param src \b in rank of source-SPE
* @retval MOC_SUCCESS No error; MPI routine completed successfully.
* @retval ... error-code
**/
int MOC_Put_data(void* source, uint32_t target, int count,
MOC_Datatype datatype, int src);
/**
* Wrapper-function for a non-blocking DMA-get operation.
* Data can be of any size, constraints of low-level DMA-functions
* are dealt by this function.
* However data-addresses have to be at least 16-byte aligned.
* Operation-completion can be derived with MOC_Test and MOC_Wait.
*
* @param target \b out pointer to buffer
* @param source \b in local-store address of data
* @param count \b in size of data in byte
* @param datatype \b in Type of Elements
* @param dest \b in rank of source spe
* @param request \b out communication request (handle)
* @retval MOC_SUCCESS No error; MPI routine completed successfully.
* @retval ... error-code
**/
int MOC_IGet_data(void* target, uint32_t source, int count,
MOC_Datatype datatype, int dest, MOC_Request *request);
/**
* Wrapper-function for a non-blocking DMA-put operation.
* Data-size should be a multiple of 16 because of low-level constraints, but
* there is no upper limit of transfer-size.
* Data-addresses have to be at least 16-byte aligned.
* Operation-completion can be derived with MOC_Test and MOC_Wait.
*
* @param source \b in pointer to data
* @param target \b in local-store address of buffer
* @param count \b in size of data in byte
* @param datatype \b in Type of Elements
* @param src \b in rank of source-SPE
* @param request \b out communication request (handle)
* @retval MOC_SUCCESS No error; MPI routine completed successfully.
* @retval ... error-code
**/
int MOC_IPut_data(void* source, uint32_t target, int count,
MOC_Datatype datatype, int src, MOC_Request *request);
/**
* Translates an effective-address of a SPU into the corresponding local-store address.
*
* @param ea \b in effective-address
* @param rank \b in rank of SPU
* @param ls \b out local-store address
**/
int MOC_Translate_EA_to_LS(uint64_t ea, int rank, uint32_t* ls);
//@}
/**
* Waits till a MOC_IGet_data or IMOC_IPut_data is completed.
*
* @param request \b in request-handle
* @return 0 when success otherwise error-code
**/
int MOC_Wait_data(MOC_Request* request);
/**
* Tests if a MOC_IGet_data or IMOC_IPut_data is completed.
*
* @param request \b in request-handle
* @param flag \b out true if operation is completed
* @return 0 when success otherwise error-code
**/
int MOC_Test_data(MOC_Request* request, int *flag);
//@}
#endif /*MOC_DATA_TRANSFER_PPU_H_*/
include/spu/moc.h
Executable
+48
View File
@@ -0,0 +1,48 @@
#ifndef MOC_H_SPU
#define MOC_H_SPU
#include "moc_dma_spu.h"
#include "moc_datatype_spu.h"
#include "../moc.h"
#include <stdint.h>
/**
* \addtogroup environment_startup
**/
//@{
/**
* @name SPU-only
**/
//@{
/**
* Initialize the MPI execution environment.
*
* \anchor moc_init_spu
*
* \attention SPU-only function
*
* \note All processes must call this routine before executing any MOC-routine.
*
* \todo Copying of user-defined information from main-memory to local buffer not implemented
*
* \param argp \b in Pointer to user-defined information in the main-memory
* \param length \b in Length of the user-defined information in bytes
* \param buffer \b out Buffer into which the user-defined information should be copied
* \retval MOC_SUCCESS No error; MPI routine completed successfully.
* \retval MOC_ERR_NO_MEMORY Insufficient Memory.
* \retval MOC_ERR_INTERN Internal Error
**/
int MOC_Init (unsigned long long argp, int length, void* buffer);
//@}
//@}
#endif /*MOC_H_*/
include/spu/moc_datatype_spu.h
Executable
+60
View File
@@ -0,0 +1,60 @@
#ifndef MOC_DATATYPE_SPU_H_
#define MOC_DATATYPE_SPU_H_
#include "../moc_datatype.h"
/**
* \addtogroup p2p_datatype
**/
//@{
/**
* @name SPU-only
**/
//@{
/**
* C-Datatype long.
* It has a size of 4 byte.
* \attention On the PPU long has a size of 8 byte. So if
* you are trying to send data of type long from the PPU to a
* SPU or vice versa there will be an error.
**/
#define MOC_LONG ((MOC_Datatype)32+2)
/**
* C-Datatype unsigned long.
* It has a size of 4 byte.
* \attention On the SPU unsigned long has a size of 8 byte. So if
* you are trying to send data of type long from the PPU to a
* SPU or vice versa there will be an error.
**/
#define MOC_UNSIGNED_LONG ((MOC_Datatype)32+3)
/**
* Cell-specific Datatype vector unsigned long long (vec_ullong2).
* It has a size of 16 byte.
* \note SPU-only. Is mapped to the PPU-Datatype vector bool char.
**/
#define MOC_VEC_ULLONG2 ((MOC_Datatype)128+0)
/**
* Cell-specific Datatype vector signed long long (vec_llong2).
* It has a size of 16 byte.
* \note SPU-only. Is mapped to the PPU-Datatype vector bool short.
**/
#define MOC_VEC_LLONG2 ((MOC_Datatype)128+1)
/**
* Cell-specific Datatype vector double (vec_double2).
* It has a size of 16 byte.
* \note SPU-only. Is mapped to the PPU-Datatype vector bool int.
**/
#define MOC_VEC_DOUBLE2 ((MOC_Datatype)128+2)
//@}
//@}
#endif /*MOC_DATATYPE_SPU_H_*/
include/spu/moc_dma_spu.h
Executable
+117
View File
@@ -0,0 +1,117 @@
#ifndef MOC_DATA_TRANSFER_SPU_H_
#define MOC_DATA_TRANSFER_SPU_H_
#include <stdint.h>
#include "moc_datatype_spu.h"
#include "../moc.h"
/**
* \addtogroup dma_access
**/
//@{
/**
* @name SPU-only
**/
//@{
/**
* Wrapper-function for a blocking DMA-get operation.
* Data can be of any size, constraints of low-level DMA-functions
* are dealt by this function.
* However data-addresses have to be at least 16-byte aligned.
*
* @param target \b out pointer to buffer
* @param source \b in ea-address of data
* @param count \b in size of data in byte
* @param datatype \b in Type of Elements
* @retval MOC_SUCCESS No error; MPI routine completed successfully.
* @retval ... error-code
**/
int MOC_Get_data(void* target, uint64_t source, int count,
MOC_Datatype datatype);
/**
* Wrapper-function for a blocking DMA-put operation.
* Data-size should be a multiple of 16 because of low-level constraints, but
* there is no upper limit of transfer-size.
* Data-addresses have to be at least 16-byte aligned.
*
* @param source \b in pointer to data
* @param target \b in ea-address of buffer
* @param count \b in size of data in byte
* @param datatype \b in Type of Elements
* @retval MOC_SUCCESS No error; MPI routine completed successfully.
* @retval ... error-code
**/
int MOC_Put_data(void* source, uint64_t target, int count,
MOC_Datatype datatype);
/**
* Wrapper-function for a non-blocking DMA-get operation.
* Data can be of any size, constraints of low-level DMA-functions
* are dealt by this function.
* However data-addresses have to be at least 16-byte aligned.
* Operation-completion can be derived with MOC_Test and MOC_Wait.
*
* @param target \b out pointer to buffer
* @param source \b in ea-address of data
* @param count \b in size of data in byte
* @param datatype \b in Type of Elements
* @param request \b out communication request (handle)
* @retval MOC_SUCCESS No error; MPI routine completed successfully.
* @retval ... error-code
**/
int MOC_IGet_data(void* target, uint64_t source, int count,
MOC_Datatype datatype, MOC_Request *request);
/**
* Wrapper-function for a non-blocking DMA-put operation.
* Data-size should be a multiple of 16 because of low-level constraints, but
* there is no upper limit of transfer-size.
* Data-addresses have to be at least 16-byte aligned.
* Operation-completion can be derived with MOC_Test and MOC_Wait.
*
* @param source \b in pointer to data
* @param target \b in ea-address of buffer
* @param count \b in size of data in byte
* @param datatype \b in Type of Elements
* @param request \b out communication request (handle)
* @retval MOC_SUCCESS No error; MPI routine completed successfully.
* @retval ... error-code
**/
int MOC_IPut_data(void* source, uint64_t target, int count,
MOC_Datatype datatype, MOC_Request *request);
/**
* Translates a local-store address of the calling SPU into the corresponding
* effective-address.
*
* @param ls \b in local-store address
* @param ea \b out effective-address
**/
int MOC_Translate_LS_to_EA(void* ls, uint64_t* ea);
//@}
/**
* Tests if a MOC_IGet_data or IMOC_IPut_data is completed.
*
* @param request \b in request-handle
* @param flag \b out true if operation is completed
* @return 0 when success otherwise error-code
**/
int MOC_Test_data(MOC_Request* request, int *flag);
/**
* Waits till a MOC_IGet_data or IMOC_IPut_data is completed.
*
* @param request \b in request-handle
* @return 0 when success otherwise error-code
**/
int MOC_Wait_data(MOC_Request* request);
//@}
#endif /*MOC_DATA_TRANSFER_SPU_H_*/