Open MPI [GFB+04] is an open source software implementation of The Message Passing Interface (MPI) standard. Before the architecture and innards of Open MPI will make any sense, a little background on the MPI standard must be discussed.
Open MPI [GFB+04] 是一个消息传递接口 (Message Passing Interface, MPI) 标准的开源软件实现。在深入Open MPI架构和内部结构之前，需要先介绍一些MPI标准的背景知识。
The Message Passing Interface (MPI)
The MPI standard is created and maintained by the MPI Forum, an open group consisting of parallel computing experts from both industry and academia. MPI defines an API that is used for a specific type of portable, high-performance inter-process communication (IPC): message passing. Specifically, the MPI document describes the reliable transfer of discrete, typed messages between MPI processes. Although the definition of an "MPI process" is subject to interpretation on a given platform, it usually corresponds to the operating system's concept of a process (e.g., a POSIX process). MPI is specifically intended to be implemented as middleware, meaning that upper-level applications call MPI functions to perform message passing.
MPI defines a high-level API, meaning that it abstracts away whatever underlying transport is actually used to pass messages between processes. The idea is that sending-process X can effectively say "take this array of 1,073 double precision values and send them to process Y". The corresponding receiving-process Y effectively says "receive an array of 1,073 double precision values from process X." A miracle occurs, and the array of 1,073 double precision values arrives in Y's waiting buffer.
Notice what is absent in this exchange: there is no concept of a connection occurring, no stream of bytes to interpret, and no network addresses exchanged. MPI abstracts all of that away, not only to hide such complexity from the upper-level application, but also to make the application portable across different environments and underlying message passing transports. Specifically, a correct MPI application is source-compatible across a wide variety of platforms and network types.
MPI defines not only point-to-point communication (e.g., send and receive), it also defines other communication patterns, such as collective communication. Collective operations are where multiple processes are involved in a single communication action. Reliable broadcast, for example, is where one process has a message at the beginning of the operation, and at the end of the operation, all processes in a group have the message. MPI also defines other concepts and communications patterns that are not described here. (As of this writing, the most recent version of the MPI standard is MPI-2.2 [For09]. Draft versions of the upcoming MPI-3 standard have been published; it may be finalized as early as late 2012.)
Uses of MPI
There are many implementations of the MPI standard that support a wide variety of platforms, operating systems, and network types. Some implementations are open source, some are closed source. Open MPI, as its name implies, is one of the open source implementations. Typical MPI transport networks include (but are not limited to): various protocols over Ethernet (e.g., TCP, iWARP, UDP, raw Ethernet frames, etc.), shared memory, and InfiniBand.
MPI implementations are typically used in so-called "high-performance computing" (HPC) environments. MPI essentially provides the IPC for simulation codes, computational algorithms, and other "big number crunching" types of applications. The input data sets on which these codes operate typically represent too much computational work for just one server; MPI jobs are spread out across tens, hundreds, or even thousands of servers, all working in concert to solve one computational problem.
典型的，MPI是在高性能计算（High Performance Computing, HPC）中使用。MPI本质上为模拟、计算算法和其他的大型数值计算应用提供IPC。通常来说，这些应用操作的输入数据代表了大量的计算，不适合于一台服务器。所以，MPI作业都是分布在几十个，几百个，甚至几千个服务器上，所有作业都是合作解决一个计算问题。
That is, the applications using MPI are both parallel in nature and highly compute-intensive. It is not unusual for all the processor cores in an MPI job to run at 100% utilization. To be clear, MPI jobs typically run in dedicated environments where the MPI processes are the only application running on the machine (in addition to bare-bones operating system functionality, of course).
As such, MPI implementations are typically focused on providing extremely high performance, measured by metrics such as:
- Extremely low latency for short message passing. As an example, a 1-byte message can be sent from a user-level Linux process on one server, through an InfiniBand switch, and received at the target user-level Linux process on a different server in a little over 1 microsecond (i.e., 0.000001 second).
- Extremely high message network injection rate for short messages. Some vendors have MPI implementations (paired with specified hardware) that can inject up to 28 million messages per second into the network.
- Quick ramp-up (as a function of message size) to the maximum bandwidth supported by the underlying transport.
- Low resource utilization. All resources used by MPI (e.g., memory, cache, and bus bandwidth) cannot be used by the application. MPI implementations therefore try to maintain a balance of low resource utilization while still providing high performance.
因此，MPI实现通常关注于提供非常高的性能，从以下尺度测量： - 短消息传递中非常低的延迟。例如，服务器上用户级Linux进程发送一条1字节的消息，通过InfiniBand交换机，被另外一台服务器的目标用户级Linux进程接受，整个过程只需1毫秒中很少一部分（比如：0.000001秒） - 短消息的极高网络注入率。一些制造商的MPI实现（配合专门的硬件）可以达到每秒向网络注入2千8百万条消息。 - 在消息大小增长时，可以快速达到底层传输支持的最大带宽。 - 低资源占用。所有MPI使用的资源（比如：内存，缓存和总线带宽）都不能被应用使用。所以，MPI实现尝试保持低资源占用和同样提供高性能的平衡。
The first version of the MPI standard, MPI-1.0, was published in 1994 [Mes93]. MPI-2.0, a set of additions on top of MPI-1, was completed in 1996 [GGHL+96].
In the first decade after MPI-1 was published, a variety of MPI implementations sprung up. Many were provided by vendors for their proprietary network interconnects. Many other implementations arose from the research and academic communities. Such implementations were typically "research-quality," meaning that their purpose was to investigate various high-performance networking concepts and provide proofs-of-concept of their work. However, some were high enough quality that they gained popularity and a number of users.
Open MPI represents the union of four research/academic, open source MPI implementations: LAM/MPI, LA/MPI (Los Alamos MPI), and FT-MPI (Fault-Tolerant MPI). The members of the PACX-MPI team joined the Open MPI group shortly after its inception.
Open MPI融合了4个科研、学术界的开源MPI实现：LAM/MPI，LA/MPI（Los Alamos MPI）和FT-MPI（Fault-Tolerant MPI）。在PACX-MPI成立之后，其成员也立刻加入了Open MPI组。
The members of these four development teams decided to collaborate when we had the collective realization that, aside from minor differences in optimizations and features, our software code bases were quite similar. Each of the four code bases had their own strengths and weaknesses, but on the whole, they more-or-less did the same things. So why compete? Why not pool our resources, work together, and make an even better MPI implementation?
After much discussion, the decision was made to abandon our four existing code bases and take only the best ideas from the prior projects. This decision was mainly predicated upon the following premises:
- Even though many of the underlying algorithms and techniques were similar among the four code bases, they each had radically different implementation architectures, and would be incredible difficult (if not impossible) to merge.
- Each of the four also had their own (significant) strengths and (significant) weaknesses. Specifically, there were features and architecture decisions from each of the four that were desirable to carry forward. Likewise, there were poorly optimized and badly designed code in each of the four that were desirable to leave behind.
The members of the four developer groups had not worked directly together before. Starting with an entirely new code base (rather than advancing one of the existing code bases) put all developers on equal ground.
Thus, Open MPI was born. Its first Subversion commit was on November 22, 2003.
For a variety of reasons (mostly related to either performance or portability), C and C++ were the only two possibilities for the primary implementation language. C++ was eventually discarded because different C++ compilers tend to lay out structs/classes in memory according to different optimization algorithms, leading to different on-the-wire network representations. C was therefore chosen as the primary implementation language, which influenced several architectural design decisions.
基于多种原因（绝大部分关系到性能或者可移植性），C 和 C++是两种可能的主要实现语言。最终，放弃C++是因为不同的C++编译器倾向于根据不同的优化算法进行数据结构或类的内存布局，这就导致了线上网络表现中的差异。因此，选择 C 作为主要的实现语言，进而影响了很多架构设计的决定。
When Open MPI was started, we knew that it would be a large, complex code base:
- In 2003, the current version of the MPI standard, MPI-2.0, defined over 300 API functions.
- Each of the four prior projects were large in themselves. For example, LAM/MPI had over 1,900 files of source code, comprising over 300,000 lines of code (including comments and blanks).
- We wanted Open MPI to support more features, environments, and networks than all four prior projects put together.
- 我们期望Open MPI能够比之前4个项目的总和支持更多的特性，环境和网络。
We therefore spent a good deal of time designing an architecture that focused on three things:
- Grouping similar functionality together in distinct abstraction layers.
- Using run-time loadable plugins and run-time parameters to choose between multiple different implementations of the same behavior.
- Not allowing abstraction to get in the way of performance.
Abstraction Layer Architecture
Open MPI has three main abstraction layers, shown in Figure 15.1:
Open, Portable Access Layer (OPAL): OPAL is the bottom layer of Open MPI's abstractions. Its abstractions are focused on individual processes (versus parallel jobs). It provides utility and glue code such as generic linked lists, string manipulation, debugging controls, and other mundane—yet necessary—functionality.
OPAL also provides Open MPI's core portability between different operating systems, such as discovering IP interfaces, sharing memory between processes on the same server, processor and memory affinity, high-precision timers, etc.
Open MPI Run-Time Environment (ORTE) (pronounced "or-tay"): An MPI implementation must provide not only the required message passing API, but also an accompanying run-time system to launch, monitor, and kill parallel jobs. In Open MPI's case, a parallel job is comprised of one or more processes that may span multiple operating system instances, and are bound together to act as a single, cohesive unit.
In simple environments with little or no distributed computational support, ORTE uses rsh or ssh to launch the individual processes in parallel jobs. More advanced, HPC-dedicated environments typically have schedulers and resource managers for fairly sharing computational resources between many users. Such environments usually provide specialized APIs to launch and regulate processes on compute servers. ORTE supports a wide variety of such managed environments, such as (but not limited to): Torque/PBS Pro, SLURM, Oracle Grid Engine, and LSF.
Open MPI (OMPI): The MPI layer is the highest abstraction layer, and is the only one exposed to applications. The MPI API is implemented in this layer, as are all the message passing semantics defined by the MPI standard.
Since portability is a primary requirement, the MPI layer supports a wide variety of network types and underlying protocols. Some networks are similar in their underlying characteristics and abstractions; some are not.
开放、可移植的访问层(Open Portable Access Layer, OPAL)：OPAL位于Open MPI抽象的最底层，它关注于各个进程个体（相对于并行作业）。它提供了一些工具和集成代码，包括：链接列表，字符串操作，debug控制和其他平凡但是必须的功能。
Open MPI运行时环境(Open MPI Run-Time Environment, ORTE,发音为“or-tay”):MPI实现不只是提供必要的消息传递API，还必须提供辅助的运行时系统以发起，监视和杀死并行作业。在Open MPI中，并行作业是指一个或者多个可能跨多个操作系统的进程，组合在一起作为一个紧密耦合的单元。
在只有很少或者没有分布式计算支持的简单环境下，ORTE使用rsh或者ssh启动并行作业中各个进程。更高级的情况下，专门的HPC环境通常会提供调度和资源管理器，以公平的在多个用户间共享计算资源。这种环境通常会提供特殊的API以在计算节点上发起和控制进程。ORTE支持众多这样的管理环境，包括（单不限于）Torque/PBS Pro, SLURM, Oracle Grid Engine和LSF。
Open MPI(OMPI):MPI层是最上面的抽象层次，唯一暴露给应用程序的层次。在这层次内，实现了MPI API并且所有的消息传递语意符合MPI标准的定义。
图15.1 Open MPI的抽象层次架构视图,包含3个主要层次：OPAL,ORTE和OMPI
Although each abstraction is layered on top of the one below it, for performance reasons the ORTE and OMPI layers can bypass the underlying abstraction layers and interact directly with the operating system and/or hardware when needed (as depicted in Figure 15.1). For example, the OMPI layer uses OS-bypass methods to communicate with certain types of NIC hardware to obtain maximum networking performance.
Each layer is built into a standalone library. The ORTE library depends on the OPAL library; the OMPI library depends on the ORTE library. Separating the layers into their own libraries has acted as a wonderful tool for preventing abstraction violations. Specifically, applications will fail to link if one layer incorrectly attempts to use a symbol in a higher layer. Over the years, this abstraction enforcement mechanism has saved many developers from inadvertently blurring the lines between the three layers.