© Stewart Weiss. CC-BY-SA.
Example
Imagine three processes that work together. One reads a file containing numbers whose sum is to be calculated.
It sends half the file to a second process and the remaining half to a third process.
These two processes add their numbers independently and send their sums to the first process, which adds them and prints them out.
If we call them processes 1, 2, and 3 and represent them as nodes, their communication pattern looks like this:
1 and 2 communicate, and 1 and 3 communicate, but not 2 and 3.
© Stewart Weiss. CC-BY-SA.
Network Topology
Suppose that in the preceding example, process 2 sends half its data to processes 4 and 5, and process 3 sends half its data to 6 and 7.
Each of 4,5,6, and 7 add up their numbers independently. Processes 4 and 5 send their sums to 2, and 6 and 7 send theirs to 3.
Then processes 2 and 3 send their sums to 1, which sums them and prints them out. The communication pattern looks like this:
© Stewart Weiss. CC-BY-SA.
Some Terminology
A path from a node S to a node T is a sequence of nodes S=N0, N1, N2, ..., Nk = T such that there is an edge from Ni to Ni+1 for each i.
- The length of the path is the number of edges. If S == T the path has length 0.
The distance between two nodes S and T is the length of the shortest path between them.
The degree of a node is the number of edges that are incident to that node, whether they are incoming or outgoing or undirected.
© Stewart Weiss. CC-BY-SA.
Mesh Network Topology
A mesh network looks like a grid in two dimensions, or a wire mesh cube in three dimensions, but is harder to visualize when there are more than 3 dimensions.
Formally, it is called a lattice.
Does not have to have same size in each dimension.
© Stewart Weiss. CC-BY-SA.
Hypercube (Binary n-Cube)
A binary n-cube, also called a hypercube, network is a network with N = 2^n nodes arranged as the vertices of a n-dimensional cube.
A hypercube is simply a generalization of an ordinary cube.
In a 3D cube, there are N = 8 = 2^3 nodes, each connected to 3 nodes, one in each dimension.
A square is a 2D cube. It has N = 4 = 2^2 nodes, each connected to 2 nodes, one in each dimension.
In general, in an n-cube, each node is connected to n other nodes, ome in each dimension.
- These arise very often in various parallel algorithms, where nodes represent tasks.
This is a 4-cube:
© Stewart Weiss. CC-BY-SA.
Butterfly Network Topology Connections
Each node is connected to the node above it and below it in its column.
Nodes are also connected to nodes not in their columns.
- Let a xor b represent the bitwise exclusive-or of a and b. In each rank, from 0 through k-1, each node [i,j] is connected to node [i+1, j xor 2^(k-i-1)].
It is easier to see how this works if you write numbers in binary using k bits for order k. Consider k=3= 011. Node [1,3] is [001,011]. 2^(k-i-1) = 2^(3-1-1) = 2= 010. The bitwise xor of j = 3 = 011 and 010 is 001 = 1, so [001,011] has an edge to [010,001] = [2,1].
There is a way to "grow" butterfly networks recursively. See the lecture notes.
© Stewart Weiss. CC-BY-SA.
Interconnection Network Properties
An interconnection network may be classified as shared or switched.
A shared network can have at most one message on it at any time. For example, a bus is a shared network, as is traditional Ethernet.
A switched network allows point-to-point messages among pairs of nodes and therefore supports the transfer of multiple concurrent messages. It is a collection of interconnected switches. Switched Ethernet is a switched network.
Shared networks are inferior to switched networks in terms of performance and scalability. In the example to the left, the switched network allows two links simultaneously.
© Stewart Weiss. CC-BY-SA.
Interconnection Network Topologies
Because interconnection networks can connect different types of devices, they have been used in different ways.
In one approach, each node is a switch, and exactly one device is connected to that switch. This is called a direct topology. The mesh to the right is a direct topology.
The convention is that switches are drawn with circles and devices such as processors, computers, or memories, are drawn with squares.
© Stewart Weiss. CC-BY-SA.
Interconnection Network Topologies
Because interconnection networks can connect different types of devices, they have been used in different ways.
In another approach, called an indirect topology, the number of switches is greater than the number of device nodes. The switches are used for routing messages from one processor or device to another. The binary tree to the right is used as a switching network in which the processors are connected only to the leaf nodes.
© Stewart Weiss. CC-BY-SA.
UMA Multiprocessors
When the shared memory is a physically central memory in one place, it is called any of:
- centralized multiprocessor,
- symmetric multiprocessor, or
- Uniform Memory Access (UMA) multiprocessor
Schematically, it looks like this:
In more detail, like this:
© Stewart Weiss. CC-BY-SA.
NUMA Multiprocessors
If a multiprocessor does not have a centralized memory, but instead has physical memory distributed across the separate CPUs, it is called either:
distributed multiprocessor, or a
Non-Uniform Memory Access (NUMA) multiprocessor
The CPUs still can access a shared set of addresses, but how they do it is different than if the memory were centralized.
Schematically, a NUMA multiprocessor looks like this:
- Data that is in the memory attached to a CPU is local to that CPU.
- Accessing local data is much faster than accessing non-local data because memory accesses that are not local must traverse the network that connects the CPUs.
- Coding efficiently for NUMA machines requires understanding where data might be located, to reduce that effects caused by local versus non-local memory accesses.
© Stewart Weiss. CC-BY-SA.
Asymmetrical Multicomputers
Some multicomputers, called asymmetrical multicomputers, are designed with a special front-end computer and back-end computers.
The front-end is the master and gateway for the machine, and the back-end CPUs are for computation. Users login to the front-end and their jobs are run on the back-end processors, and all I/O takes place through the front-end machine.
- The front-end is a bottleneck
- Does not scale well
- Other problems as well
© Stewart Weiss. CC-BY-SA.
Symmetrical Multicomputers
A symmetrical multicomputer is one in which all of the hosts are identical and are connected to each other through an interconnection network.
Users can login to any host and the file system and I/O devices are equally accessible from every host.
- No bottleneck at front-end , and highly scalable,
- Not suitable environment for running large scale parallel programs, because there is little control over loads on hosts