SGI 10-Gigabit Ethernet Network Adapter Manuale Utente Scaricare il pdf (Pagina 48)

CCGrid '11

Communication in the Memory Semantics (RDMA Model)

InfiniBand Device

Memory

InfiniBand Device

Send Recv

Memory

Segment

Send WQE contains information about the

send buffer (multiple segments) and the

receive buffer (single segment)

Processor

Send Recv

Memory

Segment

Hardware ACK

Memory

Segment

Memory

Segment

Initiator processor is involved only to:

1. Post send WQE

2. Pull out completed CQE from the send CQ

No involvement from the target processor

1 2 ... 43 44 45 46 47 48 49 50 51 52 53 ... 149 150

A Tutorial at CCGrid ’11 1
Computing Systems 3
Cluster Computing Environment 4
(http://www.top500.org) 5
Grid Computing Environment 6
Compute cluster 8
Cloud Computing Environments 9
Hadoop Architecture 10
Memcached Architecture 11
• Hardware components 13
• Software components 13
• Ex: TCP/IP, UDP/IP 14
– E.g., PCI, PCI-X 15
– One bit per wire 15
– Not scalable: 15
Myrinet (1993 -) 1 Gbit/sec 16
IB Trade Association 19
• Network speed bottlenecks 21
• I/O interface bottlenecks 21
(not shown) 24
• Fast Messages (FM) 27
• Myricom GM 27
IB Hardware Acceleration 28
• Interrupt Coalescing 29
• Jumbo Frames 29
• Hardware Checksum Engines 29
TOE and iWARP Accelerators 30
2006 (v3.0), 2008 (v3.1) 34
PCI-Express (PCIe) 34
2003 (Gen1), 2007 (Gen2) 34
IB, HSE and their Convergence 36
Traditional Ethernet 37
IB Overview 38
Components: Channel Adapters 39
• Switches: intra-subnet 40
– Not directly addressable 41
IB Communication Model 43
Queue Pair Model 44
Memory Registration 45
Memory Protection 46
(Send/Receive Model) 47
InfiniBand Device 48
Hardware ACK 48
Hardware Protocol Offload 51
• Buffering and Flow Control 52
• Switching and Multicast 52
Buffering and Flow Control 53
Virtual Lanes 54
Service Levels and QoS 55
Traffic Segregation Benefits 56
Identifiers) 57
Switch Complex 58
– 3D Torus (Sandia Red Sky) 59
More on Multipathing 60
IB Multicast Example 61
IB Transport Services 63
Reliability 64
Transport Layer Capabilities 65
Data Segmentation 66
Transaction Ordering 67
Message-level Flow-Control 68
Concepts in IB Management 71
Subnet Manager 72
HSE Overview 74
Differences 75
– Multi Stream Semantics 76
Offloaded TCP 83
Myrinet Express (MX) 85
Datagram Bypass Layer (DBL) 86
• Solarflare approach: 87
Ethernet 90
IB Network 90
IB Verbs 90
Application 90
Hardware 90
IB Transport 91
IB iWARP/HSE RoE RoCE 92
IB Hardware Products 94
Tyan Thunder S2935 Board 95
IB Hardware Products (contd.) 96
– Nortel Networks 97
• Support for VPI and RoCE 98
– www.openfabrics.org 99
– OFED 1.6 is underway 99
Verbs Interface 100
(libibverbs) 100
• Within the hardware: 101
OpenFabrics Software Stack 102
InfiniBand in the Top500 103
Cray Interconnect 104
NUMAlink 104
SP Switch 104
CCGrid '11 106
• Other small clusters 107
• Integrated Systems 107
Other HSE Installations 108
Presentation Overview 109
1/10 GigE 110
InfiniBand 110
Case Studies 111
Latency (us) 112
Message Size (bytes) 112
Bandwidth (MBps) 113
MVAPICH/MVAPICH2 Software 115
One-way Latency: MPI over IB 116
Bandwidth: MPI over IB 117
Bandwidth: MPI over iWARP 119
Convergent Technologies: 121
Traditional Model 123
Possible SDP Model 123
InfiniBand CA 123
SDP vs. IPoIB (IB QDR) 124
IB on the WAN 126
Features 127
• Supports SONET OC-192 or 128
“appear” local 128
Sunnyvale 129
700 miles 130
3300 miles 4300 miles 130
Cluster B 131
Communication Options in Grid 132
Interface 133
Modern WAN 133
Performance of Memory Based 134
Data Transfer 134
ADTS-64MB 135
IPoIB-64MB 135
Application Level Performance 136
Memcached Design Using Verbs 139
Memcached Get Latency 140
Memcached Get TPS 141
Bandwidth with C version 142
Hadoop Sort Benchmark 145
Concluding Remarks 147
Funding Acknowledgments 148
Personnel Acknowledgments 149
Web Pointers 150

Commenti su questo manuale

Nessun commento

SGI 10-Gigabit Ethernet Network Adapter Manuale Utente Pagina 48

Commenti su questo manuale

Prodotti e manuali riguardandi Networking SGI 10-Gigabit Ethernet Network Adapter