SGI 10-Gigabit Ethernet Network Adapter Manuale Utente Scaricare il pdf (Pagina 45)

1. Registration Request

• Send virtual address and length

2. Kernel handles virtual->physical

mapping and pins region into

physical memory

• Process cannot map memory

that it does not own (security !)

3. HCA caches the virtual to physical

mapping and issues a handle

• Includes an l_key and r_key

4. Handle is returned to application

CCGrid '11

Memory Registration

Before we do any communication:

All memory used for communication must

be registered

Process

Kernel

HCA/RNIC

1 2 ... 40 41 42 43 44 45 46 47 48 49 50 ... 149 150

A Tutorial at CCGrid ’11 1
Computing Systems 3
Cluster Computing Environment 4
(http://www.top500.org) 5
Grid Computing Environment 6
Compute cluster 8
Cloud Computing Environments 9
Hadoop Architecture 10
Memcached Architecture 11
• Hardware components 13
• Software components 13
• Ex: TCP/IP, UDP/IP 14
– E.g., PCI, PCI-X 15
– One bit per wire 15
– Not scalable: 15
Myrinet (1993 -) 1 Gbit/sec 16
IB Trade Association 19
• Network speed bottlenecks 21
• I/O interface bottlenecks 21
(not shown) 24
• Fast Messages (FM) 27
• Myricom GM 27
IB Hardware Acceleration 28
• Interrupt Coalescing 29
• Jumbo Frames 29
• Hardware Checksum Engines 29
TOE and iWARP Accelerators 30
2006 (v3.0), 2008 (v3.1) 34
PCI-Express (PCIe) 34
2003 (Gen1), 2007 (Gen2) 34
IB, HSE and their Convergence 36
Traditional Ethernet 37
IB Overview 38
Components: Channel Adapters 39
• Switches: intra-subnet 40
– Not directly addressable 41
IB Communication Model 43
Queue Pair Model 44
Memory Registration 45
Memory Protection 46
(Send/Receive Model) 47
InfiniBand Device 48
Hardware ACK 48
Hardware Protocol Offload 51
• Buffering and Flow Control 52
• Switching and Multicast 52
Buffering and Flow Control 53
Virtual Lanes 54
Service Levels and QoS 55
Traffic Segregation Benefits 56
Identifiers) 57
Switch Complex 58
– 3D Torus (Sandia Red Sky) 59
More on Multipathing 60
IB Multicast Example 61
IB Transport Services 63
Reliability 64
Transport Layer Capabilities 65
Data Segmentation 66
Transaction Ordering 67
Message-level Flow-Control 68
Concepts in IB Management 71
Subnet Manager 72
HSE Overview 74
Differences 75
– Multi Stream Semantics 76
Offloaded TCP 83
Myrinet Express (MX) 85
Datagram Bypass Layer (DBL) 86
• Solarflare approach: 87
Ethernet 90
IB Network 90
IB Verbs 90
Application 90
Hardware 90
IB Transport 91
IB iWARP/HSE RoE RoCE 92
IB Hardware Products 94
Tyan Thunder S2935 Board 95
IB Hardware Products (contd.) 96
– Nortel Networks 97
• Support for VPI and RoCE 98
– www.openfabrics.org 99
– OFED 1.6 is underway 99
Verbs Interface 100
(libibverbs) 100
• Within the hardware: 101
OpenFabrics Software Stack 102
InfiniBand in the Top500 103
Cray Interconnect 104
NUMAlink 104
SP Switch 104
CCGrid '11 106
• Other small clusters 107
• Integrated Systems 107
Other HSE Installations 108
Presentation Overview 109
1/10 GigE 110
InfiniBand 110
Case Studies 111
Latency (us) 112
Message Size (bytes) 112
Bandwidth (MBps) 113
MVAPICH/MVAPICH2 Software 115
One-way Latency: MPI over IB 116
Bandwidth: MPI over IB 117
Bandwidth: MPI over iWARP 119
Convergent Technologies: 121
Traditional Model 123
Possible SDP Model 123
InfiniBand CA 123
SDP vs. IPoIB (IB QDR) 124
IB on the WAN 126
Features 127
• Supports SONET OC-192 or 128
“appear” local 128
Sunnyvale 129
700 miles 130
3300 miles 4300 miles 130
Cluster B 131
Communication Options in Grid 132
Interface 133
Modern WAN 133
Performance of Memory Based 134
Data Transfer 134
ADTS-64MB 135
IPoIB-64MB 135
Application Level Performance 136
Memcached Design Using Verbs 139
Memcached Get Latency 140
Memcached Get TPS 141
Bandwidth with C version 142
Hadoop Sort Benchmark 145
Concluding Remarks 147
Funding Acknowledgments 148
Personnel Acknowledgments 149
Web Pointers 150

Commenti su questo manuale

Nessun commento

SGI 10-Gigabit Ethernet Network Adapter Manuale Utente Pagina 45

Commenti su questo manuale

Prodotti e manuali riguardandi Networking SGI 10-Gigabit Ethernet Network Adapter