1 ixl FreeBSD* Base Driver and ixlv VF Driver for the
2 Intel XL710 Ethernet Controller Family
3
4 /*$FreeBSD$*/
5 ================================================================
6
7 August 26, 2014
8
9
10 Contents
11 ========
12
13 - Overview
14 - Supported Adapters
15 - The VF Driver
16 - Building and Installation
17 - Additional Configurations
18 - Known Limitations
19
20
21 Overview
22 ========
23
24 This file describes the IXL FreeBSD* Base driver and the IXLV VF Driver
25 for the XL710 Ethernet Family of Adapters. The Driver has been developed
26 for use with FreeBSD 10.0 or later, but should be compatible with any
27 supported release.
28
29 For questions related to hardware requirements, refer to the documentation
30 supplied with your Intel XL710 adapter. All hardware requirements listed
31 apply for use with FreeBSD.
32
33
34 Supported Adapters
35 ==================
36
37 The drivers in this release are compatible with XL710 and X710-based
38 Intel Ethernet Network Connections.
39
40
41 SFP+ Devices with Pluggable Optics
42 ----------------------------------
43
44 SR Modules
45 ----------
46 Intel DUAL RATE 1G/10G SFP+ SR (bailed) FTLX8571D3BCV-IT
47 Intel DUAL RATE 1G/10G SFP+ SR (bailed) AFBR-703SDZ-IN2
48
49 LR Modules
50 ----------
51 Intel DUAL RATE 1G/10G SFP+ LR (bailed) FTLX1471D3BCV-IT
52 Intel DUAL RATE 1G/10G SFP+ LR (bailed) AFCT-701SDZ-IN2
53
54 QSFP+ Modules
55 -------------
56 Intel TRIPLE RATE 1G/10G/40G QSFP+ SR (bailed) E40GQSFPSR
57 Intel TRIPLE RATE 1G/10G/40G QSFP+ LR (bailed) E40GQSFPLR
58 QSFP+ 1G speed is not supported on XL710 based devices.
59
60 X710/XL710 Based SFP+ adapters support all passive and active limiting direct
61 attach cables that comply with SFF-8431 v4.1 and SFF-8472 v10.4 specifications.
62
63 The VF Driver
64 ==================
65 The VF driver is normally used in a virtualized environment where a host
66 driver manages SRIOV, and provides a VF device to the guest. With this
67 first release the only host environment tested was using Linux QEMU/KVM.
68 Support is planned for Xen and VMWare hosts at a later time.
69
70 In the FreeBSD guest the IXLV driver would be loaded and will function
71 using the VF device assigned to it.
72
73 The VF driver provides most of the same functionality as the CORE driver,
74 but is actually a slave to the Host, access to many controls are actually
75 accomplished by a request to the Host via what is called the "Admin queue".
76 These are startup and initialization events however, once in operation
77 the device is self-contained and should achieve near native performance.
78
79 Some notable limitations of the VF environment: for security reasons
80 the driver is never permitted to be promiscuous, therefore a tcpdump
81 will not behave the same with the interface. Second, media info is not
82 available from the PF, so it will always appear as auto.
83
84 Tarball Building and Installation
85 =========================
86
87 NOTE: You must have kernel sources installed to compile the driver tarball.
88
89 These instructions assume a standalone driver tarball, building the driver
90 already in the kernel source is simply a matter of adding the device entry
91 to the kernel config file, or building in the ixl or ixlv module directory.
92
93 In the instructions below, x.x.x is the driver version
94 as indicated in the name of the driver tarball. The example is
95 for ixl, the same procedure applies for ixlv.
96
97 1. Move the base driver tar file to the directory of your choice.
98 For example, use /home/username/ixl or /usr/local/src/ixl.
99
100 2. Untar/unzip the archive:
101 tar xfz ixl-x.x.x.tar.gz
102
103 3. To install man page:
104 cd ixl-x.x.x
105 gzip -c ixl.4 > /usr/share/man/man4/ixl.4.gz
106
107 4. To load the driver onto a running system:
108 cd ixl-x.x.x/src
109 make load
110
111 5. To assign an IP address to the interface, enter the following:
112 ifconfig ixl<interface_num> <IP_address>
113
114 6. Verify that the interface works. Enter the following, where <IP_address>
115 is the IP address for another machine on the same subnet as the interface
116 that is being tested:
117
118 ping <IP_address>
119
120 7. If you want the driver to load automatically when the system is booted:
121
122 cd ixl-x.x.x/src
123 make
124 make install
125
126 Edit /boot/loader.conf, and add the following line:
127 if_ixl_load="YES"
128
129 Edit /etc/rc.conf, and create the appropriate
130 ifconfig_ixl<interface_num> entry:
131
132 ifconfig_ixl<interface_num>="<ifconfig_settings>"
133
134 Example usage:
135
136 ifconfig_ixl0="inet 192.168.10.1 netmask 255.255.255.0"
137
138 NOTE: For assistance, see the ifconfig man page.
139
140
141
142 Configuration and Tuning
143 =========================
144
145 Both drivers supports Transmit/Receive Checksum Offload for IPv4 and IPv6,
146 TSO forIPv4 and IPv6, LRO, and Jumbo Frames on all 40 Gigabit adapters.
147
148 Jumbo Frames
149 ------------
150 To enable Jumbo Frames, use the ifconfig utility to increase
151 the MTU beyond 1500 bytes.
152
153 - The Jumbo Frames setting on the switch must be set to at least
154 22 byteslarger than that of the adapter.
155
156 - The maximum MTU setting for Jumbo Frames is 9706. This value
157 coincides with the maximum jumbo frames size of 9728.
158 To modify the setting, enter the following:
159
160 ifconfig ixl<interface_num> <hostname or IP address> mtu 9000
161
162 - To confirm an interface's MTU value, use the ifconfig command.
163 To confirm the MTU used between two specific devices, use:
164
165 route get <destination_IP_address>
166
167 VLANs
168 -----
169 To create a new VLAN pseudo-interface:
170
171 ifconfig <vlan_name> create
172
173 To associate the VLAN pseudo-interface with a physical interface
174 and assign a VLAN ID, IP address, and netmask:
175
176 ifconfig <vlan_name> <ip_address> netmask <subnet_mask> vlan
177 <vlan_id> vlandev <physical_interface>
178
179 Example:
180
181 ifconfig vlan10 10.0.0.1 netmask 255.255.255.0 vlan 10 vlandev ixl0
182
183 In this example, all packets will be marked on egress with
184 802.1Q VLAN tags, specifying a VLAN ID of 10.
185
186 To remove a VLAN pseudo-interface:
187
188 ifconfig <vlan_name> destroy
189
190
191 Checksum Offload
192 ----------------
193
194 Checksum offloading supports IPv4 and IPv6 with TCP and UDP packets
195 and is supported for both transmit and receive. Checksum offloading
196 for transmit and recieve is enabled by default for both IPv4 and IPv6.
197
198 Checksum offloading can be enabled or disabled using ifconfig.
199 Transmit and receive offloading for IPv4 and Ipv6 are enabled
200 and disabled seperately.
201
202 NOTE: TSO requires Tx checksum, so when Tx checksum
203 is disabled, TSO will also be disabled.
204
205 To enable Tx checksum offloading for ipv4:
206
207 ifconfig ixl<interface_num> txcsum4
208
209 To disable Tx checksum offloading for ipv4:
210
211 ifconfig ixl<interface_num> -txcsum4
212 (NOTE: This will disable TSO4)
213
214 To enable Rx checksum offloading for ipv6:
215
216 ifconfig ixl<interface_num> rxcsum6
217
218 To disable Rx checksum offloading for ipv6:
219
220 ifconfig ixl<interface_num> -rxcsum6
221 (NOTE: This will disable TSO6)
222
223
224 To confirm the current settings:
225
226 ifconfig ixl<interface_num>
227
228
229 TSO
230 ---
231
232 TSO supports both IPv4 and IPv6 and is enabled by default. TSO can
233 be disabled and enabled using the ifconfig utility.
234
235 NOTE: TSO requires Tx checksum, so when Tx checksum is
236 disabled, TSO will also be disabled.
237
238 To disable TSO IPv4:
239
240 ifconfig ixl<interface_num> -tso4
241
242 To enable TSO IPv4:
243
244 ifconfig ixl<interface_num> tso4
245
246 To disable TSO IPv6:
247
248 ifconfig ixl<interface_num> -tso6
249
250 To enable TSO IPv6:
251
252 ifconfig ixl<interface_num> tso6
253
254 To disable BOTH TSO IPv4 and IPv6:
255
256 ifconfig ixl<interface_num> -tso
257
258 To enable BOTH TSO IPv4 and IPv6:
259
260 ifconfig ixl<interface_num> tso
261
262
263 LRO
264 ---
265
266 Large Receive Offload is enabled by default. It can be enabled
267 or disabled by using the ifconfig utility.
268
269 NOTE: LRO should be disabled when forwarding packets.
270
271 To disable LRO:
272
273 ifconfig ixl<interface_num> -lro
274
275 To enable LRO:
276
277 ifconfig ixl<interface_num> lro
278
279
280 Flow Control (IXL only)
281 ------------
282 Flow control is disabled by default. To change flow control settings use sysctl.
283
284 To enable flow control to Rx pause frames:
285
286 sysctl dev.ixl.<interface_num>.fc=1
287
288 To enable flow control to Tx pause frames:
289
290 sysctl dev.ixl.<interface_num>.fc=2
291
292 To enable flow control to Rx and Tx pause frames:
293
294 sysctl dev.ixl.<interface_num>.fc=3
295
296 To disable flow control:
297
298 sysctl dev.ixl.<interface_num>.fc=0
299
300
301 NOTE: You must have a flow control capable link partner.
302
303 NOTE: The VF driver does not have access to flow control, it must be
304 managed from the host side.
305
306
307 Important system configuration changes:
308 =======================================
309
310 -Change the file /etc/sysctl.conf, and add the line:
311
312 hw.intr_storm_threshold: 0 (the default is 1000)
313
314 -Best throughput results are seen with a large MTU; use 9706 if possible.
315
316 -The default number of descriptors per ring is 1024, increasing this may
317 improve performance depending on the use case.
318
319 -The VF driver uses a relatively large buf ring, this was found to eliminate
320 UDP transmit errors, it is a tuneable, and if no UDP traffic is used it can
321 be reduced. It is memory used per queue.
322
323
324 Known Limitations
325 =================
326
327 Network Memory Buffer allocation
328 --------------------------------
329 FreeBSD may have a low number of network memory buffers (mbufs) by default.
330 If your mbuf value is too low, it may cause the driver to fail to initialize
331 and/or cause the system to become unresponsive. You can check to see if the
332 system is mbuf-starved by running 'netstat -m'. Increase the number of mbufs
333 by editing the lines below in /etc/sysctl.conf:
334
335 kern.ipc.nmbclusters
336 kern.ipc.nmbjumbop
337 kern.ipc.nmbjumbo9
338 kern.ipc.nmbjumbo16
339 kern.ipc.nmbufs
340
341 The amount of memory that you allocate is system specific, and may
342 require some trial and error.
343
344 Also, increasing the follwing in /etc/sysctl.conf could help increase
345 network performance:
346
347 kern.ipc.maxsockbuf
348 net.inet.tcp.sendspace
349 net.inet.tcp.recvspace
350 net.inet.udp.maxdgram
351 net.inet.udp.recvspace
352
353
354 UDP Stress Test Dropped Packet Issue
355 ------------------------------------
356 Under small packet UDP stress test with the ixl driver, the FreeBSD system
357 may drop UDP packets due to the fullness of socket buffers. You may want to
358 change the driver's Flow Control variables to the minimum value for
359 controlling packet reception.
360
361
362 Disable LRO when routing/bridging
363 ---------------------------------
364 LRO must be turned off when forwarding traffic.
365
366
367 Lower than expected performance
368 -------------------------------
369 Some PCIe x8 slots are actually configured as x4 slots. These slots have
370 insufficient bandwidth for full line rate with dual port and quad port
371 devices.
372
373 In addition, if you put a PCIe Generation 3-capable adapter into a PCIe
374 Generation 2 slot, you cannot get full bandwidth. The driver detects this
375 situation and writes the following message in the system log:
376
377 "PCI-Express bandwidth available for this card is not sufficient for
378 optimal performance. For optimal performance a x8 PCI-Express slot
379 is required."
380
381 If this error occurs, moving your adapter to a true PCIe Generation 3 x8
382 slot will resolve the issue.
383
384
385 Support
386 =======
387
388 For general information and support, go to the Intel support website at:
389
390 http://support.intel.com
391
392 If an issue is identified with the released source code on the supported kernel
393 with a supported adapter, email the specific information related to the issue
394 to freebsdnic@mailbox.intel.com.
395
396
397 License
398 =======
399
400 This software program is released under the terms of a license agreement
401 between you ('Licensee') and Intel. Do not use or load this software or any
402 associated materials (collectively, the 'Software') until you have carefully
403 read the full terms and conditions of the LICENSE located in this software
404 package. By loadingor using the Software, you agree to the terms of this
405 Agreement. If you do not agree with the terms of this Agreement, do not
406 install or use the Software.
407
408 * Other names and brands may be claimed as the property of others.
409
410