Print this page
9832 Original bug discovered as 9560 has friends IPv4 packets coming in as IPv6 creating chaos
| Split |
Close |
| Expand all |
| Collapse all |
--- old/usr/src/uts/common/io/mac/mac_sched.c
+++ new/usr/src/uts/common/io/mac/mac_sched.c
1 1 /*
2 2 * CDDL HEADER START
3 3 *
4 4 * The contents of this file are subject to the terms of the
5 5 * Common Development and Distribution License (the "License").
6 6 * You may not use this file except in compliance with the License.
7 7 *
8 8 * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
9 9 * or http://www.opensolaris.org/os/licensing.
10 10 * See the License for the specific language governing permissions
11 11 * and limitations under the License.
12 12 *
13 13 * When distributing Covered Code, include this CDDL HEADER in each
|
↓ open down ↓ |
13 lines elided |
↑ open up ↑ |
14 14 * file and include the License file at usr/src/OPENSOLARIS.LICENSE.
15 15 * If applicable, add the following below this CDDL HEADER, with the
16 16 * fields enclosed by brackets "[]" replaced with your own identifying
17 17 * information: Portions Copyright [yyyy] [name of copyright owner]
18 18 *
19 19 * CDDL HEADER END
20 20 */
21 21 /*
22 22 * Copyright 2010 Sun Microsystems, Inc. All rights reserved.
23 23 * Use is subject to license terms.
24 - * Copyright 2017 Joyent, Inc.
24 + * Copyright 2018 Joyent, Inc.
25 25 * Copyright 2013 Nexenta Systems, Inc. All rights reserved.
26 26 */
27 27
28 28 /*
29 29 * MAC data path
30 30 *
31 31 * The MAC data path is concerned with the flow of traffic from mac clients --
32 32 * DLS, IP, etc. -- to various GLDv3 device drivers -- e1000g, vnic, aggr,
33 33 * ixgbe, etc. -- and from the GLDv3 device drivers back to clients.
34 34 *
35 35 * -----------
36 36 * Terminology
37 37 * -----------
38 38 *
39 39 * MAC uses a lot of different, but related terms that are associated with the
40 40 * design and structure of the data path. Before we cover other aspects, first
41 41 * let's review the terminology that MAC uses.
42 42 *
43 43 * MAC
44 44 *
45 45 * This driver. It interfaces with device drivers and provides abstractions
46 46 * that the rest of the system consumes. All data links -- things managed
47 47 * with dladm(1M), are accessed through MAC.
48 48 *
49 49 * GLDv3 DEVICE DRIVER
50 50 *
51 51 * A GLDv3 device driver refers to a driver, both for pseudo-devices and
52 52 * real devices, which implement the GLDv3 driver API. Common examples of
53 53 * these are igb and ixgbe, which are drivers for various Intel networking
54 54 * cards. These devices may or may not have various features, such as
55 55 * hardware rings and checksum offloading. For MAC, a GLDv3 device is the
56 56 * final point for the transmission of a packet and the starting point for
57 57 * the receipt of a packet.
58 58 *
59 59 * FLOWS
60 60 *
61 61 * At a high level, a flow refers to a series of packets that are related.
62 62 * Often times the term is used in the context of TCP to indicate a unique
63 63 * TCP connection and the traffic over it. However, a flow can exist at
64 64 * other levels of the system as well. MAC has a notion of a default flow
65 65 * which is used for all unicast traffic addressed to the address of a MAC
66 66 * device. For example, when a VNIC is created, a default flow is created
67 67 * for the VNIC's MAC address. In addition, flows are created for broadcast
68 68 * groups and a user may create a flow with flowadm(1M).
69 69 *
70 70 * CLASSIFICATION
71 71 *
72 72 * Classification refers to the notion of identifying an incoming frame
73 73 * based on its destination address and optionally its source addresses and
74 74 * doing different processing based on that information. Classification can
75 75 * be done in both hardware and software. In general, we usually only
76 76 * classify based on the layer two destination, eg. for Ethernet, the
77 77 * destination MAC address.
78 78 *
79 79 * The system also will do classification based on layer three and layer
80 80 * four properties. This is used to support things like flowadm(1M), which
81 81 * allows setting QoS and other properties on a per-flow basis.
82 82 *
83 83 * RING
84 84 *
85 85 * Conceptually, a ring represents a series of framed messages, often in a
86 86 * contiguous chunk of memory that acts as a circular buffer. Rings come in
87 87 * a couple of forms. Generally they are either a hardware construct (hw
88 88 * ring) or they are a software construct (sw ring) maintained by MAC.
89 89 *
90 90 * HW RING
91 91 *
92 92 * A hardware ring is a set of resources provided by a GLDv3 device driver
93 93 * (even if it is a pseudo-device). A hardware ring comes in two different
94 94 * forms: receive (rx) rings and transmit (tx) rings. An rx hw ring is
95 95 * something that has a unique DMA (direct memory access) region and
96 96 * generally supports some form of classification (though it isn't always
97 97 * used), as well as a means of generating an interrupt specific to that
98 98 * ring. For example, the device may generate a specific MSI-X for a PCI
99 99 * express device. A tx ring is similar, except that it is dedicated to
100 100 * transmission. It may also be a vector for enabling features such as VLAN
101 101 * tagging and large transmit offloading. It usually has its own dedicated
102 102 * interrupts for transmit being completed.
103 103 *
104 104 * SW RING
105 105 *
106 106 * A software ring is a construction of MAC. It represents the same thing
107 107 * that a hardware ring generally does, a collection of frames. However,
108 108 * instead of being in a contiguous ring of memory, they're instead linked
109 109 * by using the mblk_t's b_next pointer. Each frame may itself be multiple
110 110 * mblk_t's linked together by the b_cont pointer. A software ring always
111 111 * represents a collection of classified packets; however, it varies as to
112 112 * whether it uses only layer two information, or a combination of that and
113 113 * additional layer three and layer four data.
114 114 *
115 115 * FANOUT
116 116 *
117 117 * Fanout is the idea of spreading out the load of processing frames based
118 118 * on the source and destination information contained in the layer two,
119 119 * three, and four headers, such that the data can then be processed in
120 120 * parallel using multiple hardware threads.
121 121 *
122 122 * A fanout algorithm hashes the headers and uses that to place different
123 123 * flows into a bucket. The most important thing is that packets that are
124 124 * in the same flow end up in the same bucket. If they do not, performance
125 125 * can be adversely affected. Consider the case of TCP. TCP severely
126 126 * penalizes a connection if the data arrives out of order. If a given flow
127 127 * is processed on different CPUs, then the data will appear out of order,
128 128 * hence the invariant that fanout always hash a given flow to the same
129 129 * bucket and thus get processed on the same CPU.
130 130 *
131 131 * RECEIVE SIDE SCALING (RSS)
132 132 *
133 133 *
134 134 * Receive side scaling is a term that isn't common in illumos, but is used
135 135 * by vendors and was popularized by Microsoft. It refers to the idea of
136 136 * spreading the incoming receive load out across multiple interrupts which
137 137 * can be directed to different CPUs. This allows a device to leverage
138 138 * hardware rings even when it doesn't support hardware classification. The
139 139 * hardware uses an algorithm to perform fanout that ensures the flow
140 140 * invariant is maintained.
141 141 *
142 142 * SOFT RING SET
143 143 *
144 144 * A soft ring set, commonly abbreviated SRS, is a collection of rings and
145 145 * is used for both transmitting and receiving. It is maintained in the
146 146 * structure mac_soft_ring_set_t. A soft ring set is usually associated
147 147 * with flows, and coordinates both the use of hardware and software rings.
148 148 * Because the use of hardware rings can change as devices such as VNICs
149 149 * come and go, we always ensure that the set has software classification
150 150 * rules that correspond to the hardware classification rules from rings.
151 151 *
152 152 * Soft ring sets are also used for the enforcement of various QoS
153 153 * properties. For example, if a bandwidth limit has been placed on a
154 154 * specific flow or device, then that will be enforced by the soft ring
155 155 * set.
156 156 *
157 157 * SERVICE ATTACHMENT POINT (SAP)
158 158 *
159 159 * The service attachment point is a DLPI (Data Link Provider Interface)
160 160 * concept; however, it comes up quite often in MAC. Most MAC devices speak
161 161 * a protocol that has some notion of different channels or message type
162 162 * identifiers. For example, Ethernet defines an EtherType which is a part
163 163 * of the Ethernet header and defines the particular protocol of the data
164 164 * payload. If the EtherType is set to 0x0800, then it defines that the
165 165 * contents of that Ethernet frame is IPv4 traffic. For Ethernet, the
166 166 * EtherType is the SAP.
167 167 *
168 168 * In DLPI, a given consumer attaches to a specific SAP. In illumos, the ip
169 169 * and arp drivers attach to the EtherTypes for IPv4, IPv6, and ARP. Using
170 170 * libdlpi(3LIB) user software can attach to arbitrary SAPs. With the
171 171 * exception of 802.1Q VLAN tagged traffic, MAC itself does not directly
172 172 * consume the SAP; however, it uses that information as part of hashing
173 173 * and it may be used as part of the construction of flows.
174 174 *
175 175 * PRIMARY MAC CLIENT
176 176 *
177 177 * The primary mac client refers to a mac client whose unicast address
178 178 * matches the address of the device itself. For example, if the system has
179 179 * instance of the e1000g driver such as e1000g0, e1000g1, etc., the
180 180 * primary mac client is the one named after the device itself. VNICs that
181 181 * are created on top of such devices are not the primary client.
182 182 *
183 183 * TRANSMIT DESCRIPTORS
184 184 *
185 185 * Transmit descriptors are a resource that most GLDv3 device drivers have.
186 186 * Generally, a GLDv3 device driver takes a frame that's meant to be output
187 187 * and puts a copy of it into a region of memory. Each region of memory
188 188 * usually has an associated descriptor that the device uses to manage
189 189 * properties of the frames. Devices have a limited number of such
190 190 * descriptors. They get reclaimed once the device finishes putting the
191 191 * frame on the wire.
192 192 *
193 193 * If the driver runs out of transmit descriptors, for example, the OS is
194 194 * generating more frames than it can put on the wire, then it will return
195 195 * them back to the MAC layer.
196 196 *
197 197 * ---------------------------------
198 198 * Rings, Classification, and Fanout
199 199 * ---------------------------------
200 200 *
201 201 * The heart of MAC is made up of rings, and not those that Elven-kings wear.
202 202 * When receiving a packet, MAC breaks the work into two different, though
203 203 * interrelated phases. The first phase is generally classification and then the
204 204 * second phase is generally fanout. When a frame comes in from a GLDv3 Device,
205 205 * MAC needs to determine where that frame should be delivered. If it's a
206 206 * unicast frame (say a normal TCP/IP packet), then it will be delivered to a
207 207 * single MAC client; however, if it's a broadcast or multicast frame, then MAC
208 208 * may need to deliver it to multiple MAC clients.
209 209 *
210 210 * On transmit, classification isn't quite as important, but may still be used.
211 211 * Unlike with the receive path, the classification is not used to determine
212 212 * devices that should transmit something, but rather is used for special
213 213 * properties of a flow, eg. bandwidth limits for a given IP address, device, or
214 214 * connection.
215 215 *
216 216 * MAC employs a software classifier and leverages hardware classification as
217 217 * well. The software classifier can leverage the full layer two information,
218 218 * source, destination, VLAN, and SAP. If the SAP indicates that IP traffic is
219 219 * being sent, it can classify based on the IP header, and finally, it also
220 220 * knows how to classify based on the local and remote ports of TCP, UDP, and
221 221 * SCTP.
222 222 *
223 223 * Hardware classifiers vary in capability. Generally all hardware classifiers
224 224 * provide the capability to classify based on the destination MAC address. Some
225 225 * hardware has additional filters built in for performing more in-depth
226 226 * classification; however, it often has much more limited resources for these
227 227 * activities as compared to the layer two destination address classification.
228 228 *
229 229 * The modus operandi in MAC is to always ensure that we have software-based
230 230 * capabilities and rules in place and then to supplement that with hardware
231 231 * resources when available. In general, simple layer two classification is
232 232 * sufficient and nothing else is used, unless a specific flow is created with
233 233 * tools such as flowadm(1M) or bandwidth limits are set on a device with
234 234 * dladm(1M).
235 235 *
236 236 * RINGS AND GROUPS
237 237 *
238 238 * To get into how rings and classification play together, it's first important
239 239 * to understand how hardware devices commonly associate rings and allow them to
240 240 * be programmed. Recall that a hardware ring should be thought of as a DMA
241 241 * buffer and an interrupt resource. Rings are then collected into groups. A
242 242 * group itself has a series of classification rules. One or more MAC addresses
243 243 * are assigned to a group.
244 244 *
245 245 * Hardware devices vary in terms of what capabilities they provide. Sometimes
246 246 * they allow for a dynamic assignment of rings to a group and sometimes they
247 247 * have a static assignment of rings to a group. For example, the ixgbe driver
248 248 * has a static assignment of rings to groups such that every group has exactly
249 249 * one ring and the number of groups is equal to the number of rings.
250 250 *
251 251 * Classification and receive side scaling both come into play with how a device
252 252 * advertises itself to MAC and how MAC uses it. If a device supports layer two
253 253 * classification of frames, then MAC will assign MAC addresses to a group as a
254 254 * form of primary classification. If a single MAC address is assigned to a
255 255 * group, a common case, then MAC will consider packets that come in from rings
256 256 * on that group to be fully classified and will not need to do any software
257 257 * classification unless a specific flow has been created.
258 258 *
259 259 * If a device supports receive side scaling, then it may advertise or support
260 260 * groups with multiple rings. In those cases, then receive side scaling will
261 261 * come into play and MAC will use that as a means of fanning out received
262 262 * frames across multiple CPUs. This can also be combined with groups that
263 263 * support layer two classification.
264 264 *
265 265 * If a device supports dynamic assignments of rings to groups, then MAC will
266 266 * change around the way that rings are assigned to various groups as devices
267 267 * come and go from the system. For example, when a VNIC is created, a new flow
268 268 * will be created for the VNIC's MAC address. If a hardware ring is available,
269 269 * MAC may opt to reassign it from one group to another.
270 270 *
271 271 * ASSIGNMENT OF HARDWARE RINGS
272 272 *
273 273 * This is a bit of a complicated subject that varies depending on the device,
274 274 * the use of aggregations, the special nature of the primary mac client. This
275 275 * section deserves being fleshed out.
276 276 *
277 277 * FANOUT
278 278 *
279 279 * illumos uses fanout to help spread out the incoming processing load of chains
280 280 * of frames away from a single CPU. If a device supports receive side scaling,
281 281 * then that provides an initial form of fanout; however, what we're concerned
282 282 * with all happens after the context of a given set of frames being classified
283 283 * to a soft ring set.
284 284 *
285 285 * After frames reach a soft ring set and account for any potential bandwidth
286 286 * related accounting, they may be fanned out based on one of the following
287 287 * three modes:
288 288 *
289 289 * o No Fanout
290 290 * o Protocol level fanout
291 291 * o Full software ring protocol fanout
292 292 *
293 293 * MAC makes the determination as to which of these modes a given soft ring set
294 294 * obtains based on parameters such as whether or not it's the primary mac
295 295 * client, whether it's on a 10 GbE or faster device, user controlled dladm(1M)
296 296 * properties, and the nature of the hardware and the resources that it has.
297 297 *
298 298 * When there is no fanout, MAC does not create any soft rings for a device and
299 299 * the device has frames delivered directly to the MAC client.
300 300 *
301 301 * Otherwise, all fanout is performed by software. MAC divides incoming frames
302 302 * into one of three buckets -- IPv4 TCP traffic, IPv4 UDP traffic, and
303 303 * everything else. Note, VLAN tagged traffic is considered other, regardless of
304 304 * the interior EtherType. Regardless of the type of fanout, these three
305 305 * categories or buckets are always used.
306 306 *
307 307 * The difference between protocol level fanout and full software ring protocol
308 308 * fanout is the number of software rings that end up getting created. The
309 309 * system always uses the same number of software rings per protocol bucket. So
310 310 * in the first case when we're just doing protocol level fanout, we just create
311 311 * one software ring each for IPv4 TCP traffic, IPv4 UDP traffic, and everything
312 312 * else.
313 313 *
314 314 * In the case where we do full software ring protocol fanout, we generally use
315 315 * mac_compute_soft_ring_count() to determine the number of rings. There are
316 316 * other combinations of properties and devices that may send us down other
317 317 * paths, but this is a common starting point. If it's a non-bandwidth enforced
318 318 * device and we're on at least a 10 GbE link, then we'll use eight soft rings
319 319 * per protocol bucket as a starting point. See mac_compute_soft_ring_count()
320 320 * for more information on the total number.
321 321 *
322 322 * For each of these rings, we create a mac_soft_ring_t and an associated worker
323 323 * thread. Particularly when doing full software ring protocol fanout, we bind
324 324 * each of the worker threads to individual CPUs.
325 325 *
326 326 * The other advantage of these software rings is that it allows upper layers to
327 327 * optionally poll on them. For example, TCP can leverage an squeue to poll on
328 328 * the software ring, see squeue.c for more information.
329 329 *
330 330 * DLS BYPASS
331 331 *
332 332 * DLS is the data link services module. It interfaces with DLPI, which is the
333 333 * primary way that other parts of the system such as IP interface with the MAC
334 334 * layer. While DLS is traditionally a STREAMS-based interface, it allows for
335 335 * certain modules such as IP to negotiate various more modern interfaces to be
336 336 * used, which are useful for higher performance and allow it to use direct
337 337 * function calls to DLS instead of using STREAMS.
338 338 *
339 339 * When we have IPv4 TCP or UDP software rings, then traffic on those rings is
340 340 * eligible for what we call the dls bypass. In those cases, rather than going
341 341 * out mac_rx_deliver() to DLS, DLS instead registers them to go directly via
342 342 * the direct callback registered with DLS, generally ip_input().
343 343 *
344 344 * HARDWARE RING POLLING
345 345 *
346 346 * GLDv3 devices with hardware rings generally deliver chains of messages
347 347 * (mblk_t chain) during the context of a single interrupt. However, interrupts
348 348 * are not the only way that these devices may be used. As part of implementing
349 349 * ring support, a GLDv3 device driver must have a way to disable the generation
350 350 * of that interrupt and allow for the operating system to poll on that ring.
351 351 *
352 352 * To implement this, every soft ring set has a worker thread and a polling
353 353 * thread. If a sufficient packet rate comes into the system, MAC will 'blank'
354 354 * (disable) interrupts on that specific ring and the polling thread will start
355 355 * consuming packets from the hardware device and deliver them to the soft ring
356 356 * set, where the worker thread will take over.
357 357 *
358 358 * Once the rate of packet intake drops down below a certain threshold, then
359 359 * polling on the hardware ring will be quiesced and interrupts will be
360 360 * re-enabled for the given ring. This effectively allows the system to shift
361 361 * how it handles a ring based on its load. At high packet rates, polling on the
362 362 * device as opposed to relying on interrupts can actually reduce overall system
363 363 * load due to the minimization of interrupt activity.
364 364 *
365 365 * Note the importance of each ring having its own interrupt source. The whole
366 366 * idea here is that we do not disable interrupts on the device as a whole, but
367 367 * rather each ring can be independently toggled.
368 368 *
369 369 * USE OF WORKER THREADS
370 370 *
371 371 * Both the soft ring set and individual soft rings have a worker thread
372 372 * associated with them that may be bound to a specific CPU in the system. Any
373 373 * such assignment will get reassessed as part of dynamic reconfiguration events
374 374 * in the system such as the onlining and offlining of CPUs and the creation of
375 375 * CPU partitions.
376 376 *
377 377 * In many cases, while in an interrupt, we try to deliver a frame all the way
378 378 * through the stack in the context of the interrupt itself. However, if the
379 379 * amount of queued frames has exceeded a threshold, then we instead defer to
380 380 * the worker thread to do this work and signal it. This is particularly useful
381 381 * when you have the soft ring set delivering frames into multiple software
382 382 * rings. If it was only delivering frames into a single software ring then
383 383 * there'd be no need to have another thread take over. However, if it's
384 384 * delivering chains of frames to multiple rings, then it's worthwhile to have
385 385 * the worker for the software ring take over so that the different software
386 386 * rings can be processed in parallel.
387 387 *
388 388 * In a similar fashion to the hardware polling thread, if we don't have a
389 389 * backlog or there's nothing to do, then the worker thread will go back to
390 390 * sleep and frames can be delivered all the way from an interrupt. This
391 391 * behavior is useful as it's designed to minimize latency and the default
392 392 * disposition of MAC is to optimize for latency.
393 393 *
394 394 * MAINTAINING CHAINS
395 395 *
396 396 * Another useful idea that MAC uses is to try and maintain frames in chains for
397 397 * as long as possible. The idea is that all of MAC can handle chains of frames
398 398 * structured as a series of mblk_t structures linked with the b_next pointer.
399 399 * When performing software classification and software fanout, MAC does not
400 400 * simply determine the destination and send the frame along. Instead, in the
401 401 * case of classification, it tries to maintain a chain for as long as possible
402 402 * before passing it along and performing additional processing.
403 403 *
404 404 * In the case of fanout, MAC first determines what the target software ring is
405 405 * for every frame in the original chain and constructs a new chain for each
406 406 * target. MAC then delivers the new chain to each software ring in succession.
407 407 *
408 408 * The whole rationale for doing this is that we want to try and maintain the
409 409 * pipe as much as possible and deliver as many frames through the stack at once
410 410 * that we can, rather than just pushing a single frame through. This can often
411 411 * help bring down latency and allows MAC to get a better sense of the overall
412 412 * activity in the system and properly engage worker threads.
413 413 *
414 414 * --------------------
415 415 * Bandwidth Management
416 416 * --------------------
417 417 *
418 418 * Bandwidth management is something that's built into the soft ring set itself.
419 419 * When bandwidth limits are placed on a flow, a corresponding soft ring set is
420 420 * toggled into bandwidth mode. This changes how we transmit and receive the
421 421 * frames in question.
422 422 *
423 423 * Bandwidth management is done on a per-tick basis. We translate the user's
424 424 * requested bandwidth from a quantity per-second into a quantity per-tick. MAC
425 425 * cannot process a frame across more than one tick, thus it sets a lower bound
426 426 * for the bandwidth cap to be a single MTU. This also means that when
427 427 * hires ticks are enabled (hz is set to 1000), that the minimum amount of
428 428 * bandwidth is higher, because the number of ticks has increased and MAC has to
429 429 * go from accepting 100 packets / sec to 1000 / sec.
430 430 *
431 431 * The bandwidth counter is reset by either the soft ring set's worker thread or
432 432 * a thread that is doing an inline transmit or receive if they discover that
433 433 * the current tick is in the future from the recorded tick.
434 434 *
435 435 * Whenever we're receiving or transmitting data, we end up leaving most of the
436 436 * work to the soft ring set's worker thread. This forces data inserted into the
437 437 * soft ring set to be effectively serialized and allows us to exhume bandwidth
438 438 * at a reasonable rate. If there is nothing in the soft ring set at the moment
439 439 * and the set has available bandwidth, then it may processed inline.
440 440 * Otherwise, the worker is responsible for taking care of the soft ring set.
441 441 *
442 442 * ---------------------
443 443 * The Receive Data Path
444 444 * ---------------------
445 445 *
446 446 * The following series of ASCII art images breaks apart the way that a frame
447 447 * comes in and is processed in MAC.
448 448 *
449 449 * Part 1 -- Initial frame receipt, SRS classification
450 450 *
451 451 * Here, a frame is received by a GLDv3 driver, generally in the context of an
452 452 * interrupt, and it ends up in mac_rx_common(). A driver calls either mac_rx or
453 453 * mac_rx_ring, depending on whether or not it supports rings and can identify
454 454 * the interrupt as having come from a specific ring. Here we determine whether
455 455 * or not it's fully classified and perform software classification as
456 456 * appropriate. From here, everything always ends up going to either entry [A]
457 457 * or entry [B] based on whether or not they have subflow processing needed. We
458 458 * leave via fanout or delivery.
459 459 *
460 460 * +===========+
461 461 * v hardware v
462 462 * v interrupt v
463 463 * +===========+
464 464 * |
465 465 * * . . appropriate
466 466 * | upcall made
467 467 * | by GLDv3 driver . . always
468 468 * | .
469 469 * +--------+ | +----------+ . +---------------+
470 470 * | GLDv3 | +---->| mac_rx |-----*--->| mac_rx_common |
471 471 * | Driver |-->--+ +----------+ +---------------+
472 472 * +--------+ | ^ |
473 473 * | | ^ v
474 474 * ^ | * . . always +----------------------+
475 475 * | | | | mac_promisc_dispatch |
476 476 * | | +-------------+ +----------------------+
477 477 * | +--->| mac_rx_ring | |
478 478 * | +-------------+ * . . hw classified
479 479 * | v or single flow?
480 480 * | |
481 481 * | +--------++--------------+
482 482 * | | | * hw class,
483 483 * | | * hw classified | subflows
484 484 * | no hw class and . * | or single | exist
485 485 * | subflows | | flow |
486 486 * | | v v
487 487 * | | +-----------+ +-----------+
488 488 * | | | goto | | goto |
489 489 * | | | entry [A] | | entry [B] |
490 490 * | | +-----------+ +-----------+
491 491 * | v ^
492 492 * | +-------------+ |
493 493 * | | mac_rx_flow | * SRS and flow found,
494 494 * | +-------------+ | call flow cb
495 495 * | | +------+
496 496 * | v |
497 497 * v +==========+ +-----------------+
498 498 * | v For each v--->| mac_rx_classify |
499 499 * +----------+ v mblk_t v +-----------------+
500 500 * | srs | +==========+
501 501 * | pollling |
502 502 * | thread |->------------------------------------------+
503 503 * +----------+ |
504 504 * v . inline
505 505 * +--------------------+ +----------+ +---------+ .
506 506 * [A]---->| mac_rx_srs_process |-->| check bw |-->| enqueue |--*---------+
507 507 * +--------------------+ | limits | | frames | |
508 508 * ^ +----------+ | to SRS | |
509 509 * | +---------+ |
510 510 * | send chain +--------+ | |
511 511 * * when clasified | signal | * BW limits, |
512 512 * | flow changes | srs |<---+ loopback, |
513 513 * | | worker | stack too |
514 514 * | +--------+ deep |
515 515 * +-----------------+ +--------+ |
516 516 * | mac_flow_lookup | | srs | +---------------------+ |
517 517 * +-----------------+ | worker |---->| mac_rx_srs_drain |<---+
518 518 * ^ | thread | | mac_rx_srs_drain_bw |
519 519 * | +--------+ +---------------------+
520 520 * | |
521 521 * +----------------------------+ * software rings
522 522 * [B]-->| mac_rx_srs_subflow_process | | for fanout?
523 523 * +----------------------------+ |
524 524 * +----------+-----------+
525 525 * | |
526 526 * v v
527 527 * +--------+ +--------+
528 528 * | goto | | goto |
529 529 * | Part 2 | | Part 3 |
530 530 * +--------+ +--------+
531 531 *
532 532 * Part 2 -- Fanout
533 533 *
534 534 * This part is concerned with using software fanout to assign frames to
535 535 * software rings and then deliver them to MAC clients or allow those rings to
536 536 * be polled upon. While there are two different primary fanout entry points,
537 537 * mac_rx_fanout and mac_rx_proto_fanout, they behave in similar ways, and aside
538 538 * from some of the individual hashing techniques used, most of the general
539 539 * flow is the same.
540 540 *
541 541 * +--------+ +-------------------+
542 542 * | From |---+--------->| mac_rx_srs_fanout |----+
543 543 * | Part 1 | | +-------------------+ | +=================+
544 544 * +--------+ | | v for each mblk_t v
545 545 * * . . protocol only +--->v assign to new v
546 546 * | fanout | v chain based on v
547 547 * | | v hash % nrings v
548 548 * | +-------------------------+ | +=================+
549 549 * +--->| mac_rx_srs_proto_fanout |----+ |
550 550 * +-------------------------+ |
551 551 * v
552 552 * +------------+ +--------------------------+ +================+
553 553 * | enqueue in |<---| mac_rx_soft_ring_process |<------v for each chain v
554 554 * | soft ring | +--------------------------+ +================+
555 555 * +------------+
556 556 * | +-----------+
557 557 * * soft ring set | soft ring |
558 558 * | empty and no | worker |
559 559 * | worker? | thread |
560 560 * | +-----------+
561 561 * +------*----------------+ |
562 562 * | . | v
563 563 * No . * . Yes | +------------------------+
564 564 * | +----<--| mac_rx_soft_ring_drain |
565 565 * | | +------------------------+
566 566 * v |
567 567 * +-----------+ v
568 568 * | signal | +---------------+
569 569 * | soft ring | | Deliver chain |
570 570 * | worker | | goto Part 3 |
571 571 * +-----------+ +---------------+
572 572 *
573 573 *
574 574 * Part 3 -- Packet Delivery
575 575 *
576 576 * Here, we go through and deliver the mblk_t chain directly to a given
577 577 * processing function. In a lot of cases this is mac_rx_deliver(). In the case
578 578 * of DLS bypass being used, then instead we end up going ahead and deliver it
579 579 * to the direct callback registered with DLS, generally ip_input.
580 580 *
581 581 *
582 582 * +---------+ +----------------+ +------------------+
583 583 * | From |---+------->| mac_rx_deliver |--->| Off to DLS, or |
584 584 * | Parts 1 | | +----------------+ | other MAC client |
585 585 * | and 2 | * DLS bypass +------------------+
586 586 * +---------+ | enabled +----------+ +-------------+
587 587 * +---------->| ip_input |--->| To IP |
588 588 * +----------+ | and beyond! |
589 589 * +-------------+
590 590 *
591 591 * ----------------------
592 592 * The Transmit Data Path
593 593 * ----------------------
594 594 *
595 595 * Before we go into the images, it's worth talking about a problem that is a
596 596 * bit different from the receive data path. GLDv3 device drivers have a finite
597 597 * amount of transmit descriptors. When they run out, they return unused frames
598 598 * back to MAC. MAC, at this point has several options about what it will do,
599 599 * which vary based upon the settings that the client uses.
600 600 *
601 601 * When a device runs out of descriptors, the next thing that MAC does is
602 602 * enqueue them off of the soft ring set or a software ring, depending on the
603 603 * configuration of the soft ring set. MAC will enqueue up to a high watermark
604 604 * of mblk_t chains, at which point it will indicate flow control back to the
605 605 * client. Once this condition is reached, any mblk_t chains that were not
606 606 * enqueued will be returned to the caller and they will have to decide what to
607 607 * do with them. There are various flags that control this behavior that a
608 608 * client may pass, which are discussed below.
609 609 *
610 610 * When this condition is hit, MAC also returns a cookie to the client in
611 611 * addition to unconsumed frames. Clients can poll on that cookie and register a
612 612 * callback with MAC to be notified when they are no longer subject to flow
613 613 * control, at which point they may continue to call mac_tx(). This flow control
614 614 * actually manages to work itself all the way up the stack, back through dls,
615 615 * to ip, through the various protocols, and to sockfs.
616 616 *
617 617 * While the behavior described above is the default, this behavior can be
618 618 * modified. There are two alternate modes, described below, which are
619 619 * controlled with flags.
620 620 *
621 621 * DROP MODE
622 622 *
623 623 * This mode is controlled by having the client pass the MAC_DROP_ON_NO_DESC
624 624 * flag. When this is passed, if a device driver runs out of transmit
625 625 * descriptors, then the MAC layer will drop any unsent traffic. The client in
626 626 * this case will never have any frames returned to it.
627 627 *
628 628 * DON'T ENQUEUE
629 629 *
630 630 * This mode is controlled by having the client pass the MAC_TX_NO_ENQUEUE flag.
631 631 * If the MAC_DROP_ON_NO_DESC flag is also passed, it takes precedence. In this
632 632 * mode, when we hit a case where a driver runs out of transmit descriptors,
633 633 * then instead of enqueuing packets in a soft ring set or software ring, we
634 634 * instead return the mblk_t chain back to the caller and immediately put the
635 635 * soft ring set into flow control mode.
636 636 *
637 637 * The following series of ASCII art images describe the transmit data path that
638 638 * MAC clients enter into based on calling into mac_tx(). A soft ring set has a
639 639 * transmission function associated with it. There are seven possible
640 640 * transmission modes, some of which share function entry points. The one that a
641 641 * soft ring set gets depends on properties such as whether there are
642 642 * transmission rings for fanout, whether the device involves aggregations,
643 643 * whether any bandwidth limits exist, etc.
644 644 *
645 645 *
646 646 * Part 1 -- Initial checks
647 647 *
648 648 * * . called by
649 649 * | MAC clients
650 650 * v . . No
651 651 * +--------+ +-----------+ . +-------------------+ +====================+
652 652 * | mac_tx |->| device |-*-->| mac_protect_check |->v Is this the simple v
653 653 * +--------+ | quiesced? | +-------------------+ v case? See [1] v
654 654 * +-----------+ | +====================+
655 655 * * . Yes * failed |
656 656 * v | frames |
657 657 * +--------------+ | +-------+---------+
658 658 * | freemsgchain |<---------+ Yes . * No . *
659 659 * +--------------+ v v
660 660 * +-----------+ +--------+
661 661 * | goto | | goto |
662 662 * | Part 2 | | SRS TX |
663 663 * | Entry [A] | | func |
664 664 * +-----------+ +--------+
665 665 * | |
666 666 * | v
667 667 * | +--------+
668 668 * +---------->| return |
669 669 * | cookie |
670 670 * +--------+
671 671 *
672 672 * [1] The simple case refers to the SRS being configured with the
673 673 * SRS_TX_DEFAULT transmission mode, having a single mblk_t (not a chain), their
674 674 * being only a single active client, and not having a backlog in the srs.
675 675 *
676 676 *
677 677 * Part 2 -- The SRS transmission functions
678 678 *
679 679 * This part is a bit more complicated. The different transmission paths often
680 680 * leverage one another. In this case, we'll draw out the more common ones
681 681 * before the parts that depend upon them. Here, we're going to start with the
682 682 * workings of mac_tx_send() a common function that most of the others end up
683 683 * calling.
684 684 *
685 685 * +-------------+
686 686 * | mac_tx_send |
687 687 * +-------------+
688 688 * |
689 689 * v
690 690 * +=============+ +==============+
691 691 * v more than v--->v check v
692 692 * v one client? v v VLAN and add v
693 693 * +=============+ v VLAN tags v
694 694 * | +==============+
695 695 * | |
696 696 * +------------------+
697 697 * |
698 698 * | [A]
699 699 * v |
700 700 * +============+ . No v
701 701 * v more than v . +==========+ +--------------------------+
702 702 * v one active v-*---->v for each v---->| mac_promisc_dispatch_one |---+
703 703 * v client? v v mblk_t v +--------------------------+ |
704 704 * +============+ +==========+ ^ |
705 705 * | | +==========+ |
706 706 * * . Yes | v hardware v<-------+
707 707 * v +------------+ v rings? v
708 708 * +==========+ | +==========+
709 709 * v for each v No . . . * |
710 710 * v mblk_t v specific | |
711 711 * +==========+ flow | +-----+-----+
712 712 * | | | |
713 713 * v | v v
714 714 * +-----------------+ | +-------+ +---------+
715 715 * | mac_tx_classify |------------+ | GLDv3 | | GLDv3 |
716 716 * +-----------------+ |TX func| | ring tx |
717 717 * | +-------+ | func |
718 718 * * Specific flow, generally | +---------+
719 719 * | bcast, mcast, loopback | |
720 720 * v +-----+-----+
721 721 * +==========+ +---------+ |
722 722 * v valid L2 v--*--->| freemsg | v
723 723 * v header v . No +---------+ +-------------------+
724 724 * +==========+ | return unconsumed |
725 725 * * . Yes | frames to the |
726 726 * v | caller |
727 727 * +===========+ +-------------------+
728 728 * v braodcast v +----------------+ ^
729 729 * v flow? v--*-->| mac_bcast_send |------------------+
730 730 * +===========+ . +----------------+ |
731 731 * | . . Yes |
732 732 * No . * v
733 733 * | +---------------------+ +---------------+ +----------+
734 734 * +->|mac_promisc_dispatch |->| mac_fix_cksum |->| flow |
735 735 * +---------------------+ +---------------+ | callback |
736 736 * +----------+
737 737 *
738 738 *
739 739 * In addition, many but not all of the routines, all rely on
740 740 * mac_tx_softring_process as an entry point.
741 741 *
742 742 *
743 743 * . No . No
744 744 * +--------------------------+ +========+ . +===========+ . +-------------+
745 745 * | mac_tx_soft_ring_process |-->v worker v-*->v out of tx v-*->| goto |
746 746 * +--------------------------+ v only? v v descr.? v | mac_tx_send |
747 747 * +========+ +===========+ +-------------+
748 748 * Yes . * * . Yes |
749 749 * . No v | v
750 750 * v=========+ . +===========+ . Yes | Yes . +==========+
751 751 * v apppend v<--*----------v out of tx v-*-------+---------*--v returned v
752 752 * v mblk_t v v descr.? v | v frames? v
753 753 * v chain v +===========+ | +==========+
754 754 * +=========+ | *. No
755 755 * | | v
756 756 * v v +------------+
757 757 * +===================+ +----------------------+ | done |
758 758 * v worker scheduled? v | mac_tx_sring_enqueue | | processing |
759 759 * v Out of tx descr? v +----------------------+ +------------+
760 760 * +===================+ |
761 761 * | | . Yes v
762 762 * * Yes * No . +============+
763 763 * | v +-*---------v drop on no v
764 764 * | +========+ v v TX desc? v
765 765 * | v wake v +----------+ +============+
766 766 * | v worker v | mac_pkt_ | * . No
767 767 * | +========+ | drop | | . Yes . No
768 768 * | | +----------+ v . .
769 769 * | | v ^ +===============+ . +========+ .
770 770 * +--+--------+---------+ | v Don't enqueue v-*->v ring v-*----+
771 771 * | | v Set? v v empty? v |
772 772 * | +---------------+ +===============+ +========+ |
773 773 * | | | | |
774 774 * | | +-------------------+ | |
775 775 * | *. Yes | +---------+ |
776 776 * | | v v v
777 777 * | | +===========+ +========+ +--------------+
778 778 * | +<-v At hiwat? v v append v | return |
779 779 * | +===========+ v mblk_t v | mblk_t chain |
780 780 * | * No v chain v | and flow |
781 781 * | v +========+ | control |
782 782 * | +=========+ | | cookie |
783 783 * | v append v v +--------------+
784 784 * | v mblk_t v +========+
785 785 * | v chain v v wake v +------------+
786 786 * | +=========+ v worker v-->| done |
787 787 * | | +========+ | processing |
788 788 * | v .. Yes +------------+
789 789 * | +=========+ . +========+
790 790 * | v first v--*-->v wake v
791 791 * | v append? v v worker v
792 792 * | +=========+ +========+
793 793 * | | |
794 794 * | No . * |
795 795 * | v |
796 796 * | +--------------+ |
797 797 * +------>| Return | |
798 798 * | flow control |<------------+
799 799 * | cookie |
800 800 * +--------------+
801 801 *
802 802 *
803 803 * The remaining images are all specific to each of the different transmission
804 804 * modes.
805 805 *
806 806 * SRS TX DEFAULT
807 807 *
808 808 * [ From Part 1 ]
809 809 * |
810 810 * v
811 811 * +-------------------------+
812 812 * | mac_tx_single_ring_mode |
813 813 * +-------------------------+
814 814 * |
815 815 * | . Yes
816 816 * v .
817 817 * +==========+ . +============+
818 818 * v SRS v-*->v Try to v---->---------------------+
819 819 * v backlog? v v enqueue in v |
820 820 * +==========+ v SRS v-->------+ * . . Queue too
821 821 * | +============+ * don't enqueue | deep or
822 822 * * . No ^ | | flag or at | drop flag
823 823 * | | v | hiwat, |
824 824 * v | | | return +---------+
825 825 * +-------------+ | | | cookie | freemsg |
826 826 * | goto |-*-----+ | | +---------+
827 827 * | mac_tx_send | . returned | | |
828 828 * +-------------+ mblk_t | | |
829 829 * | | | |
830 830 * | | | |
831 831 * * . . all mblk_t * queued, | |
832 832 * v consumed | may return | |
833 833 * +-------------+ | tx cookie | |
834 834 * | SRS TX func |<------------+------------+----------------+
835 835 * | completed |
836 836 * +-------------+
837 837 *
838 838 * SRS_TX_SERIALIZE
839 839 *
840 840 * +------------------------+
841 841 * | mac_tx_serializer_mode |
842 842 * +------------------------+
843 843 * |
844 844 * | . No
845 845 * v .
846 846 * +============+ . +============+ +-------------+ +============+
847 847 * v srs being v-*->v set SRS v--->| goto |-->v remove SRS v
848 848 * v processed? v v proc flags v | mac_tx_send | v proc flag v
849 849 * +============+ +============+ +-------------+ +============+
850 850 * | |
851 851 * * Yes |
852 852 * v . No v
853 853 * +--------------------+ . +==========+
854 854 * | mac_tx_srs_enqueue | +------------------------*-----<--v returned v
855 855 * +--------------------+ | v frames? v
856 856 * | | . Yes +==========+
857 857 * | | . |
858 858 * | | . +=========+ v
859 859 * v +-<-*-v queued v +--------------------+
860 860 * +-------------+ | v frames? v<----| mac_tx_srs_enqueue |
861 861 * | SRS TX func | | +=========+ +--------------------+
862 862 * | completed, |<------+ * . Yes
863 863 * | may return | | v
864 864 * | cookie | | +========+
865 865 * +-------------+ +-<---v wake v
866 866 * v worker v
867 867 * +========+
868 868 *
869 869 *
870 870 * SRS_TX_FANOUT
871 871 *
872 872 * . Yes
873 873 * +--------------------+ +=============+ . +--------------------------+
874 874 * | mac_tx_fanout_mode |--->v Have fanout v-*-->| goto |
875 875 * +--------------------+ v hint? v | mac_rx_soft_ring_process |
876 876 * +=============+ +--------------------------+
877 877 * * . No |
878 878 * v ^
879 879 * +===========+ |
880 880 * +--->v for each v +===============+
881 881 * | v mblk_t v v pick softring v
882 882 * same * +===========+ v from hash v
883 883 * hash | | +===============+
884 884 * | v |
885 885 * | +--------------+ |
886 886 * +---| mac_pkt_hash |--->*------------+
887 887 * +--------------+ . different
888 888 * hash or
889 889 * done proc.
890 890 * SRS_TX_AGGR chain
891 891 *
892 892 * +------------------+ +================================+
893 893 * | mac_tx_aggr_mode |--->v Use aggr capab function to v
894 894 * +------------------+ v find appropriate tx ring. v
895 895 * v Applies hash based on aggr v
896 896 * v policy, see mac_tx_aggr_mode() v
897 897 * +================================+
898 898 * |
899 899 * v
900 900 * +-------------------------------+
901 901 * | goto |
902 902 * | mac_rx_srs_soft_ring_process |
903 903 * +-------------------------------+
904 904 *
905 905 *
906 906 * SRS_TX_BW, SRS_TX_BW_FANOUT, SRS_TX_BW_AGGR
907 907 *
908 908 * Note, all three of these tx functions start from the same place --
909 909 * mac_tx_bw_mode().
910 910 *
911 911 * +----------------+
912 912 * | mac_tx_bw_mode |
913 913 * +----------------+
914 914 * |
915 915 * v . No . No . Yes
916 916 * +==============+ . +============+ . +=============+ . +=========+
917 917 * v Out of BW? v--*->v SRS empty? v--*->v reset BW v-*->v Bump BW v
918 918 * +==============+ +============+ v tick count? v v Usage v
919 919 * | | +=============+ +=========+
920 920 * | +---------+ | |
921 921 * | | +--------------------+ |
922 922 * | | | +----------------------+
923 923 * v | v v
924 924 * +===============+ | +==========+ +==========+ +------------------+
925 925 * v Don't enqueue v | v set bw v v Is aggr? v--*-->| goto |
926 926 * v flag set? v | v enforced v +==========+ . | mac_tx_aggr_mode |-+
927 927 * +===============+ | +==========+ | . +------------------+ |
928 928 * | Yes .* | | No . * . |
929 929 * | | | | | . Yes |
930 930 * * . No | | v | |
931 931 * | +---------+ | +========+ v +======+ |
932 932 * | | freemsg | | v append v +============+ . Yes v pick v |
933 933 * | +---------+ | v mblk_t v v Is fanout? v--*---->v ring v |
934 934 * | | | v chain v +============+ +======+ |
935 935 * +------+ | +========+ | | |
936 936 * v | | v v |
937 937 * +---------+ | v +-------------+ +--------------------+ |
938 938 * | return | | +========+ | goto | | goto | |
939 939 * | flow | | v wakeup v | mac_tx_send | | mac_tx_fanout_mode | |
940 940 * | control | | v worker v +-------------+ +--------------------+ |
941 941 * | cookie | | +========+ | | |
942 942 * +---------+ | | | +------+------+
943 943 * | v | |
944 944 * | +---------+ | v
945 945 * | | return | +============+ +------------+
946 946 * | | flow | v unconsumed v-------+ | done |
947 947 * | | control | v frames? v | | processing |
948 948 * | | cookie | +============+ | +------------+
949 949 * | +---------+ | |
950 950 * | Yes * |
951 951 * | | |
952 952 * | +===========+ |
953 953 * | v subtract v |
954 954 * | v unused bw v |
955 955 * | +===========+ |
956 956 * | | |
957 957 * | v |
958 958 * | +--------------------+ |
959 959 * +------------->| mac_tx_srs_enqueue | |
960 960 * +--------------------+ |
961 961 * | |
962 962 * | |
963 963 * +------------+ |
964 964 * | return fc | |
965 965 * | cookie and |<------+
966 966 * | mblk_t |
967 967 * +------------+
968 968 */
969 969
970 970 #include <sys/types.h>
971 971 #include <sys/callb.h>
972 972 #include <sys/sdt.h>
973 973 #include <sys/strsubr.h>
974 974 #include <sys/strsun.h>
975 975 #include <sys/vlan.h>
976 976 #include <sys/stack.h>
977 977 #include <sys/archsystm.h>
978 978 #include <inet/ipsec_impl.h>
979 979 #include <inet/ip_impl.h>
980 980 #include <inet/sadb.h>
981 981 #include <inet/ipsecesp.h>
982 982 #include <inet/ipsecah.h>
983 983 #include <inet/ip6.h>
984 984
985 985 #include <sys/mac_impl.h>
986 986 #include <sys/mac_client_impl.h>
987 987 #include <sys/mac_client_priv.h>
988 988 #include <sys/mac_soft_ring.h>
989 989 #include <sys/mac_flow_impl.h>
990 990
991 991 static mac_tx_cookie_t mac_tx_single_ring_mode(mac_soft_ring_set_t *, mblk_t *,
992 992 uintptr_t, uint16_t, mblk_t **);
993 993 static mac_tx_cookie_t mac_tx_serializer_mode(mac_soft_ring_set_t *, mblk_t *,
994 994 uintptr_t, uint16_t, mblk_t **);
995 995 static mac_tx_cookie_t mac_tx_fanout_mode(mac_soft_ring_set_t *, mblk_t *,
996 996 uintptr_t, uint16_t, mblk_t **);
997 997 static mac_tx_cookie_t mac_tx_bw_mode(mac_soft_ring_set_t *, mblk_t *,
998 998 uintptr_t, uint16_t, mblk_t **);
999 999 static mac_tx_cookie_t mac_tx_aggr_mode(mac_soft_ring_set_t *, mblk_t *,
1000 1000 uintptr_t, uint16_t, mblk_t **);
1001 1001
1002 1002 typedef struct mac_tx_mode_s {
1003 1003 mac_tx_srs_mode_t mac_tx_mode;
1004 1004 mac_tx_func_t mac_tx_func;
1005 1005 } mac_tx_mode_t;
1006 1006
1007 1007 /*
1008 1008 * There are seven modes of operation on the Tx side. These modes get set
1009 1009 * in mac_tx_srs_setup(). Except for the experimental TX_SERIALIZE mode,
1010 1010 * none of the other modes are user configurable. They get selected by
1011 1011 * the system depending upon whether the link (or flow) has multiple Tx
1012 1012 * rings or a bandwidth configured, or if the link is an aggr, etc.
1013 1013 *
1014 1014 * When the Tx SRS is operating in aggr mode (st_mode) or if there are
1015 1015 * multiple Tx rings owned by Tx SRS, then each Tx ring (pseudo or
1016 1016 * otherwise) will have a soft ring associated with it. These soft rings
1017 1017 * are stored in srs_tx_soft_rings[] array.
1018 1018 *
1019 1019 * Additionally in the case of aggr, there is the st_soft_rings[] array
1020 1020 * in the mac_srs_tx_t structure. This array is used to store the same
1021 1021 * set of soft rings that are present in srs_tx_soft_rings[] array but
1022 1022 * in a different manner. The soft ring associated with the pseudo Tx
1023 1023 * ring is saved at mr_index (of the pseudo ring) in st_soft_rings[]
1024 1024 * array. This helps in quickly getting the soft ring associated with the
1025 1025 * Tx ring when aggr_find_tx_ring() returns the pseudo Tx ring that is to
1026 1026 * be used for transmit.
1027 1027 */
1028 1028 mac_tx_mode_t mac_tx_mode_list[] = {
1029 1029 {SRS_TX_DEFAULT, mac_tx_single_ring_mode},
1030 1030 {SRS_TX_SERIALIZE, mac_tx_serializer_mode},
1031 1031 {SRS_TX_FANOUT, mac_tx_fanout_mode},
1032 1032 {SRS_TX_BW, mac_tx_bw_mode},
1033 1033 {SRS_TX_BW_FANOUT, mac_tx_bw_mode},
1034 1034 {SRS_TX_AGGR, mac_tx_aggr_mode},
1035 1035 {SRS_TX_BW_AGGR, mac_tx_bw_mode}
1036 1036 };
1037 1037
1038 1038 /*
1039 1039 * Soft Ring Set (SRS) - The Run time code that deals with
1040 1040 * dynamic polling from the hardware, bandwidth enforcement,
1041 1041 * fanout etc.
1042 1042 *
1043 1043 * We try to use H/W classification on NIC and assign traffic for
1044 1044 * a MAC address to a particular Rx ring or ring group. There is a
1045 1045 * 1-1 mapping between a SRS and a Rx ring. The SRS dynamically
1046 1046 * switches the underlying Rx ring between interrupt and
1047 1047 * polling mode and enforces any specified B/W control.
1048 1048 *
1049 1049 * There is always a SRS created and tied to each H/W and S/W rule.
1050 1050 * Whenever we create a H/W rule, we always add the the same rule to
1051 1051 * S/W classifier and tie a SRS to it.
1052 1052 *
1053 1053 * In case a B/W control is specified, it is broken into bytes
1054 1054 * per ticks and as soon as the quota for a tick is exhausted,
1055 1055 * the underlying Rx ring is forced into poll mode for remainder of
1056 1056 * the tick. The SRS poll thread only polls for bytes that are
1057 1057 * allowed to come in the SRS. We typically let 4x the configured
1058 1058 * B/W worth of packets to come in the SRS (to prevent unnecessary
1059 1059 * drops due to bursts) but only process the specified amount.
1060 1060 *
1061 1061 * A MAC client (e.g. a VNIC or aggr) can have 1 or more
1062 1062 * Rx rings (and corresponding SRSs) assigned to it. The SRS
1063 1063 * in turn can have softrings to do protocol level fanout or
1064 1064 * softrings to do S/W based fanout or both. In case the NIC
1065 1065 * has no Rx rings, we do S/W classification to respective SRS.
1066 1066 * The S/W classification rule is always setup and ready. This
1067 1067 * allows the MAC layer to reassign Rx rings whenever needed
1068 1068 * but packets still continue to flow via the default path and
1069 1069 * getting S/W classified to correct SRS.
1070 1070 *
1071 1071 * The SRS's are used on both Tx and Rx side. They use the same
1072 1072 * data structure but the processing routines have slightly different
1073 1073 * semantics due to the fact that Rx side needs to do dynamic
1074 1074 * polling etc.
1075 1075 *
1076 1076 * Dynamic Polling Notes
1077 1077 * =====================
1078 1078 *
1079 1079 * Each Soft ring set is capable of switching its Rx ring between
1080 1080 * interrupt and poll mode and actively 'polls' for packets in
1081 1081 * poll mode. If the SRS is implementing a B/W limit, it makes
1082 1082 * sure that only Max allowed packets are pulled in poll mode
1083 1083 * and goes to poll mode as soon as B/W limit is exceeded. As
1084 1084 * such, there are no overheads to implement B/W limits.
1085 1085 *
1086 1086 * In poll mode, its better to keep the pipeline going where the
1087 1087 * SRS worker thread keeps processing packets and poll thread
1088 1088 * keeps bringing more packets (specially if they get to run
1089 1089 * on different CPUs). This also prevents the overheads associated
1090 1090 * by excessive signalling (on NUMA machines, this can be
1091 1091 * pretty devastating). The exception is latency optimized case
1092 1092 * where worker thread does no work and interrupt and poll thread
1093 1093 * are allowed to do their own drain.
1094 1094 *
1095 1095 * We use the following policy to control Dynamic Polling:
1096 1096 * 1) We switch to poll mode anytime the processing
1097 1097 * thread causes a backlog to build up in SRS and
1098 1098 * its associated Soft Rings (sr_poll_pkt_cnt > 0).
1099 1099 * 2) As long as the backlog stays under the low water
1100 1100 * mark (sr_lowat), we poll the H/W for more packets.
1101 1101 * 3) If the backlog (sr_poll_pkt_cnt) exceeds low
1102 1102 * water mark, we stay in poll mode but don't poll
1103 1103 * the H/W for more packets.
1104 1104 * 4) Anytime in polling mode, if we poll the H/W for
1105 1105 * packets and find nothing plus we have an existing
1106 1106 * backlog (sr_poll_pkt_cnt > 0), we stay in polling
1107 1107 * mode but don't poll the H/W for packets anymore
1108 1108 * (let the polling thread go to sleep).
1109 1109 * 5) Once the backlog is relived (packets are processed)
1110 1110 * we reenable polling (by signalling the poll thread)
1111 1111 * only when the backlog dips below sr_poll_thres.
1112 1112 * 6) sr_hiwat is used exclusively when we are not
1113 1113 * polling capable and is used to decide when to
1114 1114 * drop packets so the SRS queue length doesn't grow
1115 1115 * infinitely.
1116 1116 *
1117 1117 * NOTE: Also see the block level comment on top of mac_soft_ring.c
1118 1118 */
1119 1119
1120 1120 /*
1121 1121 * mac_latency_optimize
1122 1122 *
1123 1123 * Controls whether the poll thread can process the packets inline
1124 1124 * or let the SRS worker thread do the processing. This applies if
1125 1125 * the SRS was not being processed. For latency sensitive traffic,
1126 1126 * this needs to be true to allow inline processing. For throughput
1127 1127 * under load, this should be false.
1128 1128 *
1129 1129 * This (and other similar) tunable should be rolled into a link
1130 1130 * or flow specific workload hint that can be set using dladm
1131 1131 * linkprop (instead of multiple such tunables).
1132 1132 */
1133 1133 boolean_t mac_latency_optimize = B_TRUE;
1134 1134
1135 1135 /*
1136 1136 * MAC_RX_SRS_ENQUEUE_CHAIN and MAC_TX_SRS_ENQUEUE_CHAIN
1137 1137 *
1138 1138 * queue a mp or chain in soft ring set and increment the
1139 1139 * local count (srs_count) for the SRS and the shared counter
1140 1140 * (srs_poll_pkt_cnt - shared between SRS and its soft rings
1141 1141 * to track the total unprocessed packets for polling to work
1142 1142 * correctly).
1143 1143 *
1144 1144 * The size (total bytes queued) counters are incremented only
1145 1145 * if we are doing B/W control.
1146 1146 */
1147 1147 #define MAC_SRS_ENQUEUE_CHAIN(mac_srs, head, tail, count, sz) { \
1148 1148 ASSERT(MUTEX_HELD(&(mac_srs)->srs_lock)); \
1149 1149 if ((mac_srs)->srs_last != NULL) \
1150 1150 (mac_srs)->srs_last->b_next = (head); \
1151 1151 else \
1152 1152 (mac_srs)->srs_first = (head); \
1153 1153 (mac_srs)->srs_last = (tail); \
1154 1154 (mac_srs)->srs_count += count; \
1155 1155 }
1156 1156
1157 1157 #define MAC_RX_SRS_ENQUEUE_CHAIN(mac_srs, head, tail, count, sz) { \
1158 1158 mac_srs_rx_t *srs_rx = &(mac_srs)->srs_rx; \
1159 1159 \
1160 1160 MAC_SRS_ENQUEUE_CHAIN(mac_srs, head, tail, count, sz); \
1161 1161 srs_rx->sr_poll_pkt_cnt += count; \
1162 1162 ASSERT(srs_rx->sr_poll_pkt_cnt > 0); \
1163 1163 if ((mac_srs)->srs_type & SRST_BW_CONTROL) { \
1164 1164 (mac_srs)->srs_size += (sz); \
1165 1165 mutex_enter(&(mac_srs)->srs_bw->mac_bw_lock); \
1166 1166 (mac_srs)->srs_bw->mac_bw_sz += (sz); \
1167 1167 mutex_exit(&(mac_srs)->srs_bw->mac_bw_lock); \
1168 1168 } \
1169 1169 }
1170 1170
1171 1171 #define MAC_TX_SRS_ENQUEUE_CHAIN(mac_srs, head, tail, count, sz) { \
1172 1172 mac_srs->srs_state |= SRS_ENQUEUED; \
1173 1173 MAC_SRS_ENQUEUE_CHAIN(mac_srs, head, tail, count, sz); \
1174 1174 if ((mac_srs)->srs_type & SRST_BW_CONTROL) { \
1175 1175 (mac_srs)->srs_size += (sz); \
1176 1176 (mac_srs)->srs_bw->mac_bw_sz += (sz); \
1177 1177 } \
1178 1178 }
1179 1179
1180 1180 /*
1181 1181 * Turn polling on routines
1182 1182 */
1183 1183 #define MAC_SRS_POLLING_ON(mac_srs) { \
1184 1184 ASSERT(MUTEX_HELD(&(mac_srs)->srs_lock)); \
1185 1185 if (((mac_srs)->srs_state & \
1186 1186 (SRS_POLLING_CAPAB|SRS_POLLING)) == SRS_POLLING_CAPAB) { \
1187 1187 (mac_srs)->srs_state |= SRS_POLLING; \
1188 1188 (void) mac_hwring_disable_intr((mac_ring_handle_t) \
1189 1189 (mac_srs)->srs_ring); \
1190 1190 (mac_srs)->srs_rx.sr_poll_on++; \
1191 1191 } \
1192 1192 }
1193 1193
1194 1194 #define MAC_SRS_WORKER_POLLING_ON(mac_srs) { \
1195 1195 ASSERT(MUTEX_HELD(&(mac_srs)->srs_lock)); \
1196 1196 if (((mac_srs)->srs_state & \
1197 1197 (SRS_POLLING_CAPAB|SRS_WORKER|SRS_POLLING)) == \
1198 1198 (SRS_POLLING_CAPAB|SRS_WORKER)) { \
1199 1199 (mac_srs)->srs_state |= SRS_POLLING; \
1200 1200 (void) mac_hwring_disable_intr((mac_ring_handle_t) \
1201 1201 (mac_srs)->srs_ring); \
1202 1202 (mac_srs)->srs_rx.sr_worker_poll_on++; \
1203 1203 } \
1204 1204 }
1205 1205
1206 1206 /*
1207 1207 * MAC_SRS_POLL_RING
1208 1208 *
1209 1209 * Signal the SRS poll thread to poll the underlying H/W ring
1210 1210 * provided it wasn't already polling (SRS_GET_PKTS was set).
1211 1211 *
1212 1212 * Poll thread gets to run only from mac_rx_srs_drain() and only
1213 1213 * if the drain was being done by the worker thread.
1214 1214 */
1215 1215 #define MAC_SRS_POLL_RING(mac_srs) { \
1216 1216 mac_srs_rx_t *srs_rx = &(mac_srs)->srs_rx; \
1217 1217 \
1218 1218 ASSERT(MUTEX_HELD(&(mac_srs)->srs_lock)); \
1219 1219 srs_rx->sr_poll_thr_sig++; \
1220 1220 if (((mac_srs)->srs_state & \
1221 1221 (SRS_POLLING_CAPAB|SRS_WORKER|SRS_GET_PKTS)) == \
1222 1222 (SRS_WORKER|SRS_POLLING_CAPAB)) { \
1223 1223 (mac_srs)->srs_state |= SRS_GET_PKTS; \
1224 1224 cv_signal(&(mac_srs)->srs_cv); \
1225 1225 } else { \
1226 1226 srs_rx->sr_poll_thr_busy++; \
1227 1227 } \
1228 1228 }
1229 1229
1230 1230 /*
1231 1231 * MAC_SRS_CHECK_BW_CONTROL
1232 1232 *
1233 1233 * Check to see if next tick has started so we can reset the
1234 1234 * SRS_BW_ENFORCED flag and allow more packets to come in the
1235 1235 * system.
1236 1236 */
1237 1237 #define MAC_SRS_CHECK_BW_CONTROL(mac_srs) { \
1238 1238 ASSERT(MUTEX_HELD(&(mac_srs)->srs_lock)); \
1239 1239 ASSERT(((mac_srs)->srs_type & SRST_TX) || \
1240 1240 MUTEX_HELD(&(mac_srs)->srs_bw->mac_bw_lock)); \
1241 1241 clock_t now = ddi_get_lbolt(); \
1242 1242 if ((mac_srs)->srs_bw->mac_bw_curr_time != now) { \
1243 1243 (mac_srs)->srs_bw->mac_bw_curr_time = now; \
1244 1244 (mac_srs)->srs_bw->mac_bw_used = 0; \
1245 1245 if ((mac_srs)->srs_bw->mac_bw_state & SRS_BW_ENFORCED) \
1246 1246 (mac_srs)->srs_bw->mac_bw_state &= ~SRS_BW_ENFORCED; \
1247 1247 } \
1248 1248 }
1249 1249
1250 1250 /*
1251 1251 * MAC_SRS_WORKER_WAKEUP
1252 1252 *
1253 1253 * Wake up the SRS worker thread to process the queue as long as
1254 1254 * no one else is processing the queue. If we are optimizing for
1255 1255 * latency, we wake up the worker thread immediately or else we
1256 1256 * wait mac_srs_worker_wakeup_ticks before worker thread gets
1257 1257 * woken up.
1258 1258 */
1259 1259 int mac_srs_worker_wakeup_ticks = 0;
1260 1260 #define MAC_SRS_WORKER_WAKEUP(mac_srs) { \
1261 1261 ASSERT(MUTEX_HELD(&(mac_srs)->srs_lock)); \
1262 1262 if (!((mac_srs)->srs_state & SRS_PROC) && \
1263 1263 (mac_srs)->srs_tid == NULL) { \
1264 1264 if (((mac_srs)->srs_state & SRS_LATENCY_OPT) || \
1265 1265 (mac_srs_worker_wakeup_ticks == 0)) \
1266 1266 cv_signal(&(mac_srs)->srs_async); \
1267 1267 else \
1268 1268 (mac_srs)->srs_tid = \
1269 1269 timeout(mac_srs_fire, (mac_srs), \
1270 1270 mac_srs_worker_wakeup_ticks); \
1271 1271 } \
1272 1272 }
1273 1273
1274 1274 #define TX_BANDWIDTH_MODE(mac_srs) \
1275 1275 ((mac_srs)->srs_tx.st_mode == SRS_TX_BW || \
1276 1276 (mac_srs)->srs_tx.st_mode == SRS_TX_BW_FANOUT || \
1277 1277 (mac_srs)->srs_tx.st_mode == SRS_TX_BW_AGGR)
1278 1278
1279 1279 #define TX_SRS_TO_SOFT_RING(mac_srs, head, hint) { \
1280 1280 if (tx_mode == SRS_TX_BW_FANOUT) \
1281 1281 (void) mac_tx_fanout_mode(mac_srs, head, hint, 0, NULL);\
1282 1282 else \
1283 1283 (void) mac_tx_aggr_mode(mac_srs, head, hint, 0, NULL); \
1284 1284 }
1285 1285
1286 1286 /*
1287 1287 * MAC_TX_SRS_BLOCK
1288 1288 *
1289 1289 * Always called from mac_tx_srs_drain() function. SRS_TX_BLOCKED
1290 1290 * will be set only if srs_tx_woken_up is FALSE. If
1291 1291 * srs_tx_woken_up is TRUE, it indicates that the wakeup arrived
1292 1292 * before we grabbed srs_lock to set SRS_TX_BLOCKED. We need to
1293 1293 * attempt to transmit again and not setting SRS_TX_BLOCKED does
1294 1294 * that.
1295 1295 */
1296 1296 #define MAC_TX_SRS_BLOCK(srs, mp) { \
1297 1297 ASSERT(MUTEX_HELD(&(srs)->srs_lock)); \
1298 1298 if ((srs)->srs_tx.st_woken_up) { \
1299 1299 (srs)->srs_tx.st_woken_up = B_FALSE; \
1300 1300 } else { \
1301 1301 ASSERT(!((srs)->srs_state & SRS_TX_BLOCKED)); \
1302 1302 (srs)->srs_state |= SRS_TX_BLOCKED; \
1303 1303 (srs)->srs_tx.st_stat.mts_blockcnt++; \
1304 1304 } \
1305 1305 }
1306 1306
1307 1307 /*
1308 1308 * MAC_TX_SRS_TEST_HIWAT
1309 1309 *
1310 1310 * Called before queueing a packet onto Tx SRS to test and set
1311 1311 * SRS_TX_HIWAT if srs_count exceeds srs_tx_hiwat.
1312 1312 */
1313 1313 #define MAC_TX_SRS_TEST_HIWAT(srs, mp, tail, cnt, sz, cookie) { \
1314 1314 boolean_t enqueue = 1; \
1315 1315 \
1316 1316 if ((srs)->srs_count > (srs)->srs_tx.st_hiwat) { \
1317 1317 /* \
1318 1318 * flow-controlled. Store srs in cookie so that it \
1319 1319 * can be returned as mac_tx_cookie_t to client \
1320 1320 */ \
1321 1321 (srs)->srs_state |= SRS_TX_HIWAT; \
1322 1322 cookie = (mac_tx_cookie_t)srs; \
1323 1323 (srs)->srs_tx.st_hiwat_cnt++; \
1324 1324 if ((srs)->srs_count > (srs)->srs_tx.st_max_q_cnt) { \
1325 1325 /* increment freed stats */ \
1326 1326 (srs)->srs_tx.st_stat.mts_sdrops += cnt; \
1327 1327 /* \
1328 1328 * b_prev may be set to the fanout hint \
1329 1329 * hence can't use freemsg directly \
1330 1330 */ \
1331 1331 mac_pkt_drop(NULL, NULL, mp_chain, B_FALSE); \
1332 1332 DTRACE_PROBE1(tx_queued_hiwat, \
1333 1333 mac_soft_ring_set_t *, srs); \
1334 1334 enqueue = 0; \
1335 1335 } \
1336 1336 } \
1337 1337 if (enqueue) \
1338 1338 MAC_TX_SRS_ENQUEUE_CHAIN(srs, mp, tail, cnt, sz); \
1339 1339 }
1340 1340
1341 1341 /* Some utility macros */
1342 1342 #define MAC_SRS_BW_LOCK(srs) \
1343 1343 if (!(srs->srs_type & SRST_TX)) \
1344 1344 mutex_enter(&srs->srs_bw->mac_bw_lock);
1345 1345
1346 1346 #define MAC_SRS_BW_UNLOCK(srs) \
1347 1347 if (!(srs->srs_type & SRST_TX)) \
1348 1348 mutex_exit(&srs->srs_bw->mac_bw_lock);
1349 1349
1350 1350 #define MAC_TX_SRS_DROP_MESSAGE(srs, mp, cookie) { \
1351 1351 mac_pkt_drop(NULL, NULL, mp, B_FALSE); \
1352 1352 /* increment freed stats */ \
1353 1353 mac_srs->srs_tx.st_stat.mts_sdrops++; \
1354 1354 cookie = (mac_tx_cookie_t)srs; \
1355 1355 }
1356 1356
1357 1357 #define MAC_TX_SET_NO_ENQUEUE(srs, mp_chain, ret_mp, cookie) { \
1358 1358 mac_srs->srs_state |= SRS_TX_WAKEUP_CLIENT; \
1359 1359 cookie = (mac_tx_cookie_t)srs; \
1360 1360 *ret_mp = mp_chain; \
1361 1361 }
1362 1362
1363 1363 /*
1364 1364 * MAC_RX_SRS_TOODEEP
1365 1365 *
1366 1366 * Macro called as part of receive-side processing to determine if handling
1367 1367 * can occur in situ (in the interrupt thread) or if it should be left to a
1368 1368 * worker thread. Note that the constant used to make this determination is
1369 1369 * not entirely made-up, and is a result of some emprical validation. That
1370 1370 * said, the constant is left as a static variable to allow it to be
1371 1371 * dynamically tuned in the field if and as needed.
1372 1372 */
1373 1373 static uintptr_t mac_rx_srs_stack_needed = 10240;
1374 1374 static uint_t mac_rx_srs_stack_toodeep;
1375 1375
1376 1376 #ifndef STACK_GROWTH_DOWN
1377 1377 #error Downward stack growth assumed.
1378 1378 #endif
1379 1379
1380 1380 #define MAC_RX_SRS_TOODEEP() (STACK_BIAS + (uintptr_t)getfp() - \
1381 1381 (uintptr_t)curthread->t_stkbase < mac_rx_srs_stack_needed && \
1382 1382 ++mac_rx_srs_stack_toodeep)
1383 1383
1384 1384
1385 1385 /*
1386 1386 * Drop the rx packet and advance to the next one in the chain.
1387 1387 */
1388 1388 static void
1389 1389 mac_rx_drop_pkt(mac_soft_ring_set_t *srs, mblk_t *mp)
1390 1390 {
1391 1391 mac_srs_rx_t *srs_rx = &srs->srs_rx;
1392 1392
1393 1393 ASSERT(mp->b_next == NULL);
1394 1394 mutex_enter(&srs->srs_lock);
1395 1395 MAC_UPDATE_SRS_COUNT_LOCKED(srs, 1);
1396 1396 MAC_UPDATE_SRS_SIZE_LOCKED(srs, msgdsize(mp));
1397 1397 mutex_exit(&srs->srs_lock);
1398 1398
1399 1399 srs_rx->sr_stat.mrs_sdrops++;
1400 1400 freemsg(mp);
1401 1401 }
1402 1402
1403 1403 /* DATAPATH RUNTIME ROUTINES */
1404 1404
1405 1405 /*
1406 1406 * mac_srs_fire
1407 1407 *
1408 1408 * Timer callback routine for waking up the SRS worker thread.
1409 1409 */
1410 1410 static void
1411 1411 mac_srs_fire(void *arg)
1412 1412 {
1413 1413 mac_soft_ring_set_t *mac_srs = (mac_soft_ring_set_t *)arg;
1414 1414
1415 1415 mutex_enter(&mac_srs->srs_lock);
1416 1416 if (mac_srs->srs_tid == NULL) {
1417 1417 mutex_exit(&mac_srs->srs_lock);
1418 1418 return;
1419 1419 }
1420 1420
1421 1421 mac_srs->srs_tid = NULL;
1422 1422 if (!(mac_srs->srs_state & SRS_PROC))
1423 1423 cv_signal(&mac_srs->srs_async);
1424 1424
1425 1425 mutex_exit(&mac_srs->srs_lock);
1426 1426 }
1427 1427
1428 1428 /*
1429 1429 * 'hint' is fanout_hint (type of uint64_t) which is given by the TCP/IP stack,
1430 1430 * and it is used on the TX path.
1431 1431 */
1432 1432 #define HASH_HINT(hint) \
1433 1433 ((hint) ^ ((hint) >> 24) ^ ((hint) >> 16) ^ ((hint) >> 8))
1434 1434
1435 1435
1436 1436 /*
1437 1437 * hash based on the src address, dst address and the port information.
1438 1438 */
1439 1439 #define HASH_ADDR(src, dst, ports) \
1440 1440 (ntohl((src) + (dst)) ^ ((ports) >> 24) ^ ((ports) >> 16) ^ \
1441 1441 ((ports) >> 8) ^ (ports))
1442 1442
1443 1443 #define COMPUTE_INDEX(key, sz) (key % sz)
1444 1444
1445 1445 #define FANOUT_ENQUEUE_MP(head, tail, cnt, bw_ctl, sz, sz0, mp) { \
1446 1446 if ((tail) != NULL) { \
1447 1447 ASSERT((tail)->b_next == NULL); \
1448 1448 (tail)->b_next = (mp); \
1449 1449 } else { \
1450 1450 ASSERT((head) == NULL); \
1451 1451 (head) = (mp); \
1452 1452 } \
1453 1453 (tail) = (mp); \
1454 1454 (cnt)++; \
1455 1455 if ((bw_ctl)) \
1456 1456 (sz) += (sz0); \
1457 1457 }
1458 1458
1459 1459 #define MAC_FANOUT_DEFAULT 0
1460 1460 #define MAC_FANOUT_RND_ROBIN 1
1461 1461 int mac_fanout_type = MAC_FANOUT_DEFAULT;
1462 1462
1463 1463 #define MAX_SR_TYPES 3
1464 1464 /* fanout types for port based hashing */
1465 1465 enum pkt_type {
1466 1466 V4_TCP = 0,
1467 1467 V4_UDP,
1468 1468 OTH,
1469 1469 UNDEF
1470 1470 };
1471 1471
1472 1472 /*
1473 1473 * Pair of local and remote ports in the transport header
1474 1474 */
1475 1475 #define PORTS_SIZE 4
1476 1476
1477 1477 /*
1478 1478 * mac_rx_srs_proto_fanout
1479 1479 *
1480 1480 * This routine delivers packets destined to an SRS into one of the
1481 1481 * protocol soft rings.
1482 1482 *
1483 1483 * Given a chain of packets we need to split it up into multiple sub chains
1484 1484 * destined into TCP, UDP or OTH soft ring. Instead of entering
1485 1485 * the soft ring one packet at a time, we want to enter it in the form of a
1486 1486 * chain otherwise we get this start/stop behaviour where the worker thread
1487 1487 * goes to sleep and then next packets comes in forcing it to wake up etc.
1488 1488 */
1489 1489 static void
1490 1490 mac_rx_srs_proto_fanout(mac_soft_ring_set_t *mac_srs, mblk_t *head)
1491 1491 {
1492 1492 struct ether_header *ehp;
1493 1493 struct ether_vlan_header *evhp;
1494 1494 uint32_t sap;
1495 1495 ipha_t *ipha;
1496 1496 uint8_t *dstaddr;
1497 1497 size_t hdrsize;
1498 1498 mblk_t *mp;
1499 1499 mblk_t *headmp[MAX_SR_TYPES];
1500 1500 mblk_t *tailmp[MAX_SR_TYPES];
1501 1501 int cnt[MAX_SR_TYPES];
1502 1502 size_t sz[MAX_SR_TYPES];
1503 1503 size_t sz1;
1504 1504 boolean_t bw_ctl;
1505 1505 boolean_t hw_classified;
1506 1506 boolean_t dls_bypass;
1507 1507 boolean_t is_ether;
1508 1508 boolean_t is_unicast;
1509 1509 enum pkt_type type;
1510 1510 mac_client_impl_t *mcip = mac_srs->srs_mcip;
1511 1511
1512 1512 is_ether = (mcip->mci_mip->mi_info.mi_nativemedia == DL_ETHER);
1513 1513 bw_ctl = ((mac_srs->srs_type & SRST_BW_CONTROL) != 0);
1514 1514
1515 1515 /*
1516 1516 * If we don't have a Rx ring, S/W classification would have done
1517 1517 * its job and its a packet meant for us. If we were polling on
1518 1518 * the default ring (i.e. there was a ring assigned to this SRS),
1519 1519 * then we need to make sure that the mac address really belongs
1520 1520 * to us.
1521 1521 */
1522 1522 hw_classified = mac_srs->srs_ring != NULL &&
1523 1523 mac_srs->srs_ring->mr_classify_type == MAC_HW_CLASSIFIER;
1524 1524
1525 1525 /*
1526 1526 * Special clients (eg. VLAN, non ether, etc) need DLS
1527 1527 * processing in the Rx path. SRST_DLS_BYPASS will be clear for
1528 1528 * such SRSs. Another way of disabling bypass is to set the
1529 1529 * MCIS_RX_BYPASS_DISABLE flag.
1530 1530 */
1531 1531 dls_bypass = ((mac_srs->srs_type & SRST_DLS_BYPASS) != 0) &&
1532 1532 ((mcip->mci_state_flags & MCIS_RX_BYPASS_DISABLE) == 0);
1533 1533
1534 1534 bzero(headmp, MAX_SR_TYPES * sizeof (mblk_t *));
1535 1535 bzero(tailmp, MAX_SR_TYPES * sizeof (mblk_t *));
1536 1536 bzero(cnt, MAX_SR_TYPES * sizeof (int));
1537 1537 bzero(sz, MAX_SR_TYPES * sizeof (size_t));
1538 1538
1539 1539 /*
1540 1540 * We got a chain from SRS that we need to send to the soft rings.
1541 1541 * Since squeues for TCP & IPv4 sap poll their soft rings (for
1542 1542 * performance reasons), we need to separate out v4_tcp, v4_udp
1543 1543 * and the rest goes in other.
1544 1544 */
1545 1545 while (head != NULL) {
1546 1546 mp = head;
1547 1547 head = head->b_next;
1548 1548 mp->b_next = NULL;
1549 1549
1550 1550 type = OTH;
1551 1551 sz1 = (mp->b_cont == NULL) ? MBLKL(mp) : msgdsize(mp);
1552 1552
1553 1553 if (is_ether) {
1554 1554 /*
1555 1555 * At this point we can be sure the packet at least
1556 1556 * has an ether header.
1557 1557 */
1558 1558 if (sz1 < sizeof (struct ether_header)) {
1559 1559 mac_rx_drop_pkt(mac_srs, mp);
1560 1560 continue;
1561 1561 }
1562 1562 ehp = (struct ether_header *)mp->b_rptr;
1563 1563
1564 1564 /*
1565 1565 * Determine if this is a VLAN or non-VLAN packet.
1566 1566 */
1567 1567 if ((sap = ntohs(ehp->ether_type)) == VLAN_TPID) {
1568 1568 evhp = (struct ether_vlan_header *)mp->b_rptr;
1569 1569 sap = ntohs(evhp->ether_type);
1570 1570 hdrsize = sizeof (struct ether_vlan_header);
1571 1571 /*
1572 1572 * Check if the VID of the packet, if any,
1573 1573 * belongs to this client.
1574 1574 */
1575 1575 if (!mac_client_check_flow_vid(mcip,
1576 1576 VLAN_ID(ntohs(evhp->ether_tci)))) {
1577 1577 mac_rx_drop_pkt(mac_srs, mp);
1578 1578 continue;
1579 1579 }
1580 1580 } else {
1581 1581 hdrsize = sizeof (struct ether_header);
1582 1582 }
1583 1583 is_unicast =
1584 1584 ((((uint8_t *)&ehp->ether_dhost)[0] & 0x01) == 0);
1585 1585 dstaddr = (uint8_t *)&ehp->ether_dhost;
1586 1586 } else {
1587 1587 mac_header_info_t mhi;
1588 1588
1589 1589 if (mac_header_info((mac_handle_t)mcip->mci_mip,
1590 1590 mp, &mhi) != 0) {
1591 1591 mac_rx_drop_pkt(mac_srs, mp);
1592 1592 continue;
1593 1593 }
1594 1594 hdrsize = mhi.mhi_hdrsize;
1595 1595 sap = mhi.mhi_bindsap;
1596 1596 is_unicast = (mhi.mhi_dsttype == MAC_ADDRTYPE_UNICAST);
1597 1597 dstaddr = (uint8_t *)mhi.mhi_daddr;
1598 1598 }
1599 1599
1600 1600 if (!dls_bypass) {
1601 1601 FANOUT_ENQUEUE_MP(headmp[type], tailmp[type],
1602 1602 cnt[type], bw_ctl, sz[type], sz1, mp);
1603 1603 continue;
1604 1604 }
1605 1605
1606 1606 if (sap == ETHERTYPE_IP) {
1607 1607 /*
1608 1608 * If we are H/W classified, but we have promisc
1609 1609 * on, then we need to check for the unicast address.
1610 1610 */
1611 1611 if (hw_classified && mcip->mci_promisc_list != NULL) {
1612 1612 mac_address_t *map;
1613 1613
1614 1614 rw_enter(&mcip->mci_rw_lock, RW_READER);
1615 1615 map = mcip->mci_unicast;
1616 1616 if (bcmp(dstaddr, map->ma_addr,
1617 1617 map->ma_len) == 0)
1618 1618 type = UNDEF;
1619 1619 rw_exit(&mcip->mci_rw_lock);
1620 1620 } else if (is_unicast) {
1621 1621 type = UNDEF;
1622 1622 }
1623 1623 }
1624 1624
1625 1625 /*
1626 1626 * This needs to become a contract with the driver for
1627 1627 * the fast path.
1628 1628 *
1629 1629 * In the normal case the packet will have at least the L2
1630 1630 * header and the IP + Transport header in the same mblk.
1631 1631 * This is usually the case when the NIC driver sends up
1632 1632 * the packet. This is also true when the stack generates
1633 1633 * a packet that is looped back and when the stack uses the
1634 1634 * fastpath mechanism. The normal case is optimized for
1635 1635 * performance and may bypass DLS. All other cases go through
1636 1636 * the 'OTH' type path without DLS bypass.
1637 1637 */
1638 1638
1639 1639 ipha = (ipha_t *)(mp->b_rptr + hdrsize);
1640 1640 if ((type != OTH) && MBLK_RX_FANOUT_SLOWPATH(mp, ipha))
1641 1641 type = OTH;
1642 1642
1643 1643 if (type == OTH) {
1644 1644 FANOUT_ENQUEUE_MP(headmp[type], tailmp[type],
1645 1645 cnt[type], bw_ctl, sz[type], sz1, mp);
1646 1646 continue;
1647 1647 }
1648 1648
1649 1649 ASSERT(type == UNDEF);
1650 1650 /*
1651 1651 * We look for at least 4 bytes past the IP header to get
1652 1652 * the port information. If we get an IP fragment, we don't
1653 1653 * have the port information, and we use just the protocol
1654 1654 * information.
1655 1655 */
1656 1656 switch (ipha->ipha_protocol) {
1657 1657 case IPPROTO_TCP:
1658 1658 type = V4_TCP;
1659 1659 mp->b_rptr += hdrsize;
1660 1660 break;
1661 1661 case IPPROTO_UDP:
1662 1662 type = V4_UDP;
1663 1663 mp->b_rptr += hdrsize;
1664 1664 break;
1665 1665 default:
1666 1666 type = OTH;
1667 1667 break;
1668 1668 }
1669 1669
1670 1670 FANOUT_ENQUEUE_MP(headmp[type], tailmp[type], cnt[type],
1671 1671 bw_ctl, sz[type], sz1, mp);
1672 1672 }
1673 1673
1674 1674 for (type = V4_TCP; type < UNDEF; type++) {
1675 1675 if (headmp[type] != NULL) {
1676 1676 mac_soft_ring_t *softring;
1677 1677
1678 1678 ASSERT(tailmp[type]->b_next == NULL);
1679 1679 switch (type) {
1680 1680 case V4_TCP:
1681 1681 softring = mac_srs->srs_tcp_soft_rings[0];
1682 1682 break;
1683 1683 case V4_UDP:
1684 1684 softring = mac_srs->srs_udp_soft_rings[0];
1685 1685 break;
1686 1686 case OTH:
1687 1687 softring = mac_srs->srs_oth_soft_rings[0];
1688 1688 }
1689 1689 mac_rx_soft_ring_process(mcip, softring,
1690 1690 headmp[type], tailmp[type], cnt[type], sz[type]);
1691 1691 }
1692 1692 }
1693 1693 }
1694 1694
1695 1695 int fanout_unaligned = 0;
1696 1696
1697 1697 /*
1698 1698 * mac_rx_srs_long_fanout
1699 1699 *
1700 1700 * The fanout routine for VLANs, and for anything else that isn't performing
1701 1701 * explicit dls bypass. Returns -1 on an error (drop the packet due to a
1702 1702 * malformed packet), 0 on success, with values written in *indx and *type.
1703 1703 */
1704 1704 static int
1705 1705 mac_rx_srs_long_fanout(mac_soft_ring_set_t *mac_srs, mblk_t *mp,
1706 1706 uint32_t sap, size_t hdrsize, enum pkt_type *type, uint_t *indx)
1707 1707 {
1708 1708 ip6_t *ip6h;
1709 1709 ipha_t *ipha;
1710 1710 uint8_t *whereptr;
1711 1711 uint_t hash;
1712 1712 uint16_t remlen;
1713 1713 uint8_t nexthdr;
1714 1714 uint16_t hdr_len;
1715 1715 uint32_t src_val, dst_val;
1716 1716 boolean_t modifiable = B_TRUE;
1717 1717 boolean_t v6;
1718 1718
1719 1719 ASSERT(MBLKL(mp) >= hdrsize);
1720 1720
1721 1721 if (sap == ETHERTYPE_IPV6) {
1722 1722 v6 = B_TRUE;
1723 1723 hdr_len = IPV6_HDR_LEN;
1724 1724 } else if (sap == ETHERTYPE_IP) {
1725 1725 v6 = B_FALSE;
1726 1726 hdr_len = IP_SIMPLE_HDR_LENGTH;
1727 1727 } else {
1728 1728 *indx = 0;
1729 1729 *type = OTH;
1730 1730 return (0);
1731 1731 }
1732 1732
1733 1733 ip6h = (ip6_t *)(mp->b_rptr + hdrsize);
1734 1734 ipha = (ipha_t *)ip6h;
1735 1735
1736 1736 if ((uint8_t *)ip6h == mp->b_wptr) {
1737 1737 /*
1738 1738 * The first mblk_t only includes the mac header.
1739 1739 * Note that it is safe to change the mp pointer here,
1740 1740 * as the subsequent operation does not assume mp
1741 1741 * points to the start of the mac header.
1742 1742 */
1743 1743 mp = mp->b_cont;
1744 1744
1745 1745 /*
1746 1746 * Make sure the IP header points to an entire one.
1747 1747 */
1748 1748 if (mp == NULL)
1749 1749 return (-1);
1750 1750
1751 1751 if (MBLKL(mp) < hdr_len) {
1752 1752 modifiable = (DB_REF(mp) == 1);
1753 1753
1754 1754 if (modifiable && !pullupmsg(mp, hdr_len))
1755 1755 return (-1);
1756 1756 }
1757 1757
1758 1758 ip6h = (ip6_t *)mp->b_rptr;
1759 1759 ipha = (ipha_t *)ip6h;
1760 1760 }
1761 1761
1762 1762 if (!modifiable || !(OK_32PTR((char *)ip6h)) ||
1763 1763 ((uint8_t *)ip6h + hdr_len > mp->b_wptr)) {
1764 1764 /*
1765 1765 * If either the IP header is not aligned, or it does not hold
1766 1766 * the complete simple structure (a pullupmsg() is not an
1767 1767 * option since it would result in an unaligned IP header),
1768 1768 * fanout to the default ring.
1769 1769 *
1770 1770 * Note that this may cause packet reordering.
1771 1771 */
1772 1772 *indx = 0;
1773 1773 *type = OTH;
1774 1774 fanout_unaligned++;
1775 1775 return (0);
1776 1776 }
1777 1777
1778 1778 /*
1779 1779 * Extract next-header, full header length, and source-hash value
1780 1780 * using v4/v6 specific fields.
1781 1781 */
|
↓ open down ↓ |
1747 lines elided |
↑ open up ↑ |
1782 1782 if (v6) {
1783 1783 remlen = ntohs(ip6h->ip6_plen);
1784 1784 nexthdr = ip6h->ip6_nxt;
1785 1785 src_val = V4_PART_OF_V6(ip6h->ip6_src);
1786 1786 dst_val = V4_PART_OF_V6(ip6h->ip6_dst);
1787 1787 /*
1788 1788 * Do src based fanout if below tunable is set to B_TRUE or
1789 1789 * when mac_ip_hdr_length_v6() fails because of malformed
1790 1790 * packets or because mblks need to be concatenated using
1791 1791 * pullupmsg().
1792 - *
1793 - * Perform a version check to prevent parsing weirdness...
1794 1792 */
1795 - if (IPH_HDR_VERSION(ip6h) != IPV6_VERSION ||
1796 - !mac_ip_hdr_length_v6(ip6h, mp->b_wptr, &hdr_len, &nexthdr,
1793 + if (!mac_ip_hdr_length_v6(ip6h, mp->b_wptr, &hdr_len, &nexthdr,
1797 1794 NULL)) {
1798 1795 goto src_dst_based_fanout;
1799 1796 }
1800 1797 } else {
1801 1798 hdr_len = IPH_HDR_LENGTH(ipha);
1802 1799 remlen = ntohs(ipha->ipha_length) - hdr_len;
1803 1800 nexthdr = ipha->ipha_protocol;
1804 1801 src_val = (uint32_t)ipha->ipha_src;
1805 1802 dst_val = (uint32_t)ipha->ipha_dst;
1806 1803 /*
1807 1804 * Catch IPv4 fragment case here. IPv6 has nexthdr == FRAG
1808 1805 * for its equivalent case.
1809 1806 */
1810 1807 if ((ntohs(ipha->ipha_fragment_offset_and_flags) &
1811 1808 (IPH_MF | IPH_OFFSET)) != 0) {
1812 1809 goto src_dst_based_fanout;
1813 1810 }
1814 1811 }
1815 1812 if (remlen < MIN_EHDR_LEN)
1816 1813 return (-1);
1817 1814 whereptr = (uint8_t *)ip6h + hdr_len;
1818 1815
1819 1816 /* If the transport is one of below, we do port/SPI based fanout */
1820 1817 switch (nexthdr) {
1821 1818 case IPPROTO_TCP:
1822 1819 case IPPROTO_UDP:
1823 1820 case IPPROTO_SCTP:
1824 1821 case IPPROTO_ESP:
1825 1822 /*
1826 1823 * If the ports or SPI in the transport header is not part of
1827 1824 * the mblk, do src_based_fanout, instead of calling
1828 1825 * pullupmsg().
1829 1826 */
1830 1827 if (mp->b_cont == NULL || whereptr + PORTS_SIZE <= mp->b_wptr)
1831 1828 break; /* out of switch... */
1832 1829 /* FALLTHRU */
1833 1830 default:
1834 1831 goto src_dst_based_fanout;
1835 1832 }
1836 1833
1837 1834 switch (nexthdr) {
1838 1835 case IPPROTO_TCP:
1839 1836 hash = HASH_ADDR(src_val, dst_val, *(uint32_t *)whereptr);
1840 1837 *indx = COMPUTE_INDEX(hash, mac_srs->srs_tcp_ring_count);
1841 1838 *type = OTH;
1842 1839 break;
1843 1840 case IPPROTO_UDP:
1844 1841 case IPPROTO_SCTP:
1845 1842 case IPPROTO_ESP:
1846 1843 if (mac_fanout_type == MAC_FANOUT_DEFAULT) {
1847 1844 hash = HASH_ADDR(src_val, dst_val,
1848 1845 *(uint32_t *)whereptr);
1849 1846 *indx = COMPUTE_INDEX(hash,
1850 1847 mac_srs->srs_udp_ring_count);
1851 1848 } else {
1852 1849 *indx = mac_srs->srs_ind % mac_srs->srs_udp_ring_count;
1853 1850 mac_srs->srs_ind++;
1854 1851 }
1855 1852 *type = OTH;
1856 1853 break;
1857 1854 }
1858 1855 return (0);
1859 1856
1860 1857 src_dst_based_fanout:
1861 1858 hash = HASH_ADDR(src_val, dst_val, (uint32_t)0);
1862 1859 *indx = COMPUTE_INDEX(hash, mac_srs->srs_oth_ring_count);
1863 1860 *type = OTH;
1864 1861 return (0);
1865 1862 }
1866 1863
1867 1864 /*
1868 1865 * mac_rx_srs_fanout
1869 1866 *
1870 1867 * This routine delivers packets destined to an SRS into a soft ring member
1871 1868 * of the set.
1872 1869 *
1873 1870 * Given a chain of packets we need to split it up into multiple sub chains
1874 1871 * destined for one of the TCP, UDP or OTH soft rings. Instead of entering
1875 1872 * the soft ring one packet at a time, we want to enter it in the form of a
1876 1873 * chain otherwise we get this start/stop behaviour where the worker thread
1877 1874 * goes to sleep and then next packets comes in forcing it to wake up etc.
1878 1875 *
1879 1876 * Note:
1880 1877 * Since we know what is the maximum fanout possible, we create a 2D array
1881 1878 * of 'softring types * MAX_SR_FANOUT' for the head, tail, cnt and sz
1882 1879 * variables so that we can enter the softrings with chain. We need the
1883 1880 * MAX_SR_FANOUT so we can allocate the arrays on the stack (a kmem_alloc
1884 1881 * for each packet would be expensive). If we ever want to have the
1885 1882 * ability to have unlimited fanout, we should probably declare a head,
1886 1883 * tail, cnt, sz with each soft ring (a data struct which contains a softring
1887 1884 * along with these members) and create an array of this uber struct so we
1888 1885 * don't have to do kmem_alloc.
1889 1886 */
1890 1887 int fanout_oth1 = 0;
1891 1888 int fanout_oth2 = 0;
1892 1889 int fanout_oth3 = 0;
1893 1890 int fanout_oth4 = 0;
1894 1891 int fanout_oth5 = 0;
1895 1892
1896 1893 static void
1897 1894 mac_rx_srs_fanout(mac_soft_ring_set_t *mac_srs, mblk_t *head)
1898 1895 {
1899 1896 struct ether_header *ehp;
1900 1897 struct ether_vlan_header *evhp;
1901 1898 uint32_t sap;
1902 1899 ipha_t *ipha;
1903 1900 uint8_t *dstaddr;
1904 1901 uint_t indx;
1905 1902 size_t ports_offset;
1906 1903 size_t ipha_len;
1907 1904 size_t hdrsize;
1908 1905 uint_t hash;
1909 1906 mblk_t *mp;
1910 1907 mblk_t *headmp[MAX_SR_TYPES][MAX_SR_FANOUT];
1911 1908 mblk_t *tailmp[MAX_SR_TYPES][MAX_SR_FANOUT];
1912 1909 int cnt[MAX_SR_TYPES][MAX_SR_FANOUT];
1913 1910 size_t sz[MAX_SR_TYPES][MAX_SR_FANOUT];
1914 1911 size_t sz1;
1915 1912 boolean_t bw_ctl;
1916 1913 boolean_t hw_classified;
1917 1914 boolean_t dls_bypass;
1918 1915 boolean_t is_ether;
1919 1916 boolean_t is_unicast;
1920 1917 int fanout_cnt;
1921 1918 enum pkt_type type;
1922 1919 mac_client_impl_t *mcip = mac_srs->srs_mcip;
1923 1920
1924 1921 is_ether = (mcip->mci_mip->mi_info.mi_nativemedia == DL_ETHER);
1925 1922 bw_ctl = ((mac_srs->srs_type & SRST_BW_CONTROL) != 0);
1926 1923
1927 1924 /*
1928 1925 * If we don't have a Rx ring, S/W classification would have done
1929 1926 * its job and its a packet meant for us. If we were polling on
1930 1927 * the default ring (i.e. there was a ring assigned to this SRS),
1931 1928 * then we need to make sure that the mac address really belongs
1932 1929 * to us.
1933 1930 */
1934 1931 hw_classified = mac_srs->srs_ring != NULL &&
1935 1932 mac_srs->srs_ring->mr_classify_type == MAC_HW_CLASSIFIER;
1936 1933
1937 1934 /*
1938 1935 * Special clients (eg. VLAN, non ether, etc) need DLS
1939 1936 * processing in the Rx path. SRST_DLS_BYPASS will be clear for
1940 1937 * such SRSs. Another way of disabling bypass is to set the
1941 1938 * MCIS_RX_BYPASS_DISABLE flag.
1942 1939 */
1943 1940 dls_bypass = ((mac_srs->srs_type & SRST_DLS_BYPASS) != 0) &&
1944 1941 ((mcip->mci_state_flags & MCIS_RX_BYPASS_DISABLE) == 0);
1945 1942
1946 1943 /*
1947 1944 * Since the softrings are never destroyed and we always
1948 1945 * create equal number of softrings for TCP, UDP and rest,
1949 1946 * its OK to check one of them for count and use it without
1950 1947 * any lock. In future, if soft rings get destroyed because
1951 1948 * of reduction in fanout, we will need to ensure that happens
1952 1949 * behind the SRS_PROC.
1953 1950 */
1954 1951 fanout_cnt = mac_srs->srs_tcp_ring_count;
1955 1952
1956 1953 bzero(headmp, MAX_SR_TYPES * MAX_SR_FANOUT * sizeof (mblk_t *));
1957 1954 bzero(tailmp, MAX_SR_TYPES * MAX_SR_FANOUT * sizeof (mblk_t *));
1958 1955 bzero(cnt, MAX_SR_TYPES * MAX_SR_FANOUT * sizeof (int));
1959 1956 bzero(sz, MAX_SR_TYPES * MAX_SR_FANOUT * sizeof (size_t));
1960 1957
1961 1958 /*
1962 1959 * We got a chain from SRS that we need to send to the soft rings.
1963 1960 * Since squeues for TCP & IPv4 sap poll their soft rings (for
1964 1961 * performance reasons), we need to separate out v4_tcp, v4_udp
1965 1962 * and the rest goes in other.
1966 1963 */
1967 1964 while (head != NULL) {
1968 1965 mp = head;
1969 1966 head = head->b_next;
1970 1967 mp->b_next = NULL;
1971 1968
1972 1969 type = OTH;
1973 1970 sz1 = (mp->b_cont == NULL) ? MBLKL(mp) : msgdsize(mp);
1974 1971
1975 1972 if (is_ether) {
1976 1973 /*
1977 1974 * At this point we can be sure the packet at least
1978 1975 * has an ether header.
1979 1976 */
1980 1977 if (sz1 < sizeof (struct ether_header)) {
1981 1978 mac_rx_drop_pkt(mac_srs, mp);
1982 1979 continue;
1983 1980 }
1984 1981 ehp = (struct ether_header *)mp->b_rptr;
1985 1982
1986 1983 /*
1987 1984 * Determine if this is a VLAN or non-VLAN packet.
1988 1985 */
1989 1986 if ((sap = ntohs(ehp->ether_type)) == VLAN_TPID) {
1990 1987 evhp = (struct ether_vlan_header *)mp->b_rptr;
1991 1988 sap = ntohs(evhp->ether_type);
1992 1989 hdrsize = sizeof (struct ether_vlan_header);
1993 1990 /*
1994 1991 * Check if the VID of the packet, if any,
1995 1992 * belongs to this client.
1996 1993 */
1997 1994 if (!mac_client_check_flow_vid(mcip,
1998 1995 VLAN_ID(ntohs(evhp->ether_tci)))) {
1999 1996 mac_rx_drop_pkt(mac_srs, mp);
2000 1997 continue;
2001 1998 }
2002 1999 } else {
2003 2000 hdrsize = sizeof (struct ether_header);
2004 2001 }
2005 2002 is_unicast =
2006 2003 ((((uint8_t *)&ehp->ether_dhost)[0] & 0x01) == 0);
2007 2004 dstaddr = (uint8_t *)&ehp->ether_dhost;
2008 2005 } else {
2009 2006 mac_header_info_t mhi;
2010 2007
2011 2008 if (mac_header_info((mac_handle_t)mcip->mci_mip,
2012 2009 mp, &mhi) != 0) {
2013 2010 mac_rx_drop_pkt(mac_srs, mp);
2014 2011 continue;
2015 2012 }
2016 2013 hdrsize = mhi.mhi_hdrsize;
2017 2014 sap = mhi.mhi_bindsap;
2018 2015 is_unicast = (mhi.mhi_dsttype == MAC_ADDRTYPE_UNICAST);
2019 2016 dstaddr = (uint8_t *)mhi.mhi_daddr;
2020 2017 }
2021 2018
2022 2019 if (!dls_bypass) {
2023 2020 if (mac_rx_srs_long_fanout(mac_srs, mp, sap,
2024 2021 hdrsize, &type, &indx) == -1) {
2025 2022 mac_rx_drop_pkt(mac_srs, mp);
2026 2023 continue;
2027 2024 }
2028 2025
2029 2026 FANOUT_ENQUEUE_MP(headmp[type][indx],
2030 2027 tailmp[type][indx], cnt[type][indx], bw_ctl,
2031 2028 sz[type][indx], sz1, mp);
2032 2029 continue;
2033 2030 }
2034 2031
2035 2032
2036 2033 /*
2037 2034 * If we are using the default Rx ring where H/W or S/W
2038 2035 * classification has not happened, we need to verify if
2039 2036 * this unicast packet really belongs to us.
2040 2037 */
2041 2038 if (sap == ETHERTYPE_IP) {
2042 2039 /*
2043 2040 * If we are H/W classified, but we have promisc
2044 2041 * on, then we need to check for the unicast address.
2045 2042 */
2046 2043 if (hw_classified && mcip->mci_promisc_list != NULL) {
2047 2044 mac_address_t *map;
2048 2045
2049 2046 rw_enter(&mcip->mci_rw_lock, RW_READER);
2050 2047 map = mcip->mci_unicast;
2051 2048 if (bcmp(dstaddr, map->ma_addr,
2052 2049 map->ma_len) == 0)
2053 2050 type = UNDEF;
2054 2051 rw_exit(&mcip->mci_rw_lock);
2055 2052 } else if (is_unicast) {
2056 2053 type = UNDEF;
2057 2054 }
2058 2055 }
2059 2056
2060 2057 /*
2061 2058 * This needs to become a contract with the driver for
2062 2059 * the fast path.
2063 2060 */
2064 2061
2065 2062 ipha = (ipha_t *)(mp->b_rptr + hdrsize);
2066 2063 if ((type != OTH) && MBLK_RX_FANOUT_SLOWPATH(mp, ipha)) {
2067 2064 type = OTH;
2068 2065 fanout_oth1++;
2069 2066 }
2070 2067
2071 2068 if (type != OTH) {
2072 2069 uint16_t frag_offset_flags;
2073 2070
2074 2071 switch (ipha->ipha_protocol) {
2075 2072 case IPPROTO_TCP:
2076 2073 case IPPROTO_UDP:
2077 2074 case IPPROTO_SCTP:
2078 2075 case IPPROTO_ESP:
2079 2076 ipha_len = IPH_HDR_LENGTH(ipha);
2080 2077 if ((uchar_t *)ipha + ipha_len + PORTS_SIZE >
2081 2078 mp->b_wptr) {
2082 2079 type = OTH;
2083 2080 break;
2084 2081 }
2085 2082 frag_offset_flags =
2086 2083 ntohs(ipha->ipha_fragment_offset_and_flags);
2087 2084 if ((frag_offset_flags &
2088 2085 (IPH_MF | IPH_OFFSET)) != 0) {
2089 2086 type = OTH;
2090 2087 fanout_oth3++;
2091 2088 break;
2092 2089 }
2093 2090 ports_offset = hdrsize + ipha_len;
2094 2091 break;
2095 2092 default:
2096 2093 type = OTH;
2097 2094 fanout_oth4++;
2098 2095 break;
2099 2096 }
2100 2097 }
2101 2098
2102 2099 if (type == OTH) {
2103 2100 if (mac_rx_srs_long_fanout(mac_srs, mp, sap,
2104 2101 hdrsize, &type, &indx) == -1) {
2105 2102 mac_rx_drop_pkt(mac_srs, mp);
2106 2103 continue;
2107 2104 }
2108 2105
2109 2106 FANOUT_ENQUEUE_MP(headmp[type][indx],
2110 2107 tailmp[type][indx], cnt[type][indx], bw_ctl,
2111 2108 sz[type][indx], sz1, mp);
2112 2109 continue;
2113 2110 }
2114 2111
2115 2112 ASSERT(type == UNDEF);
2116 2113
2117 2114 /*
2118 2115 * XXX-Sunay: We should hold srs_lock since ring_count
2119 2116 * below can change. But if we are always called from
2120 2117 * mac_rx_srs_drain and SRS_PROC is set, then we can
2121 2118 * enforce that ring_count can't be changed i.e.
2122 2119 * to change fanout type or ring count, the calling
2123 2120 * thread needs to be behind SRS_PROC.
2124 2121 */
2125 2122 switch (ipha->ipha_protocol) {
2126 2123 case IPPROTO_TCP:
2127 2124 /*
2128 2125 * Note that for ESP, we fanout on SPI and it is at the
2129 2126 * same offset as the 2x16-bit ports. So it is clumped
2130 2127 * along with TCP, UDP and SCTP.
2131 2128 */
2132 2129 hash = HASH_ADDR(ipha->ipha_src, ipha->ipha_dst,
2133 2130 *(uint32_t *)(mp->b_rptr + ports_offset));
2134 2131 indx = COMPUTE_INDEX(hash, mac_srs->srs_tcp_ring_count);
2135 2132 type = V4_TCP;
2136 2133 mp->b_rptr += hdrsize;
2137 2134 break;
2138 2135 case IPPROTO_UDP:
2139 2136 case IPPROTO_SCTP:
2140 2137 case IPPROTO_ESP:
2141 2138 if (mac_fanout_type == MAC_FANOUT_DEFAULT) {
2142 2139 hash = HASH_ADDR(ipha->ipha_src, ipha->ipha_dst,
2143 2140 *(uint32_t *)(mp->b_rptr + ports_offset));
2144 2141 indx = COMPUTE_INDEX(hash,
2145 2142 mac_srs->srs_udp_ring_count);
2146 2143 } else {
2147 2144 indx = mac_srs->srs_ind %
2148 2145 mac_srs->srs_udp_ring_count;
2149 2146 mac_srs->srs_ind++;
2150 2147 }
2151 2148 type = V4_UDP;
2152 2149 mp->b_rptr += hdrsize;
2153 2150 break;
2154 2151 default:
2155 2152 indx = 0;
2156 2153 type = OTH;
2157 2154 }
2158 2155
2159 2156 FANOUT_ENQUEUE_MP(headmp[type][indx], tailmp[type][indx],
2160 2157 cnt[type][indx], bw_ctl, sz[type][indx], sz1, mp);
2161 2158 }
2162 2159
2163 2160 for (type = V4_TCP; type < UNDEF; type++) {
2164 2161 int i;
2165 2162
2166 2163 for (i = 0; i < fanout_cnt; i++) {
2167 2164 if (headmp[type][i] != NULL) {
2168 2165 mac_soft_ring_t *softring;
2169 2166
2170 2167 ASSERT(tailmp[type][i]->b_next == NULL);
2171 2168 switch (type) {
2172 2169 case V4_TCP:
2173 2170 softring =
2174 2171 mac_srs->srs_tcp_soft_rings[i];
2175 2172 break;
2176 2173 case V4_UDP:
2177 2174 softring =
2178 2175 mac_srs->srs_udp_soft_rings[i];
2179 2176 break;
2180 2177 case OTH:
2181 2178 softring =
2182 2179 mac_srs->srs_oth_soft_rings[i];
2183 2180 break;
2184 2181 }
2185 2182 mac_rx_soft_ring_process(mcip,
2186 2183 softring, headmp[type][i], tailmp[type][i],
2187 2184 cnt[type][i], sz[type][i]);
2188 2185 }
2189 2186 }
2190 2187 }
2191 2188 }
2192 2189
2193 2190 #define SRS_BYTES_TO_PICKUP 150000
2194 2191 ssize_t max_bytes_to_pickup = SRS_BYTES_TO_PICKUP;
2195 2192
2196 2193 /*
2197 2194 * mac_rx_srs_poll_ring
2198 2195 *
2199 2196 * This SRS Poll thread uses this routine to poll the underlying hardware
2200 2197 * Rx ring to get a chain of packets. It can inline process that chain
2201 2198 * if mac_latency_optimize is set (default) or signal the SRS worker thread
2202 2199 * to do the remaining processing.
2203 2200 *
2204 2201 * Since packets come in the system via interrupt or poll path, we also
2205 2202 * update the stats and deal with promiscous clients here.
2206 2203 */
2207 2204 void
2208 2205 mac_rx_srs_poll_ring(mac_soft_ring_set_t *mac_srs)
2209 2206 {
2210 2207 kmutex_t *lock = &mac_srs->srs_lock;
2211 2208 kcondvar_t *async = &mac_srs->srs_cv;
2212 2209 mac_srs_rx_t *srs_rx = &mac_srs->srs_rx;
2213 2210 mblk_t *head, *tail, *mp;
2214 2211 callb_cpr_t cprinfo;
2215 2212 ssize_t bytes_to_pickup;
2216 2213 size_t sz;
2217 2214 int count;
2218 2215 mac_client_impl_t *smcip;
2219 2216
2220 2217 CALLB_CPR_INIT(&cprinfo, lock, callb_generic_cpr, "mac_srs_poll");
2221 2218 mutex_enter(lock);
2222 2219
2223 2220 start:
2224 2221 for (;;) {
2225 2222 if (mac_srs->srs_state & SRS_PAUSE)
2226 2223 goto done;
2227 2224
2228 2225 CALLB_CPR_SAFE_BEGIN(&cprinfo);
2229 2226 cv_wait(async, lock);
2230 2227 CALLB_CPR_SAFE_END(&cprinfo, lock);
2231 2228
2232 2229 if (mac_srs->srs_state & SRS_PAUSE)
2233 2230 goto done;
2234 2231
2235 2232 check_again:
2236 2233 if (mac_srs->srs_type & SRST_BW_CONTROL) {
2237 2234 /*
2238 2235 * We pick as many bytes as we are allowed to queue.
2239 2236 * Its possible that we will exceed the total
2240 2237 * packets queued in case this SRS is part of the
2241 2238 * Rx ring group since > 1 poll thread can be pulling
2242 2239 * upto the max allowed packets at the same time
2243 2240 * but that should be OK.
2244 2241 */
2245 2242 mutex_enter(&mac_srs->srs_bw->mac_bw_lock);
2246 2243 bytes_to_pickup =
2247 2244 mac_srs->srs_bw->mac_bw_drop_threshold -
2248 2245 mac_srs->srs_bw->mac_bw_sz;
2249 2246 /*
2250 2247 * We shouldn't have been signalled if we
2251 2248 * have 0 or less bytes to pick but since
2252 2249 * some of the bytes accounting is driver
2253 2250 * dependant, we do the safety check.
2254 2251 */
2255 2252 if (bytes_to_pickup < 0)
2256 2253 bytes_to_pickup = 0;
2257 2254 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
2258 2255 } else {
2259 2256 /*
2260 2257 * ToDO: Need to change the polling API
2261 2258 * to add a packet count and a flag which
2262 2259 * tells the driver whether we want packets
2263 2260 * based on a count, or bytes, or all the
2264 2261 * packets queued in the driver/HW. This
2265 2262 * way, we never have to check the limits
2266 2263 * on poll path. We truly let only as many
2267 2264 * packets enter the system as we are willing
2268 2265 * to process or queue.
2269 2266 *
2270 2267 * Something along the lines of
2271 2268 * pkts_to_pickup = mac_soft_ring_max_q_cnt -
2272 2269 * mac_srs->srs_poll_pkt_cnt
2273 2270 */
2274 2271
2275 2272 /*
2276 2273 * Since we are not doing B/W control, pick
2277 2274 * as many packets as allowed.
2278 2275 */
2279 2276 bytes_to_pickup = max_bytes_to_pickup;
2280 2277 }
2281 2278
2282 2279 /* Poll the underlying Hardware */
2283 2280 mutex_exit(lock);
2284 2281 head = MAC_HWRING_POLL(mac_srs->srs_ring, (int)bytes_to_pickup);
2285 2282 mutex_enter(lock);
2286 2283
2287 2284 ASSERT((mac_srs->srs_state & SRS_POLL_THR_OWNER) ==
2288 2285 SRS_POLL_THR_OWNER);
2289 2286
2290 2287 mp = tail = head;
2291 2288 count = 0;
2292 2289 sz = 0;
2293 2290 while (mp != NULL) {
2294 2291 tail = mp;
2295 2292 sz += msgdsize(mp);
2296 2293 mp = mp->b_next;
2297 2294 count++;
2298 2295 }
2299 2296
2300 2297 if (head != NULL) {
2301 2298 tail->b_next = NULL;
2302 2299 smcip = mac_srs->srs_mcip;
2303 2300
2304 2301 SRS_RX_STAT_UPDATE(mac_srs, pollbytes, sz);
2305 2302 SRS_RX_STAT_UPDATE(mac_srs, pollcnt, count);
2306 2303
2307 2304 /*
2308 2305 * If there are any promiscuous mode callbacks
2309 2306 * defined for this MAC client, pass them a copy
2310 2307 * if appropriate and also update the counters.
2311 2308 */
2312 2309 if (smcip != NULL) {
2313 2310 if (smcip->mci_mip->mi_promisc_list != NULL) {
2314 2311 mutex_exit(lock);
2315 2312 mac_promisc_dispatch(smcip->mci_mip,
2316 2313 head, NULL);
2317 2314 mutex_enter(lock);
2318 2315 }
2319 2316 }
2320 2317 if (mac_srs->srs_type & SRST_BW_CONTROL) {
2321 2318 mutex_enter(&mac_srs->srs_bw->mac_bw_lock);
2322 2319 mac_srs->srs_bw->mac_bw_polled += sz;
2323 2320 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
2324 2321 }
2325 2322 MAC_RX_SRS_ENQUEUE_CHAIN(mac_srs, head, tail,
2326 2323 count, sz);
2327 2324 if (count <= 10)
2328 2325 srs_rx->sr_stat.mrs_chaincntundr10++;
2329 2326 else if (count > 10 && count <= 50)
2330 2327 srs_rx->sr_stat.mrs_chaincnt10to50++;
2331 2328 else
2332 2329 srs_rx->sr_stat.mrs_chaincntover50++;
2333 2330 }
2334 2331
2335 2332 /*
2336 2333 * We are guaranteed that SRS_PROC will be set if we
2337 2334 * are here. Also, poll thread gets to run only if
2338 2335 * the drain was being done by a worker thread although
2339 2336 * its possible that worker thread is still running
2340 2337 * and poll thread was sent down to keep the pipeline
2341 2338 * going instead of doing a complete drain and then
2342 2339 * trying to poll the NIC.
2343 2340 *
2344 2341 * So we need to check SRS_WORKER flag to make sure
2345 2342 * that the worker thread is not processing the queue
2346 2343 * in parallel to us. The flags and conditions are
2347 2344 * protected by the srs_lock to prevent any race. We
2348 2345 * ensure that we don't drop the srs_lock from now
2349 2346 * till the end and similarly we don't drop the srs_lock
2350 2347 * in mac_rx_srs_drain() till similar condition check
2351 2348 * are complete. The mac_rx_srs_drain() needs to ensure
2352 2349 * that SRS_WORKER flag remains set as long as its
2353 2350 * processing the queue.
2354 2351 */
2355 2352 if (!(mac_srs->srs_state & SRS_WORKER) &&
2356 2353 (mac_srs->srs_first != NULL)) {
2357 2354 /*
2358 2355 * We have packets to process and worker thread
2359 2356 * is not running. Check to see if poll thread is
2360 2357 * allowed to process.
2361 2358 */
2362 2359 if (mac_srs->srs_state & SRS_LATENCY_OPT) {
2363 2360 mac_srs->srs_drain_func(mac_srs, SRS_POLL_PROC);
2364 2361 if (!(mac_srs->srs_state & SRS_PAUSE) &&
2365 2362 srs_rx->sr_poll_pkt_cnt <=
2366 2363 srs_rx->sr_lowat) {
2367 2364 srs_rx->sr_poll_again++;
2368 2365 goto check_again;
2369 2366 }
2370 2367 /*
2371 2368 * We are already above low water mark
2372 2369 * so stay in the polling mode but no
2373 2370 * need to poll. Once we dip below
2374 2371 * the polling threshold, the processing
2375 2372 * thread (soft ring) will signal us
2376 2373 * to poll again (MAC_UPDATE_SRS_COUNT)
2377 2374 */
2378 2375 srs_rx->sr_poll_drain_no_poll++;
2379 2376 mac_srs->srs_state &= ~(SRS_PROC|SRS_GET_PKTS);
2380 2377 /*
2381 2378 * In B/W control case, its possible
2382 2379 * that the backlog built up due to
2383 2380 * B/W limit being reached and packets
2384 2381 * are queued only in SRS. In this case,
2385 2382 * we should schedule worker thread
2386 2383 * since no one else will wake us up.
2387 2384 */
2388 2385 if ((mac_srs->srs_type & SRST_BW_CONTROL) &&
2389 2386 (mac_srs->srs_tid == NULL)) {
2390 2387 mac_srs->srs_tid =
2391 2388 timeout(mac_srs_fire, mac_srs, 1);
2392 2389 srs_rx->sr_poll_worker_wakeup++;
2393 2390 }
2394 2391 } else {
2395 2392 /*
2396 2393 * Wakeup the worker thread for more processing.
2397 2394 * We optimize for throughput in this case.
2398 2395 */
2399 2396 mac_srs->srs_state &= ~(SRS_PROC|SRS_GET_PKTS);
2400 2397 MAC_SRS_WORKER_WAKEUP(mac_srs);
2401 2398 srs_rx->sr_poll_sig_worker++;
2402 2399 }
2403 2400 } else if ((mac_srs->srs_first == NULL) &&
2404 2401 !(mac_srs->srs_state & SRS_WORKER)) {
2405 2402 /*
2406 2403 * There is nothing queued in SRS and
2407 2404 * no worker thread running. Plus we
2408 2405 * didn't get anything from the H/W
2409 2406 * as well (head == NULL);
2410 2407 */
2411 2408 ASSERT(head == NULL);
2412 2409 mac_srs->srs_state &=
2413 2410 ~(SRS_PROC|SRS_GET_PKTS);
2414 2411
2415 2412 /*
2416 2413 * If we have a packets in soft ring, don't allow
2417 2414 * more packets to come into this SRS by keeping the
2418 2415 * interrupts off but not polling the H/W. The
2419 2416 * poll thread will get signaled as soon as
2420 2417 * srs_poll_pkt_cnt dips below poll threshold.
2421 2418 */
2422 2419 if (srs_rx->sr_poll_pkt_cnt == 0) {
2423 2420 srs_rx->sr_poll_intr_enable++;
2424 2421 MAC_SRS_POLLING_OFF(mac_srs);
2425 2422 } else {
2426 2423 /*
2427 2424 * We know nothing is queued in SRS
2428 2425 * since we are here after checking
2429 2426 * srs_first is NULL. The backlog
2430 2427 * is entirely due to packets queued
2431 2428 * in Soft ring which will wake us up
2432 2429 * and get the interface out of polling
2433 2430 * mode once the backlog dips below
2434 2431 * sr_poll_thres.
2435 2432 */
2436 2433 srs_rx->sr_poll_no_poll++;
2437 2434 }
2438 2435 } else {
2439 2436 /*
2440 2437 * Worker thread is already running.
2441 2438 * Nothing much to do. If the polling
2442 2439 * was enabled, worker thread will deal
2443 2440 * with that.
2444 2441 */
2445 2442 mac_srs->srs_state &= ~SRS_GET_PKTS;
2446 2443 srs_rx->sr_poll_goto_sleep++;
2447 2444 }
2448 2445 }
2449 2446 done:
2450 2447 mac_srs->srs_state |= SRS_POLL_THR_QUIESCED;
2451 2448 cv_signal(&mac_srs->srs_async);
2452 2449 /*
2453 2450 * If this is a temporary quiesce then wait for the restart signal
2454 2451 * from the srs worker. Then clear the flags and signal the srs worker
2455 2452 * to ensure a positive handshake and go back to start.
2456 2453 */
2457 2454 while (!(mac_srs->srs_state & (SRS_CONDEMNED | SRS_POLL_THR_RESTART)))
2458 2455 cv_wait(async, lock);
2459 2456 if (mac_srs->srs_state & SRS_POLL_THR_RESTART) {
2460 2457 ASSERT(!(mac_srs->srs_state & SRS_CONDEMNED));
2461 2458 mac_srs->srs_state &=
2462 2459 ~(SRS_POLL_THR_QUIESCED | SRS_POLL_THR_RESTART);
2463 2460 cv_signal(&mac_srs->srs_async);
2464 2461 goto start;
2465 2462 } else {
2466 2463 mac_srs->srs_state |= SRS_POLL_THR_EXITED;
2467 2464 cv_signal(&mac_srs->srs_async);
2468 2465 CALLB_CPR_EXIT(&cprinfo);
2469 2466 thread_exit();
2470 2467 }
2471 2468 }
2472 2469
2473 2470 /*
2474 2471 * mac_srs_pick_chain
2475 2472 *
2476 2473 * In Bandwidth control case, checks how many packets can be processed
2477 2474 * and return them in a sub chain.
2478 2475 */
2479 2476 static mblk_t *
2480 2477 mac_srs_pick_chain(mac_soft_ring_set_t *mac_srs, mblk_t **chain_tail,
2481 2478 size_t *chain_sz, int *chain_cnt)
2482 2479 {
2483 2480 mblk_t *head = NULL;
2484 2481 mblk_t *tail = NULL;
2485 2482 size_t sz;
2486 2483 size_t tsz = 0;
2487 2484 int cnt = 0;
2488 2485 mblk_t *mp;
2489 2486
2490 2487 ASSERT(MUTEX_HELD(&mac_srs->srs_lock));
2491 2488 mutex_enter(&mac_srs->srs_bw->mac_bw_lock);
2492 2489 if (((mac_srs->srs_bw->mac_bw_used + mac_srs->srs_size) <=
2493 2490 mac_srs->srs_bw->mac_bw_limit) ||
2494 2491 (mac_srs->srs_bw->mac_bw_limit == 0)) {
2495 2492 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
2496 2493 head = mac_srs->srs_first;
2497 2494 mac_srs->srs_first = NULL;
2498 2495 *chain_tail = mac_srs->srs_last;
2499 2496 mac_srs->srs_last = NULL;
2500 2497 *chain_sz = mac_srs->srs_size;
2501 2498 *chain_cnt = mac_srs->srs_count;
2502 2499 mac_srs->srs_count = 0;
2503 2500 mac_srs->srs_size = 0;
2504 2501 return (head);
2505 2502 }
2506 2503
2507 2504 /*
2508 2505 * Can't clear the entire backlog.
2509 2506 * Need to find how many packets to pick
2510 2507 */
2511 2508 ASSERT(MUTEX_HELD(&mac_srs->srs_bw->mac_bw_lock));
2512 2509 while ((mp = mac_srs->srs_first) != NULL) {
2513 2510 sz = msgdsize(mp);
2514 2511 if ((tsz + sz + mac_srs->srs_bw->mac_bw_used) >
2515 2512 mac_srs->srs_bw->mac_bw_limit) {
2516 2513 if (!(mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED))
2517 2514 mac_srs->srs_bw->mac_bw_state |=
2518 2515 SRS_BW_ENFORCED;
2519 2516 break;
2520 2517 }
2521 2518
2522 2519 /*
2523 2520 * The _size & cnt is decremented from the softrings
2524 2521 * when they send up the packet for polling to work
2525 2522 * properly.
2526 2523 */
2527 2524 tsz += sz;
2528 2525 cnt++;
2529 2526 mac_srs->srs_count--;
2530 2527 mac_srs->srs_size -= sz;
2531 2528 if (tail != NULL)
2532 2529 tail->b_next = mp;
2533 2530 else
2534 2531 head = mp;
2535 2532 tail = mp;
2536 2533 mac_srs->srs_first = mac_srs->srs_first->b_next;
2537 2534 }
2538 2535 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
2539 2536 if (mac_srs->srs_first == NULL)
2540 2537 mac_srs->srs_last = NULL;
2541 2538
2542 2539 if (tail != NULL)
2543 2540 tail->b_next = NULL;
2544 2541 *chain_tail = tail;
2545 2542 *chain_cnt = cnt;
2546 2543 *chain_sz = tsz;
2547 2544
2548 2545 return (head);
2549 2546 }
2550 2547
2551 2548 /*
2552 2549 * mac_rx_srs_drain
2553 2550 *
2554 2551 * The SRS drain routine. Gets to run to clear the queue. Any thread
2555 2552 * (worker, interrupt, poll) can call this based on processing model.
2556 2553 * The first thing we do is disable interrupts if possible and then
2557 2554 * drain the queue. we also try to poll the underlying hardware if
2558 2555 * there is a dedicated hardware Rx ring assigned to this SRS.
2559 2556 *
2560 2557 * There is a equivalent drain routine in bandwidth control mode
2561 2558 * mac_rx_srs_drain_bw. There is some code duplication between the two
2562 2559 * routines but they are highly performance sensitive and are easier
2563 2560 * to read/debug if they stay separate. Any code changes here might
2564 2561 * also apply to mac_rx_srs_drain_bw as well.
2565 2562 */
2566 2563 void
2567 2564 mac_rx_srs_drain(mac_soft_ring_set_t *mac_srs, uint_t proc_type)
2568 2565 {
2569 2566 mblk_t *head;
2570 2567 mblk_t *tail;
2571 2568 timeout_id_t tid;
2572 2569 int cnt = 0;
2573 2570 mac_client_impl_t *mcip = mac_srs->srs_mcip;
2574 2571 mac_srs_rx_t *srs_rx = &mac_srs->srs_rx;
2575 2572
2576 2573 ASSERT(MUTEX_HELD(&mac_srs->srs_lock));
2577 2574 ASSERT(!(mac_srs->srs_type & SRST_BW_CONTROL));
2578 2575
2579 2576 /* If we are blanked i.e. can't do upcalls, then we are done */
2580 2577 if (mac_srs->srs_state & (SRS_BLANK | SRS_PAUSE)) {
2581 2578 ASSERT((mac_srs->srs_type & SRST_NO_SOFT_RINGS) ||
2582 2579 (mac_srs->srs_state & SRS_PAUSE));
2583 2580 goto out;
2584 2581 }
2585 2582
2586 2583 if (mac_srs->srs_first == NULL)
2587 2584 goto out;
2588 2585
2589 2586 if (!(mac_srs->srs_state & SRS_LATENCY_OPT) &&
2590 2587 (srs_rx->sr_poll_pkt_cnt <= srs_rx->sr_lowat)) {
2591 2588 /*
2592 2589 * In the normal case, the SRS worker thread does no
2593 2590 * work and we wait for a backlog to build up before
2594 2591 * we switch into polling mode. In case we are
2595 2592 * optimizing for throughput, we use the worker thread
2596 2593 * as well. The goal is to let worker thread process
2597 2594 * the queue and poll thread to feed packets into
2598 2595 * the queue. As such, we should signal the poll
2599 2596 * thread to try and get more packets.
2600 2597 *
2601 2598 * We could have pulled this check in the POLL_RING
2602 2599 * macro itself but keeping it explicit here makes
2603 2600 * the architecture more human understandable.
2604 2601 */
2605 2602 MAC_SRS_POLL_RING(mac_srs);
2606 2603 }
2607 2604
2608 2605 again:
2609 2606 head = mac_srs->srs_first;
2610 2607 mac_srs->srs_first = NULL;
2611 2608 tail = mac_srs->srs_last;
2612 2609 mac_srs->srs_last = NULL;
2613 2610 cnt = mac_srs->srs_count;
2614 2611 mac_srs->srs_count = 0;
2615 2612
2616 2613 ASSERT(head != NULL);
2617 2614 ASSERT(tail != NULL);
2618 2615
2619 2616 if ((tid = mac_srs->srs_tid) != NULL)
2620 2617 mac_srs->srs_tid = NULL;
2621 2618
2622 2619 mac_srs->srs_state |= (SRS_PROC|proc_type);
2623 2620
2624 2621
2625 2622 /*
2626 2623 * mcip is NULL for broadcast and multicast flows. The promisc
2627 2624 * callbacks for broadcast and multicast packets are delivered from
2628 2625 * mac_rx() and we don't need to worry about that case in this path
2629 2626 */
2630 2627 if (mcip != NULL) {
2631 2628 if (mcip->mci_promisc_list != NULL) {
2632 2629 mutex_exit(&mac_srs->srs_lock);
2633 2630 mac_promisc_client_dispatch(mcip, head);
2634 2631 mutex_enter(&mac_srs->srs_lock);
2635 2632 }
2636 2633 if (MAC_PROTECT_ENABLED(mcip, MPT_IPNOSPOOF)) {
2637 2634 mutex_exit(&mac_srs->srs_lock);
2638 2635 mac_protect_intercept_dynamic(mcip, head);
2639 2636 mutex_enter(&mac_srs->srs_lock);
2640 2637 }
2641 2638 }
2642 2639
2643 2640 /*
2644 2641 * Check if SRS itself is doing the processing
2645 2642 * This direct path does not apply when subflows are present. In this
2646 2643 * case, packets need to be dispatched to a soft ring according to the
2647 2644 * flow's bandwidth and other resources contraints.
2648 2645 */
2649 2646 if (mac_srs->srs_type & SRST_NO_SOFT_RINGS) {
2650 2647 mac_direct_rx_t proc;
2651 2648 void *arg1;
2652 2649 mac_resource_handle_t arg2;
2653 2650
2654 2651 /*
2655 2652 * This is the case when a Rx is directly
2656 2653 * assigned and we have a fully classified
2657 2654 * protocol chain. We can deal with it in
2658 2655 * one shot.
2659 2656 */
2660 2657 proc = srs_rx->sr_func;
2661 2658 arg1 = srs_rx->sr_arg1;
2662 2659 arg2 = srs_rx->sr_arg2;
2663 2660
2664 2661 mac_srs->srs_state |= SRS_CLIENT_PROC;
2665 2662 mutex_exit(&mac_srs->srs_lock);
2666 2663 if (tid != NULL) {
2667 2664 (void) untimeout(tid);
2668 2665 tid = NULL;
2669 2666 }
2670 2667
2671 2668 proc(arg1, arg2, head, NULL);
2672 2669 /*
2673 2670 * Decrement the size and count here itelf
2674 2671 * since the packet has been processed.
2675 2672 */
2676 2673 mutex_enter(&mac_srs->srs_lock);
2677 2674 MAC_UPDATE_SRS_COUNT_LOCKED(mac_srs, cnt);
2678 2675 if (mac_srs->srs_state & SRS_CLIENT_WAIT)
2679 2676 cv_signal(&mac_srs->srs_client_cv);
2680 2677 mac_srs->srs_state &= ~SRS_CLIENT_PROC;
2681 2678 } else {
2682 2679 /* Some kind of softrings based fanout is required */
2683 2680 mutex_exit(&mac_srs->srs_lock);
2684 2681 if (tid != NULL) {
2685 2682 (void) untimeout(tid);
2686 2683 tid = NULL;
2687 2684 }
2688 2685
2689 2686 /*
2690 2687 * Since the fanout routines can deal with chains,
2691 2688 * shoot the entire chain up.
2692 2689 */
2693 2690 if (mac_srs->srs_type & SRST_FANOUT_SRC_IP)
2694 2691 mac_rx_srs_fanout(mac_srs, head);
2695 2692 else
2696 2693 mac_rx_srs_proto_fanout(mac_srs, head);
2697 2694 mutex_enter(&mac_srs->srs_lock);
2698 2695 }
2699 2696
2700 2697 if (!(mac_srs->srs_state & (SRS_BLANK|SRS_PAUSE)) &&
2701 2698 (mac_srs->srs_first != NULL)) {
2702 2699 /*
2703 2700 * More packets arrived while we were clearing the
2704 2701 * SRS. This can be possible because of one of
2705 2702 * three conditions below:
2706 2703 * 1) The driver is using multiple worker threads
2707 2704 * to send the packets to us.
2708 2705 * 2) The driver has a race in switching
2709 2706 * between interrupt and polling mode or
2710 2707 * 3) Packets are arriving in this SRS via the
2711 2708 * S/W classification as well.
2712 2709 *
2713 2710 * We should switch to polling mode and see if we
2714 2711 * need to send the poll thread down. Also, signal
2715 2712 * the worker thread to process whats just arrived.
2716 2713 */
2717 2714 MAC_SRS_POLLING_ON(mac_srs);
2718 2715 if (srs_rx->sr_poll_pkt_cnt <= srs_rx->sr_lowat) {
2719 2716 srs_rx->sr_drain_poll_sig++;
2720 2717 MAC_SRS_POLL_RING(mac_srs);
2721 2718 }
2722 2719
2723 2720 /*
2724 2721 * If we didn't signal the poll thread, we need
2725 2722 * to deal with the pending packets ourselves.
2726 2723 */
2727 2724 if (proc_type == SRS_WORKER) {
2728 2725 srs_rx->sr_drain_again++;
2729 2726 goto again;
2730 2727 } else {
2731 2728 srs_rx->sr_drain_worker_sig++;
2732 2729 cv_signal(&mac_srs->srs_async);
2733 2730 }
2734 2731 }
2735 2732
2736 2733 out:
2737 2734 if (mac_srs->srs_state & SRS_GET_PKTS) {
2738 2735 /*
2739 2736 * Poll thread is already running. Leave the
2740 2737 * SRS_RPOC set and hand over the control to
2741 2738 * poll thread.
2742 2739 */
2743 2740 mac_srs->srs_state &= ~proc_type;
2744 2741 srs_rx->sr_drain_poll_running++;
2745 2742 return;
2746 2743 }
2747 2744
2748 2745 /*
2749 2746 * Even if there are no packets queued in SRS, we
2750 2747 * need to make sure that the shared counter is
2751 2748 * clear and any associated softrings have cleared
2752 2749 * all the backlog. Otherwise, leave the interface
2753 2750 * in polling mode and the poll thread will get
2754 2751 * signalled once the count goes down to zero.
2755 2752 *
2756 2753 * If someone is already draining the queue (SRS_PROC is
2757 2754 * set) when the srs_poll_pkt_cnt goes down to zero,
2758 2755 * then it means that drain is already running and we
2759 2756 * will turn off polling at that time if there is
2760 2757 * no backlog.
2761 2758 *
2762 2759 * As long as there are packets queued either
2763 2760 * in soft ring set or its soft rings, we will leave
2764 2761 * the interface in polling mode (even if the drain
2765 2762 * was done being the interrupt thread). We signal
2766 2763 * the poll thread as well if we have dipped below
2767 2764 * low water mark.
2768 2765 *
2769 2766 * NOTE: We can't use the MAC_SRS_POLLING_ON macro
2770 2767 * since that turn polling on only for worker thread.
2771 2768 * Its not worth turning polling on for interrupt
2772 2769 * thread (since NIC will not issue another interrupt)
2773 2770 * unless a backlog builds up.
2774 2771 */
2775 2772 if ((srs_rx->sr_poll_pkt_cnt > 0) &&
2776 2773 (mac_srs->srs_state & SRS_POLLING_CAPAB)) {
2777 2774 mac_srs->srs_state &= ~(SRS_PROC|proc_type);
2778 2775 srs_rx->sr_drain_keep_polling++;
2779 2776 MAC_SRS_POLLING_ON(mac_srs);
2780 2777 if (srs_rx->sr_poll_pkt_cnt <= srs_rx->sr_lowat)
2781 2778 MAC_SRS_POLL_RING(mac_srs);
2782 2779 return;
2783 2780 }
2784 2781
2785 2782 /* Nothing else to do. Get out of poll mode */
2786 2783 MAC_SRS_POLLING_OFF(mac_srs);
2787 2784 mac_srs->srs_state &= ~(SRS_PROC|proc_type);
2788 2785 srs_rx->sr_drain_finish_intr++;
2789 2786 }
2790 2787
2791 2788 /*
2792 2789 * mac_rx_srs_drain_bw
2793 2790 *
2794 2791 * The SRS BW drain routine. Gets to run to clear the queue. Any thread
2795 2792 * (worker, interrupt, poll) can call this based on processing model.
2796 2793 * The first thing we do is disable interrupts if possible and then
2797 2794 * drain the queue. we also try to poll the underlying hardware if
2798 2795 * there is a dedicated hardware Rx ring assigned to this SRS.
2799 2796 *
2800 2797 * There is a equivalent drain routine in non bandwidth control mode
2801 2798 * mac_rx_srs_drain. There is some code duplication between the two
2802 2799 * routines but they are highly performance sensitive and are easier
2803 2800 * to read/debug if they stay separate. Any code changes here might
2804 2801 * also apply to mac_rx_srs_drain as well.
2805 2802 */
2806 2803 void
2807 2804 mac_rx_srs_drain_bw(mac_soft_ring_set_t *mac_srs, uint_t proc_type)
2808 2805 {
2809 2806 mblk_t *head;
2810 2807 mblk_t *tail;
2811 2808 timeout_id_t tid;
2812 2809 size_t sz = 0;
2813 2810 int cnt = 0;
2814 2811 mac_client_impl_t *mcip = mac_srs->srs_mcip;
2815 2812 mac_srs_rx_t *srs_rx = &mac_srs->srs_rx;
2816 2813 clock_t now;
2817 2814
2818 2815 ASSERT(MUTEX_HELD(&mac_srs->srs_lock));
2819 2816 ASSERT(mac_srs->srs_type & SRST_BW_CONTROL);
2820 2817 again:
2821 2818 /* Check if we are doing B/W control */
2822 2819 mutex_enter(&mac_srs->srs_bw->mac_bw_lock);
2823 2820 now = ddi_get_lbolt();
2824 2821 if (mac_srs->srs_bw->mac_bw_curr_time != now) {
2825 2822 mac_srs->srs_bw->mac_bw_curr_time = now;
2826 2823 mac_srs->srs_bw->mac_bw_used = 0;
2827 2824 if (mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED)
2828 2825 mac_srs->srs_bw->mac_bw_state &= ~SRS_BW_ENFORCED;
2829 2826 } else if (mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED) {
2830 2827 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
2831 2828 goto done;
2832 2829 } else if (mac_srs->srs_bw->mac_bw_used >
2833 2830 mac_srs->srs_bw->mac_bw_limit) {
2834 2831 mac_srs->srs_bw->mac_bw_state |= SRS_BW_ENFORCED;
2835 2832 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
2836 2833 goto done;
2837 2834 }
2838 2835 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
2839 2836
2840 2837 /* If we are blanked i.e. can't do upcalls, then we are done */
2841 2838 if (mac_srs->srs_state & (SRS_BLANK | SRS_PAUSE)) {
2842 2839 ASSERT((mac_srs->srs_type & SRST_NO_SOFT_RINGS) ||
2843 2840 (mac_srs->srs_state & SRS_PAUSE));
2844 2841 goto done;
2845 2842 }
2846 2843
2847 2844 sz = 0;
2848 2845 cnt = 0;
2849 2846 if ((head = mac_srs_pick_chain(mac_srs, &tail, &sz, &cnt)) == NULL) {
2850 2847 /*
2851 2848 * We couldn't pick up a single packet.
2852 2849 */
2853 2850 mutex_enter(&mac_srs->srs_bw->mac_bw_lock);
2854 2851 if ((mac_srs->srs_bw->mac_bw_used == 0) &&
2855 2852 (mac_srs->srs_size != 0) &&
2856 2853 !(mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED)) {
2857 2854 /*
2858 2855 * Seems like configured B/W doesn't
2859 2856 * even allow processing of 1 packet
2860 2857 * per tick.
2861 2858 *
2862 2859 * XXX: raise the limit to processing
2863 2860 * at least 1 packet per tick.
2864 2861 */
2865 2862 mac_srs->srs_bw->mac_bw_limit +=
2866 2863 mac_srs->srs_bw->mac_bw_limit;
2867 2864 mac_srs->srs_bw->mac_bw_drop_threshold +=
2868 2865 mac_srs->srs_bw->mac_bw_drop_threshold;
2869 2866 cmn_err(CE_NOTE, "mac_rx_srs_drain: srs(%p) "
2870 2867 "raised B/W limit to %d since not even a "
2871 2868 "single packet can be processed per "
2872 2869 "tick %d\n", (void *)mac_srs,
2873 2870 (int)mac_srs->srs_bw->mac_bw_limit,
2874 2871 (int)msgdsize(mac_srs->srs_first));
2875 2872 }
2876 2873 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
2877 2874 goto done;
2878 2875 }
2879 2876
2880 2877 ASSERT(head != NULL);
2881 2878 ASSERT(tail != NULL);
2882 2879
2883 2880 /* zero bandwidth: drop all and return to interrupt mode */
2884 2881 mutex_enter(&mac_srs->srs_bw->mac_bw_lock);
2885 2882 if (mac_srs->srs_bw->mac_bw_limit == 0) {
2886 2883 srs_rx->sr_stat.mrs_sdrops += cnt;
2887 2884 ASSERT(mac_srs->srs_bw->mac_bw_sz >= sz);
2888 2885 mac_srs->srs_bw->mac_bw_sz -= sz;
2889 2886 mac_srs->srs_bw->mac_bw_drop_bytes += sz;
2890 2887 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
2891 2888 mac_pkt_drop(NULL, NULL, head, B_FALSE);
2892 2889 goto leave_poll;
2893 2890 } else {
2894 2891 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
2895 2892 }
2896 2893
2897 2894 if ((tid = mac_srs->srs_tid) != NULL)
2898 2895 mac_srs->srs_tid = NULL;
2899 2896
2900 2897 mac_srs->srs_state |= (SRS_PROC|proc_type);
2901 2898 MAC_SRS_WORKER_POLLING_ON(mac_srs);
2902 2899
2903 2900 /*
2904 2901 * mcip is NULL for broadcast and multicast flows. The promisc
2905 2902 * callbacks for broadcast and multicast packets are delivered from
2906 2903 * mac_rx() and we don't need to worry about that case in this path
2907 2904 */
2908 2905 if (mcip != NULL) {
2909 2906 if (mcip->mci_promisc_list != NULL) {
2910 2907 mutex_exit(&mac_srs->srs_lock);
2911 2908 mac_promisc_client_dispatch(mcip, head);
2912 2909 mutex_enter(&mac_srs->srs_lock);
2913 2910 }
2914 2911 if (MAC_PROTECT_ENABLED(mcip, MPT_IPNOSPOOF)) {
2915 2912 mutex_exit(&mac_srs->srs_lock);
2916 2913 mac_protect_intercept_dynamic(mcip, head);
2917 2914 mutex_enter(&mac_srs->srs_lock);
2918 2915 }
2919 2916 }
2920 2917
2921 2918 /*
2922 2919 * Check if SRS itself is doing the processing
2923 2920 * This direct path does not apply when subflows are present. In this
2924 2921 * case, packets need to be dispatched to a soft ring according to the
2925 2922 * flow's bandwidth and other resources contraints.
2926 2923 */
2927 2924 if (mac_srs->srs_type & SRST_NO_SOFT_RINGS) {
2928 2925 mac_direct_rx_t proc;
2929 2926 void *arg1;
2930 2927 mac_resource_handle_t arg2;
2931 2928
2932 2929 /*
2933 2930 * This is the case when a Rx is directly
2934 2931 * assigned and we have a fully classified
2935 2932 * protocol chain. We can deal with it in
2936 2933 * one shot.
2937 2934 */
2938 2935 proc = srs_rx->sr_func;
2939 2936 arg1 = srs_rx->sr_arg1;
2940 2937 arg2 = srs_rx->sr_arg2;
2941 2938
2942 2939 mac_srs->srs_state |= SRS_CLIENT_PROC;
2943 2940 mutex_exit(&mac_srs->srs_lock);
2944 2941 if (tid != NULL) {
2945 2942 (void) untimeout(tid);
2946 2943 tid = NULL;
2947 2944 }
2948 2945
2949 2946 proc(arg1, arg2, head, NULL);
2950 2947 /*
2951 2948 * Decrement the size and count here itelf
2952 2949 * since the packet has been processed.
2953 2950 */
2954 2951 mutex_enter(&mac_srs->srs_lock);
2955 2952 MAC_UPDATE_SRS_COUNT_LOCKED(mac_srs, cnt);
2956 2953 MAC_UPDATE_SRS_SIZE_LOCKED(mac_srs, sz);
2957 2954
2958 2955 if (mac_srs->srs_state & SRS_CLIENT_WAIT)
2959 2956 cv_signal(&mac_srs->srs_client_cv);
2960 2957 mac_srs->srs_state &= ~SRS_CLIENT_PROC;
2961 2958 } else {
2962 2959 /* Some kind of softrings based fanout is required */
2963 2960 mutex_exit(&mac_srs->srs_lock);
2964 2961 if (tid != NULL) {
2965 2962 (void) untimeout(tid);
2966 2963 tid = NULL;
2967 2964 }
2968 2965
2969 2966 /*
2970 2967 * Since the fanout routines can deal with chains,
2971 2968 * shoot the entire chain up.
2972 2969 */
2973 2970 if (mac_srs->srs_type & SRST_FANOUT_SRC_IP)
2974 2971 mac_rx_srs_fanout(mac_srs, head);
2975 2972 else
2976 2973 mac_rx_srs_proto_fanout(mac_srs, head);
2977 2974 mutex_enter(&mac_srs->srs_lock);
2978 2975 }
2979 2976
2980 2977 /*
2981 2978 * Send the poll thread to pick up any packets arrived
2982 2979 * so far. This also serves as the last check in case
2983 2980 * nothing else is queued in the SRS. The poll thread
2984 2981 * is signalled only in the case the drain was done
2985 2982 * by the worker thread and SRS_WORKER is set. The
2986 2983 * worker thread can run in parallel as long as the
2987 2984 * SRS_WORKER flag is set. We we have nothing else to
2988 2985 * process, we can exit while leaving SRS_PROC set
2989 2986 * which gives the poll thread control to process and
2990 2987 * cleanup once it returns from the NIC.
2991 2988 *
2992 2989 * If we have nothing else to process, we need to
2993 2990 * ensure that we keep holding the srs_lock till
2994 2991 * all the checks below are done and control is
2995 2992 * handed to the poll thread if it was running.
2996 2993 */
2997 2994 mutex_enter(&mac_srs->srs_bw->mac_bw_lock);
2998 2995 if (!(mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED)) {
2999 2996 if (mac_srs->srs_first != NULL) {
3000 2997 if (proc_type == SRS_WORKER) {
3001 2998 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
3002 2999 if (srs_rx->sr_poll_pkt_cnt <=
3003 3000 srs_rx->sr_lowat)
3004 3001 MAC_SRS_POLL_RING(mac_srs);
3005 3002 goto again;
3006 3003 } else {
3007 3004 cv_signal(&mac_srs->srs_async);
3008 3005 }
3009 3006 }
3010 3007 }
3011 3008 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
3012 3009
3013 3010 done:
3014 3011
3015 3012 if (mac_srs->srs_state & SRS_GET_PKTS) {
3016 3013 /*
3017 3014 * Poll thread is already running. Leave the
3018 3015 * SRS_RPOC set and hand over the control to
3019 3016 * poll thread.
3020 3017 */
3021 3018 mac_srs->srs_state &= ~proc_type;
3022 3019 return;
3023 3020 }
3024 3021
3025 3022 /*
3026 3023 * If we can't process packets because we have exceeded
3027 3024 * B/W limit for this tick, just set the timeout
3028 3025 * and leave.
3029 3026 *
3030 3027 * Even if there are no packets queued in SRS, we
3031 3028 * need to make sure that the shared counter is
3032 3029 * clear and any associated softrings have cleared
3033 3030 * all the backlog. Otherwise, leave the interface
3034 3031 * in polling mode and the poll thread will get
3035 3032 * signalled once the count goes down to zero.
3036 3033 *
3037 3034 * If someone is already draining the queue (SRS_PROC is
3038 3035 * set) when the srs_poll_pkt_cnt goes down to zero,
3039 3036 * then it means that drain is already running and we
3040 3037 * will turn off polling at that time if there is
3041 3038 * no backlog. As long as there are packets queued either
3042 3039 * is soft ring set or its soft rings, we will leave
3043 3040 * the interface in polling mode.
3044 3041 */
3045 3042 mutex_enter(&mac_srs->srs_bw->mac_bw_lock);
3046 3043 if ((mac_srs->srs_state & SRS_POLLING_CAPAB) &&
3047 3044 ((mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED) ||
3048 3045 (srs_rx->sr_poll_pkt_cnt > 0))) {
3049 3046 MAC_SRS_POLLING_ON(mac_srs);
3050 3047 mac_srs->srs_state &= ~(SRS_PROC|proc_type);
3051 3048 if ((mac_srs->srs_first != NULL) &&
3052 3049 (mac_srs->srs_tid == NULL))
3053 3050 mac_srs->srs_tid = timeout(mac_srs_fire,
3054 3051 mac_srs, 1);
3055 3052 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
3056 3053 return;
3057 3054 }
3058 3055 mutex_exit(&mac_srs->srs_bw->mac_bw_lock);
3059 3056
3060 3057 leave_poll:
3061 3058
3062 3059 /* Nothing else to do. Get out of poll mode */
3063 3060 MAC_SRS_POLLING_OFF(mac_srs);
3064 3061 mac_srs->srs_state &= ~(SRS_PROC|proc_type);
3065 3062 }
3066 3063
3067 3064 /*
3068 3065 * mac_srs_worker
3069 3066 *
3070 3067 * The SRS worker routine. Drains the queue when no one else is
3071 3068 * processing it.
3072 3069 */
3073 3070 void
3074 3071 mac_srs_worker(mac_soft_ring_set_t *mac_srs)
3075 3072 {
3076 3073 kmutex_t *lock = &mac_srs->srs_lock;
3077 3074 kcondvar_t *async = &mac_srs->srs_async;
3078 3075 callb_cpr_t cprinfo;
3079 3076 boolean_t bw_ctl_flag;
3080 3077
3081 3078 CALLB_CPR_INIT(&cprinfo, lock, callb_generic_cpr, "srs_worker");
3082 3079 mutex_enter(lock);
3083 3080
3084 3081 start:
3085 3082 for (;;) {
3086 3083 bw_ctl_flag = B_FALSE;
3087 3084 if (mac_srs->srs_type & SRST_BW_CONTROL) {
3088 3085 MAC_SRS_BW_LOCK(mac_srs);
3089 3086 MAC_SRS_CHECK_BW_CONTROL(mac_srs);
3090 3087 if (mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED)
3091 3088 bw_ctl_flag = B_TRUE;
3092 3089 MAC_SRS_BW_UNLOCK(mac_srs);
3093 3090 }
3094 3091 /*
3095 3092 * The SRS_BW_ENFORCED flag may change since we have dropped
3096 3093 * the mac_bw_lock. However the drain function can handle both
3097 3094 * a drainable SRS or a bandwidth controlled SRS, and the
3098 3095 * effect of scheduling a timeout is to wakeup the worker
3099 3096 * thread which in turn will call the drain function. Since
3100 3097 * we release the srs_lock atomically only in the cv_wait there
3101 3098 * isn't a fear of waiting for ever.
3102 3099 */
3103 3100 while (((mac_srs->srs_state & SRS_PROC) ||
3104 3101 (mac_srs->srs_first == NULL) || bw_ctl_flag ||
3105 3102 (mac_srs->srs_state & SRS_TX_BLOCKED)) &&
3106 3103 !(mac_srs->srs_state & SRS_PAUSE)) {
3107 3104 /*
3108 3105 * If we have packets queued and we are here
3109 3106 * because B/W control is in place, we better
3110 3107 * schedule the worker wakeup after 1 tick
3111 3108 * to see if bandwidth control can be relaxed.
3112 3109 */
3113 3110 if (bw_ctl_flag && mac_srs->srs_tid == NULL) {
3114 3111 /*
3115 3112 * We need to ensure that a timer is already
3116 3113 * scheduled or we force schedule one for
3117 3114 * later so that we can continue processing
3118 3115 * after this quanta is over.
3119 3116 */
3120 3117 mac_srs->srs_tid = timeout(mac_srs_fire,
3121 3118 mac_srs, 1);
3122 3119 }
3123 3120 wait:
3124 3121 CALLB_CPR_SAFE_BEGIN(&cprinfo);
3125 3122 cv_wait(async, lock);
3126 3123 CALLB_CPR_SAFE_END(&cprinfo, lock);
3127 3124
3128 3125 if (mac_srs->srs_state & SRS_PAUSE)
3129 3126 goto done;
3130 3127 if (mac_srs->srs_state & SRS_PROC)
3131 3128 goto wait;
3132 3129
3133 3130 if (mac_srs->srs_first != NULL &&
3134 3131 mac_srs->srs_type & SRST_BW_CONTROL) {
3135 3132 MAC_SRS_BW_LOCK(mac_srs);
3136 3133 if (mac_srs->srs_bw->mac_bw_state &
3137 3134 SRS_BW_ENFORCED) {
3138 3135 MAC_SRS_CHECK_BW_CONTROL(mac_srs);
3139 3136 }
3140 3137 bw_ctl_flag = mac_srs->srs_bw->mac_bw_state &
3141 3138 SRS_BW_ENFORCED;
3142 3139 MAC_SRS_BW_UNLOCK(mac_srs);
3143 3140 }
3144 3141 }
3145 3142
3146 3143 if (mac_srs->srs_state & SRS_PAUSE)
3147 3144 goto done;
3148 3145 mac_srs->srs_drain_func(mac_srs, SRS_WORKER);
3149 3146 }
3150 3147 done:
3151 3148 /*
3152 3149 * The Rx SRS quiesce logic first cuts off packet supply to the SRS
3153 3150 * from both hard and soft classifications and waits for such threads
3154 3151 * to finish before signaling the worker. So at this point the only
3155 3152 * thread left that could be competing with the worker is the poll
3156 3153 * thread. In the case of Tx, there shouldn't be any thread holding
3157 3154 * SRS_PROC at this point.
3158 3155 */
3159 3156 if (!(mac_srs->srs_state & SRS_PROC)) {
3160 3157 mac_srs->srs_state |= SRS_PROC;
3161 3158 } else {
3162 3159 ASSERT((mac_srs->srs_type & SRST_TX) == 0);
3163 3160 /*
3164 3161 * Poll thread still owns the SRS and is still running
3165 3162 */
3166 3163 ASSERT((mac_srs->srs_poll_thr == NULL) ||
3167 3164 ((mac_srs->srs_state & SRS_POLL_THR_OWNER) ==
3168 3165 SRS_POLL_THR_OWNER));
3169 3166 }
3170 3167 mac_srs_worker_quiesce(mac_srs);
3171 3168 /*
3172 3169 * Wait for the SRS_RESTART or SRS_CONDEMNED signal from the initiator
3173 3170 * of the quiesce operation
3174 3171 */
3175 3172 while (!(mac_srs->srs_state & (SRS_CONDEMNED | SRS_RESTART)))
3176 3173 cv_wait(&mac_srs->srs_async, &mac_srs->srs_lock);
3177 3174
3178 3175 if (mac_srs->srs_state & SRS_RESTART) {
3179 3176 ASSERT(!(mac_srs->srs_state & SRS_CONDEMNED));
3180 3177 mac_srs_worker_restart(mac_srs);
3181 3178 mac_srs->srs_state &= ~SRS_PROC;
3182 3179 goto start;
3183 3180 }
3184 3181
3185 3182 if (!(mac_srs->srs_state & SRS_CONDEMNED_DONE))
3186 3183 mac_srs_worker_quiesce(mac_srs);
3187 3184
3188 3185 mac_srs->srs_state &= ~SRS_PROC;
3189 3186 /* The macro drops the srs_lock */
3190 3187 CALLB_CPR_EXIT(&cprinfo);
3191 3188 thread_exit();
3192 3189 }
3193 3190
3194 3191 /*
3195 3192 * mac_rx_srs_subflow_process
3196 3193 *
3197 3194 * Receive side routine called from interrupt path when there are
3198 3195 * sub flows present on this SRS.
3199 3196 */
3200 3197 /* ARGSUSED */
3201 3198 void
3202 3199 mac_rx_srs_subflow_process(void *arg, mac_resource_handle_t srs,
3203 3200 mblk_t *mp_chain, boolean_t loopback)
3204 3201 {
3205 3202 flow_entry_t *flent = NULL;
3206 3203 flow_entry_t *prev_flent = NULL;
3207 3204 mblk_t *mp = NULL;
3208 3205 mblk_t *tail = NULL;
3209 3206 mac_soft_ring_set_t *mac_srs = (mac_soft_ring_set_t *)srs;
3210 3207 mac_client_impl_t *mcip;
3211 3208
3212 3209 mcip = mac_srs->srs_mcip;
3213 3210 ASSERT(mcip != NULL);
3214 3211
3215 3212 /*
3216 3213 * We need to determine the SRS for every packet
3217 3214 * by walking the flow table, if we don't get any,
3218 3215 * then we proceed using the SRS we came with.
3219 3216 */
3220 3217 mp = tail = mp_chain;
3221 3218 while (mp != NULL) {
3222 3219
3223 3220 /*
3224 3221 * We will increment the stats for the mactching subflow.
3225 3222 * when we get the bytes/pkt count for the classified packets
3226 3223 * later in mac_rx_srs_process.
3227 3224 */
3228 3225 (void) mac_flow_lookup(mcip->mci_subflow_tab, mp,
3229 3226 FLOW_INBOUND, &flent);
3230 3227
3231 3228 if (mp == mp_chain || flent == prev_flent) {
3232 3229 if (prev_flent != NULL)
3233 3230 FLOW_REFRELE(prev_flent);
3234 3231 prev_flent = flent;
3235 3232 flent = NULL;
3236 3233 tail = mp;
3237 3234 mp = mp->b_next;
3238 3235 continue;
3239 3236 }
3240 3237 tail->b_next = NULL;
3241 3238 /*
3242 3239 * A null indicates, this is for the mac_srs itself.
3243 3240 * XXX-venu : probably assert for fe_rx_srs_cnt == 0.
3244 3241 */
3245 3242 if (prev_flent == NULL || prev_flent->fe_rx_srs_cnt == 0) {
3246 3243 mac_rx_srs_process(arg,
3247 3244 (mac_resource_handle_t)mac_srs, mp_chain,
3248 3245 loopback);
3249 3246 } else {
3250 3247 (prev_flent->fe_cb_fn)(prev_flent->fe_cb_arg1,
3251 3248 prev_flent->fe_cb_arg2, mp_chain, loopback);
3252 3249 FLOW_REFRELE(prev_flent);
3253 3250 }
3254 3251 prev_flent = flent;
3255 3252 flent = NULL;
3256 3253 mp_chain = mp;
3257 3254 tail = mp;
3258 3255 mp = mp->b_next;
3259 3256 }
3260 3257 /* Last chain */
3261 3258 ASSERT(mp_chain != NULL);
3262 3259 if (prev_flent == NULL || prev_flent->fe_rx_srs_cnt == 0) {
3263 3260 mac_rx_srs_process(arg,
3264 3261 (mac_resource_handle_t)mac_srs, mp_chain, loopback);
3265 3262 } else {
3266 3263 (prev_flent->fe_cb_fn)(prev_flent->fe_cb_arg1,
3267 3264 prev_flent->fe_cb_arg2, mp_chain, loopback);
3268 3265 FLOW_REFRELE(prev_flent);
3269 3266 }
3270 3267 }
3271 3268
3272 3269 /*
3273 3270 * mac_rx_srs_process
3274 3271 *
3275 3272 * Receive side routine called from the interrupt path.
3276 3273 *
3277 3274 * loopback is set to force a context switch on the loopback
3278 3275 * path between MAC clients.
3279 3276 */
3280 3277 /* ARGSUSED */
3281 3278 void
3282 3279 mac_rx_srs_process(void *arg, mac_resource_handle_t srs, mblk_t *mp_chain,
3283 3280 boolean_t loopback)
3284 3281 {
3285 3282 mac_soft_ring_set_t *mac_srs = (mac_soft_ring_set_t *)srs;
3286 3283 mblk_t *mp, *tail, *head;
3287 3284 int count = 0;
3288 3285 int count1;
3289 3286 size_t sz = 0;
3290 3287 size_t chain_sz, sz1;
3291 3288 mac_bw_ctl_t *mac_bw;
3292 3289 mac_srs_rx_t *srs_rx = &mac_srs->srs_rx;
3293 3290
3294 3291 /*
3295 3292 * Set the tail, count and sz. We set the sz irrespective
3296 3293 * of whether we are doing B/W control or not for the
3297 3294 * purpose of updating the stats.
3298 3295 */
3299 3296 mp = tail = mp_chain;
3300 3297 while (mp != NULL) {
3301 3298 tail = mp;
3302 3299 count++;
3303 3300 sz += msgdsize(mp);
3304 3301 mp = mp->b_next;
3305 3302 }
3306 3303
3307 3304 mutex_enter(&mac_srs->srs_lock);
3308 3305
3309 3306 if (loopback) {
3310 3307 SRS_RX_STAT_UPDATE(mac_srs, lclbytes, sz);
3311 3308 SRS_RX_STAT_UPDATE(mac_srs, lclcnt, count);
3312 3309
3313 3310 } else {
3314 3311 SRS_RX_STAT_UPDATE(mac_srs, intrbytes, sz);
3315 3312 SRS_RX_STAT_UPDATE(mac_srs, intrcnt, count);
3316 3313 }
3317 3314
3318 3315 /*
3319 3316 * If the SRS in already being processed; has been blanked;
3320 3317 * can be processed by worker thread only; or the B/W limit
3321 3318 * has been reached, then queue the chain and check if
3322 3319 * worker thread needs to be awakend.
3323 3320 */
3324 3321 if (mac_srs->srs_type & SRST_BW_CONTROL) {
3325 3322 mac_bw = mac_srs->srs_bw;
3326 3323 ASSERT(mac_bw != NULL);
3327 3324 mutex_enter(&mac_bw->mac_bw_lock);
3328 3325 mac_bw->mac_bw_intr += sz;
3329 3326 if (mac_bw->mac_bw_limit == 0) {
3330 3327 /* zero bandwidth: drop all */
3331 3328 srs_rx->sr_stat.mrs_sdrops += count;
3332 3329 mac_bw->mac_bw_drop_bytes += sz;
3333 3330 mutex_exit(&mac_bw->mac_bw_lock);
3334 3331 mutex_exit(&mac_srs->srs_lock);
3335 3332 mac_pkt_drop(NULL, NULL, mp_chain, B_FALSE);
3336 3333 return;
3337 3334 } else {
3338 3335 if ((mac_bw->mac_bw_sz + sz) <=
3339 3336 mac_bw->mac_bw_drop_threshold) {
3340 3337 mutex_exit(&mac_bw->mac_bw_lock);
3341 3338 MAC_RX_SRS_ENQUEUE_CHAIN(mac_srs, mp_chain,
3342 3339 tail, count, sz);
3343 3340 } else {
3344 3341 mp = mp_chain;
3345 3342 chain_sz = 0;
3346 3343 count1 = 0;
3347 3344 tail = NULL;
3348 3345 head = NULL;
3349 3346 while (mp != NULL) {
3350 3347 sz1 = msgdsize(mp);
3351 3348 if (mac_bw->mac_bw_sz + chain_sz + sz1 >
3352 3349 mac_bw->mac_bw_drop_threshold)
3353 3350 break;
3354 3351 chain_sz += sz1;
3355 3352 count1++;
3356 3353 tail = mp;
3357 3354 mp = mp->b_next;
3358 3355 }
3359 3356 mutex_exit(&mac_bw->mac_bw_lock);
3360 3357 if (tail != NULL) {
3361 3358 head = tail->b_next;
3362 3359 tail->b_next = NULL;
3363 3360 MAC_RX_SRS_ENQUEUE_CHAIN(mac_srs,
3364 3361 mp_chain, tail, count1, chain_sz);
3365 3362 sz -= chain_sz;
3366 3363 count -= count1;
3367 3364 } else {
3368 3365 /* Can't pick up any */
3369 3366 head = mp_chain;
3370 3367 }
3371 3368 if (head != NULL) {
3372 3369 /* Drop any packet over the threshold */
3373 3370 srs_rx->sr_stat.mrs_sdrops += count;
3374 3371 mutex_enter(&mac_bw->mac_bw_lock);
3375 3372 mac_bw->mac_bw_drop_bytes += sz;
3376 3373 mutex_exit(&mac_bw->mac_bw_lock);
3377 3374 freemsgchain(head);
3378 3375 }
3379 3376 }
3380 3377 MAC_SRS_WORKER_WAKEUP(mac_srs);
3381 3378 mutex_exit(&mac_srs->srs_lock);
3382 3379 return;
3383 3380 }
3384 3381 }
3385 3382
3386 3383 /*
3387 3384 * If the total number of packets queued in the SRS and
3388 3385 * its associated soft rings exceeds the max allowed,
3389 3386 * then drop the chain. If we are polling capable, this
3390 3387 * shouldn't be happening.
3391 3388 */
3392 3389 if (!(mac_srs->srs_type & SRST_BW_CONTROL) &&
3393 3390 (srs_rx->sr_poll_pkt_cnt > srs_rx->sr_hiwat)) {
3394 3391 mac_bw = mac_srs->srs_bw;
3395 3392 srs_rx->sr_stat.mrs_sdrops += count;
3396 3393 mutex_enter(&mac_bw->mac_bw_lock);
3397 3394 mac_bw->mac_bw_drop_bytes += sz;
3398 3395 mutex_exit(&mac_bw->mac_bw_lock);
3399 3396 freemsgchain(mp_chain);
3400 3397 mutex_exit(&mac_srs->srs_lock);
3401 3398 return;
3402 3399 }
3403 3400
3404 3401 MAC_RX_SRS_ENQUEUE_CHAIN(mac_srs, mp_chain, tail, count, sz);
3405 3402
3406 3403 if (!(mac_srs->srs_state & SRS_PROC)) {
3407 3404 /*
3408 3405 * If we are coming via loopback, if we are not optimizing for
3409 3406 * latency, or if our stack is running deep, we should signal
3410 3407 * the worker thread.
3411 3408 */
3412 3409 if (loopback || !(mac_srs->srs_state & SRS_LATENCY_OPT) ||
3413 3410 MAC_RX_SRS_TOODEEP()) {
3414 3411 /*
3415 3412 * For loopback, We need to let the worker take
3416 3413 * over as we don't want to continue in the same
3417 3414 * thread even if we can. This could lead to stack
3418 3415 * overflows and may also end up using
3419 3416 * resources (cpu) incorrectly.
3420 3417 */
3421 3418 cv_signal(&mac_srs->srs_async);
3422 3419 } else {
3423 3420 /*
3424 3421 * Seems like no one is processing the SRS and
3425 3422 * there is no backlog. We also inline process
3426 3423 * our packet if its a single packet in non
3427 3424 * latency optimized case (in latency optimized
3428 3425 * case, we inline process chains of any size).
3429 3426 */
3430 3427 mac_srs->srs_drain_func(mac_srs, SRS_PROC_FAST);
3431 3428 }
3432 3429 }
3433 3430 mutex_exit(&mac_srs->srs_lock);
3434 3431 }
3435 3432
3436 3433 /* TX SIDE ROUTINES (RUNTIME) */
3437 3434
3438 3435 /*
3439 3436 * mac_tx_srs_no_desc
3440 3437 *
3441 3438 * This routine is called by Tx single ring default mode
3442 3439 * when Tx ring runs out of descs.
3443 3440 */
3444 3441 mac_tx_cookie_t
3445 3442 mac_tx_srs_no_desc(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain,
3446 3443 uint16_t flag, mblk_t **ret_mp)
3447 3444 {
3448 3445 mac_tx_cookie_t cookie = NULL;
3449 3446 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx;
3450 3447 boolean_t wakeup_worker = B_TRUE;
3451 3448 uint32_t tx_mode = srs_tx->st_mode;
3452 3449 int cnt, sz;
3453 3450 mblk_t *tail;
3454 3451
3455 3452 ASSERT(tx_mode == SRS_TX_DEFAULT || tx_mode == SRS_TX_BW);
3456 3453 if (flag & MAC_DROP_ON_NO_DESC) {
3457 3454 MAC_TX_SRS_DROP_MESSAGE(mac_srs, mp_chain, cookie);
3458 3455 } else {
3459 3456 if (mac_srs->srs_first != NULL)
3460 3457 wakeup_worker = B_FALSE;
3461 3458 MAC_COUNT_CHAIN(mac_srs, mp_chain, tail, cnt, sz);
3462 3459 if (flag & MAC_TX_NO_ENQUEUE) {
3463 3460 /*
3464 3461 * If TX_QUEUED is not set, queue the
3465 3462 * packet and let mac_tx_srs_drain()
3466 3463 * set the TX_BLOCKED bit for the
3467 3464 * reasons explained above. Otherwise,
3468 3465 * return the mblks.
3469 3466 */
3470 3467 if (wakeup_worker) {
3471 3468 MAC_TX_SRS_ENQUEUE_CHAIN(mac_srs,
3472 3469 mp_chain, tail, cnt, sz);
3473 3470 } else {
3474 3471 MAC_TX_SET_NO_ENQUEUE(mac_srs,
3475 3472 mp_chain, ret_mp, cookie);
3476 3473 }
3477 3474 } else {
3478 3475 MAC_TX_SRS_TEST_HIWAT(mac_srs, mp_chain,
3479 3476 tail, cnt, sz, cookie);
3480 3477 }
3481 3478 if (wakeup_worker)
3482 3479 cv_signal(&mac_srs->srs_async);
3483 3480 }
3484 3481 return (cookie);
3485 3482 }
3486 3483
3487 3484 /*
3488 3485 * mac_tx_srs_enqueue
3489 3486 *
3490 3487 * This routine is called when Tx SRS is operating in either serializer
3491 3488 * or bandwidth mode. In serializer mode, a packet will get enqueued
3492 3489 * when a thread cannot enter SRS exclusively. In bandwidth mode,
3493 3490 * packets gets queued if allowed byte-count limit for a tick is
3494 3491 * exceeded. The action that gets taken when MAC_DROP_ON_NO_DESC and
3495 3492 * MAC_TX_NO_ENQUEUE is set is different than when operaing in either
3496 3493 * the default mode or fanout mode. Here packets get dropped or
3497 3494 * returned back to the caller only after hi-watermark worth of data
3498 3495 * is queued.
3499 3496 */
3500 3497 static mac_tx_cookie_t
3501 3498 mac_tx_srs_enqueue(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain,
3502 3499 uint16_t flag, uintptr_t fanout_hint, mblk_t **ret_mp)
3503 3500 {
3504 3501 mac_tx_cookie_t cookie = NULL;
3505 3502 int cnt, sz;
3506 3503 mblk_t *tail;
3507 3504 boolean_t wakeup_worker = B_TRUE;
3508 3505
3509 3506 /*
3510 3507 * Ignore fanout hint if we don't have multiple tx rings.
3511 3508 */
3512 3509 if (!MAC_TX_SOFT_RINGS(mac_srs))
3513 3510 fanout_hint = 0;
3514 3511
3515 3512 if (mac_srs->srs_first != NULL)
3516 3513 wakeup_worker = B_FALSE;
3517 3514 MAC_COUNT_CHAIN(mac_srs, mp_chain, tail, cnt, sz);
3518 3515 if (flag & MAC_DROP_ON_NO_DESC) {
3519 3516 if (mac_srs->srs_count > mac_srs->srs_tx.st_hiwat) {
3520 3517 MAC_TX_SRS_DROP_MESSAGE(mac_srs, mp_chain, cookie);
3521 3518 } else {
3522 3519 MAC_TX_SRS_ENQUEUE_CHAIN(mac_srs,
3523 3520 mp_chain, tail, cnt, sz);
3524 3521 }
3525 3522 } else if (flag & MAC_TX_NO_ENQUEUE) {
3526 3523 if ((mac_srs->srs_count > mac_srs->srs_tx.st_hiwat) ||
3527 3524 (mac_srs->srs_state & SRS_TX_WAKEUP_CLIENT)) {
3528 3525 MAC_TX_SET_NO_ENQUEUE(mac_srs, mp_chain,
3529 3526 ret_mp, cookie);
3530 3527 } else {
3531 3528 mp_chain->b_prev = (mblk_t *)fanout_hint;
3532 3529 MAC_TX_SRS_ENQUEUE_CHAIN(mac_srs,
3533 3530 mp_chain, tail, cnt, sz);
3534 3531 }
3535 3532 } else {
3536 3533 /*
3537 3534 * If you are BW_ENFORCED, just enqueue the
3538 3535 * packet. srs_worker will drain it at the
3539 3536 * prescribed rate. Before enqueueing, save
3540 3537 * the fanout hint.
3541 3538 */
3542 3539 mp_chain->b_prev = (mblk_t *)fanout_hint;
3543 3540 MAC_TX_SRS_TEST_HIWAT(mac_srs, mp_chain,
3544 3541 tail, cnt, sz, cookie);
3545 3542 }
3546 3543 if (wakeup_worker)
3547 3544 cv_signal(&mac_srs->srs_async);
3548 3545 return (cookie);
3549 3546 }
3550 3547
3551 3548 /*
3552 3549 * There are seven tx modes:
3553 3550 *
3554 3551 * 1) Default mode (SRS_TX_DEFAULT)
3555 3552 * 2) Serialization mode (SRS_TX_SERIALIZE)
3556 3553 * 3) Fanout mode (SRS_TX_FANOUT)
3557 3554 * 4) Bandwdith mode (SRS_TX_BW)
3558 3555 * 5) Fanout and Bandwidth mode (SRS_TX_BW_FANOUT)
3559 3556 * 6) aggr Tx mode (SRS_TX_AGGR)
3560 3557 * 7) aggr Tx bw mode (SRS_TX_BW_AGGR)
3561 3558 *
3562 3559 * The tx mode in which an SRS operates is decided in mac_tx_srs_setup()
3563 3560 * based on the number of Tx rings requested for an SRS and whether
3564 3561 * bandwidth control is requested or not.
3565 3562 *
3566 3563 * The default mode (i.e., no fanout/no bandwidth) is used when the
3567 3564 * underlying NIC does not have Tx rings or just one Tx ring. In this mode,
3568 3565 * the SRS acts as a pass-thru. Packets will go directly to mac_tx_send().
3569 3566 * When the underlying Tx ring runs out of Tx descs, it starts queueing up
3570 3567 * packets in SRS. When flow-control is relieved, the srs_worker drains
3571 3568 * the queued packets and informs blocked clients to restart sending
3572 3569 * packets.
3573 3570 *
3574 3571 * In the SRS_TX_SERIALIZE mode, all calls to mac_tx() are serialized. This
3575 3572 * mode is used when the link has no Tx rings or only one Tx ring.
3576 3573 *
3577 3574 * In the SRS_TX_FANOUT mode, packets will be fanned out to multiple
3578 3575 * Tx rings. Each Tx ring will have a soft ring associated with it.
3579 3576 * These soft rings will be hung off the Tx SRS. Queueing if it happens
3580 3577 * due to lack of Tx desc will be in individual soft ring (and not srs)
3581 3578 * associated with Tx ring.
3582 3579 *
3583 3580 * In the TX_BW mode, tx srs will allow packets to go down to Tx ring
3584 3581 * only if bw is available. Otherwise the packets will be queued in
3585 3582 * SRS. If fanout to multiple Tx rings is configured, the packets will
3586 3583 * be fanned out among the soft rings associated with the Tx rings.
3587 3584 *
3588 3585 * In SRS_TX_AGGR mode, mac_tx_aggr_mode() routine is called. This routine
3589 3586 * invokes an aggr function, aggr_find_tx_ring(), to find a pseudo Tx ring
3590 3587 * belonging to a port on which the packet has to be sent. Aggr will
3591 3588 * always have a pseudo Tx ring associated with it even when it is an
3592 3589 * aggregation over a single NIC that has no Tx rings. Even in such a
3593 3590 * case, the single pseudo Tx ring will have a soft ring associated with
3594 3591 * it and the soft ring will hang off the SRS.
3595 3592 *
3596 3593 * If a bandwidth is specified for an aggr, SRS_TX_BW_AGGR mode is used.
3597 3594 * In this mode, the bandwidth is first applied on the outgoing packets
3598 3595 * and later mac_tx_addr_mode() function is called to send the packet out
3599 3596 * of one of the pseudo Tx rings.
3600 3597 *
3601 3598 * Four flags are used in srs_state for indicating flow control
3602 3599 * conditions : SRS_TX_BLOCKED, SRS_TX_HIWAT, SRS_TX_WAKEUP_CLIENT.
3603 3600 * SRS_TX_BLOCKED indicates out of Tx descs. SRS expects a wakeup from the
3604 3601 * driver below.
3605 3602 * SRS_TX_HIWAT indicates packet count enqueued in Tx SRS exceeded Tx hiwat
3606 3603 * and flow-control pressure is applied back to clients. The clients expect
3607 3604 * wakeup when flow-control is relieved.
3608 3605 * SRS_TX_WAKEUP_CLIENT get set when (flag == MAC_TX_NO_ENQUEUE) and mblk
3609 3606 * got returned back to client either due to lack of Tx descs or due to bw
3610 3607 * control reasons. The clients expect a wakeup when condition is relieved.
3611 3608 *
3612 3609 * The fourth argument to mac_tx() is the flag. Normally it will be 0 but
3613 3610 * some clients set the following values too: MAC_DROP_ON_NO_DESC,
3614 3611 * MAC_TX_NO_ENQUEUE
3615 3612 * Mac clients that do not want packets to be enqueued in the mac layer set
3616 3613 * MAC_DROP_ON_NO_DESC value. The packets won't be queued in the Tx SRS or
3617 3614 * Tx soft rings but instead get dropped when the NIC runs out of desc. The
3618 3615 * behaviour of this flag is different when the Tx is running in serializer
3619 3616 * or bandwidth mode. Under these (Serializer, bandwidth) modes, the packet
3620 3617 * get dropped when Tx high watermark is reached.
3621 3618 * There are some mac clients like vsw, aggr that want the mblks to be
3622 3619 * returned back to clients instead of being queued in Tx SRS (or Tx soft
3623 3620 * rings) under flow-control (i.e., out of desc or exceeding bw limits)
3624 3621 * conditions. These clients call mac_tx() with MAC_TX_NO_ENQUEUE flag set.
3625 3622 * In the default and Tx fanout mode, the un-transmitted mblks will be
3626 3623 * returned back to the clients when the driver runs out of Tx descs.
3627 3624 * SRS_TX_WAKEUP_CLIENT (or S_RING_WAKEUP_CLIENT) will be set in SRS (or
3628 3625 * soft ring) so that the clients can be woken up when Tx desc become
3629 3626 * available. When running in serializer or bandwidth mode mode,
3630 3627 * SRS_TX_WAKEUP_CLIENT will be set when tx hi-watermark is reached.
3631 3628 */
3632 3629
3633 3630 mac_tx_func_t
3634 3631 mac_tx_get_func(uint32_t mode)
3635 3632 {
3636 3633 return (mac_tx_mode_list[mode].mac_tx_func);
3637 3634 }
3638 3635
3639 3636 /* ARGSUSED */
3640 3637 static mac_tx_cookie_t
3641 3638 mac_tx_single_ring_mode(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain,
3642 3639 uintptr_t fanout_hint, uint16_t flag, mblk_t **ret_mp)
3643 3640 {
3644 3641 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx;
3645 3642 mac_tx_stats_t stats;
3646 3643 mac_tx_cookie_t cookie = NULL;
3647 3644
3648 3645 ASSERT(srs_tx->st_mode == SRS_TX_DEFAULT);
3649 3646
3650 3647 /* Regular case with a single Tx ring */
3651 3648 /*
3652 3649 * SRS_TX_BLOCKED is set when underlying NIC runs
3653 3650 * out of Tx descs and messages start getting
3654 3651 * queued. It won't get reset until
3655 3652 * tx_srs_drain() completely drains out the
3656 3653 * messages.
3657 3654 */
3658 3655 if ((mac_srs->srs_state & SRS_ENQUEUED) != 0) {
3659 3656 /* Tx descs/resources not available */
3660 3657 mutex_enter(&mac_srs->srs_lock);
3661 3658 if ((mac_srs->srs_state & SRS_ENQUEUED) != 0) {
3662 3659 cookie = mac_tx_srs_no_desc(mac_srs, mp_chain,
3663 3660 flag, ret_mp);
3664 3661 mutex_exit(&mac_srs->srs_lock);
3665 3662 return (cookie);
3666 3663 }
3667 3664 /*
3668 3665 * While we were computing mblk count, the
3669 3666 * flow control condition got relieved.
3670 3667 * Continue with the transmission.
3671 3668 */
3672 3669 mutex_exit(&mac_srs->srs_lock);
3673 3670 }
3674 3671
3675 3672 mp_chain = mac_tx_send(srs_tx->st_arg1, srs_tx->st_arg2,
3676 3673 mp_chain, &stats);
3677 3674
3678 3675 /*
3679 3676 * Multiple threads could be here sending packets.
3680 3677 * Under such conditions, it is not possible to
3681 3678 * automically set SRS_TX_BLOCKED bit to indicate
3682 3679 * out of tx desc condition. To atomically set
3683 3680 * this, we queue the returned packet and do
3684 3681 * the setting of SRS_TX_BLOCKED in
3685 3682 * mac_tx_srs_drain().
3686 3683 */
3687 3684 if (mp_chain != NULL) {
3688 3685 mutex_enter(&mac_srs->srs_lock);
3689 3686 cookie = mac_tx_srs_no_desc(mac_srs, mp_chain, flag, ret_mp);
3690 3687 mutex_exit(&mac_srs->srs_lock);
3691 3688 return (cookie);
3692 3689 }
3693 3690 SRS_TX_STATS_UPDATE(mac_srs, &stats);
3694 3691
3695 3692 return (NULL);
3696 3693 }
3697 3694
3698 3695 /*
3699 3696 * mac_tx_serialize_mode
3700 3697 *
3701 3698 * This is an experimental mode implemented as per the request of PAE.
3702 3699 * In this mode, all callers attempting to send a packet to the NIC
3703 3700 * will get serialized. Only one thread at any time will access the
3704 3701 * NIC to send the packet out.
3705 3702 */
3706 3703 /* ARGSUSED */
3707 3704 static mac_tx_cookie_t
3708 3705 mac_tx_serializer_mode(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain,
3709 3706 uintptr_t fanout_hint, uint16_t flag, mblk_t **ret_mp)
3710 3707 {
3711 3708 mac_tx_stats_t stats;
3712 3709 mac_tx_cookie_t cookie = NULL;
3713 3710 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx;
3714 3711
3715 3712 /* Single ring, serialize below */
3716 3713 ASSERT(srs_tx->st_mode == SRS_TX_SERIALIZE);
3717 3714 mutex_enter(&mac_srs->srs_lock);
3718 3715 if ((mac_srs->srs_first != NULL) ||
3719 3716 (mac_srs->srs_state & SRS_PROC)) {
3720 3717 /*
3721 3718 * In serialization mode, queue all packets until
3722 3719 * TX_HIWAT is set.
3723 3720 * If drop bit is set, drop if TX_HIWAT is set.
3724 3721 * If no_enqueue is set, still enqueue until hiwat
3725 3722 * is set and return mblks after TX_HIWAT is set.
3726 3723 */
3727 3724 cookie = mac_tx_srs_enqueue(mac_srs, mp_chain,
3728 3725 flag, NULL, ret_mp);
3729 3726 mutex_exit(&mac_srs->srs_lock);
3730 3727 return (cookie);
3731 3728 }
3732 3729 /*
3733 3730 * No packets queued, nothing on proc and no flow
3734 3731 * control condition. Fast-path, ok. Do inline
3735 3732 * processing.
3736 3733 */
3737 3734 mac_srs->srs_state |= SRS_PROC;
3738 3735 mutex_exit(&mac_srs->srs_lock);
3739 3736
3740 3737 mp_chain = mac_tx_send(srs_tx->st_arg1, srs_tx->st_arg2,
3741 3738 mp_chain, &stats);
3742 3739
3743 3740 mutex_enter(&mac_srs->srs_lock);
3744 3741 mac_srs->srs_state &= ~SRS_PROC;
3745 3742 if (mp_chain != NULL) {
3746 3743 cookie = mac_tx_srs_enqueue(mac_srs,
3747 3744 mp_chain, flag, NULL, ret_mp);
3748 3745 }
3749 3746 if (mac_srs->srs_first != NULL) {
3750 3747 /*
3751 3748 * We processed inline our packet and a new
3752 3749 * packet/s got queued while we were
3753 3750 * processing. Wakeup srs worker
3754 3751 */
3755 3752 cv_signal(&mac_srs->srs_async);
3756 3753 }
3757 3754 mutex_exit(&mac_srs->srs_lock);
3758 3755
3759 3756 if (cookie == NULL)
3760 3757 SRS_TX_STATS_UPDATE(mac_srs, &stats);
3761 3758
3762 3759 return (cookie);
3763 3760 }
3764 3761
3765 3762 /*
3766 3763 * mac_tx_fanout_mode
3767 3764 *
3768 3765 * In this mode, the SRS will have access to multiple Tx rings to send
3769 3766 * the packet out. The fanout hint that is passed as an argument is
3770 3767 * used to find an appropriate ring to fanout the traffic. Each Tx
3771 3768 * ring, in turn, will have a soft ring associated with it. If a Tx
3772 3769 * ring runs out of Tx desc's the returned packet will be queued in
3773 3770 * the soft ring associated with that Tx ring. The srs itself will not
3774 3771 * queue any packets.
3775 3772 */
3776 3773
3777 3774 #define MAC_TX_SOFT_RING_PROCESS(chain) { \
3778 3775 index = COMPUTE_INDEX(hash, mac_srs->srs_tx_ring_count), \
3779 3776 softring = mac_srs->srs_tx_soft_rings[index]; \
3780 3777 cookie = mac_tx_soft_ring_process(softring, chain, flag, ret_mp); \
3781 3778 DTRACE_PROBE2(tx__fanout, uint64_t, hash, uint_t, index); \
3782 3779 }
3783 3780
3784 3781 static mac_tx_cookie_t
3785 3782 mac_tx_fanout_mode(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain,
3786 3783 uintptr_t fanout_hint, uint16_t flag, mblk_t **ret_mp)
3787 3784 {
3788 3785 mac_soft_ring_t *softring;
3789 3786 uint64_t hash;
3790 3787 uint_t index;
3791 3788 mac_tx_cookie_t cookie = NULL;
3792 3789
3793 3790 ASSERT(mac_srs->srs_tx.st_mode == SRS_TX_FANOUT ||
3794 3791 mac_srs->srs_tx.st_mode == SRS_TX_BW_FANOUT);
3795 3792 if (fanout_hint != 0) {
3796 3793 /*
3797 3794 * The hint is specified by the caller, simply pass the
3798 3795 * whole chain to the soft ring.
3799 3796 */
3800 3797 hash = HASH_HINT(fanout_hint);
3801 3798 MAC_TX_SOFT_RING_PROCESS(mp_chain);
3802 3799 } else {
3803 3800 mblk_t *last_mp, *cur_mp, *sub_chain;
3804 3801 uint64_t last_hash = 0;
3805 3802 uint_t media = mac_srs->srs_mcip->mci_mip->mi_info.mi_media;
3806 3803
3807 3804 /*
3808 3805 * Compute the hash from the contents (headers) of the
3809 3806 * packets of the mblk chain. Split the chains into
3810 3807 * subchains of the same conversation.
3811 3808 *
3812 3809 * Since there may be more than one ring used for
3813 3810 * sub-chains of the same call, and since the caller
3814 3811 * does not maintain per conversation state since it
3815 3812 * passed a zero hint, unsent subchains will be
3816 3813 * dropped.
3817 3814 */
3818 3815
3819 3816 flag |= MAC_DROP_ON_NO_DESC;
3820 3817 ret_mp = NULL;
3821 3818
3822 3819 ASSERT(ret_mp == NULL);
3823 3820
3824 3821 sub_chain = NULL;
3825 3822 last_mp = NULL;
3826 3823
3827 3824 for (cur_mp = mp_chain; cur_mp != NULL;
3828 3825 cur_mp = cur_mp->b_next) {
3829 3826 hash = mac_pkt_hash(media, cur_mp, MAC_PKT_HASH_L4,
3830 3827 B_TRUE);
3831 3828 if (last_hash != 0 && hash != last_hash) {
3832 3829 /*
3833 3830 * Starting a different subchain, send current
3834 3831 * chain out.
3835 3832 */
3836 3833 ASSERT(last_mp != NULL);
3837 3834 last_mp->b_next = NULL;
3838 3835 MAC_TX_SOFT_RING_PROCESS(sub_chain);
3839 3836 sub_chain = NULL;
3840 3837 }
3841 3838
3842 3839 /* add packet to subchain */
3843 3840 if (sub_chain == NULL)
3844 3841 sub_chain = cur_mp;
3845 3842 last_mp = cur_mp;
3846 3843 last_hash = hash;
3847 3844 }
3848 3845
3849 3846 if (sub_chain != NULL) {
3850 3847 /* send last subchain */
3851 3848 ASSERT(last_mp != NULL);
3852 3849 last_mp->b_next = NULL;
3853 3850 MAC_TX_SOFT_RING_PROCESS(sub_chain);
3854 3851 }
3855 3852
3856 3853 cookie = NULL;
3857 3854 }
3858 3855
3859 3856 return (cookie);
3860 3857 }
3861 3858
3862 3859 /*
3863 3860 * mac_tx_bw_mode
3864 3861 *
3865 3862 * In the bandwidth mode, Tx srs will allow packets to go down to Tx ring
3866 3863 * only if bw is available. Otherwise the packets will be queued in
3867 3864 * SRS. If the SRS has multiple Tx rings, then packets will get fanned
3868 3865 * out to a Tx rings.
3869 3866 */
3870 3867 static mac_tx_cookie_t
3871 3868 mac_tx_bw_mode(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain,
3872 3869 uintptr_t fanout_hint, uint16_t flag, mblk_t **ret_mp)
3873 3870 {
3874 3871 int cnt, sz;
3875 3872 mblk_t *tail;
3876 3873 mac_tx_cookie_t cookie = NULL;
3877 3874 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx;
3878 3875 clock_t now;
3879 3876
3880 3877 ASSERT(TX_BANDWIDTH_MODE(mac_srs));
3881 3878 ASSERT(mac_srs->srs_type & SRST_BW_CONTROL);
3882 3879 mutex_enter(&mac_srs->srs_lock);
3883 3880 if (mac_srs->srs_bw->mac_bw_limit == 0) {
3884 3881 /*
3885 3882 * zero bandwidth, no traffic is sent: drop the packets,
3886 3883 * or return the whole chain if the caller requests all
3887 3884 * unsent packets back.
3888 3885 */
3889 3886 if (flag & MAC_TX_NO_ENQUEUE) {
3890 3887 cookie = (mac_tx_cookie_t)mac_srs;
3891 3888 *ret_mp = mp_chain;
3892 3889 } else {
3893 3890 MAC_TX_SRS_DROP_MESSAGE(mac_srs, mp_chain, cookie);
3894 3891 }
3895 3892 mutex_exit(&mac_srs->srs_lock);
3896 3893 return (cookie);
3897 3894 } else if ((mac_srs->srs_first != NULL) ||
3898 3895 (mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED)) {
3899 3896 cookie = mac_tx_srs_enqueue(mac_srs, mp_chain, flag,
3900 3897 fanout_hint, ret_mp);
3901 3898 mutex_exit(&mac_srs->srs_lock);
3902 3899 return (cookie);
3903 3900 }
3904 3901 MAC_COUNT_CHAIN(mac_srs, mp_chain, tail, cnt, sz);
3905 3902 now = ddi_get_lbolt();
3906 3903 if (mac_srs->srs_bw->mac_bw_curr_time != now) {
3907 3904 mac_srs->srs_bw->mac_bw_curr_time = now;
3908 3905 mac_srs->srs_bw->mac_bw_used = 0;
3909 3906 } else if (mac_srs->srs_bw->mac_bw_used >
3910 3907 mac_srs->srs_bw->mac_bw_limit) {
3911 3908 mac_srs->srs_bw->mac_bw_state |= SRS_BW_ENFORCED;
3912 3909 MAC_TX_SRS_ENQUEUE_CHAIN(mac_srs,
3913 3910 mp_chain, tail, cnt, sz);
3914 3911 /*
3915 3912 * Wakeup worker thread. Note that worker
3916 3913 * thread has to be woken up so that it
3917 3914 * can fire up the timer to be woken up
3918 3915 * on the next tick. Also once
3919 3916 * BW_ENFORCED is set, it can only be
3920 3917 * reset by srs_worker thread. Until then
3921 3918 * all packets will get queued up in SRS
3922 3919 * and hence this this code path won't be
3923 3920 * entered until BW_ENFORCED is reset.
3924 3921 */
3925 3922 cv_signal(&mac_srs->srs_async);
3926 3923 mutex_exit(&mac_srs->srs_lock);
3927 3924 return (cookie);
3928 3925 }
3929 3926
3930 3927 mac_srs->srs_bw->mac_bw_used += sz;
3931 3928 mutex_exit(&mac_srs->srs_lock);
3932 3929
3933 3930 if (srs_tx->st_mode == SRS_TX_BW_FANOUT) {
3934 3931 mac_soft_ring_t *softring;
3935 3932 uint_t indx, hash;
3936 3933
3937 3934 hash = HASH_HINT(fanout_hint);
3938 3935 indx = COMPUTE_INDEX(hash,
3939 3936 mac_srs->srs_tx_ring_count);
3940 3937 softring = mac_srs->srs_tx_soft_rings[indx];
3941 3938 return (mac_tx_soft_ring_process(softring, mp_chain, flag,
3942 3939 ret_mp));
3943 3940 } else if (srs_tx->st_mode == SRS_TX_BW_AGGR) {
3944 3941 return (mac_tx_aggr_mode(mac_srs, mp_chain,
3945 3942 fanout_hint, flag, ret_mp));
3946 3943 } else {
3947 3944 mac_tx_stats_t stats;
3948 3945
3949 3946 mp_chain = mac_tx_send(srs_tx->st_arg1, srs_tx->st_arg2,
3950 3947 mp_chain, &stats);
3951 3948
3952 3949 if (mp_chain != NULL) {
3953 3950 mutex_enter(&mac_srs->srs_lock);
3954 3951 MAC_COUNT_CHAIN(mac_srs, mp_chain, tail, cnt, sz);
3955 3952 if (mac_srs->srs_bw->mac_bw_used > sz)
3956 3953 mac_srs->srs_bw->mac_bw_used -= sz;
3957 3954 else
3958 3955 mac_srs->srs_bw->mac_bw_used = 0;
3959 3956 cookie = mac_tx_srs_enqueue(mac_srs, mp_chain, flag,
3960 3957 fanout_hint, ret_mp);
3961 3958 mutex_exit(&mac_srs->srs_lock);
3962 3959 return (cookie);
3963 3960 }
3964 3961 SRS_TX_STATS_UPDATE(mac_srs, &stats);
3965 3962
3966 3963 return (NULL);
3967 3964 }
3968 3965 }
3969 3966
3970 3967 /*
3971 3968 * mac_tx_aggr_mode
3972 3969 *
3973 3970 * This routine invokes an aggr function, aggr_find_tx_ring(), to find
3974 3971 * a (pseudo) Tx ring belonging to a port on which the packet has to
3975 3972 * be sent. aggr_find_tx_ring() first finds the outgoing port based on
3976 3973 * L2/L3/L4 policy and then uses the fanout_hint passed to it to pick
3977 3974 * a Tx ring from the selected port.
3978 3975 *
3979 3976 * Note that a port can be deleted from the aggregation. In such a case,
3980 3977 * the aggregation layer first separates the port from the rest of the
3981 3978 * ports making sure that port (and thus any Tx rings associated with
3982 3979 * it) won't get selected in the call to aggr_find_tx_ring() function.
3983 3980 * Later calls are made to mac_group_rem_ring() passing pseudo Tx ring
3984 3981 * handles one by one which in turn will quiesce the Tx SRS and remove
3985 3982 * the soft ring associated with the pseudo Tx ring. Unlike Rx side
3986 3983 * where a cookie is used to protect against mac_rx_ring() calls on
3987 3984 * rings that have been removed, no such cookie is needed on the Tx
3988 3985 * side as the pseudo Tx ring won't be available anymore to
3989 3986 * aggr_find_tx_ring() once the port has been removed.
3990 3987 */
3991 3988 static mac_tx_cookie_t
3992 3989 mac_tx_aggr_mode(mac_soft_ring_set_t *mac_srs, mblk_t *mp_chain,
3993 3990 uintptr_t fanout_hint, uint16_t flag, mblk_t **ret_mp)
3994 3991 {
3995 3992 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx;
3996 3993 mac_tx_ring_fn_t find_tx_ring_fn;
3997 3994 mac_ring_handle_t ring = NULL;
3998 3995 void *arg;
3999 3996 mac_soft_ring_t *sringp;
4000 3997
4001 3998 find_tx_ring_fn = srs_tx->st_capab_aggr.mca_find_tx_ring_fn;
4002 3999 arg = srs_tx->st_capab_aggr.mca_arg;
4003 4000 if (find_tx_ring_fn(arg, mp_chain, fanout_hint, &ring) == NULL)
4004 4001 return (NULL);
4005 4002 sringp = srs_tx->st_soft_rings[((mac_ring_t *)ring)->mr_index];
4006 4003 return (mac_tx_soft_ring_process(sringp, mp_chain, flag, ret_mp));
4007 4004 }
4008 4005
4009 4006 void
4010 4007 mac_tx_invoke_callbacks(mac_client_impl_t *mcip, mac_tx_cookie_t cookie)
4011 4008 {
4012 4009 mac_cb_t *mcb;
4013 4010 mac_tx_notify_cb_t *mtnfp;
4014 4011
4015 4012 /* Wakeup callback registered clients */
4016 4013 MAC_CALLBACK_WALKER_INC(&mcip->mci_tx_notify_cb_info);
4017 4014 for (mcb = mcip->mci_tx_notify_cb_list; mcb != NULL;
4018 4015 mcb = mcb->mcb_nextp) {
4019 4016 mtnfp = (mac_tx_notify_cb_t *)mcb->mcb_objp;
4020 4017 mtnfp->mtnf_fn(mtnfp->mtnf_arg, cookie);
4021 4018 }
4022 4019 MAC_CALLBACK_WALKER_DCR(&mcip->mci_tx_notify_cb_info,
4023 4020 &mcip->mci_tx_notify_cb_list);
4024 4021 }
4025 4022
4026 4023 /* ARGSUSED */
4027 4024 void
4028 4025 mac_tx_srs_drain(mac_soft_ring_set_t *mac_srs, uint_t proc_type)
4029 4026 {
4030 4027 mblk_t *head, *tail;
4031 4028 size_t sz;
4032 4029 uint32_t tx_mode;
4033 4030 uint_t saved_pkt_count;
4034 4031 mac_tx_stats_t stats;
4035 4032 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx;
4036 4033 clock_t now;
4037 4034
4038 4035 saved_pkt_count = 0;
4039 4036 ASSERT(mutex_owned(&mac_srs->srs_lock));
4040 4037 ASSERT(!(mac_srs->srs_state & SRS_PROC));
4041 4038
4042 4039 mac_srs->srs_state |= SRS_PROC;
4043 4040
4044 4041 tx_mode = srs_tx->st_mode;
4045 4042 if (tx_mode == SRS_TX_DEFAULT || tx_mode == SRS_TX_SERIALIZE) {
4046 4043 if (mac_srs->srs_first != NULL) {
4047 4044 head = mac_srs->srs_first;
4048 4045 tail = mac_srs->srs_last;
4049 4046 saved_pkt_count = mac_srs->srs_count;
4050 4047 mac_srs->srs_first = NULL;
4051 4048 mac_srs->srs_last = NULL;
4052 4049 mac_srs->srs_count = 0;
4053 4050 mutex_exit(&mac_srs->srs_lock);
4054 4051
4055 4052 head = mac_tx_send(srs_tx->st_arg1, srs_tx->st_arg2,
4056 4053 head, &stats);
4057 4054
4058 4055 mutex_enter(&mac_srs->srs_lock);
4059 4056 if (head != NULL) {
4060 4057 /* Device out of tx desc, set block */
4061 4058 if (head->b_next == NULL)
4062 4059 VERIFY(head == tail);
4063 4060 tail->b_next = mac_srs->srs_first;
4064 4061 mac_srs->srs_first = head;
4065 4062 mac_srs->srs_count +=
4066 4063 (saved_pkt_count - stats.mts_opackets);
4067 4064 if (mac_srs->srs_last == NULL)
4068 4065 mac_srs->srs_last = tail;
4069 4066 MAC_TX_SRS_BLOCK(mac_srs, head);
4070 4067 } else {
4071 4068 srs_tx->st_woken_up = B_FALSE;
4072 4069 SRS_TX_STATS_UPDATE(mac_srs, &stats);
4073 4070 }
4074 4071 }
4075 4072 } else if (tx_mode == SRS_TX_BW) {
4076 4073 /*
4077 4074 * We are here because the timer fired and we have some data
4078 4075 * to tranmit. Also mac_tx_srs_worker should have reset
4079 4076 * SRS_BW_ENFORCED flag
4080 4077 */
4081 4078 ASSERT(!(mac_srs->srs_bw->mac_bw_state & SRS_BW_ENFORCED));
4082 4079 head = tail = mac_srs->srs_first;
4083 4080 while (mac_srs->srs_first != NULL) {
4084 4081 tail = mac_srs->srs_first;
4085 4082 tail->b_prev = NULL;
4086 4083 mac_srs->srs_first = tail->b_next;
4087 4084 if (mac_srs->srs_first == NULL)
4088 4085 mac_srs->srs_last = NULL;
4089 4086 mac_srs->srs_count--;
4090 4087 sz = msgdsize(tail);
4091 4088 mac_srs->srs_size -= sz;
4092 4089 saved_pkt_count++;
4093 4090 MAC_TX_UPDATE_BW_INFO(mac_srs, sz);
4094 4091
4095 4092 if (mac_srs->srs_bw->mac_bw_used <
4096 4093 mac_srs->srs_bw->mac_bw_limit)
4097 4094 continue;
4098 4095
4099 4096 now = ddi_get_lbolt();
4100 4097 if (mac_srs->srs_bw->mac_bw_curr_time != now) {
4101 4098 mac_srs->srs_bw->mac_bw_curr_time = now;
4102 4099 mac_srs->srs_bw->mac_bw_used = sz;
4103 4100 continue;
4104 4101 }
4105 4102 mac_srs->srs_bw->mac_bw_state |= SRS_BW_ENFORCED;
4106 4103 break;
4107 4104 }
4108 4105
4109 4106 ASSERT((head == NULL && tail == NULL) ||
4110 4107 (head != NULL && tail != NULL));
4111 4108 if (tail != NULL) {
4112 4109 tail->b_next = NULL;
4113 4110 mutex_exit(&mac_srs->srs_lock);
4114 4111
4115 4112 head = mac_tx_send(srs_tx->st_arg1, srs_tx->st_arg2,
4116 4113 head, &stats);
4117 4114
4118 4115 mutex_enter(&mac_srs->srs_lock);
4119 4116 if (head != NULL) {
4120 4117 uint_t size_sent;
4121 4118
4122 4119 /* Device out of tx desc, set block */
4123 4120 if (head->b_next == NULL)
4124 4121 VERIFY(head == tail);
4125 4122 tail->b_next = mac_srs->srs_first;
4126 4123 mac_srs->srs_first = head;
4127 4124 mac_srs->srs_count +=
4128 4125 (saved_pkt_count - stats.mts_opackets);
4129 4126 if (mac_srs->srs_last == NULL)
4130 4127 mac_srs->srs_last = tail;
4131 4128 size_sent = sz - stats.mts_obytes;
4132 4129 mac_srs->srs_size += size_sent;
4133 4130 mac_srs->srs_bw->mac_bw_sz += size_sent;
4134 4131 if (mac_srs->srs_bw->mac_bw_used > size_sent) {
4135 4132 mac_srs->srs_bw->mac_bw_used -=
4136 4133 size_sent;
4137 4134 } else {
4138 4135 mac_srs->srs_bw->mac_bw_used = 0;
4139 4136 }
4140 4137 MAC_TX_SRS_BLOCK(mac_srs, head);
4141 4138 } else {
4142 4139 srs_tx->st_woken_up = B_FALSE;
4143 4140 SRS_TX_STATS_UPDATE(mac_srs, &stats);
4144 4141 }
4145 4142 }
4146 4143 } else if (tx_mode == SRS_TX_BW_FANOUT || tx_mode == SRS_TX_BW_AGGR) {
4147 4144 mblk_t *prev;
4148 4145 uint64_t hint;
4149 4146
4150 4147 /*
4151 4148 * We are here because the timer fired and we
4152 4149 * have some quota to tranmit.
4153 4150 */
4154 4151 prev = NULL;
4155 4152 head = tail = mac_srs->srs_first;
4156 4153 while (mac_srs->srs_first != NULL) {
4157 4154 tail = mac_srs->srs_first;
4158 4155 mac_srs->srs_first = tail->b_next;
4159 4156 if (mac_srs->srs_first == NULL)
4160 4157 mac_srs->srs_last = NULL;
4161 4158 mac_srs->srs_count--;
4162 4159 sz = msgdsize(tail);
4163 4160 mac_srs->srs_size -= sz;
4164 4161 mac_srs->srs_bw->mac_bw_used += sz;
4165 4162 if (prev == NULL)
4166 4163 hint = (ulong_t)tail->b_prev;
4167 4164 if (hint != (ulong_t)tail->b_prev) {
4168 4165 prev->b_next = NULL;
4169 4166 mutex_exit(&mac_srs->srs_lock);
4170 4167 TX_SRS_TO_SOFT_RING(mac_srs, head, hint);
4171 4168 head = tail;
4172 4169 hint = (ulong_t)tail->b_prev;
4173 4170 mutex_enter(&mac_srs->srs_lock);
4174 4171 }
4175 4172
4176 4173 prev = tail;
4177 4174 tail->b_prev = NULL;
4178 4175 if (mac_srs->srs_bw->mac_bw_used <
4179 4176 mac_srs->srs_bw->mac_bw_limit)
4180 4177 continue;
4181 4178
4182 4179 now = ddi_get_lbolt();
4183 4180 if (mac_srs->srs_bw->mac_bw_curr_time != now) {
4184 4181 mac_srs->srs_bw->mac_bw_curr_time = now;
4185 4182 mac_srs->srs_bw->mac_bw_used = 0;
4186 4183 continue;
4187 4184 }
4188 4185 mac_srs->srs_bw->mac_bw_state |= SRS_BW_ENFORCED;
4189 4186 break;
4190 4187 }
4191 4188 ASSERT((head == NULL && tail == NULL) ||
4192 4189 (head != NULL && tail != NULL));
4193 4190 if (tail != NULL) {
4194 4191 tail->b_next = NULL;
4195 4192 mutex_exit(&mac_srs->srs_lock);
4196 4193 TX_SRS_TO_SOFT_RING(mac_srs, head, hint);
4197 4194 mutex_enter(&mac_srs->srs_lock);
4198 4195 }
4199 4196 }
4200 4197 /*
4201 4198 * SRS_TX_FANOUT case not considered here because packets
4202 4199 * won't be queued in the SRS for this case. Packets will
4203 4200 * be sent directly to soft rings underneath and if there
4204 4201 * is any queueing at all, it would be in Tx side soft
4205 4202 * rings.
4206 4203 */
4207 4204
4208 4205 /*
4209 4206 * When srs_count becomes 0, reset SRS_TX_HIWAT and
4210 4207 * SRS_TX_WAKEUP_CLIENT and wakeup registered clients.
4211 4208 */
4212 4209 if (mac_srs->srs_count == 0 && (mac_srs->srs_state &
4213 4210 (SRS_TX_HIWAT | SRS_TX_WAKEUP_CLIENT | SRS_ENQUEUED))) {
4214 4211 mac_client_impl_t *mcip = mac_srs->srs_mcip;
4215 4212 boolean_t wakeup_required = B_FALSE;
4216 4213
4217 4214 if (mac_srs->srs_state &
4218 4215 (SRS_TX_HIWAT|SRS_TX_WAKEUP_CLIENT)) {
4219 4216 wakeup_required = B_TRUE;
4220 4217 }
4221 4218 mac_srs->srs_state &= ~(SRS_TX_HIWAT |
4222 4219 SRS_TX_WAKEUP_CLIENT | SRS_ENQUEUED);
4223 4220 mutex_exit(&mac_srs->srs_lock);
4224 4221 if (wakeup_required) {
4225 4222 mac_tx_invoke_callbacks(mcip, (mac_tx_cookie_t)mac_srs);
4226 4223 /*
4227 4224 * If the client is not the primary MAC client, then we
4228 4225 * need to send the notification to the clients upper
4229 4226 * MAC, i.e. mci_upper_mip.
4230 4227 */
4231 4228 mac_tx_notify(mcip->mci_upper_mip != NULL ?
4232 4229 mcip->mci_upper_mip : mcip->mci_mip);
4233 4230 }
4234 4231 mutex_enter(&mac_srs->srs_lock);
4235 4232 }
4236 4233 mac_srs->srs_state &= ~SRS_PROC;
4237 4234 }
4238 4235
4239 4236 /*
4240 4237 * Given a packet, get the flow_entry that identifies the flow
4241 4238 * to which that packet belongs. The flow_entry will contain
4242 4239 * the transmit function to be used to send the packet. If the
4243 4240 * function returns NULL, the packet should be sent using the
4244 4241 * underlying NIC.
4245 4242 */
4246 4243 static flow_entry_t *
4247 4244 mac_tx_classify(mac_impl_t *mip, mblk_t *mp)
4248 4245 {
4249 4246 flow_entry_t *flent = NULL;
4250 4247 mac_client_impl_t *mcip;
4251 4248 int err;
4252 4249
4253 4250 /*
4254 4251 * Do classification on the packet.
4255 4252 */
4256 4253 err = mac_flow_lookup(mip->mi_flow_tab, mp, FLOW_OUTBOUND, &flent);
4257 4254 if (err != 0)
4258 4255 return (NULL);
4259 4256
4260 4257 /*
4261 4258 * This flent might just be an additional one on the MAC client,
4262 4259 * i.e. for classification purposes (different fdesc), however
4263 4260 * the resources, SRS et. al., are in the mci_flent, so if
4264 4261 * this isn't the mci_flent, we need to get it.
4265 4262 */
4266 4263 if ((mcip = flent->fe_mcip) != NULL && mcip->mci_flent != flent) {
4267 4264 FLOW_REFRELE(flent);
4268 4265 flent = mcip->mci_flent;
4269 4266 FLOW_TRY_REFHOLD(flent, err);
4270 4267 if (err != 0)
4271 4268 return (NULL);
4272 4269 }
4273 4270
4274 4271 return (flent);
4275 4272 }
4276 4273
4277 4274 /*
4278 4275 * This macro is only meant to be used by mac_tx_send().
4279 4276 */
4280 4277 #define CHECK_VID_AND_ADD_TAG(mp) { \
4281 4278 if (vid_check) { \
4282 4279 int err = 0; \
4283 4280 \
4284 4281 MAC_VID_CHECK(src_mcip, (mp), err); \
4285 4282 if (err != 0) { \
4286 4283 freemsg((mp)); \
4287 4284 (mp) = next; \
4288 4285 oerrors++; \
4289 4286 continue; \
4290 4287 } \
4291 4288 } \
4292 4289 if (add_tag) { \
4293 4290 (mp) = mac_add_vlan_tag((mp), 0, vid); \
4294 4291 if ((mp) == NULL) { \
4295 4292 (mp) = next; \
4296 4293 oerrors++; \
4297 4294 continue; \
4298 4295 } \
4299 4296 } \
4300 4297 }
4301 4298
4302 4299 mblk_t *
4303 4300 mac_tx_send(mac_client_handle_t mch, mac_ring_handle_t ring, mblk_t *mp_chain,
4304 4301 mac_tx_stats_t *stats)
4305 4302 {
4306 4303 mac_client_impl_t *src_mcip = (mac_client_impl_t *)mch;
4307 4304 mac_impl_t *mip = src_mcip->mci_mip;
4308 4305 uint_t obytes = 0, opackets = 0, oerrors = 0;
4309 4306 mblk_t *mp = NULL, *next;
4310 4307 boolean_t vid_check, add_tag;
4311 4308 uint16_t vid = 0;
4312 4309
4313 4310 if (mip->mi_nclients > 1) {
4314 4311 vid_check = MAC_VID_CHECK_NEEDED(src_mcip);
4315 4312 add_tag = MAC_TAG_NEEDED(src_mcip);
4316 4313 if (add_tag)
4317 4314 vid = mac_client_vid(mch);
4318 4315 } else {
4319 4316 ASSERT(mip->mi_nclients == 1);
4320 4317 vid_check = add_tag = B_FALSE;
4321 4318 }
4322 4319
4323 4320 /*
4324 4321 * Fastpath: if there's only one client, we simply send
4325 4322 * the packet down to the underlying NIC.
4326 4323 */
4327 4324 if (mip->mi_nactiveclients == 1) {
4328 4325 DTRACE_PROBE2(fastpath,
4329 4326 mac_client_impl_t *, src_mcip, mblk_t *, mp_chain);
4330 4327
4331 4328 mp = mp_chain;
4332 4329 while (mp != NULL) {
4333 4330 next = mp->b_next;
4334 4331 mp->b_next = NULL;
4335 4332 opackets++;
4336 4333 obytes += (mp->b_cont == NULL ? MBLKL(mp) :
4337 4334 msgdsize(mp));
4338 4335
4339 4336 CHECK_VID_AND_ADD_TAG(mp);
4340 4337 MAC_TX(mip, ring, mp, src_mcip);
4341 4338
4342 4339 /*
4343 4340 * If the driver is out of descriptors and does a
4344 4341 * partial send it will return a chain of unsent
4345 4342 * mblks. Adjust the accounting stats.
4346 4343 */
4347 4344 if (mp != NULL) {
4348 4345 opackets--;
4349 4346 obytes -= msgdsize(mp);
4350 4347 mp->b_next = next;
4351 4348 break;
4352 4349 }
4353 4350 mp = next;
4354 4351 }
4355 4352 goto done;
4356 4353 }
4357 4354
4358 4355 /*
4359 4356 * No fastpath, we either have more than one MAC client
4360 4357 * defined on top of the same MAC, or one or more MAC
4361 4358 * client promiscuous callbacks.
4362 4359 */
4363 4360 DTRACE_PROBE3(slowpath, mac_client_impl_t *,
4364 4361 src_mcip, int, mip->mi_nclients, mblk_t *, mp_chain);
4365 4362
4366 4363 mp = mp_chain;
4367 4364 while (mp != NULL) {
4368 4365 flow_entry_t *dst_flow_ent;
4369 4366 void *flow_cookie;
4370 4367 size_t pkt_size;
4371 4368 mblk_t *mp1;
4372 4369
4373 4370 next = mp->b_next;
4374 4371 mp->b_next = NULL;
4375 4372 opackets++;
4376 4373 pkt_size = (mp->b_cont == NULL ? MBLKL(mp) : msgdsize(mp));
4377 4374 obytes += pkt_size;
4378 4375 CHECK_VID_AND_ADD_TAG(mp);
4379 4376
4380 4377 /*
4381 4378 * Find the destination.
4382 4379 */
4383 4380 dst_flow_ent = mac_tx_classify(mip, mp);
4384 4381
4385 4382 if (dst_flow_ent != NULL) {
4386 4383 size_t hdrsize;
4387 4384 int err = 0;
4388 4385
4389 4386 if (mip->mi_info.mi_nativemedia == DL_ETHER) {
4390 4387 struct ether_vlan_header *evhp =
4391 4388 (struct ether_vlan_header *)mp->b_rptr;
4392 4389
4393 4390 if (ntohs(evhp->ether_tpid) == ETHERTYPE_VLAN)
4394 4391 hdrsize = sizeof (*evhp);
4395 4392 else
4396 4393 hdrsize = sizeof (struct ether_header);
4397 4394 } else {
4398 4395 mac_header_info_t mhi;
4399 4396
4400 4397 err = mac_header_info((mac_handle_t)mip,
4401 4398 mp, &mhi);
4402 4399 if (err == 0)
4403 4400 hdrsize = mhi.mhi_hdrsize;
4404 4401 }
4405 4402
4406 4403 /*
4407 4404 * Got a matching flow. It's either another
4408 4405 * MAC client, or a broadcast/multicast flow.
4409 4406 * Make sure the packet size is within the
4410 4407 * allowed size. If not drop the packet and
4411 4408 * move to next packet.
4412 4409 */
4413 4410 if (err != 0 ||
4414 4411 (pkt_size - hdrsize) > mip->mi_sdu_max) {
4415 4412 oerrors++;
4416 4413 DTRACE_PROBE2(loopback__drop, size_t, pkt_size,
4417 4414 mblk_t *, mp);
4418 4415 freemsg(mp);
4419 4416 mp = next;
4420 4417 FLOW_REFRELE(dst_flow_ent);
4421 4418 continue;
4422 4419 }
4423 4420 flow_cookie = mac_flow_get_client_cookie(dst_flow_ent);
4424 4421 if (flow_cookie != NULL) {
4425 4422 /*
4426 4423 * The vnic_bcast_send function expects
4427 4424 * to receive the sender MAC client
4428 4425 * as value for arg2.
4429 4426 */
4430 4427 mac_bcast_send(flow_cookie, src_mcip, mp,
4431 4428 B_TRUE);
4432 4429 } else {
4433 4430 /*
4434 4431 * loopback the packet to a local MAC
4435 4432 * client. We force a context switch
4436 4433 * if both source and destination MAC
4437 4434 * clients are used by IP, i.e.
4438 4435 * bypass is set.
4439 4436 */
4440 4437 boolean_t do_switch;
4441 4438 mac_client_impl_t *dst_mcip =
4442 4439 dst_flow_ent->fe_mcip;
4443 4440
4444 4441 /*
4445 4442 * Check if there are promiscuous mode
4446 4443 * callbacks defined. This check is
4447 4444 * done here in the 'else' case and
4448 4445 * not in other cases because this
4449 4446 * path is for local loopback
4450 4447 * communication which does not go
4451 4448 * through MAC_TX(). For paths that go
4452 4449 * through MAC_TX(), the promisc_list
4453 4450 * check is done inside the MAC_TX()
4454 4451 * macro.
4455 4452 */
4456 4453 if (mip->mi_promisc_list != NULL)
4457 4454 mac_promisc_dispatch(mip, mp, src_mcip);
4458 4455
4459 4456 do_switch = ((src_mcip->mci_state_flags &
4460 4457 dst_mcip->mci_state_flags &
4461 4458 MCIS_CLIENT_POLL_CAPABLE) != 0);
4462 4459
4463 4460 if ((mp1 = mac_fix_cksum(mp)) != NULL) {
4464 4461 (dst_flow_ent->fe_cb_fn)(
4465 4462 dst_flow_ent->fe_cb_arg1,
4466 4463 dst_flow_ent->fe_cb_arg2,
4467 4464 mp1, do_switch);
4468 4465 }
4469 4466 }
4470 4467 FLOW_REFRELE(dst_flow_ent);
4471 4468 } else {
4472 4469 /*
4473 4470 * Unknown destination, send via the underlying
4474 4471 * NIC.
4475 4472 */
4476 4473 MAC_TX(mip, ring, mp, src_mcip);
4477 4474 if (mp != NULL) {
4478 4475 /*
4479 4476 * Adjust for the last packet that
4480 4477 * could not be transmitted
4481 4478 */
4482 4479 opackets--;
4483 4480 obytes -= pkt_size;
4484 4481 mp->b_next = next;
4485 4482 break;
4486 4483 }
4487 4484 }
4488 4485 mp = next;
4489 4486 }
4490 4487
4491 4488 done:
4492 4489 stats->mts_obytes = obytes;
4493 4490 stats->mts_opackets = opackets;
4494 4491 stats->mts_oerrors = oerrors;
4495 4492 return (mp);
4496 4493 }
4497 4494
4498 4495 /*
4499 4496 * mac_tx_srs_ring_present
4500 4497 *
4501 4498 * Returns whether the specified ring is part of the specified SRS.
4502 4499 */
4503 4500 boolean_t
4504 4501 mac_tx_srs_ring_present(mac_soft_ring_set_t *srs, mac_ring_t *tx_ring)
4505 4502 {
4506 4503 int i;
4507 4504 mac_soft_ring_t *soft_ring;
4508 4505
4509 4506 if (srs->srs_tx.st_arg2 == tx_ring)
4510 4507 return (B_TRUE);
4511 4508
4512 4509 for (i = 0; i < srs->srs_tx_ring_count; i++) {
4513 4510 soft_ring = srs->srs_tx_soft_rings[i];
4514 4511 if (soft_ring->s_ring_tx_arg2 == tx_ring)
4515 4512 return (B_TRUE);
4516 4513 }
4517 4514
4518 4515 return (B_FALSE);
4519 4516 }
4520 4517
4521 4518 /*
4522 4519 * mac_tx_srs_get_soft_ring
4523 4520 *
4524 4521 * Returns the TX soft ring associated with the given ring, if present.
4525 4522 */
4526 4523 mac_soft_ring_t *
4527 4524 mac_tx_srs_get_soft_ring(mac_soft_ring_set_t *srs, mac_ring_t *tx_ring)
4528 4525 {
4529 4526 int i;
4530 4527 mac_soft_ring_t *soft_ring;
4531 4528
4532 4529 if (srs->srs_tx.st_arg2 == tx_ring)
4533 4530 return (NULL);
4534 4531
4535 4532 for (i = 0; i < srs->srs_tx_ring_count; i++) {
4536 4533 soft_ring = srs->srs_tx_soft_rings[i];
4537 4534 if (soft_ring->s_ring_tx_arg2 == tx_ring)
4538 4535 return (soft_ring);
4539 4536 }
4540 4537
4541 4538 return (NULL);
4542 4539 }
4543 4540
4544 4541 /*
4545 4542 * mac_tx_srs_wakeup
4546 4543 *
4547 4544 * Called when Tx desc become available. Wakeup the appropriate worker
4548 4545 * thread after resetting the SRS_TX_BLOCKED/S_RING_BLOCK bit in the
4549 4546 * state field.
4550 4547 */
4551 4548 void
4552 4549 mac_tx_srs_wakeup(mac_soft_ring_set_t *mac_srs, mac_ring_handle_t ring)
4553 4550 {
4554 4551 int i;
4555 4552 mac_soft_ring_t *sringp;
4556 4553 mac_srs_tx_t *srs_tx = &mac_srs->srs_tx;
4557 4554
4558 4555 mutex_enter(&mac_srs->srs_lock);
4559 4556 /*
4560 4557 * srs_tx_ring_count == 0 is the single ring mode case. In
4561 4558 * this mode, there will not be Tx soft rings associated
4562 4559 * with the SRS.
4563 4560 */
4564 4561 if (!MAC_TX_SOFT_RINGS(mac_srs)) {
4565 4562 if (srs_tx->st_arg2 == ring &&
4566 4563 mac_srs->srs_state & SRS_TX_BLOCKED) {
4567 4564 mac_srs->srs_state &= ~SRS_TX_BLOCKED;
4568 4565 srs_tx->st_stat.mts_unblockcnt++;
4569 4566 cv_signal(&mac_srs->srs_async);
4570 4567 }
4571 4568 /*
4572 4569 * A wakeup can come before tx_srs_drain() could
4573 4570 * grab srs lock and set SRS_TX_BLOCKED. So
4574 4571 * always set woken_up flag when we come here.
4575 4572 */
4576 4573 srs_tx->st_woken_up = B_TRUE;
4577 4574 mutex_exit(&mac_srs->srs_lock);
4578 4575 return;
4579 4576 }
4580 4577
4581 4578 /*
4582 4579 * If you are here, it is for FANOUT, BW_FANOUT,
4583 4580 * AGGR_MODE or AGGR_BW_MODE case
4584 4581 */
4585 4582 for (i = 0; i < mac_srs->srs_tx_ring_count; i++) {
4586 4583 sringp = mac_srs->srs_tx_soft_rings[i];
4587 4584 mutex_enter(&sringp->s_ring_lock);
4588 4585 if (sringp->s_ring_tx_arg2 == ring) {
4589 4586 if (sringp->s_ring_state & S_RING_BLOCK) {
4590 4587 sringp->s_ring_state &= ~S_RING_BLOCK;
4591 4588 sringp->s_st_stat.mts_unblockcnt++;
4592 4589 cv_signal(&sringp->s_ring_async);
4593 4590 }
4594 4591 sringp->s_ring_tx_woken_up = B_TRUE;
4595 4592 }
4596 4593 mutex_exit(&sringp->s_ring_lock);
4597 4594 }
4598 4595 mutex_exit(&mac_srs->srs_lock);
4599 4596 }
4600 4597
4601 4598 /*
4602 4599 * Once the driver is done draining, send a MAC_NOTE_TX notification to unleash
4603 4600 * the blocked clients again.
4604 4601 */
4605 4602 void
4606 4603 mac_tx_notify(mac_impl_t *mip)
4607 4604 {
4608 4605 i_mac_notify(mip, MAC_NOTE_TX);
4609 4606 }
4610 4607
4611 4608 /*
4612 4609 * RX SOFTRING RELATED FUNCTIONS
4613 4610 *
4614 4611 * These functions really belong in mac_soft_ring.c and here for
4615 4612 * a short period.
4616 4613 */
4617 4614
4618 4615 #define SOFT_RING_ENQUEUE_CHAIN(ringp, mp, tail, cnt, sz) { \
4619 4616 /* \
4620 4617 * Enqueue our mblk chain. \
4621 4618 */ \
4622 4619 ASSERT(MUTEX_HELD(&(ringp)->s_ring_lock)); \
4623 4620 \
4624 4621 if ((ringp)->s_ring_last != NULL) \
4625 4622 (ringp)->s_ring_last->b_next = (mp); \
4626 4623 else \
4627 4624 (ringp)->s_ring_first = (mp); \
4628 4625 (ringp)->s_ring_last = (tail); \
4629 4626 (ringp)->s_ring_count += (cnt); \
4630 4627 ASSERT((ringp)->s_ring_count > 0); \
4631 4628 if ((ringp)->s_ring_type & ST_RING_BW_CTL) { \
4632 4629 (ringp)->s_ring_size += sz; \
4633 4630 } \
4634 4631 }
4635 4632
4636 4633 /*
4637 4634 * Default entry point to deliver a packet chain to a MAC client.
4638 4635 * If the MAC client has flows, do the classification with these
4639 4636 * flows as well.
4640 4637 */
4641 4638 /* ARGSUSED */
4642 4639 void
4643 4640 mac_rx_deliver(void *arg1, mac_resource_handle_t mrh, mblk_t *mp_chain,
4644 4641 mac_header_info_t *arg3)
4645 4642 {
4646 4643 mac_client_impl_t *mcip = arg1;
4647 4644
4648 4645 if (mcip->mci_nvids == 1 &&
4649 4646 !(mcip->mci_state_flags & MCIS_STRIP_DISABLE)) {
4650 4647 /*
4651 4648 * If the client has exactly one VID associated with it
4652 4649 * and striping of VLAN header is not disabled,
4653 4650 * remove the VLAN tag from the packet before
4654 4651 * passing it on to the client's receive callback.
4655 4652 * Note that this needs to be done after we dispatch
4656 4653 * the packet to the promiscuous listeners of the
4657 4654 * client, since they expect to see the whole
4658 4655 * frame including the VLAN headers.
4659 4656 */
4660 4657 mp_chain = mac_strip_vlan_tag_chain(mp_chain);
4661 4658 }
4662 4659
4663 4660 mcip->mci_rx_fn(mcip->mci_rx_arg, mrh, mp_chain, B_FALSE);
4664 4661 }
4665 4662
4666 4663 /*
4667 4664 * mac_rx_soft_ring_process
4668 4665 *
4669 4666 * process a chain for a given soft ring. The number of packets queued
4670 4667 * in the SRS and its associated soft rings (including this one) is
4671 4668 * very small (tracked by srs_poll_pkt_cnt), then allow the entering
4672 4669 * thread (interrupt or poll thread) to do inline processing. This
4673 4670 * helps keep the latency down under low load.
4674 4671 *
4675 4672 * The proc and arg for each mblk is already stored in the mblk in
4676 4673 * appropriate places.
4677 4674 */
4678 4675 /* ARGSUSED */
4679 4676 void
4680 4677 mac_rx_soft_ring_process(mac_client_impl_t *mcip, mac_soft_ring_t *ringp,
4681 4678 mblk_t *mp_chain, mblk_t *tail, int cnt, size_t sz)
4682 4679 {
4683 4680 mac_direct_rx_t proc;
4684 4681 void *arg1;
4685 4682 mac_resource_handle_t arg2;
4686 4683 mac_soft_ring_set_t *mac_srs = ringp->s_ring_set;
4687 4684
4688 4685 ASSERT(ringp != NULL);
4689 4686 ASSERT(mp_chain != NULL);
4690 4687 ASSERT(tail != NULL);
4691 4688 ASSERT(MUTEX_NOT_HELD(&ringp->s_ring_lock));
4692 4689
4693 4690 mutex_enter(&ringp->s_ring_lock);
4694 4691 ringp->s_ring_total_inpkt += cnt;
4695 4692 ringp->s_ring_total_rbytes += sz;
4696 4693 if ((mac_srs->srs_rx.sr_poll_pkt_cnt <= 1) &&
4697 4694 !(ringp->s_ring_type & ST_RING_WORKER_ONLY)) {
4698 4695 /* If on processor or blanking on, then enqueue and return */
4699 4696 if (ringp->s_ring_state & S_RING_BLANK ||
4700 4697 ringp->s_ring_state & S_RING_PROC) {
4701 4698 SOFT_RING_ENQUEUE_CHAIN(ringp, mp_chain, tail, cnt, sz);
4702 4699 mutex_exit(&ringp->s_ring_lock);
4703 4700 return;
4704 4701 }
4705 4702 proc = ringp->s_ring_rx_func;
4706 4703 arg1 = ringp->s_ring_rx_arg1;
4707 4704 arg2 = ringp->s_ring_rx_arg2;
4708 4705 /*
4709 4706 * See if anything is already queued. If we are the
4710 4707 * first packet, do inline processing else queue the
4711 4708 * packet and do the drain.
4712 4709 */
4713 4710 if (ringp->s_ring_first == NULL) {
4714 4711 /*
4715 4712 * Fast-path, ok to process and nothing queued.
4716 4713 */
4717 4714 ringp->s_ring_run = curthread;
4718 4715 ringp->s_ring_state |= (S_RING_PROC);
4719 4716
4720 4717 mutex_exit(&ringp->s_ring_lock);
4721 4718
4722 4719 /*
4723 4720 * We are the chain of 1 packet so
4724 4721 * go through this fast path.
4725 4722 */
4726 4723 ASSERT(mp_chain->b_next == NULL);
4727 4724
4728 4725 (*proc)(arg1, arg2, mp_chain, NULL);
4729 4726
4730 4727 ASSERT(MUTEX_NOT_HELD(&ringp->s_ring_lock));
4731 4728 /*
4732 4729 * If we have a soft ring set which is doing
4733 4730 * bandwidth control, we need to decrement
4734 4731 * srs_size and count so it the SRS can have a
4735 4732 * accurate idea of what is the real data
4736 4733 * queued between SRS and its soft rings. We
4737 4734 * decrement the counters only when the packet
4738 4735 * gets processed by both SRS and the soft ring.
4739 4736 */
4740 4737 mutex_enter(&mac_srs->srs_lock);
4741 4738 MAC_UPDATE_SRS_COUNT_LOCKED(mac_srs, cnt);
4742 4739 MAC_UPDATE_SRS_SIZE_LOCKED(mac_srs, sz);
4743 4740 mutex_exit(&mac_srs->srs_lock);
4744 4741
4745 4742 mutex_enter(&ringp->s_ring_lock);
4746 4743 ringp->s_ring_run = NULL;
4747 4744 ringp->s_ring_state &= ~S_RING_PROC;
4748 4745 if (ringp->s_ring_state & S_RING_CLIENT_WAIT)
4749 4746 cv_signal(&ringp->s_ring_client_cv);
4750 4747
4751 4748 if ((ringp->s_ring_first == NULL) ||
4752 4749 (ringp->s_ring_state & S_RING_BLANK)) {
4753 4750 /*
4754 4751 * We processed inline our packet and
4755 4752 * nothing new has arrived or our
4756 4753 * receiver doesn't want to receive
4757 4754 * any packets. We are done.
4758 4755 */
4759 4756 mutex_exit(&ringp->s_ring_lock);
4760 4757 return;
4761 4758 }
4762 4759 } else {
4763 4760 SOFT_RING_ENQUEUE_CHAIN(ringp,
4764 4761 mp_chain, tail, cnt, sz);
4765 4762 }
4766 4763
4767 4764 /*
4768 4765 * We are here because either we couldn't do inline
4769 4766 * processing (because something was already
4770 4767 * queued), or we had a chain of more than one
4771 4768 * packet, or something else arrived after we were
4772 4769 * done with inline processing.
4773 4770 */
4774 4771 ASSERT(MUTEX_HELD(&ringp->s_ring_lock));
4775 4772 ASSERT(ringp->s_ring_first != NULL);
4776 4773
4777 4774 ringp->s_ring_drain_func(ringp);
4778 4775 mutex_exit(&ringp->s_ring_lock);
4779 4776 return;
4780 4777 } else {
4781 4778 /* ST_RING_WORKER_ONLY case */
4782 4779 SOFT_RING_ENQUEUE_CHAIN(ringp, mp_chain, tail, cnt, sz);
4783 4780 mac_soft_ring_worker_wakeup(ringp);
4784 4781 mutex_exit(&ringp->s_ring_lock);
4785 4782 }
4786 4783 }
4787 4784
4788 4785 /*
4789 4786 * TX SOFTRING RELATED FUNCTIONS
4790 4787 *
4791 4788 * These functions really belong in mac_soft_ring.c and here for
4792 4789 * a short period.
4793 4790 */
4794 4791
4795 4792 #define TX_SOFT_RING_ENQUEUE_CHAIN(ringp, mp, tail, cnt, sz) { \
4796 4793 ASSERT(MUTEX_HELD(&ringp->s_ring_lock)); \
4797 4794 ringp->s_ring_state |= S_RING_ENQUEUED; \
4798 4795 SOFT_RING_ENQUEUE_CHAIN(ringp, mp_chain, tail, cnt, sz); \
4799 4796 }
4800 4797
4801 4798 /*
4802 4799 * mac_tx_sring_queued
4803 4800 *
4804 4801 * When we are out of transmit descriptors and we already have a
4805 4802 * queue that exceeds hiwat (or the client called us with
4806 4803 * MAC_TX_NO_ENQUEUE or MAC_DROP_ON_NO_DESC flag), return the
4807 4804 * soft ring pointer as the opaque cookie for the client enable
4808 4805 * flow control.
4809 4806 */
4810 4807 static mac_tx_cookie_t
4811 4808 mac_tx_sring_enqueue(mac_soft_ring_t *ringp, mblk_t *mp_chain, uint16_t flag,
4812 4809 mblk_t **ret_mp)
4813 4810 {
4814 4811 int cnt;
4815 4812 size_t sz;
4816 4813 mblk_t *tail;
4817 4814 mac_soft_ring_set_t *mac_srs = ringp->s_ring_set;
4818 4815 mac_tx_cookie_t cookie = NULL;
4819 4816 boolean_t wakeup_worker = B_TRUE;
4820 4817
4821 4818 ASSERT(MUTEX_HELD(&ringp->s_ring_lock));
4822 4819 MAC_COUNT_CHAIN(mac_srs, mp_chain, tail, cnt, sz);
4823 4820 if (flag & MAC_DROP_ON_NO_DESC) {
4824 4821 mac_pkt_drop(NULL, NULL, mp_chain, B_FALSE);
4825 4822 /* increment freed stats */
4826 4823 ringp->s_ring_drops += cnt;
4827 4824 cookie = (mac_tx_cookie_t)ringp;
4828 4825 } else {
4829 4826 if (ringp->s_ring_first != NULL)
4830 4827 wakeup_worker = B_FALSE;
4831 4828
4832 4829 if (flag & MAC_TX_NO_ENQUEUE) {
4833 4830 /*
4834 4831 * If QUEUED is not set, queue the packet
4835 4832 * and let mac_tx_soft_ring_drain() set
4836 4833 * the TX_BLOCKED bit for the reasons
4837 4834 * explained above. Otherwise, return the
4838 4835 * mblks.
4839 4836 */
4840 4837 if (wakeup_worker) {
4841 4838 TX_SOFT_RING_ENQUEUE_CHAIN(ringp,
4842 4839 mp_chain, tail, cnt, sz);
4843 4840 } else {
4844 4841 ringp->s_ring_state |= S_RING_WAKEUP_CLIENT;
4845 4842 cookie = (mac_tx_cookie_t)ringp;
4846 4843 *ret_mp = mp_chain;
4847 4844 }
4848 4845 } else {
4849 4846 boolean_t enqueue = B_TRUE;
4850 4847
4851 4848 if (ringp->s_ring_count > ringp->s_ring_tx_hiwat) {
4852 4849 /*
4853 4850 * flow-controlled. Store ringp in cookie
4854 4851 * so that it can be returned as
4855 4852 * mac_tx_cookie_t to client
4856 4853 */
4857 4854 ringp->s_ring_state |= S_RING_TX_HIWAT;
4858 4855 cookie = (mac_tx_cookie_t)ringp;
4859 4856 ringp->s_ring_hiwat_cnt++;
4860 4857 if (ringp->s_ring_count >
4861 4858 ringp->s_ring_tx_max_q_cnt) {
4862 4859 /* increment freed stats */
4863 4860 ringp->s_ring_drops += cnt;
4864 4861 /*
4865 4862 * b_prev may be set to the fanout hint
4866 4863 * hence can't use freemsg directly
4867 4864 */
4868 4865 mac_pkt_drop(NULL, NULL,
4869 4866 mp_chain, B_FALSE);
4870 4867 DTRACE_PROBE1(tx_queued_hiwat,
4871 4868 mac_soft_ring_t *, ringp);
4872 4869 enqueue = B_FALSE;
4873 4870 }
4874 4871 }
4875 4872 if (enqueue) {
4876 4873 TX_SOFT_RING_ENQUEUE_CHAIN(ringp, mp_chain,
4877 4874 tail, cnt, sz);
4878 4875 }
4879 4876 }
4880 4877 if (wakeup_worker)
4881 4878 cv_signal(&ringp->s_ring_async);
4882 4879 }
4883 4880 return (cookie);
4884 4881 }
4885 4882
4886 4883
4887 4884 /*
4888 4885 * mac_tx_soft_ring_process
4889 4886 *
4890 4887 * This routine is called when fanning out outgoing traffic among
4891 4888 * multipe Tx rings.
4892 4889 * Note that a soft ring is associated with a h/w Tx ring.
4893 4890 */
4894 4891 mac_tx_cookie_t
4895 4892 mac_tx_soft_ring_process(mac_soft_ring_t *ringp, mblk_t *mp_chain,
4896 4893 uint16_t flag, mblk_t **ret_mp)
4897 4894 {
4898 4895 mac_soft_ring_set_t *mac_srs = ringp->s_ring_set;
4899 4896 int cnt;
4900 4897 size_t sz;
4901 4898 mblk_t *tail;
4902 4899 mac_tx_cookie_t cookie = NULL;
4903 4900
4904 4901 ASSERT(ringp != NULL);
4905 4902 ASSERT(mp_chain != NULL);
4906 4903 ASSERT(MUTEX_NOT_HELD(&ringp->s_ring_lock));
4907 4904 /*
4908 4905 * The following modes can come here: SRS_TX_BW_FANOUT,
4909 4906 * SRS_TX_FANOUT, SRS_TX_AGGR, SRS_TX_BW_AGGR.
4910 4907 */
4911 4908 ASSERT(MAC_TX_SOFT_RINGS(mac_srs));
4912 4909 ASSERT(mac_srs->srs_tx.st_mode == SRS_TX_FANOUT ||
4913 4910 mac_srs->srs_tx.st_mode == SRS_TX_BW_FANOUT ||
4914 4911 mac_srs->srs_tx.st_mode == SRS_TX_AGGR ||
4915 4912 mac_srs->srs_tx.st_mode == SRS_TX_BW_AGGR);
4916 4913
4917 4914 if (ringp->s_ring_type & ST_RING_WORKER_ONLY) {
4918 4915 /* Serialization mode */
4919 4916
4920 4917 mutex_enter(&ringp->s_ring_lock);
4921 4918 if (ringp->s_ring_count > ringp->s_ring_tx_hiwat) {
4922 4919 cookie = mac_tx_sring_enqueue(ringp, mp_chain,
4923 4920 flag, ret_mp);
4924 4921 mutex_exit(&ringp->s_ring_lock);
4925 4922 return (cookie);
4926 4923 }
4927 4924 MAC_COUNT_CHAIN(mac_srs, mp_chain, tail, cnt, sz);
4928 4925 TX_SOFT_RING_ENQUEUE_CHAIN(ringp, mp_chain, tail, cnt, sz);
4929 4926 if (ringp->s_ring_state & (S_RING_BLOCK | S_RING_PROC)) {
4930 4927 /*
4931 4928 * If ring is blocked due to lack of Tx
4932 4929 * descs, just return. Worker thread
4933 4930 * will get scheduled when Tx desc's
4934 4931 * become available.
4935 4932 */
4936 4933 mutex_exit(&ringp->s_ring_lock);
4937 4934 return (cookie);
4938 4935 }
4939 4936 mac_soft_ring_worker_wakeup(ringp);
4940 4937 mutex_exit(&ringp->s_ring_lock);
4941 4938 return (cookie);
4942 4939 } else {
4943 4940 /* Default fanout mode */
4944 4941 /*
4945 4942 * S_RING_BLOCKED is set when underlying NIC runs
4946 4943 * out of Tx descs and messages start getting
4947 4944 * queued. It won't get reset until
4948 4945 * tx_srs_drain() completely drains out the
4949 4946 * messages.
4950 4947 */
4951 4948 mac_tx_stats_t stats;
4952 4949
4953 4950 if (ringp->s_ring_state & S_RING_ENQUEUED) {
4954 4951 /* Tx descs/resources not available */
4955 4952 mutex_enter(&ringp->s_ring_lock);
4956 4953 if (ringp->s_ring_state & S_RING_ENQUEUED) {
4957 4954 cookie = mac_tx_sring_enqueue(ringp, mp_chain,
4958 4955 flag, ret_mp);
4959 4956 mutex_exit(&ringp->s_ring_lock);
4960 4957 return (cookie);
4961 4958 }
4962 4959 /*
4963 4960 * While we were computing mblk count, the
4964 4961 * flow control condition got relieved.
4965 4962 * Continue with the transmission.
4966 4963 */
4967 4964 mutex_exit(&ringp->s_ring_lock);
4968 4965 }
4969 4966
4970 4967 mp_chain = mac_tx_send(ringp->s_ring_tx_arg1,
4971 4968 ringp->s_ring_tx_arg2, mp_chain, &stats);
4972 4969
4973 4970 /*
4974 4971 * Multiple threads could be here sending packets.
4975 4972 * Under such conditions, it is not possible to
4976 4973 * automically set S_RING_BLOCKED bit to indicate
4977 4974 * out of tx desc condition. To atomically set
4978 4975 * this, we queue the returned packet and do
4979 4976 * the setting of S_RING_BLOCKED in
4980 4977 * mac_tx_soft_ring_drain().
4981 4978 */
4982 4979 if (mp_chain != NULL) {
4983 4980 mutex_enter(&ringp->s_ring_lock);
4984 4981 cookie =
4985 4982 mac_tx_sring_enqueue(ringp, mp_chain, flag, ret_mp);
4986 4983 mutex_exit(&ringp->s_ring_lock);
4987 4984 return (cookie);
4988 4985 }
4989 4986 SRS_TX_STATS_UPDATE(mac_srs, &stats);
4990 4987 SOFTRING_TX_STATS_UPDATE(ringp, &stats);
4991 4988
4992 4989 return (NULL);
4993 4990 }
4994 4991 }
|
↓ open down ↓ |
3188 lines elided |
↑ open up ↑ |
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX