Posts‎ > ‎

Catalyst 2960/3560/3750 Output Drops (Part 1)

posted Jul 12, 2014, 1:23 PM by Marc Kerscher   [ updated Jul 12, 2014, 4:19 PM ]
As of late I have been researching output drops and trying to understand them on the switches that I'm responsible for. If you have ever GOOGL'ed the Output drops, you get a lot of links, but none seem to really explain why these are occurring. Example below:

Switch#sh inter fa0/7
FastEthernet0/7 is up, line protocol is up (connected)
  Hardware is Fast Ethernet, address is 9c4e.2087.3d88 (bia 9c4e.2087.3d88)
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 11/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, media type is 10/100BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:01:44
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 148


And the biggest questions is why? Especially on a LAN switch, where utilization is normally low.

Below is my topology:
(Note: the 3560 has a basic configuration on it, basically enabling MLS QOS)



Initially I was trying to use IPERF or IPERF3 (TCP and UDP) trying to cause these output drops, but unless I was over subscribing the links (trying to push 200 mbps into a 100 mbps port) I had no luck, but this is not really the source of the problem as port utilization was normally low. Finally I used the best bandwidth grabbing application out there ( :-) ) also known as FTP. So I installed an FTP server on the Ubuntu PC and generated a 20 MB files. Low and behold just trying to download the 20 MB file, I generated these errors on Fa0/7. Further testing using a generated 10 MB and 5 MB file also generated these errors. For verification I did the test several times, table below:

 Drops / File Size Try 1Try 2 Try 3 
 5 MB12    11 12
 10 MB 26 26 25
 20 MB 53 51 46

As you can see it does not need to be a large file to be generating the drops and what application file is at minimum 5 MBs today. Even worse is that the packet gets to the local switch and drops at the last hop, wasting a lot of bandwidth along the way. Looking further into the switch shows the cause:

Switch#sh inter fa0/7
FastEthernet0/7 is up, line protocol is up (connected)
  Hardware is Fast Ethernet, address is 9c4e.2087.3d88 (bia 9c4e.2087.3d88)
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, media type is 10/100BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:01, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 54
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  30 second input rate 3000 bits/sec, 6 packets/sec
  30 second output rate 209000 bits/sec, 16 packets/sec
     7003 packets input, 501972 bytes, 0 no buffer
     Received 46 broadcasts (45 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 45 multicast, 0 pause input
     0 input packets with dribble condition detected
     14705 packets output, 21042722 bytes, 0 underruns

Switch#sh controller ethernet-controller port-asic st
===========================================================================
 Switch 1, PortASIC 0 Statistics
---------------------------------------------------------------------------
         0 RxQ-0, wt-0 enqueue frames            0 RxQ-0, wt-0 drop frames
     15671 RxQ-0, wt-1 enqueue frames            0 RxQ-0, wt-1 drop frames
         0 RxQ-0, wt-2 enqueue frames            0 RxQ-0, wt-2 drop frames

         0 RxQ-1, wt-0 enqueue frames            0 RxQ-1, wt-0 drop frames
        17 RxQ-1, wt-1 enqueue frames            0 RxQ-1, wt-1 drop frames
      1401 RxQ-1, wt-2 enqueue frames            0 RxQ-1, wt-2 drop frames

     21343 RxQ-2, wt-0 enqueue frames            0 RxQ-2, wt-0 drop frames
         0 RxQ-2, wt-1 enqueue frames            0 RxQ-2, wt-1 drop frames
        45 RxQ-2, wt-2 enqueue frames            0 RxQ-2, wt-2 drop frames

         0 RxQ-3, wt-0 enqueue frames            0 RxQ-3, wt-0 drop frames
         0 RxQ-3, wt-1 enqueue frames            0 RxQ-3, wt-1 drop frames
         0 RxQ-3, wt-2 enqueue frames            0 RxQ-3, wt-2 drop frames


        54 TxQueue Drop Stats
         0 TxBufferFull Drop Count               0 Rx Fcs Error Frames
         0 TxBufferFrameDesc BadCrc16            0 Rx Invalid Oversize Frames
 

Switch#sh platform port-asic stats nonzero-drop

Port-asic Port Drop Statistics - Summary
========================================

  Port  9 TxQueue Drop Stats: 54

  Supervisor TxQueue Drop Statistics

Port-asic Port Drop Statistics - Details
========================================
  RxQueue Drop Statistics

  Port 0 TxQueue Drop Statistics

  Port 1 TxQueue Drop Statistics

  Port 2 TxQueue Drop Statistics

  Port 3 TxQueue Drop Statistics

  Port 4 TxQueue Drop Statistics

  Port 5 TxQueue Drop Statistics

  Port 6 TxQueue Drop Statistics

  Port 7 TxQueue Drop Statistics

  Port 8 TxQueue Drop Statistics

  Port 9 TxQueue Drop Statistics
    Queue 1
      Weight 0 Frames 54

All of the output from the commands point to Transmit Queue drops. As everyone knows the Catalyst switches have 4 egress queues. But a little less know is the fact that there are software queues that are inserted before the packets are put out on the wire. At this point I have not been able to actually find out the amount of physical memory assigned to each queue, but from the tests it does not seem to be a whole lot. So when traffic comes in from a gigabit port and goes into a 100 megabit port, these errors seem to occur to adjust for the speed difference.

In the second part of this article I will go into adjustments that can be made to reduce/eliminate these kinds of errors.