CCIE Wireless Techtorial v3: TECCCIE-3106

posted Nov 5, 2015, 6:17 AM by Marc Kerscher   [ updated Nov 5, 2015, 6:18 AM ]

I have been looking for this document for a while for my studies.
 Using some guessing I have found it:

Catalyst 2960/3560/3750 Output Drops (Part 2)

posted Jul 12, 2014, 5:34 PM by Marc Kerscher   [ updated Jul 12, 2014, 5:42 PM ]

This is the second part of the article, trying to adjust buffers to remove the output drops and prevent re-transmits. A bit of trial and error will be required to make it work. Let's start of with the settings generated by AUTO-QOS:

mls qos queue-set output 1 threshold 1 100 100 50 200
mls qos queue-set output 1 threshold 2 125 125 100 400
mls qos queue-set output 1 threshold 3 100 100 100 400
mls qos queue-set output 1 threshold 4 60 150 50 200
mls qos queue-set output 1 buffers 15 25 40 20

interface FastEthernet0/6
 srr-queue bandwidth share 1 30 35 5
 priority-queue out

This is Cisco's best effort as to guess the best QOS settings (baseline), now depending on whether you are using 4 / 8 / 12 qos model, these will all not fit into the AUTO-QOS and changes will need to be made. The best way to start is pick a couple of ports that have lots of the output queue errors and take a look at the QOS breakdown:

Switch#sh mls qos inter fa0/6 stat
FastEthernet0/6 (All statistics are in packets)


Sample output queues enqueued:
 queue:    threshold1   threshold2   threshold3
 queue 0:           0           0         51252
 queue 1:           0           0        847872
 queue 2:           0           0         45743
 queue 3:           0      355323        782472

  output queues dropped:
 queue:    threshold1   threshold2   threshold3
 queue 0:           0           0             0
 queue 1:           0           0           746
 queue 2:           0           0          7720
 queue 3:           0       67464         73897

Analyzing the data:

PacketsT1T2T3% of Traffic
DropsT1T2T3% of T1 Drops% of T2 Drops% of T3 Drops

Using the information above I would make the changes as follows. The PACKETS section shows the traffic breakdown, so I would adjust the following parameters to match:

mls qos queue-set output 1 buffers 5 40 5 50
srr-queue bandwidth share 1 40 5 50
(I tried to make both of them close to match the packets, rather than the default AUTO-QOS parameters)

Now using the DROPS I would make the following changes:

mls qos queue-set output 1 threshold 1 100 100 50 100
(Leave the priority queue alone)
mls qos queue-set output 1 threshold 2 80 90 200 800
(Since there was a lot of traffic, I increased the reserve and maximum)
mls qos queue-set output 1 threshold 3 80 90 50 100
(Not a lot of traffic here reducing some of the numbers)
mls qos queue-set output 1 threshold 4 80 150 200 800
(Since there was a lot of traffic even on T2, I increased several parameters)

Again this is based on the analysis above, but still might need some more adjustments, make the changes wait for a day and see if you need to change the 3rd and 4th parameter. 

From my reading Cisco recommends using Q1 for station ports and Q2 for uplinks. One might have to do the same analysis for the uplinks. 

Whether this could be a global or site settings will depend on the consistency of traffic. 

Catalyst 2960/3560/3750 Output Drops (Part 1)

posted Jul 12, 2014, 1:23 PM by Marc Kerscher   [ updated Jul 12, 2014, 4:19 PM ]

As of late I have been researching output drops and trying to understand them on the switches that I'm responsible for. If you have ever GOOGL'ed the Output drops, you get a lot of links, but none seem to really explain why these are occurring. Example below:

Switch#sh inter fa0/7
FastEthernet0/7 is up, line protocol is up (connected)
  Hardware is Fast Ethernet, address is 9c4e.2087.3d88 (bia 9c4e.2087.3d88)
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 11/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, media type is 10/100BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:01:44
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 148

And the biggest questions is why? Especially on a LAN switch, where utilization is normally low.

Below is my topology:
(Note: the 3560 has a basic configuration on it, basically enabling MLS QOS)

Initially I was trying to use IPERF or IPERF3 (TCP and UDP) trying to cause these output drops, but unless I was over subscribing the links (trying to push 200 mbps into a 100 mbps port) I had no luck, but this is not really the source of the problem as port utilization was normally low. Finally I used the best bandwidth grabbing application out there ( :-) ) also known as FTP. So I installed an FTP server on the Ubuntu PC and generated a 20 MB files. Low and behold just trying to download the 20 MB file, I generated these errors on Fa0/7. Further testing using a generated 10 MB and 5 MB file also generated these errors. For verification I did the test several times, table below:

 Drops / File Size Try 1Try 2 Try 3 
 5 MB12    11 12
 10 MB 26 26 25
 20 MB 53 51 46

As you can see it does not need to be a large file to be generating the drops and what application file is at minimum 5 MBs today. Even worse is that the packet gets to the local switch and drops at the last hop, wasting a lot of bandwidth along the way. Looking further into the switch shows the cause:

Switch#sh inter fa0/7
FastEthernet0/7 is up, line protocol is up (connected)
  Hardware is Fast Ethernet, address is 9c4e.2087.3d88 (bia 9c4e.2087.3d88)
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, media type is 10/100BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:01, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 54
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  30 second input rate 3000 bits/sec, 6 packets/sec
  30 second output rate 209000 bits/sec, 16 packets/sec
     7003 packets input, 501972 bytes, 0 no buffer
     Received 46 broadcasts (45 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 45 multicast, 0 pause input
     0 input packets with dribble condition detected
     14705 packets output, 21042722 bytes, 0 underruns

Switch#sh controller ethernet-controller port-asic st
 Switch 1, PortASIC 0 Statistics
         0 RxQ-0, wt-0 enqueue frames            0 RxQ-0, wt-0 drop frames
     15671 RxQ-0, wt-1 enqueue frames            0 RxQ-0, wt-1 drop frames
         0 RxQ-0, wt-2 enqueue frames            0 RxQ-0, wt-2 drop frames

         0 RxQ-1, wt-0 enqueue frames            0 RxQ-1, wt-0 drop frames
        17 RxQ-1, wt-1 enqueue frames            0 RxQ-1, wt-1 drop frames
      1401 RxQ-1, wt-2 enqueue frames            0 RxQ-1, wt-2 drop frames

     21343 RxQ-2, wt-0 enqueue frames            0 RxQ-2, wt-0 drop frames
         0 RxQ-2, wt-1 enqueue frames            0 RxQ-2, wt-1 drop frames
        45 RxQ-2, wt-2 enqueue frames            0 RxQ-2, wt-2 drop frames

         0 RxQ-3, wt-0 enqueue frames            0 RxQ-3, wt-0 drop frames
         0 RxQ-3, wt-1 enqueue frames            0 RxQ-3, wt-1 drop frames
         0 RxQ-3, wt-2 enqueue frames            0 RxQ-3, wt-2 drop frames

        54 TxQueue Drop Stats
         0 TxBufferFull Drop Count               0 Rx Fcs Error Frames
         0 TxBufferFrameDesc BadCrc16            0 Rx Invalid Oversize Frames

Switch#sh platform port-asic stats nonzero-drop

Port-asic Port Drop Statistics - Summary

  Port  9 TxQueue Drop Stats: 54

  Supervisor TxQueue Drop Statistics

Port-asic Port Drop Statistics - Details
  RxQueue Drop Statistics

  Port 0 TxQueue Drop Statistics

  Port 1 TxQueue Drop Statistics

  Port 2 TxQueue Drop Statistics

  Port 3 TxQueue Drop Statistics

  Port 4 TxQueue Drop Statistics

  Port 5 TxQueue Drop Statistics

  Port 6 TxQueue Drop Statistics

  Port 7 TxQueue Drop Statistics

  Port 8 TxQueue Drop Statistics

  Port 9 TxQueue Drop Statistics
    Queue 1
      Weight 0 Frames 54

All of the output from the commands point to Transmit Queue drops. As everyone knows the Catalyst switches have 4 egress queues. But a little less know is the fact that there are software queues that are inserted before the packets are put out on the wire. At this point I have not been able to actually find out the amount of physical memory assigned to each queue, but from the tests it does not seem to be a whole lot. So when traffic comes in from a gigabit port and goes into a 100 megabit port, these errors seem to occur to adjust for the speed difference.

In the second part of this article I will go into adjustments that can be made to reduce/eliminate these kinds of errors.

Simulating Multicast

posted Mar 23, 2014, 10:03 AM by Marc Kerscher   [ updated Mar 23, 2014, 10:04 AM ]


over the past couple of weekends I have been working on getting a better understanding of multicast. Looking at all the potential multicast streaming capable applications, I chose to go with VLC (, as I recall using it in the past.

I spend countless hours, just trying to get a stream working using their latest version, using a W2k3 Server and Windows 7 PC relying on PIM Dense mode. Trying to save other the headaches that I went through, I could not get it to work. However I went back to a much earlier version (see picture below) and I have no problems streaming and receiving my favorite (mp3,m4a) songs over the lab network.

On the server I use the Wizard to stream files and on the PC File Open network stream.
(Ensure you use the same Multicast address and do not forget to apply proper TTL)

Cisco 4402/4 Wireless Controller Corrupt Image

posted Mar 23, 2014, 9:48 AM by Marc Kerscher   [ updated Mar 23, 2014, 9:49 AM ]


I have seen several Cisco Wireless Controllers out there on Ebay with corrupt images. With all the work that I have done on a 2106, I took one of my 4402 apart, just to see where the flash was located. Much to my surprise I found that there is a CF 256MB flash card connected to the motherboard. I did notice some grey gooey stuff that either insures that the CF flash does not come loose or is there for warranty purposes. However the good news is that the process of recovering a corrupt flash using the methods that I have posted for the 2106 also do apply. 

Ettercap (Part 2), ARP Poison and Cisco switches

posted Oct 21, 2013, 9:52 AM by Marc Kerscher   [ updated Oct 21, 2013, 9:56 AM ]

This is part two of my series on how to manage Ettercap on Cisco switches. As posted in my previous article my lab is setup as follows:

I'm using the Unbuntu 12.04 LTS server as the ettercap machine, that will ARP poison WinXP Client 1 (top one), using the following command:

ettercap -T -w dump -M ARP / //

Now before going any further, let's see what ettercap is actually doing on the wire:

It looks like all the active IPs on the LAN (only a couple) get ARPd to the ettercap device. Also notice that this ARP is repeated every 10 seconds.

Screenshots of the ARP cache on the windows machine:

BEFORE                                                                    AFTER

Even the layer 3 devices are affected:

Now to the point of how can this be avoided. Cisco switches have a feature called dynamic arp inspection, which is built on to of dhcp snooping, which you have to enable, too. WARNING !!! Now you need to be careful in implementing these, as it's very easy to bring down your network if you are not careful. 

The commands on the switch are straight forward:

ip dhcp snooping vlan 10
no ip dhcp snooping information option (this command is needed if you have an IOS dhcp server)
ip dhcp snooping
ip arp inspection vlan 10

These are the global commands, now there are port commands that need to be placed on the uplink ports between switches:

interface FastEthernet0/24
 ip arp inspection trust
 ip dhcp snooping trust

Also if you are running etherchannels, they also need the commands:

interface Port-channel23
 ip arp inspection trust
 ip dhcp snooping trust

Turning these two features on will generate output as shown below:

The switch is now actively dropping ARPs associated with ettercap. Just not enough ARPs to trigger the port shutdown feature.  

Looking at the XP client, no more traffic and after a little while even the local ARP is clear:

Looking at the Layer 3 switches, their arp is still poisoned. ICMP to the default gateway does not work. Looks like the XP client is lost/confused. (side affect of running hacking tools / DOS) Even turning off ettercap does not help. 

I will give all the machines a reboot. That updated the ARP and XP client is back. Now let's launch ettercap on a Cisco switch with DHCP snooping and DAI running:

PORT DISABLED !!! Now the reason why the port was disabled, was due to the fact that once ettercap laucnhes, it sends out ARPs for all devices on the subnet, in this case the /24. By default ARP inspection triggers at 15 packet per second. 

Copyright Kerscher Computing LLC 2013 

Ettercap (Part 1), STP and Cisco switches

posted Oct 17, 2013, 11:48 AM by Marc Kerscher   [ updated Oct 18, 2013, 11:30 AM ]

I'm currently studying for the GIAC Security Essentials (GSEC) ( certification. Along with reading any materials that I can find, I also plan to explore the tools that are referenced and seeing how Cisco switches can deal with them. First of here is my lab environment:

I'm using the Ubuntu 12.04 LTS server to run ettercap, which I had no problems installing via apt-get. To run ettercap ARP poison (more on that in another post) against the first WinXP box I use the following command:

ettercap -T -w dump -M ARP / //

Now as I mentioned I will get into ARP poison later on in a different article, initially I want to focus on the stp_mangler plugin. Enabling this feature will have the Linux box acting as root bridge with a priority of 0:


Probably not something I would like to have happen on my network. The VMWare ports connecting to the 3560 switch are all configured as station ports on vlan 10:

interface FastEthernet0/11
 switchport access vlan 10
 switchport mode access
 load-interval 30
 spanning-tree portfast

The easiest would be to enable portfast bpduguard globally on the switch. (Make sure that all the switch to switch connections are not running spanning-tree portfast (which they should not)). The command is:

spanning-tree portfast bpduguard default

Note that enabling bpduguard does not automatically shutdown the port, much to my surprise. However a shut/no shut does the magic:

Now the port is err-disabled. As show above a shut/no shut will re-enable the port. You can also adjust the errdisable setting for the switch to re-enable the port automatically.

Copyright Kerscher Computing LLC 2013

Cisco NME-NAM-120S

posted Sep 4, 2013, 11:12 AM by Marc Kerscher   [ updated Sep 4, 2013, 11:12 AM ]

I acquired a NME-NAM module as a Proof of Concept. Of course like all other things acquired on Ebay, they never seem to have the latest code on them. I especially love the switches and routers that still have the factory code on them :-). Does anyone upgrade the code!!! Anyways I was able to get a copy of the 5.1 code for it, which seems to have lots of nice features. So I tried to get it updated via the maintenance partition using TFTP and got the following error:

 File upgrade.bin is not a valid NM-NAM application image!

Initially I though I had a bad image, however when using FTP as a transfer method it works just fine.

Dell C1100

posted Aug 7, 2013, 3:48 AM by Marc Kerscher   [ updated Aug 7, 2013, 3:52 AM ]

I have been very excited to get my hands on one of the Dell C1100s. Since this server does not have a CD/DVD drive, it makes updating the firmware and bmc at bit of a challenge before running VMWare on it. Found the following article very useful: 

Cisco CSR1000v up and running

posted Aug 7, 2013, 3:42 AM by Marc Kerscher   [ updated Aug 7, 2013, 3:51 AM ]

Got the Dell C1100 (off Ebay) yesterday and boy does it look nice :-) . Installed the mezzanine SAS controller, new wiring for it, a couple of additional Ethernet ports and finally the USB stick that holds VMWare ESXi. Sadly retired the Dell 1950 G3 server. Once all this was completed the CSR came up without any problems. 

1-10 of 14