Cisco 1131 to 2106 Wireless Controller Registration Issue

Cisco 1131 Access Points Failing to Register to 2106 Wireless Controller

Issue:

Two of four 1131 APs not registering to WLC 2106 after replacing legacy switches with new 3650 switching

Troubleshooting:

Power cycling of APs, by unplugging from network and by cycling POE on the switchport, still failing to register.
Confirmed switchport configurations were correct for an AP and matched the configurations on the decommissioned switches.
Debugging of CAPWAP from the WLC showed the APs were attempting to join the WLC but the connections were being closed by the controller due to DTLS errors
debug capwap events enable
debug capwap detail enable
debug capwap error enable

"Discarding non-ClientHello Handshake OR DTLS encrypted packet from 10.43.98.98:55643) since DTLS session is not established "

Because of the DTLS errors, started looking at certificate issues. Debugging certificates showed the APs installed certificate expired April 2016
debug pm pki enable

"sshpmGetIssuerHandles: Current time outside AP cert validity interval: make sure the controller time is set"

AP certificates installed from factory are valid for 10 years. We can check the manufacture year by looking at the serial number of the AP. The first two numbers correspond to the manufacturing year:

First two numbers of this serial are 10, placing the manufacture date in 2006, which corresponds with the validity of the cert we see above (2006/4/15 to 2016/4/15).
Checking an AP which was working shows it was manufactured in 2007, so its certificate was still valid (11 = 2007)

Based on this info, it appears that replacing the switching was the first time the APs have attempted to re-register to the WLC after their certificates expired in April 2016. The old switches and the WLC all had uptimes of over 1year, so they haven’t had to re-register recently. Following the switch upgrades, the APs manufactured in 2007 registered without issue, but the two APs built in 2006 could not reconnect.

Solution/Workaround:

The permanent solution would be upgrading the WLC and AP code versions, 7.0.98 is from 2011, but this requires a current Smartnet contract. The WLC 2106 was also announced end-of-life in 2011, so it’s currently out of support as of 2013. As a workaround, I changed the system time on the WLC to April 10 2016, and confirmed the APs were able to register correctly. This is not a permanent solution, as the time will need to be constantly set back to prevent future issues.

Cisco 3750 – High CPU – TTY Background

Cisco 3750 - High CPU - TTY Background

Working from an old 3750 stack, noticed the SSH session was definitely sluggish, inputting commands and receiving output not responsive at all.
Because I use exec prompt timestamp under all my console/VTY ports, whenever I run show commands it includes some brief info on the devices CPU and time/date settings.


MDF-01#sh int status
Load for five secs: 57%/1%; one minute: 63%; five minutes: 63%
Time source is NTP, 11:12:17.547 EST Wed Jan 20 2016
Port Name Status Vlan Duplex Speed Type
Gi1/0/1 Uplink to RTR connected 254 a-full a-1000 10/100/1000BaseTX

Checking on the CPU processes and history, one process stands out as unusual for using much CPU.

MDF-01#sh proc cpu sorted
Load for five secs: 76%/0%; one minute: 68%; five minutes: 65%
Time source is NTP, 11:24:14.362 EST Wed Jan 20 2016
CPU utilization for five seconds: 76%/0%; one minute: 68%; five minutes: 65%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
37 2507934490 68679384 36516 45.20% 50.41% 50.66% 0 TTY Background
245 9984 1415 7055 18.84% 5.05% 1.39% 2 SSH Process
74 1260867175 270578900 4659 1.43% 1.13% 1.10% 0 RedEarth Tx Mana
4 951951800 42048527 22639 1.27% 1.33% 1.20% 0 Check heaps

1

Show proc cpu history [This is the 1 hour summary graph ]

2

Cisco bug ID CSCdy01705 [https://tools.cisco.com/bugsearch/bug/CSCdy01705] points to an issue using logging synchronous on the console port. According to Cisco doc, TTY Background process “is a generic process used by all terminal lines (console, aux, async, and so on). Normally there should not be any impact on the performance of the router since this process has a lower priority compared to the other processes that need to be scheduled by the Cisco IOS software”

After disabling logging synchronous under the line console, CPU dropped off immediately.

MDF-01#conf t
MDF-01(config)#line con 0
MDF-01(config-line)#no logg sync
MDF-01(config-line)#do sh proc cpu sort | i TTY
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
37 2507975461 68680981 36516 0.00% 0.00% 0.23% 0 TTY Background

Checking the CPU after 20 minutes:

3

Cisco RSPAN on 3560/3750

SPAN (Switched Port Analyzer) is also called port-mirroring. It forwards a copy of traffic from one/multiple interfaces to another interface, usually for traffic monitoring.

RSPAN is Remote SPAN, used to forward traffic to a port connected to a remote switch.

ERSPAN can be used to send mirrored traffic across layer-3 boundaries to overcome the limitations of SPAN/RSPAN, but is only supported on a limited set of hardware (Catalyst 6500, Nexus, ASR-series)

In this example we'll be mirroring traffic from an IP phone connected to an access switch, over to a server connected to an upstream switch.

Because we're using RSPAN, we need to create a remote-span VLAN. This is a special VLAN that will be used as the destination for the mirrored traffic, and must exist on all switches in between the source and destination. Traffic to the RSPAN VLAN is flooded out all trunk ports carrying the RSPAN VLAN, so take care to prune the VLAN off inter-switch links where it's not needed if you're going to be mirroring a lot of traffic.

In this example we'll start at the access switch (source switch), by creating the remote-VLAN. Make sure to use the remote-span parameter after creating the VLAN, or the switch will not mirror traffic.

AccessSwitch#conf t
AccessSwitch(config)#vlan 700
AccessSwitch(config-vlan)#name Voice-Monitor
AccessSwitch(config-vlan)#remote-span

Continue reading

VPN Client – Cannot match peerless map when peer found in previous map entry

I recently had to connect a VM to a customers VPN, but found the VPN Client (legacy client, not Anyconnect) would simply disconnect after entering in credentials. Entering invalid credentials would re-prompt, so I knew I had the right user/password.

 

Logged into the customer's ASA and checking logs while attempting a connection showed valid Phase 1 completing, but this error stuck out:

Skipping dynamic map XXX sequence XX: cannot match peerless map when peer found in previous map entry.

I knew I had seen this before and couldn't remember the solution offhand, but the interesting bit was "peer found in previous map entry".

In this instance, the network I was working from already had a site-to-site VPN tunnel to the customer (but for separate/unrelated subnets, hence needing the VPN client).

The issue here is that the ASA recognizes my public IP as already being associated with the site-to-site VPN, but can't match it to the incoming VPN client requests, causing the client connection to fail.

We can work around this issue by updating the dynamic cryptomap in use:

Verify the dynamic cryptomap in use, in my case it was outside_dyn_map that had been auto-generated from the ASDM. Checking this configuration found the typical snippet for a VPN client:

crypto dynamic-map outside_dyn_map 20 set transform-set ESP-3DES-SHA
crypto dynamic-map outside_dyn_map 20 set security-association lifetime seconds 28800
crypto dynamic-map outside_dyn_map 20 set security-association lifetime kilobytes 4608000

By simply adding another entry in the dynamic match, but specifying the peer of the site-to-site VPN, we can allow both VPN client and site-to-site to coexist:

crypto dynamic-map outside_dyn_map 10 set peer X.X.X.X
crypto dynamic-map outside_dyn_map 10 set transform-set ESP-3DES-SHA
crypto dynamic-map outside_dyn_map 10 set security-association lifetime seconds 28800
crypto dynamic-map outside_dyn_map 10 set security-association lifetime kilobytes 4608000