Cisco 3750 – High CPU – TTY Background

Cisco 3750 - High CPU - TTY Background

Working from an old 3750 stack, noticed the SSH session was definitely sluggish, inputting commands and receiving output not responsive at all.
Because I use exec prompt timestamp under all my console/VTY ports, whenever I run show commands it includes some brief info on the devices CPU and time/date settings.


MDF-01#sh int status
Load for five secs: 57%/1%; one minute: 63%; five minutes: 63%
Time source is NTP, 11:12:17.547 EST Wed Jan 20 2016
Port Name Status Vlan Duplex Speed Type
Gi1/0/1 Uplink to RTR connected 254 a-full a-1000 10/100/1000BaseTX

Checking on the CPU processes and history, one process stands out as unusual for using much CPU.

MDF-01#sh proc cpu sorted
Load for five secs: 76%/0%; one minute: 68%; five minutes: 65%
Time source is NTP, 11:24:14.362 EST Wed Jan 20 2016
CPU utilization for five seconds: 76%/0%; one minute: 68%; five minutes: 65%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
37 2507934490 68679384 36516 45.20% 50.41% 50.66% 0 TTY Background
245 9984 1415 7055 18.84% 5.05% 1.39% 2 SSH Process
74 1260867175 270578900 4659 1.43% 1.13% 1.10% 0 RedEarth Tx Mana
4 951951800 42048527 22639 1.27% 1.33% 1.20% 0 Check heaps

1

Show proc cpu history [This is the 1 hour summary graph ]

2

Cisco bug ID CSCdy01705 [https://tools.cisco.com/bugsearch/bug/CSCdy01705] points to an issue using logging synchronous on the console port. According to Cisco doc, TTY Background process “is a generic process used by all terminal lines (console, aux, async, and so on). Normally there should not be any impact on the performance of the router since this process has a lower priority compared to the other processes that need to be scheduled by the Cisco IOS software”

After disabling logging synchronous under the line console, CPU dropped off immediately.

MDF-01#conf t
MDF-01(config)#line con 0
MDF-01(config-line)#no logg sync
MDF-01(config-line)#do sh proc cpu sort | i TTY
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
37 2507975461 68680981 36516 0.00% 0.00% 0.23% 0 TTY Background

Checking the CPU after 20 minutes:

3

Cisco RSPAN on 3560/3750

SPAN (Switched Port Analyzer) is also called port-mirroring. It forwards a copy of traffic from one/multiple interfaces to another interface, usually for traffic monitoring.

RSPAN is Remote SPAN, used to forward traffic to a port connected to a remote switch.

ERSPAN can be used to send mirrored traffic across layer-3 boundaries to overcome the limitations of SPAN/RSPAN, but is only supported on a limited set of hardware (Catalyst 6500, Nexus, ASR-series)

In this example we'll be mirroring traffic from an IP phone connected to an access switch, over to a server connected to an upstream switch.

Because we're using RSPAN, we need to create a remote-span VLAN. This is a special VLAN that will be used as the destination for the mirrored traffic, and must exist on all switches in between the source and destination. Traffic to the RSPAN VLAN is flooded out all trunk ports carrying the RSPAN VLAN, so take care to prune the VLAN off inter-switch links where it's not needed if you're going to be mirroring a lot of traffic.

In this example we'll start at the access switch (source switch), by creating the remote-VLAN. Make sure to use the remote-span parameter after creating the VLAN, or the switch will not mirror traffic.

AccessSwitch#conf t
AccessSwitch(config)#vlan 700
AccessSwitch(config-vlan)#name Voice-Monitor
AccessSwitch(config-vlan)#remote-span

Continue reading