11-21-2011 04:23 AM
Hi
We are running one linux PC with 3 VLC streams for our company Add, last couple days I notices that our multicast source router, where this VLC PC connected is running on high cpu utilization, and I did the following troubleshooting
After I disconneted this PC form network, the CPU process is normal.
The PC is running on Linux, what could be the probelm.
A-DISTT1#sh processes cpu sorted | exclude 0.00%
CPU utilization for five seconds: 92%/91%; one minute: 25%; five minutes: 22%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
395 97350452 389572702 249 0.15% 0.74% 0.69% 0 PIM Process
155 2974377924264655218 0 0.15% 0.96% 1.57% 0 IP Input
396 57168908 68962609 828 0.15% 0.39% 0.26% 0 MLSM Process
3 12376 2411 5133 0.07% 0.41% 0.47% 1 SSH Process
350 121028320 392568888 308 0.07% 0.53% 0.53% 0 Port manager pe
A-DISTT2#sh processes cpu sorted | exclude 0.00%
CPU utilization for five seconds: 45%/33%; one minute: 38%; five minutes: 23%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
155 347591256 330574202 1051 10.87% 2.20% 1.46% 0 IP Input
395 154942084 526796334 294 0.47% 0.70% 0.53% 0 PIM Process
397 34611708 35864894 965 0.15% 0.09% 0.09% 0 MLSM Stat Proces
396 39788256 153516402 259 0.15% 0.21% 0.16% 0 MLSM Process
A-DISTT1#sh processes cpu history
2222222111112222211111222222222211111111112222211111111111
5577777999991111166666333332222299999555556666633333555552
100
90
80
70
60
50
40
30 ******* *****
20 *********************************************** *****
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
2159989999899998899999999999999889997999989999999999999999
6247377575427883157615748084586688842776862785125887682872
100 * **** *** *** ** * * *** ** **** *** ****** **
90 ******* **** ******************* *********************
80 ********************************* *********************
70 *******************************************************
60 *******************************************************
50 ************************************#************#***#**
40 *##*##****###****#*#####*####**#****###****#*##*##*####*
30 * *#######*#########################*####*################
20 # *#######################################################
10 ##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
9224236422232233272499999999999999999999999999999999999999999999999999
8770918257566815646499999999999999999999999999999999999899999999999999
100 * **************************************************
90 * **************************************************
80 * **************************************************
70 * * * **************************************************
60 * * * **************************************************
50 * * * **************************************************
40 * * ** * * * **#################################################
30 *********************#################################################
20 #*******************##################################################
10 ######################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
-------------------------------------------------------------------------------------
A-DISTT2#sh processes cpu history
111
1114444444444333339999911111888884444466666444444444422222
100
90
80
70
60
50
40
30
20
10 *** ***** ***** *****
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
9989999999999999999999979999999998889998899998999899999
9798707677387474683822447660644006864779868968758886289656
100 ** **** ** * ** * ** * *** *** **** *** *****
90 ** ******************** ********* *************** *****
80 *******************************************************
70 *******************************************************
60 *******************************************************
50 ******************************************#************
40 ##*#####*###***##***###*####**###***###*####**###***##*
30 #######################################################
20 #######################################################
10 ***#######################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
9922417242123212422199999999999999999999999999999999999999999999999999
9733570000945192502989999999999999999999999999999999999999999999899999
100 ** **************************************************
90 ** **************************************************
80 ** **************************************************
70 ** * **************************************************
60 ** * **************************************************
50 ** * * * **************************************************
40 ** * * * * * *#################################################
30 #* * * * * * *#################################################
20 #********************#################################################
10 ###########*##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
You can analysis by seeing the packet drops
A-DISTT1#sh int | in up | drop
Vlan120 is up, line protocol is up
Input queue: 0/75/1452/1452 (size/max/drops/flushes); Total output drops: 0
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
source and destination and where the packet drop is ocurring
A-DISTT1#sh buffers input-interface vlan 120 header
Buffer information for Big buffer at 0x45A30684
data_area 0x826A144, refcount 1, next 0x5026A528, flags 0x200
linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1
if_input 0x47B08364 (Vlan120), if_output 0x0 (None)
inputtime 00:00:00.000 (elapsed never)
outputtime 21w5d (elapsed 21:33:01.704), oqnumber 65535
datagramstart 0x826A1BA, datagramsize 1358, maximum size 1740
mac_start 0x826A1BA, addr_start 0x826A1BA, info_start 0x0
network_start 0x826A1C8, transport_start 0x826A1DC, caller_pc 0x41FFDF6C
source: 192.168.120.200, destination: 239.232.3.2, id: 0xFA3C, ttl: 1,
TOS: 0 prot: 17, source port 39598, destination port 5000
Buffer information for Big buffer at 0x45A5FA00
data_area 0x8324744, refcount 1, next 0x5027A814, flags 0x200
linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1
if_input 0x47B08364 (Vlan120), if_output 0x0 (None)
inputtime 00:00:00.000 (elapsed never)
outputtime 17w5d (elapsed 4w0d), oqnumber 65535
datagramstart 0x83247BA, datagramsize 1358, maximum size 1740
mac_start 0x83247BA, addr_start 0x83247BA, info_start 0x0
network_start 0x83247C8, transport_start 0x83247DC, caller_pc 0x41FFDF6C
source: 192.168.120.200, destination: 239.232.3.3, id: 0x156B, ttl: 1,
TOS: 0 prot: 17, source port 50428, destination port 5000
A-DISTT2#sh buffers input-interface vlan 120 header
Buffer information for Big buffer at 0x45A76AF0
data_area 0x839B744, refcount 1, next 0x502883FC, flags 0x200
linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1
if_input 0x52C86C6C (Vlan120), if_output 0x0 (None)
inputtime 00:00:00.000 (elapsed never)
outputtime 14w5d (elapsed 21:21:01.044), oqnumber 65535
datagramstart 0x839B7BA, datagramsize 1358, maximum size 1740
mac_start 0x839B7BA, addr_start 0x839B7BA, info_start 0x0
network_start 0x839B7C8, transport_start 0x839B7DC, caller_pc 0x41FFDF6C
source: 192.168.120.200, destination: 239.232.3.2, id: 0xDDD3, ttl: 1,
TOS: 0 prot: 17, source port 46214, destination port 5000
Buffer information for Big buffer at 0x45A77208
data_area 0x839D344, refcount 1, next 0x502827C4, flags 0x200
linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1
if_input 0x52C86C6C (Vlan120), if_output 0x0 (None)
inputtime 00:00:00.000 (elapsed never)
outputtime 14w5d (elapsed 16:51:56.084), oqnumber 65535
datagramstart 0x839D3BA, datagramsize 1358, maximum size 1740
mac_start 0x839D3BA, addr_start 0x839D3BA, info_start 0x0
network_start 0x839D3C8, transport_start 0x839D3DC, caller_pc 0x41FFDF6C
source: 192.168.120.200, destination: 239.232.3.3, id: 0xF9AD, ttl: 1,
TOS: 0 prot: 17, source port 39010, destination port 5000
11-22-2011 07:26 AM
Hi there,
the issue is that the mcast streams sourced by the PC have TTL=1
source: 192.168.120.200, destination: 239.232.3.2, id: 0xFA3C, ttl: 1,
...
source: 192.168.120.200, destination: 239.232.3.3, id: 0x156B, ttl: 1,
...
source: 192.168.120.200, destination: 239.232.3.2, id: 0xDDD3, ttl: 1,
...
source: 192.168.120.200, destination: 239.232.3.3, id: 0xF9AD, ttl: 1,
Those packets cannot be handled in hardware, and therefore require CPU assistance > High CPU, as packets with TTL=1 cannot be routed and must be sent to the CPU for it to generate the TTL expired ICMP message.
Easiest solution is to configure your VLC streams to use a TTL>1 in order to avoid TTL expiration in vlan120.
Problem is pretty common on all Cisco switches, not only on cat6k.
regards,
Riccardo
11-25-2011 06:36 AM
Hi there,
please share with us your comments or flag this question as answered.
Riccardo
12-01-2011 09:03 AM
along with the above comments, for future reference you might start adding the following line to the interface
storm-control multicast level 10.00 (10 being a %, change to your preference)
good practice for multicast storm control.
Jude
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: