We've recently replaced a 7200NPE-G2 with 7600-RSP720. We had around 1Gbps GRE tunnels terminated on the previous box with 85% CPU utilization comming from around 30 tunnel interfaces. Also running OSPF and some route-maps. Everyting was doing good. For scaling our connectivity scenario we decided to replace it with a newer and more powerful platform and 7600-RSP720 was our final choice.
After replacing the old 7200 with the new 7600 we surprisingly noticed high cpu utilization (95%) which all came from interrupts and not any specific process. I tried to use some mls configuration commands but there was no success. For the test scenario we've been using GE ports on the RSP720 and not any other line card interfaces.
I double checked the Cisco 7600 IOS guide document, I'm using commands which is supported by PFC3/DFC3 for hardware switching, which is "tunnel source", "tunnel destination" and "tunnel more gre". the only strange paragraph which i suspect is related to this problem is but it's never been detailed anywhere for the correct implementation, i tried different ip addresses for source tunnels but no difference
"Each hardware-assisted tunnel must have a unique source. Hardware-assisted tunnels cannot share a source even if the destinations are different. Use secondary addresses on loopback interfaces or create multiple loopback interfaces. Failure to use unique source addresses may result in control plane failures during software path congestion."
checking "show ip cef not-cef-switched":
IPv4 CEF Packets passed on to next switching layer
Slot No_adj No_encap Unsupp'ted Redirect Receive Options Access Frag
RP 0 0 3919465 0 195682 0 331065 0
5/0 0 0 0 0 0 0 0 0
1/0 0 0 0 0 0 0 0 0
also "show ip cef switching statistics"
Reason Drop Punt Punt2Host
RP LES Packet destined for us 0 136317 59304
RP LES Encapsulation resource 0 1944707 0
RP LES No adjacency 8 0 0
RP LES TTL expired 0 0 1643028
RP LES Fragmentation failed 78770 0 0
RP LES Features 12836183 0 330963
RP LES Neighbor resolution req 5 2 0
RP LES Total 12914966 2081026 2033295
All Total 12914966 2081026 2033295
ip unnumbered GigabitEthernet5/2.19
ip mtu 1476
ip flow ingress
ip policy route-map redirect
ip ospf cost 100
ip ospf 1 area 1
keepalive 3 3
tunnel source yy.yy.yy.yy
tunnel destination xx.xx.xx.xx
mls flow ip interface-full
mls rp ip route-map
mls ip multicast flow-stat-timer 9
no mls acl tcam share-global
mls cef error action reset
I also tried the configuration without route-map but still the same.
Anyone had such experince with this problem before?
Thanks in advance
Meanwhile, taking this approach going to consume so many IP addresses as each tunnel will need a unique IP address. Is there any work around on this? like using different loopbacks which all unnumbered to the same IP interface? Or is this the matter of different source IP only?
Giuseppe is correct and unfortunately there is no config workaround as it is not a config oddity but a hardware limitation. Each tunnel must have a unquie source IP due to the way the GRE decap path works on the 6K ASICs.
The hardware ASICs on the 6k cannot demux GRE traffic based on both SIP and DIP. It can only be done based on DIP so when GRE packets come into the 6k they must have unique DIP for each tunnel else all incoming traffic in decap path for tunnles that share IP addreses will be punted to the RP for software switching.
>> "Each hardware-assisted tunnel must have a unique source. Hardware-assisted tunnels cannot share a source even if the destinations are different. Use secondary addresses on loopback interfaces or create multiple loopback interfaces. Failure to use unique source addresses may result in control plane failures during software path congestion."
are your GRE tunnels using different tunnel source ip addresses or they share the same source?
this can make quite a difference
Hope to help