Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements
Webcast-Catalyst9k
New Member

HSRP Packet Issues

I am at a site and have an interesting HSRP situation between two 7200 routers. These routers are running v15.0(1)M3 (AdvSecurity) IOS and configured interfaces are both G0/2 on each router.

They are laid out as shown in the attached drawing, nothing out of the ordinary there.

Configs are as follows

R1

interface GigabitEthernet0/2

description Nunya

ip address x.x.x.2 x.x.x.x.x

ip access-group 101 in

ip flow ingress

duplex auto

speed auto

media-type rj45

negotiation auto

standby 100 ip x.x.x.1

standby 100 priority 110

standby 100 preempt delay minimum 30

R2

interface GigabitEthernet0/2

description Nunya

ip address x.x.x.3 x.x.x.x

ip access-group 101 in

duplex auto

speed auto

media-type rj45

negotiation auto

standby 100 x.x.x.1

standby 100 priority 105

standby 100 preempt delay minimum 30


R1#sh standby br

                     P indicates configured to preempt.

                     |

Interface   Grp  Pri   P   State      Active       Standby         Virtual IP

Gi0/2       100  110  P   Active     local          unknown        x.x.x.1

R2#sh standby brief

                     P indicates configured to preempt.

                     |

Interface   Grp  Pri   P     State      Active          Standby         Virtual IP

Gi0/2       100  105  P    Standby   x.x.x.2         local              x.x.x.1


Debug output from R1

Jan 20 2014 09:30:54.178 CST: HSRP: Gi0/2 Grp 100 Hello  out x.x.x.2 Active  pri 110 vIP x.x.x.1

Jan 20 2014 09:30:59.154 CST: HSRP: Gi0/2 Grp 100 Hello  out x.x.x.2 Active  pri 110 vIP x.x.x.1

Jan 20 2014 09:31:01.795 CST: HSRP: Gi0/2 Grp 100 Hello  out x.x.x.2 Active  pri 110 vIP x.x.x.1

Jan 20 2014 09:31:04.723 CST: HSRP: Gi0/2 Grp 100 Hello  out x.x.x.2 Active  pri 110 vIP x.x.x.1

Jan 20 2014 09:31:07.155 CST: HSRP: Gi0/2 Grp 100 Hello  out x.x.x.2 Active  pri 110 vIP x.x.x.1

Debug output from R2

Jan 20 2014 09:31:23.447 CST: HSRP: Gi0/2 Grp 100 Hello  out x.x.x.3 Standby pri 105 vIP x.x.x.1

Jan 20 2014 09:31:23.459 CST: HSRP: Gi0/2 Grp 100 Hello  in  x.x.x.2 Active  pri 110 vIP x.x.x.1

Jan 20 2014 09:31:25.879 CST: HSRP: Gi0/2 Grp 100 Hello  out x.x.x.3 Standby pri 105 vIP x.x.x.1

Jan 20 2014 09:31:25.971 CST: HSRP: Gi0/2 Grp 100 Hello  in  x.x.x.2 Active  pri 110 vIP x.x.x.1

Jan 20 2014 09:31:28.451 CST: HSRP: Gi0/2 Grp 100 Hello  in  x.x.x.2 Active  pri 110 vIP x.x.x.1

Jan 20 2014 09:31:28.455 CST: HSRP: Gi0/2 Grp 100 Hello  out x.x.x.3 Standby pri 105 vIP x.x.x.1

Jan 20 2014 09:31:29.127 CST: HSRP: Gi0/2 Interface adv out, Passive, active 0 passive 1

Here is what I have done. I have specifically added a permit statement to ACL 101 on R1 for 224.0.0.2 port 1985, it still does nothing. I then added the same to R2 just to see the hit count increase, it did of course although the ACL is not needed, more of a visual way for me to track it. On the end of each ACL 101 there is a "permit ip any any"

I made sure both sides had appropriate priorities, preempt statements. The routers have been rebooted and the next thing I could remove HSRP all together from G0/2 on R1 and add it back. It's simply an odd issue, is it buggy IOS perhaps?  Switches are configured the same, can find nothing wrong there.

Everyone's tags (6)
23 REPLIES
Hall of Fame Super Gold

HSRP Packet Issues

The debug is pretty clear that R1 sees outbound HSRP but no inbound. My first question would be what does CDP show on each router? Does R1 see R2 as a neighbor on G0/2? My second question would be whether the routers can traceroute to each other and if so is the response coming back from G0/2?

HTH

Rick

Hall of Fame Super Blue

Re: HSRP Packet Issues

David

Apologies for interrupting the thread.

Rick

Could i ask you a favour. I have been involved in a thread where i seem to be going round in circles and cannot understand exactly how things are working.

If possible could you have a look at it and see if it makes sense to you because it doesn't to me but it could be my lack of understanding -

https://supportforums.cisco.com/thread/2262246?tstart=0

Many thanks.

Jon

New Member

Re: HSRP Packet Issues

Sorry, should have put that data in the first post

CDP shows

R1#sh cdp neighbors

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge

                  S - Switch, H - Host, I - IGMP, r - Repeater

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID

Sw1

                 Gig 0/2            168          S I      WS-C2960G Gig 0/15

R2#sh cdp neighbors

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge

                  S - Switch, H - Host, I - IGMP, r - Repeater

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID

Sw2

                 Gig 0/2            150          S I      WS-C2960G Gig 0/15

R1#traceroute x.x.x.3

Type escape sequence to abort.

Tracing the route to x.x.x.3

  1  *  *

    x.x.x.3 0 msec

R1#

R1#ping x.x.x.3

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to x.x.x.3, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

R1#

R2#traceroute x.x.x.2

Type escape sequence to abort.

Tracing the route to x.x.x.2

  1 x.x.x.2 0 msec *  0 msec

R2#

R2#ping x.x.x.2

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to x.x.x.2, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

There is obviously an issue with pinging the vIP as shown here

R1#ping x.x.x.1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to x.x.x.1, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

R1#

R2#ping x.x.x.1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to x.x.x.1, timeout is 2 seconds:

.....

Success rate is 0 percent (0/5)

R2#

Re: HSRP Packet Issues

David,

could you do one more ping on R2:

ping 224.0.0.2 source gi0/2

just to see if R1 responds?

Regards

Rolf

New Member

Re: HSRP Packet Issues

R1#ping 224.0.0.2 source g0/2

Type escape sequence to abort.

Sending 1, 100-byte ICMP Echos to 224.0.0.2, timeout is 2 seconds:

Packet sent with a source address of x.x.x.2

Reply to request 0 from x.x.x.3, 1 ms

R1#

R2#ping 224.0.0.2 source g0/2

Type escape sequence to abort.

Sending 1, 100-byte ICMP Echos to 224.0.0.2, timeout is 2 seconds:

Packet sent with a source address of x.x.x.3

Hall of Fame Super Gold

HSRP Packet Issues

That certainly is interesting and suggests a one way issue with multi cast. Your earlier test shows that we have good two way communication for unicast. It might be interesting to see the output of show ip interface g0/2 from both routers.

I also wonder if there might be something in the configuration of the switches that might cause this.

HTH

Rick

New Member

Re: HSRP Packet Issues

Yeah, an interesting issue to say the least, it's going to be something i've over looked. I can feel it. ha

R1#

sh ip interface g0/2

GigabitEthernet0/2 is up, line protocol is up

  Internet address is x.x.x.2/x

  Broadcast address is 255.255.255.255

  Address determined by non-volatile memory

  MTU is 1500 bytes

  Helper address is not set

  Directed broadcast forwarding is disabled

  Multicast reserved groups joined: 224.0.0.2

  Outgoing access list is not set

  Inbound  access list is 101

  Proxy ARP is enabled

  Local Proxy ARP is disabled

  Security level is default

  Split horizon is enabled

  ICMP redirects are always sent

  ICMP unreachables are always sent

  ICMP mask replies are never sent

  IP fast switching is enabled

  IP fast switching on the same interface is disabled

  IP Flow switching is disabled

  IP CEF switching is enabled

  IP CEF switching turbo vector

  IP CEF turbo switching turbo vector

  IP multicast fast switching is enabled

  IP multicast distributed fast switching is disabled

  IP route-cache flags are Fast, CEF

  Router Discovery is disabled

  IP output packet accounting is disabled

  IP access violation accounting is disabled

  TCP/IP header compression is disabled

  RTP/IP header compression is disabled

  Policy routing is disabled

  Network address translation is disabled

  BGP Policy Mapping is disabled

  Input features: Ingress-NetFlow, Access List, MCI Check

  Output features: Post-Ingress-NetFlow

  WCCP Redirect outbound is disabled

  WCCP Redirect inbound is disabled

  WCCP Redirect exclude is disabled

R2#sh ip int g0/2

GigabitEthernet0/2 is up, line protocol is up

  Internet address is x.x.x.3/x

  Broadcast address is 255.255.255.255

  Address determined by non-volatile memory

  MTU is 1500 bytes

  Helper address is not set

  Directed broadcast forwarding is disabled

  Multicast reserved groups joined: 224.0.0.2

  Outgoing access list is not set

  Inbound  access list is 101

  Proxy ARP is enabled

  Local Proxy ARP is disabled

  Security level is default

  Split horizon is enabled

  ICMP redirects are always sent

  ICMP unreachables are always sent

  ICMP mask replies are never sent

  IP fast switching is enabled

  IP fast switching on the same interface is disabled

  IP Flow switching is disabled

  IP CEF switching is enabled

  IP CEF switching turbo vector

  IP CEF turbo switching turbo vector

  IP multicast fast switching is enabled

  IP multicast distributed fast switching is disabled

  IP route-cache flags are Fast, CEF

  Router Discovery is disabled

  IP output packet accounting is disabled

  IP access violation accounting is disabled

  TCP/IP header compression is disabled

  RTP/IP header compression is disabled

  Policy routing is disabled

  Network address translation is disabled

  BGP Policy Mapping is disabled

  Input features: Access List, MCI Check

  WCCP Redirect outbound is disabled

  WCCP Redirect inbound is disabled

  WCCP Redirect exclude is disabled

Hall of Fame Super Gold

HSRP Packet Issues

David

Thanks for this output. But what I asked for was show ip interface and not just show interface.

HTH

Rick

New Member

Re: HSRP Packet Issues

Oops, there you go..edited post above

Hall of Fame Super Gold

HSRP Packet Issues

David

Thanks for the updated output. I had hoped that it would have some insight into the issue. But other than demonstrating that both have "Multicast reserved groups joined: 224.0.0.2"  it does not have much clue (at least that I can detect). One more request: would you post the access-list 101 from both routers?

HTH

Rick

New Member

Re: HSRP Packet Issues

I agree, I see nothing at this point and almost at a loss..here are the ACL's

R1#sh access-lists

Standard IP access list 1

    10 permit x.x.x.x, wildcard bits 0.0.0.255

Standard IP access list 2

    10 permit x.x.x.x, wildcard bits 0.0.0.255

Extended IP access list 101

    10 permit tcp x.x.x.x 0.0.1.255 x.x.x.x 0.0.1.255 eq 22 (7498 matches)

    20 permit udp x.x.x.x 0.0.1.255 x.x.x.x 0.0.1.255 eq ntp

    30 permit udp host x.x.x.x x. x.x.x 0.0.1.255 eq ntp (2095 matches)

    40 permit udp x.x.x.x 0.0.1.255 host x.x.x.x eq ntp (2265 matches)

    50 permit udp x.x.x.x 0.0.1.255 x.x.x.x 0.0.1.255 eq snmp (475234 matches)

    70 deny tcp any host x.x.x.x eq 22 (43 matches)

    80 deny udp any host x.x.x.x eq snmp (6 matches)

    100 deny tcp any host x.x.x.x eq 22 (92 matches)

    110 deny udp any host x.x.x.x eq ntp

    120 deny udp any host x.x.x.1 eq snmp (3 matches)

    130 deny tcp any host x.x.x.x.2 eq 22 (204 matches)

    140 deny udp any host x.x.x.x.2 eq ntp (1 match)

    150 deny udp any host x.x.x.2 eq snmp (3 matches)

    160 permit ip any any (732299596 matches)

R2#sh access-lists

Standard IP access list 1

    10 permit x.x.x.x, wildcard bits 0.0.0.255 (1 match)

Standard IP access list 2

    10 permit x.x.x.x, wildcard bits 0.0.0.255

Extended IP access list 101

    10 permit tcp x.x.x.x 0.0.1.255 x.x.x.x0.0.1.255 eq 22 (10586 matches)

    20 permit udp x.x.x.x 0.0.1.255 x.x.x.x 0.0.1.255 eq ntp

    30 permit udp host x.x.x.x x.x.x.x 0.0.1.255 eq ntp (2 matches)

    40 permit udp x.x.x.x 0.0.1.255 host x.x.x.x eq ntp (31 matches)

    50 permit udp x.x.x.x 0.0.1.255 x.x.x.x 0.0.1.255 eq snmp (494446 matches)

    60 permit udp host x.x.x.2 host 224.0.0.2 eq 1985 (97910 matches)

    70 deny tcp any host x.x.x.x eq 22

    80 deny udp any host x.x.x.x eq ntp

    90 deny udp any host x.x.x.x eq snmp

    100 deny tcp any host x.x.x.1 eq 22

    110 deny udp any host x.x.x.1 eq ntp

    120 deny udp any host x.x.x.1 eq snmp

    130 deny tcp any host x.x.x.3 eq 22 (88 matches)

    140 deny udp any host x.x.x.3 eq ntp

    150 deny udp any host x.x.x.3 eq snmp (3 matches)

    160 permit ip any any (90446403 matches)

Highlighted the ACL I put in place to get a visual on the hit count for the multicast traffic for HSRP, added and removed from R1 with no hits of course

Hall of Fame Super Blue

Re: HSRP Packet Issues

David

Have you checked your switch configurations eg. specifically do you have any port acls applied to any of the interfaces that are part of the path between the routers.

Also worth checking if the switches are using VACLs which could be blocking multicast one way.

Jon

New Member

Re: HSRP Packet Issues

Yeah, I have checked the switches..there are several ACL's on the 2nd switch to which R2 is connected but nothing affecting this issue

Sw2#sh ip int g0/9

GigabitEthernet0/9 is up, line protocol is up

  Inbound  access list is not set

access-list 103 deny tcp host x.x.x.x eq 1723 any

access-list 103 permit ip any any

access-list 178 deny udp any eq ntp host x.x.x.x

access-list 178 permit ip any any

Connection to R1

interface GigabitEthernet0/9

description R1

switchport access vlan x

switchport mode access

spanning-tree portfast

spanning-tree bpduguard enable


Connection to R2

interface GigabitEthernet0/9

description R2

switchport access vlan x

switchport mode access

spanning-tree portfast

spanning-tree bpduguard enable


I also want to mention that there have been some ARP issues with these routers recently, wondering if this IOS is buggy? (C7200P-ADVSECURITYK9-M) Version 15.0(1)M3

Hall of Fame Super Gold

Re: HSRP Packet Issues

David

Thanks for the additional information. I am wondering about the possibility that something on some switch is causing the issue. Perhaps some CGMP/IGMP config? I am wondering if we can try some other multicast traffic and see if it is impacted. Perhaps something like trying to run EIGRP or OSPF on these two router interfaces? We do not need to advertise anything, but it would be interesting to see if R1 receives the multicast hello from R2.

HTH

Rick

New Member

Re: HSRP Packet Issues

Excellent idea actually..so this is obviously a multicast issue as shown below

R1

R1#

Jan 22 2014 09:34:09.864 CST: EIGRP: Neighbor(x.x.x.3) not yet found

R1#

Jan 22 2014 09:34:14.864 CST: EIGRP: Neighbor(x.x.x.3) not yet found

R1#

Jan 22 2014 09:34:19.864 CST: EIGRP: Neighbor(x.x.x.3) not yet found

R2

Jan 22 2014 09:34:00.354 CST: EIGRP: New peer x.x.x.2

R2#

Jan 22 2014 09:34:00.354 CST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor x.x.x.2 (GigabitEthernet0/2) is up: new adjacency

R2#sh ip eigrp neighbors

EIGRP-IPv4 Neighbors for AS(1)

H   Address                 Interface       Hold Uptime   SRTT   RTO  Q  Seq

                                            (sec)         (ms)       Cnt Num

0   x.x.x.2            Gi0/2             13 00:00:38    1  5000  1  0


Hall of Fame Super Blue

Re: HSRP Packet Issues

David  / Rick

Great idea Rick. I think this has to be an acl issue because IGMP snooping certainly does not filter link local multicast addresses.

It's actually very hard to filter that specific multicast range

Whether CGMP does or doesn't i can't say but it would have to be quite an old switch to be running that.

Jon

Hall of Fame Super Gold

Re: HSRP Packet Issues

The drawing in the original post shows 2 switches connecting the 2 routers. So I believe that we need to look more closely at the switches to see if one of them is the cause of this strange behavior.

HTH

Rick

New Member

Re: HSRP Packet Issues

These routers and switches are "External", so the two routers are connected to access ports on the two WS-C2960G-24TC-L via G0/9, switches are running 12.2(46)SE (LANBASE)

VLAN 301 is the access vlan on each port. CDP neighbor shows the other switch, guest router, vpn etc. Etherchannel is used between the two switches

SW1

interface Port-channel1

switchport mode trunk

interface GigabitEthernet0/22

switchport mode trunk

channel-group 1 mode active

end

interface GigabitEthernet0/23

switchport mode trunk

channel-group 1 mode active

SW1#sh cdp neighbors

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge

                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID

SW2

                 Gig 0/23          126           S I      WS-C2960G Gig 0/23

SW2

                 Gig 0/22          129           S I      WS-C2960G Gig 0/22

R1

                 Gig 0/9           173            R       7204VXR   Gig 0/2


SW2

interface Port-channel1

switchport mode trunk

interface GigabitEthernet0/22

switchport mode trunk

channel-group 1 mode active

interface GigabitEthernet0/23

switchport mode trunk

channel-group 1 mode active

SW2#sh cdp neighbors

Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge

                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID

SW1

                 Gig 0/23          159           S I      WS-C2960G Gig 0/23

SW1

                 Gig 0/22          159           S I      WS-C2960G Gig 0/22

R2

                 Gig 0/9           175            R       7204VXR   Gig 0/2

New Member

Re: HSRP Packet Issues

Interesting enough, I can now see HSRP is up and running fine. I did nothing that is not posted here, I simply logged into the routers today to pick up where I left off and was shocked at the out of sh standby br, more like a "wtf?" moment.

Can anyone say odd? Wow!

R1#sh standby br

                     P indicates configured to preempt.

                     |

Interface   Grp  Pri P State   Active          Standby         Virtual IP

Gi0/2       100  110 P Active  local           x.x.x.3            x.x.x.1

R2#sh standby br

                     P indicates configured to preempt.

                     |

Interface   Grp   Pri   P      State           Active          Standby         Virtual IP

Gi0/2       100   105   P     Standby       x.x.x.2         local               x.x.x.1

R2 can now ping multicast as well

R2#ping 224.0.0.2

Type escape sequence to abort.

Sending 1, 100-byte ICMP Echos to 224.0.0.2, timeout is 2 seconds:

Reply to request 0 from x.x.x.2, 1 ms

ACL's show hits now

R1

60 permit udp host x.x.x.3 host 224.0.0.2 eq 1985 (142878 matches)

R2

60 permit udp host x.x.x.2 host 224.0.0.2 eq 1985 (260476 matches)

If you remember, I added these ACL's to get a visual of the hit counts going up as the traffic came in. Had previously added it to R1 but no hits ever came across. Removed the ACL but added it back last week and left it, this one has me folks. Something buggy for sure.



Hall of Fame Super Gold

Re: HSRP Packet Issues

David

That surely is unexpected but is good news 

Is there any chance that the router rebooted, or interface reset, or anything like that which might have re-initialized the multicast processing?

HTH

Rick

New Member

Re: HSRP Packet Issues

Nope, sure I shut the ports down on the switch and brought them up during the troubleshooting last week but this did not show any immediate resolution to this issue.

Also, on this R2 there is an ARP issue. A few entries have had previously been manually configured. I don't see the x.x.x.1 entry in the table so with that said, R2 cannot ping the vIP address x.x.x.1

Hall of Fame Super Gold

Re: HSRP Packet Issues

David

Probably the router not being able to ping the virtual IP is not much of a problem (in fact I remember when HSRP was a fairly new feature it was normal behavior that the router could not ping the virtual IP - but that changed many releases ago). So I believe that this is one more indicator that there is something quite buggy about this version of code and HSRP.

HTH

Rick

New Member

Re: HSRP Packet Issues

Agreed! I'm filing this one under "buggy IOS". Thanks for all the insight..this was an interesting one for sure.

1014
Views
0
Helpful
23
Replies
CreatePlease to create content