Strange multicast problem when using Alteris Deployment Server

Unanswered Question
May 11th, 2010
User Badges:

Hi,


We use the PXE boot funtion on our desktop PCs and laptops. Multicast TFTP is enabled within the BIOS to grab a boot file and only recently we have been witnessing an "Open TFTP Timeout" during the bootup process.


I have checked over the IGMP / IP PIM configuration across our network and everything looks fine. However when I check the IGMP group that the client is trying to join I see address 224.1.1.2. Yet when I look at the IGMP group on the port to which the Altiris Deployment server connects I only see group 225.1.2.3.


A colleague has shown me the configuration for the MTFTP server on the deployment server and the group address is set at 224.1.1.0.


Shouldn't the group address on the Altiris deployment server also be 224.1.1.2 or is the 225.1.2.3 address within the 224.1.1.0 IGMP group and the problem is in fact something to do with the network ?


Any help would be appreciated.


Chris

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Giuseppe Larosa Tue, 05/11/2010 - 07:28
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Chris,


>> However when I check the IGMP group that the client is trying to join I  see address 224.1.1.2. Yet when I look at the IGMP group on the port to  which the Altiris Deployment server connects I only see group 225.1.2.3.


it looks like the client is trying to get traffic from a different group, but when you check IGMP membership you can check receiver interest.


So server is interested in receiving traffic on 224.1.1.2


can 225.1.2.3 fit with 224.1.1.0?


it is unlikely as you can see first byte 225 is different then 224.1.1.0


check with

sh ip mroute 225.1.2.3

if there is any source sending packets to this group


(I mean (S,225.1.2.3) entries the (*, 225.1.2.3) should be present in any case)


sh ip pim rp mapping 225.1.2.3


there is an RP for this address?



if there is not you have a problem


Hope to help

Giuseppe

cbeswick Tue, 05/11/2010 - 07:48
User Badges:

Hi Giuseppe and thanks for your reply.


When I look on the first hop distribution switch (the switch block that contains the server) I see the following:


(*, 225.1.2.3), 4d17h/00:02:55, RP 172.16.255.4, flags: SJC

Incoming interface: Vlan227, RPF nbr 172.16.227.250, Partial-SC

Outgoing interface list:

Vlan192, Forward/Sparse-Dense, 4d17h/00:02:55, H



The RP mapping shows the following:


ci_t2_c200_sfdist_sw1#sh ip pim rp mapping 225.1.2.3

PIM Group-to-RP Mappings

Group(s) 224.0.0.0/4

RP 172.16.255.4 (?), v2v1

Info source: 172.16.255.7 (?), elected via Auto-RP

Uptime: 4d17h, expires: 00:02:50



When I check the RP itself at 172.16.255.4, I see the following:


(*, 225.1.2.3), 4w0d/00:02:57, RP 172.16.255.4, flags: S

Incoming interface: Null, RPF nbr 0.0.0.0

Outgoing interface list:

Vlan227, Forward/Sparse-Dense, 4d17h/00:02:57


Everything looks fine for the group.


I have just looked at the IGMP group the client is trying to join and this has now changed to 224.1.1.3 all by itself. I have checked over the core switch and can see the following:


ci_t2_65_c200_core2#sh ip mroute 224.1.1.3

IP Multicast Routing Table

Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,

L - Local, P - Pruned, R - RP-bit set, F - Register flag,

T - SPT-bit set, J - Join SPT, M - MSDP created entry,

X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,

U - URD, I - Received Source Specific Host Report,

Z - Multicast Tunnel, z - MDT-data group sender,

Y - Joined MDT-data group, y - Sending to MDT-data group

V - RD & Vector, v - Vector

Outgoing interface flags: H - Hardware switched, A - Assert winner

Timers: Uptime/Expires

Interface state: Interface, Next-Hop or VCD, State/Mode


(*, 224.1.1.3), 00:23:23/00:03:24, RP 172.16.255.4, flags: S

Incoming interface: Null, RPF nbr 0.0.0.0

Outgoing interface list:

Vlan232, Forward/Sparse-Dense, 00:00:05/00:03:24

Vlan224, Forward/Sparse-Dense, 00:09:32/00:02:49


(172.16.192.49, 224.1.1.3), 00:23:23/00:02:16, flags:

Incoming interface: Vlan228, RPF nbr 172.16.228.251

Outgoing interface list:

Vlan232, Forward/Sparse-Dense, 00:00:05/00:03:24

Vlan224, Forward/Sparse-Dense, 00:09:32/00:02:49



Device 172.16.192.49 is the server address....So the server is a member of the 224.1.1.3 group, yet when I check on the switch port it is 225.1.2.3.


I just do not understand this.

Giuseppe Larosa Tue, 05/11/2010 - 08:23
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Chris,


>>

Device 172.16.192.49 is the server address....So the server is a  member of the 224.1.1.3 group, yet when I check on the switch port it is  225.1.2.3.


I  just do not understand this.


note:

a source S for group G does not need to be a member of group G, it is simply a source of packets sent with destination G


then the server can be a multicast receiver on its own, and it can have joined a totally different multicast group G2 not related to the first.


This is what happens here


the following output shows the source is active:


(172.16.192.49,  224.1.1.3), 00:23:23/00:02:16, flags:

Incoming interface: Vlan228,  RPF nbr 172.16.228.251

Outgoing interface list:

Vlan232,  Forward/Sparse-Dense, 00:00:05/00:03:24

Vlan224,  Forward/Sparse-Dense, 00:09:32/00:02:49


Can the host receive on 224.1.1.3?



Hope to help

Giuseppe

cbeswick Wed, 05/12/2010 - 00:06
User Badges:

Can the host receive on 224.1.1.3?


Do you mean the server 172.16.192.49 ? How would I check this on the network ? If it is a source of the multicast group (S,G), would it not be able to automatically recieve on the same group ?

cbeswick Wed, 05/12/2010 - 00:24
User Badges:

I think I may have found a problem.


When I trace the S,G multicast tree across the backbone I see the following:


1) On the Server distribution switch (switch block containing the server 172.16.192.49)


ci_t1_c050_sfdist_sw1#sh ip mroute 224.1.1.3
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report,
       Z - Multicast Tunnel, z - MDT-data group sender,
       Y - Joined MDT-data group, y - Sending to MDT-data group
       V - RD & Vector, v - Vector
Outgoing interface flags: H - Hardware switched, A - Assert winner
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode


(*, 224.1.1.3), 16:53:00/stopped, RP 172.16.255.4, flags: SP
  Incoming interface: Vlan228, RPF nbr 172.16.228.250, RPF-MFD
  Outgoing interface list: Null


(172.16.192.49, 224.1.1.3), 16:53:00/00:03:21, flags: T
  Incoming interface: Vlan192, RPF nbr 0.0.0.0, RPF-MFD
  Outgoing interface list:
    Vlan228, Forward/Sparse-Dense, 16:08:12/00:02:49, H


2) On the Core switch (also the RP for the group)


ci_t2_65_c200_core2#sh ip mroute 224.1.1.3
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report,
       Z - Multicast Tunnel, z - MDT-data group sender,
       Y - Joined MDT-data group, y - Sending to MDT-data group
       V - RD & Vector, v - Vector
Outgoing interface flags: H - Hardware switched, A - Assert winner
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode


(*, 224.1.1.3), 16:54:57/00:03:18, RP 172.16.255.4, flags: S
  Incoming interface: Null, RPF nbr 0.0.0.0
  Outgoing interface list:
    Vlan232, Forward/Sparse-Dense, 00:00:37/00:02:52
    Vlan222, Forward/Sparse-Dense, 00:05:46/00:00:38
    Vlan224, Forward/Sparse-Dense, 16:08:53/00:03:18


(172.16.192.49, 224.1.1.3), 16:54:57/00:01:53, flags:
  Incoming interface: Vlan228, RPF nbr 172.16.228.251
  Outgoing interface list:
    Vlan232, Forward/Sparse-Dense, 00:00:37/00:02:52
    Vlan222, Forward/Sparse-Dense, 00:05:46/00:00:38
   Vlan224, Forward/Sparse-Dense, 16:08:53/00:03:18


3) On the distribution switch containing the clients using PXE Boot and the Multicast TFTP:


ci_t1_3a1_dist_sw1#sh ip mroute 224.1.1.3
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report,
       Z - Multicast Tunnel, z - MDT-data group sender,
       Y - Joined MDT-data group, y - Sending to MDT-data group
       V - RD & Vector, v - Vector
Outgoing interface flags: H - Hardware switched, A - Assert winner
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode


(*, 224.1.1.3), 16:42:50/00:02:38, RP 172.16.255.4, flags: SJC
  Incoming interface: Vlan224, RPF nbr 172.16.224.250, Partial-SC
  Outgoing interface list:
    Vlan5, Forward/Sparse-Dense, 00:00:34/00:02:38, H
    Vlan67, Forward/Sparse-Dense, 00:01:10/00:01:49, H
    Vlan8, Forward/Sparse-Dense, 00:02:33/00:00:41, H
    Vlan6, Forward/Sparse-Dense, 00:07:53/00:02:22, H
    Vlan68, Forward/Sparse-Dense, 16:10:37/00:02:28, H


On the above switch you can see the end hosts joining the group 224.1.1.3 but there is no S,G entry for the server.


We had an issue a few weeks ago where a rogue DHCP device dished out an ip address used by one of our HSRP groups and caused a few issues, this seems to coincide with when the multicasting stopped for this particular service. Could this have caused an issue with the multicast stream ?


Do you think a clearing of the mroute cache on the backbone switches might resolve the issue ?

Giuseppe Larosa Wed, 05/12/2010 - 05:56
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Chris,

cumulative answer to your two posts


>> How would I check this on the network ? If it is a source of the  multicast group (S,G), would it not be able to automatically recieve on  the same group ?


no, as noted in my previous post a source is not a member of the group, it can be but being a source is not enough to be a member of group G


I meant on the intended receiver, that is what counts



>> We had an issue a few weeks ago where a rogue DHCP device dished out an  ip address used by one of our HSRP groups and caused a few issues, this  seems to coincide with when the multicasting stopped for this particular  service. Could this have caused an issue with the multicast stream ?


Yes, there can be issues in interaction of HSRP and multicast.

I agree that there can be a relationship between the two events


Edit:

Incoming interface: Vlan224, RPF nbr 172.16.224.250,  Partial-SC


this partial-SC flag is the sign of a problem


you could use debug ip pim to see error messages



can you check on last switch


with

sh ip rpf

sh ip rpf 172.16.192.49


what should be the RPF neighbor for RP and for source?


Hope to help

Giuseppe

cbeswick Wed, 05/12/2010 - 06:14
User Badges:

Hi ,


Output of show commands on the last distribution switch before hitting the receiver:


ci_t1_3a1_dist_sw1#sh ip rpf 172.16.255.4
  RPF information for ? (172.16.255.4)
  RPF interface: Vlan224
  RPF neighbor: ? (172.16.224.250)
  RPF route/mask: 172.16.255.4/32
  RPF type: unicast (ospf 1)
  RPF recursion count: 0
  Doing distance-preferred lookups across tables


ci_t1_3a1_dist_sw1#sh ip rpf 172.16.192.49
  RPF information for ? (172.16.192.49)
  RPF interface: Vlan224
  RPF neighbor: ? (172.16.224.250)
  RPF route/mask: 172.16.192.0/24
  RPF type: unicast (ospf 1)
  RPF recursion count: 0
  Doing distance-preferred lookups across tables


I haven't used this command before. The RP is 172.16.255.4 for the group yet the RPF neighbour is an interface on the RP itself. We have dual distribution switches / cores throughout the network so RPF should be blocking the stream from replicating across multiple paths, should I therefore expect the RFP neighbour to be the redundant / diverse path ?. Does the above indicate that Reverse Path Forward mechanism is actually blocking the path to the RP ? If so this would explain why the receivers aren't seeing the stream.


Giuseppe Larosa Wed, 05/12/2010 - 06:20
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Chris,

RPF checks look like fine


let's go on with


sh ip pim neighbor

on last switch

and

sh ip pim int


edit:

another thought

there is a companion switch in client vlan that could be the PIM DR on user segments


non PIM DR switch should have only the (*,G) entry


you could try to change HSRP active router on user lan segment to see if there is any effect



Hope to help

Giuseppe

cbeswick Wed, 05/12/2010 - 06:24
User Badges:

As requested (many thanks for your help!)


ci_t1_3a1_dist_sw1#sh ip pim neighbor
PIM Neighbor Table
Mode: B - Bidir Capable, DR - Designated Router, N - Default DR Priority,
      P - Proxy Capable, S - State Refresh Capable
Neighbor          Interface                Uptime/Expires    Ver   DR
Address                                                            Prio/Mode
172.16.2.252      Vlan2                    6w6d/00:01:19     v2    1 / S P
172.16.3.252      Vlan3                    6w6d/00:01:34     v2    1 / S P
172.16.4.252      Vlan4                    6w6d/00:01:28     v2    1 / S P
172.16.5.252      Vlan5                    6w6d/00:01:28     v2    1 / S P
172.16.6.252      Vlan6                    6w6d/00:01:41     v2    1 / S P
172.16.7.252      Vlan7                    6w6d/00:01:15     v2    1 / S P
172.16.8.252      Vlan8                    6w6d/00:01:39     v2    1 / S P
172.16.10.252     Vlan10                   6w6d/00:01:17     v2    1 / S P
172.16.18.252     Vlan18                   6w6d/00:01:31     v2    1 / S P
172.16.20.252     Vlan20                   6w6d/00:01:27     v2    1 / S P
172.16.25.252     Vlan25                   6w6d/00:01:33     v2    1 / S P
172.16.27.252     Vlan27                   6w6d/00:01:30     v2    1 / S P
172.16.64.252     Vlan64                   6w6d/00:01:38     v2    1 / S P
172.16.65.252     Vlan65                   6w6d/00:01:31     v2    1 / S P
172.16.66.252     Vlan66                   6w6d/00:01:40     v2    1 / S P
172.16.67.252     Vlan67                   6w6d/00:01:20     v2    1 / S P
172.16.68.252     Vlan68                   6w6d/00:01:18     v2    1 / S P
172.16.69.252     Vlan69                   6w6d/00:01:16     v2    1 / S P
172.16.70.252     Vlan70                   6w6d/00:01:39     v2    1 / S P
172.16.71.252     Vlan71                   6w6d/00:01:43     v2    1 / S P
172.16.72.252     Vlan72                   6w6d/00:01:34     v2    1 / S P
172.16.73.252     Vlan73                   6w6d/00:01:32     v2    1 / S P
172.16.74.252     Vlan74                   6w6d/00:01:20     v2    1 / S P
172.16.76.252     Vlan76                   6w6d/00:01:39     v2    1 / S P
172.16.82.252     Vlan82                   6w6d/00:01:33     v2    1 / S P
172.16.83.252     Vlan83                   6w6d/00:01:30     v2    1 / S P
172.16.84.252     Vlan84                   6w6d/00:01:18     v2    1 / S P
172.16.85.252     Vlan85                   6w6d/00:01:22     v2    1 / S P
172.16.88.252     Vlan88                   6w6d/00:01:37     v2    1 / S P
172.16.89.252     Vlan89                   6w6d/00:01:23     v2    1 / S P
172.16.90.252     Vlan90                   6w6d/00:01:17     v2    1 / S P
172.16.91.252     Vlan91                   6w6d/00:01:43     v2    1 / S P
172.16.92.252     Vlan92                   6w6d/00:01:17     v2    1 / S P
172.16.93.252     Vlan93                   6w6d/00:01:21     v2    1 / S P
172.16.94.252     Vlan94                   6w6d/00:01:18     v2    1 / S P
172.16.95.252     Vlan95                   6w6d/00:01:24     v2    1 / S P
172.16.96.252     Vlan96                   6w6d/00:01:21     v2    1 / S P
172.16.97.252     Vlan97                   6w6d/00:01:19     v2    1 / S P
172.16.98.252     Vlan98                   6w6d/00:01:16     v2    1 / S P
172.16.99.252     Vlan99                   6w6d/00:01:43     v2    1 / S P
172.16.102.252    Vlan102                  6w6d/00:01:37     v2    1 / S P
172.16.104.252    Vlan104                  6w6d/00:01:16     v2    1 / S P
172.16.105.252    Vlan105                  6w6d/00:01:26     v2    1 / S P
172.16.106.252    Vlan106                  3w2d/00:01:42     v2    1 / S P
172.16.223.250    Vlan223                  4w0d/00:01:37     v2    1 / S P
172.16.224.250    Vlan224                  4w1d/00:01:41     v2    1 / S P


Vlans 223 and 224 (3rd octet) are the SVIs that form OSPF adjancies with the two Cores.

Giuseppe Larosa Wed, 05/12/2010 - 06:31
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Chris,


I think you should use the debug ip pim


to see what happens on the distribution switch


have a PC joining the group and see what happens



Hope to help

Giuseppe

cbeswick Wed, 05/12/2010 - 06:47
User Badges:

I let an existing entry in the mroute cache time out before turning on a test machine, output below.


ci_t1_3a1_dist_sw1#debug ip pim 224.1.1.3
PIM debugging is on


004216: May 12 13:36:52: PIM(0): Received RP-Reachable on Vlan224 from 172.16.255.4
004217: May 12 13:36:52: PIM(0): Received RP-Reachable on Vlan224 from 172.16.255.4
004218: May 12 13:36:52:      for group 224.1.1.3
004219: May 12 13:36:52: PIM(0): Update RP expiration timer (270 sec) for 224.1.1.3
004220: May 12 13:36:52: PIM(0): Forward RP-reachability for 224.1.1.3 on Vlan68


004221: May 12 13:37:15: PIM(0): Building Periodic (*,G) Join / (S,G,RP-bit) Prune message for 224.1.1.3
004222: May 12 13:37:15: PIM(0): Insert (*,224.1.1.3) join in nbr 172.16.224.250's queue
004223: May 12 13:37:15: PIM(0): Building Join/Prune packet for nbr 172.16.224.250
004224: May 12 13:37:15: PIM(0):  Adding v2 (172.16.255.4/32, 224.1.1.3), WC-bit, RPT-bit, S-bit Join
004225: May 12 13:37:15: PIM(0): Send v2 join/prune to 172.16.224.250 (Vlan224)


004226: May 12 13:38:14: PIM(0): Building Periodic (*,G) Join / (S,G,RP-bit) Prune message for 224.1.1.3
004227: May 12 13:38:14: PIM(0): Insert (*,224.1.1.3) join in nbr 172.16.224.250's queue
004228: May 12 13:38:14: PIM(0): Building Join/Prune packet for nbr 172.16.224.250
004229: May 12 13:38:14: PIM(0):  Adding v2 (172.16.255.4/32, 224.1.1.3), WC-bit, RPT-bit, S-bit Join
004230: May 12 13:38:14: PIM(0): Send v2 join/prune to 172.16.224.250 (Vlan224)


004231: May 12 13:38:22: PIM(0): Received RP-Reachable on Vlan224 from 172.16.255.4
004232: May 12 13:38:22: PIM(0): Received RP-Reachable on Vlan224 from 172.16.255.4
004233: May 12 13:38:22:      for group 224.1.1.3
004234: May 12 13:38:22: PIM(0): Update RP expiration timer (270 sec) for 224.1.1.3
004235: May 12 13:38:22: PIM(0): Forward RP-reachability for 224.1.1.3 on Vlan68


004236: May 12 13:38:23: PIM(0): Insert (172.16.255.4,224.1.1.3) prune in nbr 172.16.224.250's queue
004237: May 12 13:38:23: PIM(0): Building Join/Prune packet for nbr 172.16.224.250
004238: May 12 13:38:23: PIM(0):  Adding v2 (172.16.255.4/32, 224.1.1.3), WC-bit, RPT-bit, S-bit Prune
004239: May 12 13:38:23: PIM(0): Send v2 join/prune to 172.16.224.250 (Vlan224)


ci_t1_3a1_dist_sw1#sh ip mroute 224.1.1.3
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report,
       Z - Multicast Tunnel, z - MDT-data group sender,
       Y - Joined MDT-data group, y - Sending to MDT-data group
       V - RD & Vector, v - Vector
Outgoing interface flags: H - Hardware switched, A - Assert winner
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode


(*, 224.1.1.3), 00:09:06/00:02:55, RP 172.16.255.4, flags: SP
  Incoming interface: Vlan224, RPF nbr 172.16.224.250, RPF-MFD
  Outgoing interface list: Null


<>


004240: May 12 13:38:55: PIM(0): Building Triggered (*,G) Join / (S,G,RP-bit) Prune message for 224.1.1.3
004241: May 12 13:38:55: PIM(0): Insert (*,224.1.1.3) join in nbr 172.16.224.250's queue
004242: May 12 13:38:55: PIM(0): Building Join/Prune packet for nbr 172.16.224.250
004243: May 12 13:38:55: PIM(0):  Adding v2 (172.16.255.4/32, 224.1.1.3), WC-bit, RPT-bit, S-bit Join
004244: May 12 13:38:55: PIM(0): Send v2 join/prune to 172.16.224.250 (Vlan224)


004245: May 12 13:39:14: PIM(0): Building Periodic (*,G) Join / (S,G,RP-bit) Prune message for 224.1.1.3
004246: May 12 13:39:14: PIM(0): Insert (*,224.1.1.3) join in nbr 172.16.224.250's queue
004247: May 12 13:39:14: PIM(0): Building Join/Prune packet for nbr 172.16.224.250
004248: May 12 13:39:14: PIM(0):  Adding v2 (172.16.255.4/32, 224.1.1.3), WC-bit, RPT-bit, S-bit Join
004249: May 12 13:39:14: PIM(0): Send v2 join/prune to 172.16.224.250 (Vlan224)


ci_t1_3a1_dist_sw1#sh ip mroute 224.1.1.3
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report,
       Z - Multicast Tunnel, z - MDT-data group sender,
       Y - Joined MDT-data group, y - Sending to MDT-data group
       V - RD & Vector, v - Vector
Outgoing interface flags: H - Hardware switched, A - Assert winner
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode


(*, 224.1.1.3), 00:17:49/00:00:25, RP 172.16.255.4, flags: SJC
  Incoming interface: Vlan224, RPF nbr 172.16.224.250, Partial-SC
  Outgoing interface list:
    Vlan68, Forward/Sparse-Dense, 00:08:16/00:00:25, H

Giuseppe Larosa Wed, 05/12/2010 - 07:04
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Chris,

if you can repeat the same test on core switch to see the same debug output on it


we see that the switch attempts to prune from shared tree and to join source based tree but something goes wrong and it does not reach forward state for source based tree


>>>>004240: May 12 13:38:55: PIM(0):  Building Triggered (*,G) Join / (S,G,RP-bit) Prune message for 224.1.1.3
004241:  May 12 13:38:55: PIM(0): Insert (*,224.1.1.3) join in nbr  172.16.224.250's queue
004242: May 12 13:38:55: PIM(0): Building  Join/Prune packet for nbr 172.16.224.250
>>>> 004243: May 12 13:38:55:  PIM(0):  Adding v2 (172.16.255.4/32, 224.1.1.3), WC-bit, RPT-bit, S-bit  Join
004244: May 12 13:38:55: PIM(0): Send v2 join/prune to  172.16.224.250 (Vlan224)


Hope to help

Giuseppe

cbeswick Thu, 05/13/2010 - 02:58
User Badges:

Hi Guiseppe,


The next debug, showing pim on the distribution switch where the receiver / client is, and the Core / RP switch:


Distribution Switch where the receiver is

---------------------------------------------------------


004250: May 13 09:48:49: PIM(0): Building Periodic (*,G) Join / (S,G,RP-bit) Prune message for 224.1.1.3
004251: May 13 09:48:49: PIM(0): Insert (*,224.1.1.3) join in nbr 172.16.224.250's queue
004252: May 13 09:48:49: PIM(0): Building Join/Prune packet for nbr 172.16.224.250
004253: May 13 09:48:49: PIM(0):  Adding v2 (172.16.255.4/32, 224.1.1.3), WC-bit, RPT-bit, S-bit Join
004254: May 13 09:48:49: PIM(0): Send v2 join/prune to 172.16.224.250 (Vlan224)


RP and Core switch

-----------------------------


000654: .May 13 09:48:49: PIM(0): Received v2 Join/Prune on Vlan222 from 172.16.222.251, to us
000655: .May 13 09:48:49: PIM(0): Join-list: (*, 224.1.1.3), RPT-bit set, WC-bit set, S-bit set
000656: .May 13 09:48:49: PIM(0): Update Vlan222/172.16.222.251 to (*, 224.1.1.3), Forward state, by PIM *G Join
000657: .May 13 09:48:49: PIM(0): Update Vlan222/172.16.222.251 to (172.16.192.49, 224.1.1.3), Forward state, by PIM *G Join
000658: .May 13 09:48:49: PIM(0): Received v2 Join/Prune on Vlan224 from 172.16.224.251, to us
000659: .May 13 09:48:49: PIM(0): Join-list: (*, 224.1.1.3), RPT-bit set, WC-bit set, S-bit set


000660: .May 13 09:48:49: PIM(0): Update Vlan224/172.16.224.251 to (*, 224.1.1.3), Forward state, by PIM *G Join
000661: .May 13 09:48:49: PIM(0): Update Vlan224/172.16.224.251 to (172.16.192.49, 224.1.1.3), Forward state, by PIM *G Join
000662: .May 13 09:48:50: PIM(0): Building Periodic (*,G) Join / (S,G,RP-bit) Prune message for 224.1.1.3


000663: .May 13 09:48:50: PIM(0): Insert (172.16.192.49,224.1.1.3) join in nbr 172.16.228.251's queue
000664: .May 13 09:48:50: PIM(0): Building Join/Prune packet for nbr 172.16.228.251
000665: .May 13 09:48:50: PIM(0):  Adding v2 (172.16.192.49/32, 224.1.1.3), S-bit Join
000666: .May 13 09:48:50: PIM(0): Send v2 join/prune to 172.16.228.251 (Vlan228)


000667: .May 13 09:48:53: PIM(0): Received v2 Register on Vlan227 from 172.16.227.251
000668: .May 13 09:48:53:      (Data-header) for 172.16.192.49, group 224.1.1.3
000669: .May 13 09:48:53: PIM(0): Send v2 Register-Stop to 172.16.227.251 for 172.16.192.49, group 224.1.1.3
000670: .May 13 09:48:53: PIM(0): Insert (172.16.192.49,224.1.1.3) join in nbr 172.16.228.251's queue
000671: .May 13 09:48:53: PIM(0): Building Join/Prune packet for nbr 172.16.228.251
000672: .May 13 09:48:53: PIM(0):  Adding v2 (172.16.192.49/32, 224.1.1.3), S-bit Join
000673: .May 13 09:48:53: PIM(0): Send v2 join/prune to 172.16.228.251 (Vlan228)

Giuseppe Larosa Thu, 05/13/2010 - 08:54
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Chris,

sorry for late answer


RP device is behaving correctly


000658: .May 13 09:48:49:  PIM(0): Received v2 Join/Prune on Vlan224 from 172.16.224.251, to us
000659:  .May 13 09:48:49: PIM(0): Join-list: (*, 224.1.1.3), RPT-bit set,  WC-bit set, S-bit set


000660:  .May 13 09:48:49: PIM(0): Update Vlan224/172.16.224.251 to (*,  224.1.1.3), Forward state, by PIM *G Join
000661: .May 13 09:48:49:  PIM(0): Update Vlan224/172.16.224.251 to (172.16.192.49, 224.1.1.3),  Forward state, by PIM


just to know: in vlan 68 where are the users there is only the last switch or there are two switches providing L3 services to users in this vlan?


Hope to help

Giuseppe

cbeswick Fri, 05/14/2010 - 00:05
User Badges:

Hi Giuseppe,


There are two distribution switches that provide layer 3 to the receivers / clients of the multicast stream. The switch I have been debugging is the one containing the active HSRP gateway for Vlan 68.


When I look at the mroute information for 224.1.1.3 on the standby switch I see the following:


ci_t1_c075_dist_sw1#sh ip mroute 224.1.1.3
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report,
       Z - Multicast Tunnel, z - MDT-data group sender,
       Y - Joined MDT-data group, y - Sending to MDT-data group
       V - RD & Vector, v - Vector
Outgoing interface flags: H - Hardware switched, A - Assert winner
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode


(*, 224.1.1.3), 00:25:08/00:01:38, RP 172.16.255.4, flags: SP
  Incoming interface: Vlan211, RPF nbr 172.16.211.250, RPF-MFD
  Outgoing interface list: Null


I just debugged PIM on the standby switch and saw the following:


003387: .May 14 07:01:27: PIM(0): Building Periodic (*,G) Join / (S,G,RP-bit) Prune message for 224.1.1.3

003388: .May 14 07:02:19: PIM(0): Received RP-Reachable on Vlan68 from 172.16.255.4
003389: .May 14 07:02:19: PIM(0): Received RP-Reachable on Vlan68 from 172.16.255.4
003390: .May 14 07:02:19:      for group 224.1.1.3
003391: .May 14 07:02:19: PIM(0): Update RP expiration timer (270 sec) for 224.1.1.3
003392: .May 14 07:02:19: PIM(0): Not RPF interface, group 224.1.1.3
003393: .May 14 07:02:19: PIM(0): Received RP-Reachable on Vlan82 from 172.16.255.4
003394: .May 14 07:02:19: PIM(0): Received RP-Reachable on Vlan82 from 172.16.255.4
003395: .May 14 07:02:19:      for group 224.1.1.3
003396: .May 14 07:02:19: PIM(0):   Duplicate RP-reachable from 172.16.255.4 for 224.1.1.3
003397: .May 14 07:02:19: PIM(0): Received RP-Reachable on Vlan8 from 172.16.255.4
003398: .May 14 07:02:19: PIM(0): Received RP-Reachable on Vlan8 from 172.16.255.4
003399: .May 14 07:02:19:      for group 224.1.1.3
003400: .May 14 07:02:19: PIM(0):   Duplicate RP-reachable from 172.16.255.4 for 224.1.1.3
003401: .May 14 07:02:19: PIM(0): Received RP-Reachable on Vlan5 from 172.16.255.4
003402: .May 14 07:02:19: PIM(0): Received RP-Reachable on Vlan5 from 172.16.255.4
003403: .May 14 07:02:19:      for group 224.1.1.3
003404: .May 14 07:02:19: PIM(0):   Duplicate RP-reachable from 172.16.255.4 for 224.1.1.3
003405: .May 14 07:02:19: PIM(0): Received RP-Reachable on Vlan7 from 172.16.255.4
ci_t1_c075_dist_sw1#
003406: .May 14 07:02:19: PIM(0): Received RP-Reachable on Vlan7 from 172.16.255.4
003407: .May 14 07:02:19:      for group 224.1.1.3
003408: .May 14 07:02:19: PIM(0):   Duplicate RP-reachable from 172.16.255.4 for 224.1.1.3
ci_t1_c075_dist_sw1#
003409: .May 14 07:02:27: PIM(0): Building Periodic (*,G) Join / (S,G,RP-bit) Prune message for 224.1.1.3


What does the "duplicate RP-reachable" mean ? Could this be a problem ?

Giuseppe Larosa Fri, 05/14/2010 - 09:13
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Chris,

I cannot see your last post


any news?


also what platform and what IOS image is running on last switch (the one affected) ?


is IGMP snooping enabled on vlan 68?


I'm thinking of possible bugs ( I know it was working before, but sometimes they are triggered by some network changes)


Hope to help

Giuseppe

cbeswick Mon, 05/17/2010 - 00:08
User Badges:

Hi Giuseppe,


I have copied in my last post again below so you can read it. We have the same IOS / Platform across the backbone, consisting of the Sup720-3B on version 12.2(33)SXH6



There are two distribution switches that provide layer 3 to the receivers / clients of the multicast stream. The switch I have been debugging is the one containing the active HSRP gateway for Vlan 68.


When I look at the mroute information for 224.1.1.3 on the standby switch I see the following:


ci_t1_c075_dist_sw1#sh ip mroute 224.1.1.3
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report,
       Z - Multicast Tunnel, z - MDT-data group sender,
       Y - Joined MDT-data group, y - Sending to MDT-data group
       V - RD & Vector, v - Vector
Outgoing interface flags: H - Hardware switched, A - Assert winner
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode


(*, 224.1.1.3), 00:25:08/00:01:38, RP 172.16.255.4, flags: SP
  Incoming interface: Vlan211, RPF nbr 172.16.211.250, RPF-MFD
  Outgoing interface list: Null


I just debugged PIM on the standby switch and saw the following:


003387: .May 14 07:01:27: PIM(0): Building Periodic (*,G) Join / (S,G,RP-bit) Prune message for 224.1.1.3

003388: .May 14 07:02:19: PIM(0): Received RP-Reachable on Vlan68 from 172.16.255.4
003389: .May 14 07:02:19: PIM(0): Received RP-Reachable on Vlan68 from 172.16.255.4
003390: .May 14 07:02:19:      for group 224.1.1.3
003391: .May 14 07:02:19: PIM(0): Update RP expiration timer (270 sec) for 224.1.1.3
003392: .May 14 07:02:19: PIM(0): Not RPF interface, group 224.1.1.3
003393: .May 14 07:02:19: PIM(0): Received RP-Reachable on Vlan82 from 172.16.255.4
003394: .May 14 07:02:19: PIM(0): Received RP-Reachable on Vlan82 from 172.16.255.4
003395: .May 14 07:02:19:      for group 224.1.1.3
003396: .May 14 07:02:19: PIM(0):   Duplicate RP-reachable from 172.16.255.4 for 224.1.1.3
003397: .May 14 07:02:19: PIM(0): Received RP-Reachable on Vlan8 from 172.16.255.4
003398: .May 14 07:02:19: PIM(0): Received RP-Reachable on Vlan8 from 172.16.255.4
003399: .May 14 07:02:19:      for group 224.1.1.3
003400: .May 14 07:02:19: PIM(0):   Duplicate RP-reachable from 172.16.255.4 for 224.1.1.3
003401: .May 14 07:02:19: PIM(0): Received RP-Reachable on Vlan5 from 172.16.255.4
003402: .May 14 07:02:19: PIM(0): Received RP-Reachable on Vlan5 from 172.16.255.4
003403: .May 14 07:02:19:      for group 224.1.1.3
003404: .May 14 07:02:19: PIM(0):   Duplicate RP-reachable from 172.16.255.4 for 224.1.1.3
003405: .May 14 07:02:19: PIM(0): Received RP-Reachable on Vlan7 from 172.16.255.4
ci_t1_c075_dist_sw1#
003406: .May 14 07:02:19: PIM(0): Received RP-Reachable on Vlan7 from 172.16.255.4
003407: .May 14 07:02:19:      for group 224.1.1.3
003408: .May 14 07:02:19: PIM(0):   Duplicate RP-reachable from 172.16.255.4 for 224.1.1.3
ci_t1_c075_dist_sw1#
003409: .May 14 07:02:27: PIM(0): Building Periodic (*,G) Join / (S,G,RP-bit) Prune message for 224.1.1.3


What does the "duplicate RP-reachable" mean ? Could this be a problem ?

Giuseppe Larosa Mon, 05/17/2010 - 09:02
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Chris,

I've discovered that the problem is related to the specific web browser I use.


it is correct that the standby switch if it is not the PIM DR in segment (you can check this with sh ip pim neighbors) and if it does not provide a better path to the  source of multicast traffic shouldn't have interface vlan68 in its OI list.


About the other messages:


003402: .May 14 07:02:19: PIM(0): Received RP-Reachable on Vlan5 from 172.16.255.4
003403: .May 14 07:02:19:      for group 224.1.1.3

003404: .May 14 07:02:19: PIM(0):   Duplicate RP-reachable from 172.16.255.4 for 224.1.1.3


my guess is that 172.16.255.4 is the loopback address of companion switch, the two devices share multiple LAN segments (different Vlans / IP subnets).


For a reason that we haven't understood up to now, the companion switch the one that I've called last switch in some of my previous posts is trapped in PIM sparse mode on the shared tree and it is not able to join the source specific tree (that partial SC in the sh ip mroute).


these messages like the one above say that actually something strange is happening there, the standby switch receives a PIM message from companion about the involved group (224.1.1.3) on other client vlans.


I think this is not the  root cause but another symptom that affected switch is not behaving  correctly.


I would agree to clear ip mroute on it as a start and if this is not enough you could think of a reload.


Another possible action plan: make the standby switch the PIM DR on segment by using the ip pim dr-priority command in interface mode and see if this solves this can give you time to handle the misbehaving switch as described in the previous sentence


Hope to help

Giuseppe

cbeswick Wed, 05/19/2010 - 07:43
User Badges:

Hi Giuseppe,


We have a change raised this evening to clear the multicast routes. I will let you know how we get on.

cbeswick Tue, 05/25/2010 - 02:35
User Badges:

Hi Giuseppe,


We cleared the ip mroute cache and the problem still remains. I will schedule a switch reload and see if this helps.

Giuseppe Larosa Tue, 05/25/2010 - 02:40
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Chris,

thanks for your update. The issue is still there.

I would consider also making the companion switch PIM DR on segment for vlan 68 (client vlan if I remember correctly)


using interface configuration command ip pim dr-priority (check with help I'm not sure of spelling)


Hope to help

Giuseppe

cbeswick Tue, 05/25/2010 - 03:08
User Badges:

Hi Giuseppe,


I forgot to mention that I have already tried swapping the DR to the other distribution switch. The problem remains with the same output.

cbeswick Tue, 06/08/2010 - 05:45
User Badges:

Well, we have finally managed to reboot the switch to no avail. The problem still remains. I am wondering if this is a bug in the code as the problem did start to manifest shortly after we completed a complete backbone upgrade to 12.2(33)SXH6

Giuseppe Larosa Tue, 06/08/2010 - 06:23
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Chris,

nice to hear from you even if this is not good news


>> Well, we have finally managed to reboot the switch to no avail. The problem still remains. I am wondering if this is a bug in the code as the problem did start to manifest shortly after we completed a complete backbone upgrade to 12.2(33)SXH6


Unfortunately this is something that cannot be excluded!


What IOS image was running before?


Consider also moving to a 12.2(33)SXI2a or later that could fix this.




Hope to help

Giuseppe

Mohamed Sobair Tue, 06/08/2010 - 06:21
User Badges:
  • Gold, 750 points or more

Hi Chris,


could you please send a small network diagram pointing out the following:


1- The RP for the Group 224.1.1.2


2- Hosts joining the group 224.1.1.2.


3- The multicast source.




Thanks,

Mohamed

cbeswick Wed, 06/09/2010 - 00:12
User Badges:

Hi Mohamed,


Thanks for your response. I have attached a modified diagram of our backbone architecture. We have what I think is a text book configuration for multicast routing using auto-rp and pim sparse/dense mode. The two core switches are auto-rp candidates, with one of the cores acting as the rp-mapping agent.


All backone routed links are configured in sparse-dense mode. The multicast group was changed to 227.1.1.3 by our server guys just incase the group it uses by default (224.1.1.2) was causing issues. The Alteris PXE boot server that advertises this group sits on the server farm, the clients (receivers) all sit on the switch block access layers.


I am beginning to think that this is something to do with the way in which the PXE boot service operates, because we have CCTV streaming across the network fine using a proper shared multicast tree, i.e. I can see clients on the access distribution layers joining the shared tree for a source which is on another switch block.


Giuseppe - we were on software release 12.2(18)SXF8 prior to our upgrade.

Mohamed Sobair Wed, 06/09/2010 - 03:59
User Badges:
  • Gold, 750 points or more

Hello Chris,


2 issues could resulted this problem:-



1- Now, You mentioned the RP is Core 1, while when you issue (sh ip mroute 224.1.1.2), the RPF interface is Core 2??? Why ?


Answer: Although you have configured 2 RPs for the same groups for redundancy purposes, but be informed that only ONE RP is going to be elected by the hosts joining the group. The pim join message will be send from the DR to the RP with the highest IP address for that group.


So, here I would explicitly make the primary RP has the highest IP address first, Secondly , I would check the RPF.


Please post: (sh ip mroute 227.x.x.x) from the Active HSRP gateway for the recievers in that group and let us see if it points to the correct RP.


Also from the RP itself (Core 1) , post (sh ip mroute 227.x.x.x) and let us see if there is a multicast source known to the RP.



2- I would make sure that the DR for the multicast groups (224.x.x.x) and (227.x.x.x) is the Active HSRP router , as from your previous output of (sh ip mroute) from the Active HSRP router, the output is not the desired, the Forwarding (Outgoing Interface is: Null) and this shouldnt be, that means its not the PIM DR for that Group. make the DR the Active HSRP by manually setting a highest ip address on the interface OR changing the DR priority to a higer value than the current DR.



Please come back with the output of those commands and let us know the result,



HTH

Mohamed

cbeswick Thu, 06/10/2010 - 05:35
User Badges:

Hi Mohamed,


The RP is actually Core 2 (the core on the right in the diagram) - my mistake.


Output from the Active HSRP gateway for Vlan 68 - this is the Vlan I am focusing on at the moment.


ci_t1_3a1_dist_sw1#sh ip mroute 227.1.1.3
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report,
       Z - Multicast Tunnel, z - MDT-data group sender,
       Y - Joined MDT-data group, y - Sending to MDT-data group
       V - RD & Vector, v - Vector
Outgoing interface flags: H - Hardware switched, A - Assert winner
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode


(*, 227.1.1.3), 1d01h/00:02:55, RP 172.16.255.4, flags: SJC
  Incoming interface: Vlan224, RPF nbr 172.16.224.250, Partial-SC
  Outgoing interface list:
    Vlan8, Forward/Sparse-Dense, 00:00:27/00:02:32, H
    Vlan74, Forward/Sparse-Dense, 00:01:27/00:01:32, H
    Vlan73, Forward/Sparse-Dense, 00:01:57/00:01:02, H
    Vlan5, Forward/Sparse-Dense, 00:02:14/00:00:45, H
    Vlan92, Forward/Sparse-Dense, 12:29:12/00:02:27, H
    Vlan6, Forward/Sparse-Dense, 1d01h/00:02:31, H
    Vlan68, Forward/Sparse-Dense, 00:00:04/00:02:55, H


Below is the output from the RP, which is Core 2 on ip 172.16.255.4:


ci_t2_65_c200_core2#sh ip mroute 227.1.1.3
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report,
       Z - Multicast Tunnel, z - MDT-data group sender,
       Y - Joined MDT-data group, y - Sending to MDT-data group
       V - RD & Vector, v - Vector
Outgoing interface flags: H - Hardware switched, A - Assert winner
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode


(*, 227.1.1.3), 1d01h/00:02:31, RP 172.16.255.4, flags: S
  Incoming interface: Null, RPF nbr 0.0.0.0
  Outgoing interface list:
    Vlan222, Forward/Sparse-Dense, 00:00:58/00:02:31
    Vlan224, Forward/Sparse-Dense, 1d01h/00:02:30

(172.16.192.49, 227.1.1.3), 1d01h/00:01:48, flags:
  Incoming interface: Vlan228, RPF nbr 172.16.228.251
  Outgoing interface list:
    Vlan222, Forward/Sparse-Dense, 00:00:58/00:02:31
    Vlan224, Forward/Sparse-Dense, 1d01h/00:02:30



As you can see, the Multicast source, 172.16.192.49 can be seen on the RP, however the distribution switch just wont join the shared tree, it lists the *,G entries, but not the S,G.

Mohamed Sobair Thu, 06/10/2010 - 07:05
User Badges:
  • Gold, 750 points or more


Hi Chris,


The output from the RP looks fine,  However, its not from the distribution Switch.


1- You should recieve the groups (224.0.0.39) and (224.0.0.40) from the output of the sh ip mroute at the distribution switch. you have only showed a shared tree of 227.x.x.x , IS there any shared/source base tree for the groups (224.0.1.39) and (224.0.1.40) respectively?



--- You will need to make sure the mapping Agent configured correctly on Core2 the RP--



2- Do you have reachability to the source of the multicast stream from the Distribution switch?



Please confirm,



Mohamed

cbeswick Thu, 06/10/2010 - 07:13
User Badges:

Hi Mohamed,


Output of the "sh ip mroute 224.0.1.39" and "sh ip mroute 224.0.1.40" below. I didn't put this in before as I was just checking the 227.1.1.3 group. Auto RP seems to working fine.


I can see the source of the multicast tree fine on the Server farm. It is only when I look at the distribution switches to which end clients are connected that the problems occur.


ci_t1_3a1_dist_sw1#sh ip mroute 224.0.1.40
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report,
       Z - Multicast Tunnel, z - MDT-data group sender,
       Y - Joined MDT-data group, y - Sending to MDT-data group
       V - RD & Vector, v - Vector
Outgoing interface flags: H - Hardware switched, A - Assert winner
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 224.0.1.40), 2d21h/stopped, RP 0.0.0.0, flags: DCL
  Incoming interface: Null, RPF nbr 0.0.0.0
  Outgoing interface list:
    Vlan89, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan70, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan64, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan94, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan90, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan65, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan95, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan98, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan67, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan93, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan91, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan73, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan97, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan99, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan8, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan27, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan82, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan106, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan74, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan71, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan66, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan96, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan84, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan88, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan102, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan72, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan10, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan25, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan85, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan69, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan83, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan5, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan7, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan92, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan105, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan6, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan4, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan76, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan20, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan2, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan3, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan18, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan104, Forward/Sparse-Dense, 2d21h/00:00:00
    Loopback0, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan223, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan224, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan68, Forward/Sparse-Dense, 2d21h/00:00:00

(172.16.255.3, 224.0.1.40), 2d21h/00:03:02, flags: LT
  Incoming interface: Vlan223, RPF nbr 172.16.223.250
  Outgoing interface list:
    Vlan89, Prune/Sparse-Dense, 00:00:01/00:02:59, A
    Vlan70, Prune/Sparse-Dense, 00:00:01/00:02:59, A
    Vlan64, Prune/Sparse-Dense, 00:00:01/00:02:59, A
    Vlan94, Prune/Sparse-Dense, 00:00:01/00:02:59, A
    Vlan90, Prune/Sparse-Dense, 00:00:01/00:02:59, A
    Vlan65, Prune/Sparse-Dense, 00:00:01/00:02:59, A
    Loopback0, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan104, Prune/Sparse-Dense, 00:00:01/00:02:59, A
    Vlan18, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan3, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan2, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan20, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan76, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan4, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan6, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan105, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan92, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan7, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan5, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan83, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan69, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan85, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan25, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan10, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan72, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan102, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan88, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan84, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan96, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan66, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan71, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan74, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan106, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan82, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan27, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan8, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan99, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan97, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan73, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan91, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan93, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan67, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan98, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan95, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan224, Prune/Sparse-Dense, 00:00:04/00:02:56, A
    Vlan68, Prune/Sparse-Dense, 00:00:04/00:02:56, A

ci_t1_3a1_dist_sw1#sh ip mroute 224.0.1.39
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report,
       Z - Multicast Tunnel, z - MDT-data group sender,
       Y - Joined MDT-data group, y - Sending to MDT-data group
       V - RD & Vector, v - Vector
Outgoing interface flags: H - Hardware switched, A - Assert winner
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 224.0.1.39), 2d21h/stopped, RP 0.0.0.0, flags: DC
  Incoming interface: Null, RPF nbr 0.0.0.0
  Outgoing interface list:
    Vlan89, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan70, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan64, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan94, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan90, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan65, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan95, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan98, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan67, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan93, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan91, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan73, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan97, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan99, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan8, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan27, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan82, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan106, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan74, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan71, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan66, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan96, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan84, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan88, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan102, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan72, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan10, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan25, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan85, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan69, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan83, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan105, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan104, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan92, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan76, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan20, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan18, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan7, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan6, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan5, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan4, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan3, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan2, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan223, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan224, Forward/Sparse-Dense, 2d21h/00:00:00
    Vlan68, Forward/Sparse-Dense, 2d21h/00:00:00

(172.16.255.4, 224.0.1.39), 1d07h/00:02:23, flags: T
  Incoming interface: Vlan224, RPF nbr 172.16.224.250
  Outgoing interface list:
    Vlan2, Prune/Sparse-Dense, 00:00:38/00:02:21, A
    Vlan3, Prune/Sparse-Dense, 00:00:38/00:02:22, A
    Vlan4, Prune/Sparse-Dense, 00:00:38/00:02:21, A
    Vlan5, Prune/Sparse-Dense, 00:00:38/00:02:22, A
    Vlan6, Prune/Sparse-Dense, 00:00:38/00:02:21, A
    Vlan7, Prune/Sparse-Dense, 00:00:38/00:02:22, A
    Vlan18, Prune/Sparse-Dense, 00:00:38/00:02:21, A
    Vlan20, Prune/Sparse-Dense, 00:01:38/00:01:22, A
    Vlan76, Prune/Sparse-Dense, 00:00:38/00:02:21, A
    Vlan92, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan104, Prune/Sparse-Dense, 00:00:40/00:02:19, A
    Vlan105, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan83, Prune/Sparse-Dense, 00:00:40/00:02:19, A
    Vlan69, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan85, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan25, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan10, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan72, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan102, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan88, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan84, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan96, Prune/Sparse-Dense, 00:02:41/00:00:19, A
    Vlan66, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan71, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan74, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan106, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan82, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan27, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan8, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan99, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan97, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan73, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan91, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan93, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan67, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan98, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan95, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan65, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan90, Prune/Sparse-Dense, 00:01:40/00:01:20, A
    Vlan94, Prune/Sparse-Dense, 00:00:40/00:02:19, A
    Vlan64, Prune/Sparse-Dense, 00:00:40/00:02:19, A
    Vlan70, Prune/Sparse-Dense, 00:00:40/00:02:19, A
    Vlan89, Prune/Sparse-Dense, 00:00:40/00:02:19, A
    Vlan223, Forward/Sparse-Dense, 1d07h/00:00:00, A
    Vlan68, Prune/Sparse-Dense, 00:00:40/00:02:19, A

(172.16.255.3, 224.0.1.39), 2d02h/00:02:11, flags: PTX
  Incoming interface: Vlan223, RPF nbr 172.16.223.250
  Outgoing interface list:
    Vlan2, Prune/Sparse-Dense, 00:01:55/00:01:04, A
    Vlan3, Prune/Sparse-Dense, 00:01:55/00:01:04, A
    Vlan4, Prune/Sparse-Dense, 00:01:55/00:01:04, A
    Vlan5, Prune/Sparse-Dense, 00:04:55/00:01:04, A
    Vlan6, Prune/Sparse-Dense, 00:01:55/00:01:04, A
    Vlan7, Prune/Sparse-Dense, 00:04:55/00:01:04, A
    Vlan18, Prune/Sparse-Dense, 00:01:55/00:01:04, A
    Vlan20, Prune/Sparse-Dense, 00:00:55/00:02:04, A
    Vlan76, Prune/Sparse-Dense, 00:01:55/00:01:04, A
    Vlan92, Prune/Sparse-Dense, 00:00:55/00:02:04, A
    Vlan104, Prune/Sparse-Dense, 00:01:55/00:01:04, A
    Vlan105, Prune/Sparse-Dense, 00:04:55/00:01:04, A
    Vlan83, Prune/Sparse-Dense, 00:02:55/00:00:04, A
    Vlan69, Prune/Sparse-Dense, 00:03:55/00:02:04, A
    Vlan85, Prune/Sparse-Dense, 00:02:55/00:00:04, A
    Vlan25, Prune/Sparse-Dense, 00:02:55/00:00:04, A
    Vlan10, Prune/Sparse-Dense, 00:02:55/00:00:05, A
    Vlan72, Prune/Sparse-Dense, 00:05:55/00:00:04, A
    Vlan102, Prune/Sparse-Dense, 00:05:55/00:00:04, A
    Vlan88, Prune/Sparse-Dense, 00:05:55/00:00:04, A
    Vlan84, Prune/Sparse-Dense, 00:05:55/00:00:04, A
    Vlan96, Prune/Sparse-Dense, 00:05:55/00:00:04, A
    Vlan66, Prune/Sparse-Dense, 00:05:55/00:00:04, A
    Vlan71, Prune/Sparse-Dense, 00:03:55/00:02:04, A
    Vlan74, Prune/Sparse-Dense, 00:05:55/00:00:04, A
    Vlan106, Prune/Sparse-Dense, 00:03:55/00:02:04, A
    Vlan82, Prune/Sparse-Dense, 00:05:57/00:00:02, A
    Vlan27, Prune/Sparse-Dense, 00:04:57/00:01:02, A
    Vlan8, Prune/Sparse-Dense, 00:05:57/00:00:02, A
    Vlan99, Prune/Sparse-Dense, 00:05:57/00:00:02, A
    Vlan97, Prune/Sparse-Dense, 00:05:57/00:00:02, A
    Vlan73, Prune/Sparse-Dense, 00:05:57/00:00:02, A
    Vlan91, Prune/Sparse-Dense, 00:05:57/00:00:02, A
    Vlan93, Prune/Sparse-Dense, 00:05:57/00:00:02, A
    Vlan67, Prune/Sparse-Dense, 00:03:57/00:02:02, A
    Vlan98, Prune/Sparse-Dense, 00:04:57/00:01:02, A
    Vlan95, Prune/Sparse-Dense, 00:05:57/00:00:02, A
    Vlan65, Prune/Sparse-Dense, 00:05:57/00:00:02, A
    Vlan90, Prune/Sparse-Dense, 00:05:57/00:00:02, A
    Vlan94, Prune/Sparse-Dense, 00:02:57/00:00:02, A
    Vlan64, Prune/Sparse-Dense, 00:02:57/00:00:02, A
    Vlan70, Prune/Sparse-Dense, 00:02:57/00:00:02, A
    Vlan89, Prune/Sparse-Dense, 00:02:57/00:00:02, A
    Vlan224, Prune/Sparse-Dense, 00:03:57/00:02:02, A
    Vlan68, Prune/Sparse-Dense, 00:02:57/00:00:02, A

ci_t1_3a1_dist_sw1#

Mohamed Sobair Thu, 06/10/2010 - 12:50
User Badges:
  • Gold, 750 points or more

Hi Chris,


Ok, could you post (sh ip route 172.16.192.49) from the distribution switch? Is it reachable? If so, what is the next hop address for it?



HTH

MOhamed

cbeswick Fri, 06/11/2010 - 00:10
User Badges:

Hi Mohamed,


ci_t2_65_c200_core2#sh ip route 172.16.192.49
Routing entry for 172.16.192.0/24
  Known via "ospf 1", distance 110, metric 20, type extern 2, forward metric 1
  Last update from 172.16.227.251 on Vlan227, 3d14h ago
  Routing Descriptor Blocks:
  * 172.16.228.251, from 172.16.255.1, 3d14h ago, via Vlan228
      Route metric is 20, traffic share count is 1
    172.16.227.251, from 172.16.255.2, 3d14h ago, via Vlan227
      Route metric is 20, traffic share count is 1


These routes point to the Server farm Distribution layer which contain the default gateway (HSRP) for Vlan 192 in which the server resides.


The incoming Interface on the previous output (Core 2) for group 227.1.1.3 is Vlan 228 which also correlates with the above.


I have also copied in a "sh ip mroute 227.1.1.3" for the active gateway for this Vlan:


ci_t1_c050_sfdist_sw1#sh ip mroute 227.1.1.3
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report,
       Z - Multicast Tunnel, z - MDT-data group sender,
       Y - Joined MDT-data group, y - Sending to MDT-data group
       V - RD & Vector, v - Vector
Outgoing interface flags: H - Hardware switched, A - Assert winner
Timers: Uptime/Expires

Interface state: Interface, Next-Hop or VCD, State/Mode


(*, 227.1.1.3), 1d19h/stopped, RP 172.16.255.4, flags: SP
  Incoming interface: Vlan228, RPF nbr 172.16.228.250, RPF-MFD
  Outgoing interface list: Null

(172.16.192.49, 227.1.1.3), 1d19h/00:03:20, flags: T
  Incoming interface: Vlan192, RPF nbr 0.0.0.0, RPF-MFD
  Outgoing interface list:
    Vlan228, Forward/Sparse-Dense, 1d19h/00:02:52, H


As you can see the group has the correct outgoing interface (Vlan 228) which takes the stream up to the RP on Core 2. This is final hop before it gets to the server at layer 2. The only switch it traverses after this one is a Cisco 2960G which has igmp snooping enabled by default.

Mohamed Sobair Sat, 06/12/2010 - 04:18
User Badges:
  • Gold, 750 points or more

Hi Chris,


Thats fine, could you just post the result for the output from (sh ip route 172.16.192.49) from this switch: ((ci_t1_c050_sfdist_sw1))




Thanks,

Mohamed

cbeswick Mon, 06/14/2010 - 00:29
User Badges:

Hi Mohamed,


ci_t1_c050_sfdist_sw1#sh ip route 172.16.192.49
Routing entry for 172.16.192.0/24
  Known via "connected", distance 0, metric 0 (connected, via interface)
  Redistributing via ospf 1
  Advertised by ospf 1 subnets
  Routing Descriptor Blocks:
  * directly connected, via Vlan192
      Route metric is 0, traffic share count is 1


I have also copied in the configuration of the SVI on this switch;


interface Vlan192
description VL192_SF_MA
ip address 172.16.192.251 255.255.255.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip pim dr-priority 2
ip pim sparse-dense-mode
ip pim snooping
ip igmp snooping fast-leave
standby 1 ip 172.16.192.254
standby 1 timers msec 250 msec 750
standby 1 priority 200
standby 1 preempt delay minimum 240
end


I have forced the interface to be the DR for the Vlan as this switch is the active gateway for the HSRP group.

Mohamed Sobair Mon, 06/14/2010 - 02:32
User Badges:
  • Gold, 750 points or more

Hi Chris,


Apologize for this,


Vlan 68 is where you reciever is located right? if so, please post the output of (sh ip route 172.16.255.4) and (sh ip mroute 227.1.1.3) from this switch: ci_t1_3a1_dist_sw1


For Vlan 68, do you have HSRP configured?



HTH


Mohamed

cbeswick Mon, 06/14/2010 - 07:56
User Badges:

Hi Mohamed,


Yes, the receivers are on Vlan 68 (and a few others):


ci_t1_3a1_dist_sw1#sh ip route 172.16.255.4
Routing entry for 172.16.255.4/32
  Known via "ospf 1", distance 110, metric 2, type intra area
  Last update from 172.16.224.250 on Vlan224, 6d22h ago
  Routing Descriptor Blocks:
  * 172.16.224.250, from 172.16.255.4, 6d22h ago, via Vlan224
      Route metric is 2, traffic share count is 1


ci_t1_3a1_dist_sw1#sh ip mroute 227.1.1.3
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report,
       Z - Multicast Tunnel, z - MDT-data group sender,
       Y - Joined MDT-data group, y - Sending to MDT-data group
       V - RD & Vector, v - Vector
Outgoing interface flags: H - Hardware switched, A - Assert winner
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode


(*, 227.1.1.3), 05:58:50/00:02:54, RP 172.16.255.4, flags: SJC
  Incoming interface: Vlan224, RPF nbr 172.16.224.250, Partial-SC
  Outgoing interface list:
    Vlan68, Forward/Sparse-Dense, 00:02:11/00:00:48, H
    Vlan6, Forward/Sparse-Dense, 05:58:28/00:02:54, H

Giuseppe Larosa Mon, 06/14/2010 - 11:20
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Mohamed, Chris,


the strange part that we had noticed in first posts is that the mroute is stucked in

Partial-SC


and the flags

SJC


meaning that the switch is attempting to join (J) the SPT but it is not able to complete this phase.


in first posts there are also examples of debug output showing the PIM activity for the group.


At least one device of the two providing unicast and multicast routing to vlan68 should be able to complete the join to SPT.


Chris had already tried to change pim dr-priority, to clear the mroute table, even to reload the system (ar least one ) with no results.

Chris has also mentioned that the two devices had another IOS image before and they were working for this vlan68 with multicast traffic.

I searched bug toolkit but I couldn't find a good match for this issue among bugs related to PIM.


Hope to help

Giuseppe

Mohamed Sobair Mon, 06/14/2010 - 11:28
User Badges:
  • Gold, 750 points or more

Hi Chris,


Ok then, what I meant is to make vlan 68 on this switch((ci_t1_3a1_dist_sw1)) the active DR for vlan 68, not vlan 192.



On this switch ci_t1_3a1_dist_sw1, make interface Vlan 68 the Active Pim DR by changing its DR priority.


after that , please post back the (sh ip mroute 227.x.x.x) from the same switch ci_t1_3a1_dist_sw1)) as well as (sh ip igmp groups).



let us see if the problem resolved or not,


Mohamed

cbeswick Tue, 06/15/2010 - 02:18
User Badges:

Hi Mohamed,


The switch in question (ci_t1_3a1_dist_sw1) is the active PIM DR for Vlan 68.


The output of the "sh ip igmp groups" is below. Please note that the Server team have changed the Multicast network back to 225.1.1.0 on the Alteris PXE server. The server has chosen address 225.1.1.6 as the new multicast address. Everything is as before but with a different multicast group address. The entry for Vlan 68 is mid table:


IGMP Connected Group Membership
Group Address    Interface                Uptime    Expires   Last Reporter   Group Accounted
239.192.1.0      Vlan10                   1w0d      00:02:43  172.16.10.3
239.255.255.253  Vlan6                    01:16:09  00:02:52  172.16.6.243
239.255.255.253  Vlan82                   02:00:18  00:02:46  172.16.82.226
239.255.255.250  Vlan85                   2d11h     00:02:48  172.16.85.126
239.255.255.250  Vlan73                   1w0d      00:02:43  172.16.73.47
239.255.255.250  Vlan94                   1w0d      00:02:03  172.16.94.74
239.255.255.250  Vlan64                   1w0d      00:02:18  172.16.64.36
239.255.255.250  Vlan70                   1w0d      00:02:34  172.16.70.24
239.255.255.250  Vlan89                   1w0d      00:02:20  172.16.89.82
239.255.255.250  Vlan65                   1w0d      00:02:57  172.16.65.51
239.255.255.250  Vlan98                   1w0d      00:02:57  172.16.98.59
239.255.255.250  Vlan95                   1w0d      00:02:43  172.16.95.54
239.255.255.250  Vlan67                   1w0d      00:02:03  172.16.67.29
239.255.255.250  Vlan93                   1w0d      00:02:46  172.16.93.54
239.255.255.250  Vlan91                   1w0d      00:02:43  172.16.91.73
239.255.255.250  Vlan88                   1w0d      00:02:48  172.16.88.72
239.255.255.250  Vlan99                   1w0d      00:02:45  172.16.99.52
239.255.255.250  Vlan97                   1w0d      00:02:43  172.16.97.96
239.255.255.250  Vlan27                   1w0d      00:02:04  172.16.27.9
239.255.255.250  Vlan8                    1w0d      00:02:13  172.16.8.34
239.255.255.250  Vlan82                   1w0d      00:02:41  172.16.82.69
239.255.255.250  Vlan74                   1w0d      00:02:18  172.16.74.24
239.255.255.250  Vlan66                   1w0d      00:02:47  172.16.66.21
239.255.255.250  Vlan71                   1w0d      00:02:57  172.16.71.45
239.255.255.250  Vlan72                   1w0d      00:02:42  172.16.72.25
239.255.255.250  Vlan102                  1w0d      00:02:41  172.16.102.1
239.255.255.250  Vlan96                   1w0d      00:02:55  172.16.96.65
239.255.255.250  Vlan68                   1w0d      00:02:02  172.16.68.33
239.255.255.250  Vlan69                   1w0d      00:02:03  172.16.69.50
239.255.255.250  Vlan7                    1w0d      00:02:41  172.16.7.57
239.255.255.250  Vlan5                    1w0d      00:02:43  172.16.5.52
239.255.255.250  Vlan4                    1w0d      00:02:49  172.16.4.25
239.255.255.250  Vlan6                    1w0d      00:02:43  172.16.6.5
239.255.255.250  Vlan3                    1w0d      00:02:43  172.16.3.21
239.255.255.250  Vlan92                   1w0d      00:02:30  172.16.92.70
239.255.255.250  Vlan2                    1w0d      00:02:42  172.16.2.221
239.255.255.250  Vlan76                   1w0d      00:02:43  172.16.76.201
239.255.255.250  Vlan104                  1w0d      00:02:44  172.16.104.147
239.220.200.201  Vlan92                   00:01:52  00:02:38  172.16.92.52
239.220.200.201  Vlan68                   00:03:17  00:02:10  172.16.68.235
225.86.67.83     Vlan68                   3d17h     00:02:13  172.16.68.235
225.86.67.83     Vlan74                   1w0d      00:02:17  172.16.74.221
224.0.23.12      Vlan83                   1w0d      00:02:42  172.16.83.36
224.1.23.12      Vlan83                   1w0d      00:02:45  172.16.83.32
239.192.8.3      Vlan74                   6d22h     00:02:17  172.16.74.10
224.0.5.128      Vlan71                   1w0d      00:02:04  172.16.71.15
234.56.93.162    Vlan6                    3d21h     00:01:49  172.16.6.11
234.56.93.162    Vlan90                   1w0d      00:02:12  172.16.90.53
234.56.93.162    Vlan89                   1w0d      00:02:19  172.16.89.66
234.56.93.162    Vlan94                   1w0d      00:02:02  172.16.94.80
234.56.93.162    Vlan98                   1w0d      00:02:58  172.16.98.53
234.56.93.162    Vlan95                   1w0d      00:02:43  172.16.95.55
234.56.93.162    Vlan93                   1w0d      00:02:46  172.16.93.55
234.56.93.162    Vlan91                   1w0d      00:02:43  172.16.91.136
234.56.93.162    Vlan99                   1w0d      00:02:45  172.16.99.52
234.56.93.162    Vlan97                   1w0d      00:02:43  172.16.97.51
234.56.93.162    Vlan68                   1w0d      00:02:06  172.16.68.83
234.56.93.162    Vlan96                   1w0d      00:02:58  172.16.96.68
234.56.93.162    Vlan88                   1w0d      00:02:48  172.16.88.51
234.56.93.162    Vlan92                   1w0d      00:02:31  172.16.92.53
225.1.1.6        Vlan84                   00:00:03  00:02:56  172.16.84.157
225.1.1.6        Vlan68                   00:00:05  00:02:54  172.16.68.93
225.1.1.6        Vlan6                    1d00h     00:02:29  172.16.6.3
224.0.255.135    Vlan2                    6d02h     00:02:46  172.16.2.24
224.0.255.135    Vlan73                   1w0d      00:02:43  172.16.73.42
224.0.255.135    Vlan64                   1w0d      00:02:17  172.16.64.36
224.0.255.135    Vlan94                   1w0d      00:02:01  172.16.94.107
224.0.255.135    Vlan70                   1w0d      00:02:33  172.16.70.24
224.0.255.135    Vlan89                   1w0d      00:02:18  172.16.89.80
224.0.255.135    Vlan65                   1w0d      00:02:56  172.16.65.57
224.0.255.135    Vlan98                   1w0d      00:02:56  172.16.98.79
224.0.255.135    Vlan95                   1w0d      00:02:44  172.16.95.52
224.0.255.135    Vlan67                   1w0d      00:02:02  172.16.67.32
224.0.255.135    Vlan99                   1w0d      00:02:44  172.16.99.52
224.0.255.135    Vlan93                   1w0d      00:02:42  172.16.93.62
224.0.255.135    Vlan91                   1w0d      00:02:42  172.16.91.75
224.0.255.135    Vlan97                   1w0d      00:02:47  172.16.97.87
224.0.255.135    Vlan27                   1w0d      00:02:02  172.16.27.6
224.0.255.135    Vlan8                    1w0d      00:02:12  172.16.8.45
224.0.255.135    Vlan82                   1w0d      00:02:40  172.16.82.246
224.0.255.135    Vlan71                   1w0d      00:02:56  172.16.71.45
224.0.255.135    Vlan74                   1w0d      00:02:16  172.16.74.5
224.0.255.135    Vlan102                  1w0d      00:02:42  172.16.102.1
224.0.255.135    Vlan72                   1w0d      00:02:42  172.16.72.1
224.0.255.135    Vlan66                   1w0d      00:02:43  172.16.66.23
224.0.255.135    Vlan88                   1w0d      00:02:47  172.16.88.77
224.0.255.135    Vlan96                   1w0d      00:02:55  172.16.96.58
224.0.255.135    Vlan68                   1w0d      00:02:02  172.16.68.15
224.0.255.135    Vlan69                   1w0d      00:02:04  172.16.69.50
224.0.255.135    Vlan7                    1w0d      00:02:40  172.16.7.57
224.0.255.135    Vlan5                    1w0d      00:02:42  172.16.5.26
224.0.255.135    Vlan4                    1w0d      00:02:50  172.16.4.25
224.0.255.135    Vlan92                   1w0d      00:02:29  172.16.92.70
224.0.255.135    Vlan3                    1w0d      00:02:42  172.16.3.21
224.0.255.135    Vlan6                    1w0d      00:02:42  172.16.6.30
224.0.255.135    Vlan76                   1w0d      00:02:40  172.16.76.105
224.0.1.1        Loopback0                1w0d      00:02:04  172.16.255.6
225.168.0.150    Vlan74                   4d22h     00:01:23  172.16.74.223
224.0.1.60       Vlan2                    1d20h     00:02:47  172.16.2.223
224.0.1.60       Vlan71                   5d00h     00:02:56  172.16.71.225
224.0.1.60       Vlan64                   1w0d      00:02:24  172.16.64.227
224.0.1.60       Vlan65                   1w0d      00:02:02  172.16.65.223
224.0.1.60       Vlan67                   1w0d      00:02:09  172.16.67.222
224.0.1.60       Vlan68                   1w0d      00:02:08  172.16.68.222
224.0.1.60       Vlan8                    1w0d      00:02:14  172.16.8.221
224.0.1.60       Vlan82                   1w0d      00:02:42  172.16.82.225
224.0.1.60       Vlan66                   1w0d      00:02:49  172.16.66.228
224.0.1.60       Vlan5                    1w0d      00:02:47  172.16.5.242
224.0.1.60       Vlan7                    1w0d      00:02:47  172.16.7.221
224.0.1.60       Vlan6                    1w0d      00:02:42  172.16.6.222
224.0.1.39       Vlan223                  1w0d      00:02:16  172.16.223.250
224.0.1.40       Loopback0                1w0d      00:02:57  172.16.255.6
224.0.1.84       Vlan83                   1w0d      00:02:45  172.16.83.33

Giuseppe Larosa Tue, 06/15/2010 - 03:29
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Chris,


with this change of multicast address used by the server, do you see any changes?



225.1.1.6        Vlan68                   00:00:05  00:02:54  172.16.68.93


is the group able to be stable in the show ip igmp group?


Hope to help

Giuseppe

cbeswick Tue, 06/15/2010 - 03:54
User Badges:

Hi Giuseppe,


Unfortunately no change. I can see the *,G and S,G information from the server farm distribution switch to the core / RP, but the distribution switches which contain the receivers just don't join the shared tree and show the "partial-sc" message with just the *,G.

Giuseppe Larosa Tue, 06/15/2010 - 08:08
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Chris,

what if in a maintanance window you shut the SVI Vlan68 on one of the two switches?


this is a test that can tell us if the presence of both devices create some confusion, as final check to decide if this is strange but working devices or you are really facing a software issue.


also it is important to check the following


sh ip rpf

sh ip rpf


because all the game of switching to source based tree is based on this check, a sort of dead lock could be caused if distrib1 has the better path to group source address and the other device distrib2 has the best path to RP address.



Hope to help

Giuseppe

cbeswick Thu, 06/24/2010 - 04:19
User Badges:

Hi,


Just to give you an update. I have a change raised today to change our multicast configuration. I am changing all layer 3 SVIs to "pim sparse-mode" and will use the "ip pim autorp listener" command to allow auto rp to still function. I will also be applying a whole host of commands that are highlighted in the best practices guide for multicasting using the "Any Soure Mutlicast" guide.


I will let you know how I get on.

cbeswick Thu, 06/24/2010 - 07:12
User Badges:

Success!!


I can now see the routers joining the Source / Group and the stream is being fully hardware switched (RPF-MFD). I had to do a lot of reading to get back up the speed and have learnt a lot as a result of this problem. Thanks to Giuseppe and Mohamed for all your help.


ci_t1_3a1_dist_sw1#sh ip mroute 225.1.1.6

IP Multicast Routing Table

Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,

L - Local, P - Pruned, R - RP-bit set, F - Register flag,

T - SPT-bit set, J - Join SPT, M - MSDP created entry,

X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,

U - URD, I - Received Source Specific Host Report,

Z - Multicast Tunnel, z - MDT-data group sender,

Y - Joined MDT-data group, y - Sending to MDT-data group

V - RD & Vector, v - Vector

Outgoing interface flags: H - Hardware switched, A - Assert winner

Timers: Uptime/Expires

Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 225.1.1.6), 00:23:56/stopped, RP 172.16.255.4, flags: SJC

Incoming interface: Vlan224, RPF nbr 172.16.224.250, Partial-SC

Outgoing interface list:

Vlan68, Forward/Sparse, 00:01:01/00:02:27, H

Vlan84, Forward/Sparse, 00:04:18/00:01:09, H

Vlan6, Forward/Sparse, 00:23:56/00:02:49, H

(172.16.192.49, 225.1.1.6), 00:23:56/00:02:50, flags: JT

Incoming interface: Vlan224, RPF nbr 172.16.224.250, RPF-MFD

Outgoing interface list:

Vlan68, Forward/Sparse, 00:01:01/00:02:27, H

Vlan84, Forward/Sparse, 00:04:18/00:01:09, H

Vlan6, Forward/Sparse, 00:23:56/00:02:49, H

Giuseppe Larosa Thu, 06/24/2010 - 07:51
User Badges:
  • Super Silver, 17500 points or more
  • Hall of Fame,

    Founding Member

Hello Chris,



(172.16.192.49, 225.1.1.6), 00:23:56/00:02:50, flags: JT

Incoming interface: Vlan224, RPF nbr 172.16.224.250, RPF-MFD

Outgoing interface list:

Vlan68, Forward/Sparse, 00:01:01/00:02:27, H

Vlan84, Forward/Sparse, 00:04:18/00:01:09, H

Vlan6, Forward/Sparse, 00:23:56/00:02:49, H


very good news so you moved to to sparse-mode PIM + autoRP listener and this solved.




Hope to help

Giuseppe

cbeswick Thu, 06/24/2010 - 07:58
User Badges:

Hi Giuseppe,


Yes. I also enabled a few other commands including :


3) Enable the "no ip dm-fallback" command
4) Enable the "mls ip multicast non-rpf aging fast" command
5) Enable the "mls ip multicast non-rpf aging global" command
6) Enable the "mls ip multicast consistency-check" command
7) Enable the "mls ip multicast consistency-check type scan-mroute" command


On the 2 core switches I disabled multicast for directly connected interfaces as according to the best practice guides, this should only be enabled on the first hop routers.


Strangely enough the new config didnt work straight away. I had to clear the mroute's on all switches before the routers serving the distribution layer could join the source tree.


I have also enabled 2 mapping agents and 2 candidate RPs on the core, with a 3 second failover on the interval. I still have a few tweeks to do, such as optimising the DR to follow HSRP, and some filters to protect the RPs, but I am almost there.


Thanks again for your help.

Mohamed Sobair Sat, 06/26/2010 - 10:43
User Badges:
  • Gold, 750 points or more

Hi Chris,


I am very glad that you make it through and your problem is resolved. However, I dont know why I was very suspecious about The DR. as from the previous strange output you had, this could be the only reason afer we checked all neccessary config.


Good luck ...




HTH

Mohamed

cbeswick Thu, 07/01/2010 - 03:14
User Badges:

Giuseppe / Mohamed,


After all that, it appears the problem hasn't gone away after all. I have however done some more debugging and found the following output on the RP:


001410: .Jul  1 10:15:41: MRT(0): Reset the z-flag for (172.16.192.49, 225.1.1.6)
001411: .Jul  1 10:15:41: MRT(0): Create (172.16.192.49,225.1.1.6), RPF Vlan228/172.16.228.251
001412: .Jul  1 10:15:41: MRT(0): WAVL Insert interface: Vlan224 in (* ,225.1.1.6) Successful
001413: .Jul  1 10:15:41: MRT(0): set min mtu for (172.16.255.4, 225.1.1.6) 0->1500
001414: .Jul  1 10:15:41: MRT(0): Add Vlan224/225.1.1.6 to the olist of (*, 225.1.1.6), Forward state - MAC not built
001415: .Jul  1 10:15:41: MRT(0): Add Vlan224/225.1.1.6 to the olist of (*, 225.1.1.6), Forward state - MAC not built
001416: .Jul  1 10:15:41: MRT(0): WAVL Insert interface: Vlan224 in (172.16.192.49,225.1.1.6) Successful
001417: .Jul  1 10:15:41: MRT(0): set min mtu for (172.16.192.49, 225.1.1.6) 18010->1500
001418: .Jul  1 10:15:41: MRT(0): Add Vlan224/225.1.1.6 to the olist of (172.16.192.49, 225.1.1.6), Forward state - MAC not built
001419: .Jul  1 10:15:41: MRT(0): Add Vlan224/225.1.1.6 to the olist of (172.16.192.49, 225.1.1.6), Forward state - MAC not built



Strangely enough, if I clear the mroute cache for group 225.1.1.6 the recievers start joining the group and everything works. The problem starts when the source tree times out and gets pruned. When this happens, the shared tree entry (*,G) remains on the distribution switch, but then I get the above messages on the RP, which is also a mapping agent. The output looks like it is failing to create a forward state because of something to do with the mtu ?


I think the reason why I suspected the fault had cleared is because I cleared all the mroute caches once I applied the new config. The only problem was that the fault returned the very next day when people tried logging onto the network....


If I clear the mroute cache on the switches, the system works, and continues to work until the source tree times out.




Actions

This Discussion