Well I have been doing a bit of reading and IGMP snooping I believe is supposed to learn and control where the forwarding goes.   But without it,  multicast traffic will just travel over all ports on that vlan.

The connection between the routers is Layer 2 so no funky multicast routing is really needed as far as I can see.

I did try turning IGMP snopping on the HP switch and applied "cgmp leave-processing" on the C3548 see below article. 

http://www.experts-exchange.com/Networking/Telecommunications/IP_Telephony/VoIP/Q_23882007.html

But it still made no difference.  

Thanks

David

David

You are right about IGMP snooping and flooding, i just thought it might be something specific to OTV.

Can you post the output of "sh otv overlay 1" from one site 1 CSR 1.

Jon

Excuse the print screen...

I'm not sure what the issue is but happy to keep looking if you are but like i say i haven;t done this before.

Up to you really.

What does a "sh ip mroute" show ?

Jon

With packet captures I can see the IGMP v3 traffic making it accross the sites and up the vmware trunk.

Example packet captured on the trunk to the ESX host at Site 2.  10.0.0.1 is in Site 1. 

172 43.157517000 10.0.0.1 224.0.0.22 IGMPv3 60 Membership Report / Join group 239.37.37.37 for any sources

Just been thinking about this again.

Each OTV device acts as an IGMP host and sends a host membership report. What happens here is that if a host (think OTV device) sees another membership report it doesn't send it's own. This is a kind of optimisation for IGMP.

However you don't want this in your setup ie. each OTV device must send a membership report.

IGMP snooping suppresses this behaviour ie. each IGMP host has to send a membership report because they never see the other IGMP hosts membership reports.

Looking at your outputs it does seem as though only one  membership report is being seen and the other suppressed. From the CSR 1000v configuration guide -

OTV uses IGMP snooping and group membership advertisements (GM-LSPs) to learn all multicast group members from remote sites. OTV also uses IGMP snooping to detect all multicast groups in a local site.

I appreciate you tried enabling it but could you perhaps look at this again.

Edit - not saying i am right but the docs do emphasise IGMP snooping.

Jon

Could you please post configs of both router's/all routers - I'm not quite clear on the topology? Is there any transit routers/devices in between? If you are not seeing the routers in the other sites, if the overlay is up in some shape or form, almost 90% of the time its a multicast problem. Just ensure you pim enabled en routes between too. Also the RP's accessible and that there are no RPF check failures.

Also show us the output of 'show ip pim neigh'

Edited: I did an implementation for new DC's with N7K's and ASR1002X's - I took the pim passive command out and enabled pim sparse as usual, all started working. Pim neighbor was not up. Jon is also correct with the igmp comments to my understanding.

Please rate useful posts & remember to mark any solved questions as answered. Thank you.

Hi Bilal,

I rekon it is multicast as well, just can't see why.  the CSR's are linked via layer 2 going through 2 physical switches.

The swithes have there own VTP domain and are linked via access ports,  unfortunately they are not great switches,  an HP and a Linksys.

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2014.02.27 09:23:27 =~=~=~=~=~=~=~=~=~=~=~=
sh run
Building configuration...

Current configuration : 3995 bytes
!
! Last configuration change at 08:37:44 UTC Thu Feb 27 2014
!
version 15.4
service timestamps debug datetime msec
service timestamps log datetime msec
no platform punt-keepalive disable-kernel-core
platform console virtual
!
hostname uk-lab2-asr-1
!
boot-start-marker
boot-end-marker
!
!
enable secret 5 $1$GbRT$mBduePO9aR7yex0LGJ8Ct1
!
aaa new-model
!
!
aaa authentication login default local
aaa authentication enable default enable
aaa authorization exec default local
!
!
!
!
!
aaa session-id common
!
!
!
!
!
!
!

ip multicast-routing distributed
no ip domain lookup
ip domain name test.co.uk
!
!
!
!
!
!
!
!
!
!
subscriber templating
!
otv site bridge-domain 352
!
otv site-identifier 0002.0002.0002
multilink bundle-name authenticated
!
crypto pki trustpoint TP-self-signed-2933701141
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-2933701141
revocation-check none
rsakeypair TP-self-signed-2933701141
!
!
crypto pki certificate chain TP-self-signed-2933701141
certificate self-signed 01
  3082022B 30820194 A0030201 02020101 300D0609 2A864886 F70D0101 05050030
  31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274
  69666963 6174652D 32393333 37303131 3431301E 170D3134 30323133 31353335
  35325A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649
  4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D32 39333337
  30313134 3130819F 300D0609 2A864886 F70D0101 01050003 818D0030 81890281
  8100DA0C DA33BF64 F53A5AF5 E2B23F64 29F3FD7F CB5DBADA F6AA3864 FACEED07
  F382C8E0 EEC46C3D 9B1DD2FF 83565E68 A08DAF22 BBB3DED6 E47EBB6F 40C4C67E
  35BC7B24 5020B052 C7884341 8B763CBB 01DBBEEA D870CE23 B40292EA 516812ED
  F2EB0F05 C85BA1A4 1A2C401A 1BD00E0E F7A1C599 6FFAA3DD A0CA4A60 A6B087B3
  B85B0203 010001A3 53305130 0F060355 1D130101 FF040530 030101FF 301F0603
  551D2304 18301680 14AC647E DBB4D148 10DCC37E 529E8C7B BF27CBA1 5F301D06
  03551D0E 04160414 AC647EDB B4D14810 DCC37E52 9E8C7BBF 27CBA15F 300D0609
  2A864886 F70D0101 05050003 8181002C F73FD7EA 7EF9DF33 BAA241F1 A643EBE7
  8296CE0D 3CA16435 ACAB21B3 E3F3EC54 F8A720A5 AD6E09C7 C304F16E A2C97100
  F0C93A86 AF5CBC34 8D0B41B5 E745049E 83A6508E BF6B864F 82B794F8 1FA69759
  25FF5142 B8852E81 50C9A543 E17A4122 134B0827 47260D0F 0B01162A E8F661BD
  404404C7 043B8598 3FE4A839 4A16F7
  quit
license udi pid CSR1000V sn 901VI2BA36Y
license boot level premium
spanning-tree extend system-id
!
username router password 0 password
!
redundancy
mode none
bridge-domain 352
!
!
!
!
!
!
ip ssh rsa keypair-name ssh-key
ip ssh version 2
ip scp server enable
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
interface Overlay1
no ip address
otv control-group 239.37.37.37
otv data-group 232.37.37.0/24
otv join-interface GigabitEthernet2
!
interface VirtualPortGroup0
ip unnumbered GigabitEthernet3
no mop enabled
no mop sysid
!
interface GigabitEthernet1
ip address 10.10.0.2 255.255.255.0
negotiation auto
!
interface GigabitEthernet2
ip address 10.0.0.11 255.255.255.0
ip pim passive
ip igmp version 3
negotiation auto
!
interface GigabitEthernet3
ip address 10.100.0.121 255.255.255.0
negotiation auto
!
interface GigabitEthernet4
no ip address
negotiation auto
service instance 352 ethernet
  encapsulation dot1q 352
  bridge-domain 352
!
service instance 400 ethernet
  encapsulation dot1q 400
  bridge-domain 400
!
!
interface GigabitEthernet5
no ip address
shutdown
negotiation auto
!
router ospf 1
router-id 172.20.2.11
network 172.20.2.0 0.0.0.255 area 0
!
router bgp 300
bgp log-neighbor-changes
bgp redistribute-internal
redistribute ospf 1
neighbor 10.0.0.1 remote-as 300
neighbor 10.0.0.2 remote-as 300
neighbor 10.0.0.12 remote-as 300
!
!
virtual-service csr_mgmt
activate
!
ip forward-protocol nd
!
ip pim send-rp-discovery scope 16
no ip http server
ip http secure-server
!
!
!
!
!
!
!
control-plane
!
!
line con 0
stopbits 1
line aux 0
stopbits 1
line vty 0 4
length 0
transport input none
line vty 5 43
!
!
end

uk-lab2-asr-1#

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2014.02.27 09:22:17 =~=~=~=~=~=~=~=~=~=~=~=
sh run
Building configuration...

Current configuration : 3994 bytes
!
! Last configuration change at 09:13:03 UTC Thu Feb 27 2014
!
version 15.4
service timestamps debug datetime msec
service timestamps log datetime msec
no platform punt-keepalive disable-kernel-core
platform console virtual
!
hostname uk-lab1-asr-1
!
boot-start-marker
boot-end-marker
!
!
enable secret 5 $1$H757$OL9WBB5XLVB.UXo1cWqRq1
!
aaa new-model
!
!
aaa authentication login default local
aaa authentication enable default enable
aaa authorization exec default local
!
!
!
!
!
aaa session-id common
!
!
!
!
!
!
!

ip multicast-routing distributed
no ip domain lookup
ip domain name test.co.uk
!
!
!
!
!
!
!
!
!
!
subscriber templating
!
otv site bridge-domain 351
!
otv site-identifier 0001.0001.0001
multilink bundle-name authenticated
!
crypto pki trustpoint TP-self-signed-3862702558
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-3862702558
revocation-check none
rsakeypair TP-self-signed-3862702558
!
!
crypto pki certificate chain TP-self-signed-3862702558
certificate self-signed 01
  3082022B 30820194 A0030201 02020101 300D0609 2A864886 F70D0101 05050030
  31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274
  69666963 6174652D 33383632 37303235 3538301E 170D3134 30323133 30393233
  34365A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649
  4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D33 38363237
  30323535 3830819F 300D0609 2A864886 F70D0101 01050003 818D0030 81890281
  810088DA 9697C97A 930C3A8D D267179F 698523A4 07F3FA3F 34BBDDBA DDDF29DB
  FF4FDBAF 4F4E1ABF 13C43D66 12EA74C7 5FCCE276 A1948142 CB86B9F4 0A05B616
  3E907BF7 96D0A47B 1A512413 1880BA79 CA3088AD 6D425FA9 1F0DC324 45B252EA
  1EEA753A 25F9A539 68E43C37 D53EE322 74597163 127A3B00 135FAB5F EDA42101
  32090203 010001A3 53305130 0F060355 1D130101 FF040530 030101FF 301F0603
  551D2304 18301680 141628C8 3E68B5C3 C2B27F93 49AF7031 4EB3AC01 52301D06
  03551D0E 04160414 1628C83E 68B5C3C2 B27F9349 AF70314E B3AC0152 300D0609
  2A864886 F70D0101 05050003 8181007D 46A40ACF 1E5A5350 A1AD2A38 D8E19A09
  F3421437 B67AC727 ACCDF152 2F279BDA C48F1ACF C0F7A50F C09F1BCE 8E67466A
  87787BD5 8200B247 D0080F4A 70E05920 F056690F D299CE7E 70F0E594 D33CAF3A
  7711ED8F 1E8F1E6C A98A476A A3118E59 6712B895 47001E33 B4BB68CB 2DA93A9A
  831986DC B9268D67 B8956C96 D39F1F
  quit
license udi pid CSR1000V sn 9JJQFT8YK0X
license accept end user agreement
license boot level premium
spanning-tree extend system-id
!
username router password 0 password
!
redundancy
mode none
bridge-domain 351
!
!
!
!
!
!
ip ssh rsa keypair-name ssh-key
ip ssh version 2
ip scp server enable
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
interface Overlay1
no ip address
otv control-group 239.37.37.37
otv data-group 232.37.37.0/24
otv join-interface GigabitEthernet2
!
interface VirtualPortGroup0
ip unnumbered GigabitEthernet3
no mop enabled
no mop sysid
!
interface GigabitEthernet1
ip address 10.20.0.2 255.255.255.0
negotiation auto
!
interface GigabitEthernet2
ip address 10.0.0.1 255.255.255.0
ip pim passive
ip igmp version 3
negotiation auto
!
interface GigabitEthernet3
ip address 10.100.0.120 255.255.255.0
negotiation auto
!
interface GigabitEthernet4
no ip address
negotiation auto
service instance 351 ethernet
  encapsulation dot1q 351
  bridge-domain 351
!
service instance 400 ethernet
  encapsulation dot1q 400
  bridge-domain 400
!
!
interface GigabitEthernet5
no ip address
shutdown
negotiation auto
!
router ospf 1
router-id 172.21.1.11
network 172.20.1.0 0.0.0.255 area 0
!
router bgp 300
bgp log-neighbor-changes
bgp redistribute-internal
redistribute ospf 1
neighbor 10.0.0.2 remote-as 300
neighbor 10.0.0.11 remote-as 300
neighbor 10.0.0.12 remote-as 300
!
!
!
ip forward-protocol nd
!
ip pim send-rp-discovery scope 16
no ip http server
ip http secure-server
!
!
!
!
!
!
!
control-plane
!
!
line con 0
stopbits 1
line aux 0
stopbits 1
line vty 0 4
length 0
transport input none
line vty 5 43
!
!
end

uk-lab1-asr-1#

And,  'show ip pim neigh' has no entries.    I have shutdown the second CSR at each site,  so there is only 1 CSR at each site.

Hi Jon,

To make it simpler I have shutdown the 2nd CSR at each site so there is only 1 at each site. 

• The C3500XL does not support IGMP snooping. 
  
• I replaced it with a Linksys switch and enabled IGMP snooping on VLAN 300 on both switches,  it still made no made no difference.
  
• I know the HP switch is new and supports IGMP version 3.   So I connected both ESX hosts on a trunk port directly to the HP removing the 2nd switch from the environmnet.  Still made no difference.
  
• I migrated the CSR's to the same ESX host and the adjaceny worked!
  
• Just FYI the VLAN 300 is passing traffic,  and BGP is peering fine.
  

I rekon it must be something to do with IGMP on the HP not playing ball,  even though I can see with packet trace it is not droping IGMP traffic.        I guess I need better lab equipement.

Thanks

David

Hello David,

I assume you can ping from and to the WAN interfaces (gi2). Could you please try to remove the command ip pim passive from both sides and enable ip pim sparse-mode instead please, when you do this, do you see the pim neighborship come up.

A thing to note though are MTU sizes. Remember OTV has overheads and you will need to support 1542 or above on the WAN I think.

"show otv log event" may give clues? The site adjacency seemed fine to me. Just the WAN to the other routers which was trouble.

Please rate useful posts & remember to mark any solved questions as answered. Thank you.

Yes Gi2 can ping.

I tried pim sparse-mode which I did before and still nothing shows in neighborship.

The adjaceny should show the routers in the other site,  right now there is nothing listed in adjacency because I turned the 2nd CSR at each site off.

The MTU seems to be set to 1500 all round,   and all the devices do support higher.    I dont see any error packets on the nic interfaces.  

Overlay Interface Overlay1

VPN name                 : None

VPN ID                   : 1

State                    : UP

AED Capable              : No, overlay DIS not elected

IPv4 control group       : 239.37.37.37

Mcast data group range(s): 232.37.37.0/24

Join interface(s)        : GigabitEthernet2

Join IPv4 address        : 10.0.0.11

Tunnel interface(s)      : Tunnel0

Encapsulation format     : GRE/IPv4

Site Bridge-Domain       : 352

Capability               : Multicast-reachable

Is Adjacency Server      : No

Adj Server Configured    : No

Prim/Sec Adj Svr(s)      : None

Nothing to helpful:

uk-lab1-asr-1#sh otv log ev
[02/27/14 15:11:25.481 UTC 1 3] OTV-APP-DB: otv-app Database initializing (Overlay table size = 4104 bytes (513 subblocks))
[02/27/14 15:11:25.483 UTC 2 3] OTV-APP-DB: Created otv_app subblock for overlay 1 VPN 1
[02/27/14 15:11:25.488 UTC 3 3] OTV-APP-OVERLAY: Updating Overlay1 state, current state DOWN

[02/27/14 15:11:25.488 UTC 4 3] OTV-APP-OVERLAY:   overlay1 state is DOWN, Missing Parameter: Control Group Address

[02/27/14 15:11:25.502 UTC 5 3] OTV-APP-UI: Set control-group on Overlay1
[02/27/14 15:11:25.503 UTC 6 3] OTV-APP-UI: Set data-group on Overlay1
[02/27/14 15:11:25.503 UTC 7 3] OTV-APP-UI: Set join-interface GigabitEthernet2 on Overlay1
[02/27/14 15:11:26.796 UTC 8 3] OTV-APP-EVC: Added site EFP 0xAF1D2E28 to list, 1 EFPs in list
[02/27/14 15:11:28.723 UTC 9 497] OTV-APP-EVENT: Event thread started
[02/27/14 15:11:29.197 UTC A 497] OTV-APP-EVENT: Processing event type OTV_APP_EV_OVERLAY_FIRST_ADD_LAST_DEL
[02/27/14 15:11:29.197 UTC B 497] OTV-APP-EVENT: Processing event type OTV_APP_EV_MLRIB_SEND_REDIST_REG_MSG
[02/27/14 15:11:29.198 UTC C 497] OTV-APP-L2RIB: Sending ucast redist register msg for VPN 1 prod 0
[02/27/14 15:11:29.198 UTC D 497] OTV-APP-L2RIB: Sending mcast redist register msg for VPN 1 prod 1
[02/27/14 15:11:29.198 UTC E 497] OTV-APP-EVENT: Processing event type OTV_APP_EV_CFG_SUPPRESS_ARP
[02/27/14 15:11:29.198 UTC F 497] OTV-APP-EVENT: Processing event type OTV_APP_EV_INSTALL_FHRP_ACL
[02/27/14 15:11:29.263 UTC 10 497] OTV-APP-OVERLAY: install FHRP ACL for Overlay1
[02/27/14 15:11:29.478 UTC 11 497] OTV-APP-EVENT: Processing event type OTV_APP_EV_WALK_TOPOLOGIES
[02/27/14 15:11:29.478 UTC 12 497] OTV-APP-EVC: Event: Walk topologies for VPN 1
[02/27/14 15:11:29.479 UTC 13 497] OTV-APP-EVENT: Processing event type OTV_APP_EV_CFG_CONTROL_GROUP
[02/27/14 15:11:29.479 UTC 14 497] OTV-APP-OVERLAY: Updating Overlay1 state, current state DOWN

[02/27/14 15:11:29.479 UTC 15 497] OTV-APP-OVERLAY:   overlay1 state is DOWN, External Interface State Down

[02/27/14 15:11:29.479 UTC 16 497] OTV-APP-EVENT: Processing event type OTV_APP_EV_CFG_DATA_GROUP
[02/27/14 15:11:29.479 UTC 17 497] OTV-APP-MCAST: Mapped 0 (S,G) to new data-groups
[02/27/14 15:11:29.479 UTC 18 497] OTV-APP-MCAST: Successfully added data-group range [232.37.37.0, 232.37.37.255] on overlay 1
[02/27/14 15:11:29.479 UTC 19 497] OTV-APP-OVERLAY: Updating Overlay1 state, current state DOWN

[02/27/14 15:11:29.479 UTC 1A 497] OTV-APP-OVERLAY:   overlay1 state is DOWN, External Interface State Down

[02/27/14 15:11:29.479 UTC 1B 497] OTV-APP-EVENT: Processing event type OTV_APP_EV_CFG_JOIN_INTF
[02/27/14 15:11:29.640 UTC 1C 497] OTV-APP-MCAST: Overlay 1 updated 0 (S,G) mappings with new DS: 10.0.0.1 (old DS: NoAddress(Type=0))
[02/27/14 15:11:29.640 UTC 1D 497] OTV-APP-OVERLAY: Overlay 1 updated core address to 10.0.0.1
[02/27/14 15:11:29.640 UTC 1E 497] OTV-APP-OVERLAY: Updating Overlay1 state, current state DOWN

[02/27/14 15:11:29.640 UTC 1F 497] OTV-APP-OVERLAY:   overlay1 state is DOWN, External Interface State Down

[02/27/14 15:11:29.640 UTC 20 497] OTV-APP-OVERLAY: Updating Overlay1 state, current state DOWN

[02/27/14 15:11:29.640 UTC 21 497] OTV-APP-OVERLAY:   overlay1 state is DOWN, External Interface State Down

[02/27/14 15:11:29.640 UTC 22 497] OTV-APP-EVENT: Processing event type OTV_APP_EV_MLRIB_EVENT_RCVD
[02/27/14 15:11:29.640 UTC 23 497] OTV-APP-EVENT: Processing event type OTV_APP_EV_MLRIB_EVENT_RCVD
[02/27/14 15:11:32.028 UTC 24 64] OTV-APP-OVERLAY: Updating Overlay1 state, current state DOWN

[02/27/14 15:11:32.028 UTC 25 64] OTV-APP-OVERLAY:   overlay1 state is UP

[02/27/14 15:11:32.049 UTC 26 64] OTV-APP-OVERLAY: Updating Overlay1 state, current state UP

[02/27/14 15:11:32.049 UTC 27 64] OTV-APP-OVERLAY:   overlay1 state is UP

[02/27/14 15:11:32.071 UTC 28 64] OTV-APP-EVC: All EFPs up setting site IDB up
40 entries printed