we have the following setup.
ISP <------- Firewall <----------- Switch
Nexus 2----------- Nexus 1
| (VPC) |
User Switch stack - 3 cisco 2950
Initally we just had one connection from the switch to the nexus. I configured vpc, thereby giving a secondary connection to each nexus. both nexus are connected using vpc. the end user switch is a stack of 3 2950 cisco switches. for some reason only for a few people, the internet does not seem to work (maybe a switch in the stack) - but when you remove the secondary connection the one of the nexus the internet seems to work for everyone. any thought s?
It seems that your WAN is connected to Nexus 2 only. Am I right? If so Nexus will need to pass traffic coming to Nexus 1 to VSL link and it then shoudl come bacj through it. In certain cases Nexus can drop traffic on VSL link due to loop avoidance mechanism. You can use peer-gateway feature on your setup. It will make Nexus 2 to send traffic to access on behalf on Nexus 1 not sending that through VSL link.
Thanks Nikolay for the reply.
yes there is only one link from wan router connected to nexus 2. We have indivigual static routes on each nexus each pointing to the internet firewall. i understand that the edge switch can load balance (vpc) between the links to nexus 1 and nexus 2. so as per your note, if the traffic is passing from the edge switch to nexus 1 (say on load balancing), then nexus 1 should be able to route back to the firewall since it has a static route. (but the fact that few users can access the internet and rest cant makes me think what you say is possible as the few users can be connected to the switch in the stack that uses the nexus 2 and the users that cant connect can be using the nexus 1 route via vpc load balancing)
still if i am wrong and the only way to sort out this is using peer gateway, how do i go about configuring peer-gateway and it which router should it be connected ?
on a side note if i configure vpc to the wan switch (at the moment only switch is connected to nexus 2) will this problem be sorted ?
Basically Peer-gateway is a feature which is configured with a single command under the VPC domain:
Though still not sure if that can help here. SO if we think of the traffic path I guess packet from 3750 coming e.g. to Nexus 1 in VLAN 101 first. Then Nexus 1 doing intervlan routing between VLAN 101 and 100 based on routing table and further switch traffic to Upper (on diagram) switch.
Interesting thing is how return traffic is coming back, Firewall shoudl send it to particular MAC of one of the Nexus switches within Vlan 100, depending on load-balancing (if present) or better routing. If there is a chance that FW sending return traffic to the hosts to the MAC of Nexus 1 than there is a chance we hit the problem which can be solved by peer-gateway. This feature is fixing the issue when Nexus2 receiving packet sent to Nexus 1 MAC which further should be sent through VPC. These packets can be dropped by VS link due to loop avoidance mechanism.
So hard to say without knowledge of the packet flow. So peer-gateway feature can be tried if your network allowing this change.
I have been going through cisco website and about peer-gateway.
I think we are in a sitation of single homed device or orphan port - where the WAN switch is just connected to the nexus 2.
Also the edge switch have hsrp. If the packet from edge switch takes the nexus 2 path then it goes to the wan switch and the problem is sorted. Say a packet that comes from the edge switch take the nexus 1 path (port channel load balancing on edge switch) then nexus 1 cant route through the peer-link (as per the rule of vpc) and thus the packet gets dropped, isnt it ?
So if we had peer gateway on both nexus, will this problem be sorted ? will the edge switch send the packets to the nexus 2 as the the wan switch is connected to the nexus 2 ? (also i beleive we have to disable ip redirects during peer-gateway implementation ) - am i correct ?
I have attached the diagram of what our present setup is ?
any thoughts on the above ? as this is in production network and affecting
i understand that the vpc peer link is used if the secondary connection to a switch fails and therby it uses the vpc peer link to get to that switch....what will happen if tehre is a device connected to nexus 1 and the edge switch routes packets to nexus 2 (due to port channel load balancing ) - will this go through the peer link ?
my network setup is similar in a way as the wan switch is connected the nexus 2 which is like connecting a device to a nexus ?
Of course this is affecting production traffic. Remember how HSRP works under vPC? - Both HSRP active and standby routers are forwarding traffic.
In order to privide outbound connectivity for traffic hitting Nexus-1 with vPC and HSRP, you will need a dedicated L3 link between Nexus-2 and Nexus-1. This way traffic hitting Nexus-1's SVI will be able to use the L3 link to send traffic toward the single uplink of your WAN switch.
If access switch will send packet to Nexus 1 - it will for sure go through peer-link as the next hop MAC learnt from it. It will not yet dropped by peer-link as the packet is not leaving VPC. But the return packet coming from server to Nexus 2. If by any chance that will have destination MAC of Nexus 1 and destination ip of PC located behind VPC - then it will be dropped on peer-link.
With a peer-link this issue will be sorted as Nexus 2 receiving packet with destination MAC of Nexus 1 will not send that on peer-link but will send through VPC on behalf of Nexus 1. So this is smth to try.
thanks both for your time however i am confused now with the concept of peer-link. when will the packets travel through the peer-link and when will it be dropped.
think in the above diagram i missed out the switch. I have attached a new diagram. the 2 nexus's and l3 wan switch are running ospf. the 2 nexus's are conencted using vpc. and the l3 wan switch is conncted to nexus 2 via l2 link. now how do i acheive the above. will peer-gateway problem fix this ? and what would be the solution to a L3 switch that is connected to a orphan port (here in my topology)
(also we need to vpc wan switch at a later stage after vpc the edge switch - what would be your recommandation ?)
any help appreciated !
Please take a look at the following link:
basically, vPC peer-link is used to sync L2 information between both N7Ks. In terms of what is going through the peer-link, traffic destined to single attach host/switch (a.k.a. orphan device) will be allowed to go through the peer-link.
Like I said before, you are missing a L3 link between the N7Ks, please take a look at the below link which explain how HSRP works under vPC. In a very short sentance, both HSRP router (active and standby) will forward traffic outbound. If the traffic is on the Nexus-B, without the L3 link, traffic will be black holed.
What do you mean by vPC WAN? Do you mean another L3 device running OSPF? If this is the case, you should use L3 Equal Cost MultiPath (ECMP). Please do not use vPC since running routing protocol over vPC is not supported.
so jerry, you think that the traffic comes to nexus 1 (in my case - pls refer topology diagram) and gets blackholed, isnt it. but what i am thinking is since the traffic comes to nexus 1 and it sees the device (L3 switch) is connected to nexus 2 directly (a.k.a orphan port), wont it pass traffic through peer-link (peer-link cant drop the traffic as the traffic would flow thorugh a non vpc port - in my case l3 switch a.k,.a orphan port) - please correct me if i am missing a point in this ?
And i meant we need to vpc the l3 switch (wan switch) - i read on the cisco website, saying any l3 device needs l3 link to vpc and cant be vpc over l2 links - but this ist he second stage of my implemenataion and not worried about it too much at this time. i am just worried about the edge switch vpc as its affecting our production network.
Yes, it will be black holed (assuming your network 10.1.1.x is passive interface, and it should be passive). If you look at your config closely, when you turn on peer-gateway, it will turn on no ip redirect automatically. Traffic will not send over the Nexus-A if it hits Nexus-B (HSRP interaction with vPC) unless you have L3 between Nexus-A and B.
BTW, the L3 switch in Nexus-A is not consider orphan port, it is a L3 interface. When we said orphan, it is refers to L2 interfaces.
I don't think peer OSPF between the 10.1.1.x network is a supported design.
the orphan port you mentioned is a l2 port. (l3 wan switch connected to the nexus 2 (Nexus B) in my case is connected using a l2 link so it should be considered as a orphan port isnt it ? so in theory it should hit the WAN switch (even though it runs ospf) but is connected to the nexus using a l2 link
and edge switch is l2 as well and connecteed through l2 link
Your L3 switch has an IP address of 10.5.5.x/24. What is on the Nexus side, is that on the interface or SVI? If it is SVI, is that extended over peer-link to the other Nexus?
sorry that was just a diagram i got from internet with their ip addressing. I have corrected this to a similar one now
all the svi exits on the l3 switch and both nexus. also there is static route on each nexus pointing to firewall.
I tried vpc on a test switch (gave connections to each nexus) and i cant seem to ping the l3 switch from my pc which was connected to the test switch. and pinging the l3 switch (10.1.1.30) didnt seem to work either from the neuxs (lost 2 packets out of every 5)
so as you said there is a blackhole. now will the peer-gateway problem sort this out or a l3 link between both nexus sort oyt his problem ?
any thoughts nikolay and jeye ?
the above is just a representation as the ip address used are different.
so do you think peer-gateway or adding a l3 link between both nexus would sort the problem ? if we add a l3 link between both nexus will it affect our exisitng network as we have svi on both nexus and l3 wan switch and other sites connected and we run ospf. does this new l3 link need to be advertised into ospf ?
So, from this post, you are using the L3 switch as your SVIs.
This interface from the "L3 switch" is consider as orphan. If you can share the Nexus 7000 config and the L3 switch, it would be great. I am still confused with topology.
In terms of the ping lost, it is normal. The Nexus has CoPP which will drop ping.
can i have your email address so that i can mail you the actual config with topology. my boss would not like to post hte configs on the internet
Did you get a resolution to this problem?
I am going to look at a network that seems to have the same problem, the next downtime slot is at the end of November amd am doing some reading up on the issue first to prepare.
When I came across this thread I was hoping that it may have had some details of the fix posted, assuming it did get fixed.
hi havent sorted this out yet but the resolution is if you have a second link from the switch to nexus (swithport or have both links routed - preferred) would sort the problem out