WCCP problem or routing

Answered Question
Sep 6th, 2010

Hi,

We have two datacenters same logical LAN.

Two ISP routers and two WAE 674 and using WCCP "egress-method negotiated-return intercept-method wccp"

See attached file.

The problem is when one of the "line" WAN interface goes down, some of the network are not reach from the LAN side and some are.

We are using BGP as routing protocol in the ISP routers.

Any suggestion for the problem?

Jan

wccp.png

I have this problem too.
0 votes
Correct Answer by Bhavin Yadav about 6 years 2 months ago

Hi Jan,

You can open a TAC case and they can assist you to retain confidentiality.

Further, about the static routes , BGP and RIP, I am not sure if I am the right person to answer that. You probably want to put this across Routing forum to get a more specific answer.

From WCCP perspective, WCCP will only change its status if the WCCP goes down by means of non-rechanble router / CE. Otherwise, if the WAN interface goes down, WCCP may still remain alive depending on your configuration because all WCCP cares is redirect traffic to CE and receive it back from CE and then forward it to gateway. If the gateway is not reachable, connectivity breaks but not WCCP. So, you want to make sure to probably add a static route in router to forward the traffic to backup gateway if the WAN link goes down.

Hope this helps.

Regards.

PS: Please mark this as Answered, if this resolves your issue.

Correct Answer by Bhavin Yadav about 6 years 2 months ago

Hi Jan,

What is suspect here is WCCP.

When you configure, it allocates buckets if you are using hash assignment. If you are using Mask, it calculates mask depending on your destination / source ip address.

Now, what I understand is - if WAN goes down, some networks are not reachable.

When LAN goes down, WCCP goes down and everything starts running smooth.

Few questions:

1. What happens if LAN goes down but WAN remains up? Does WCCP remain active / usable state?

2. When WAN goes down and LAN remains up, your WCCP is still UP and hence, it continues to forward packets out of same WAN interface but  because that interface is down, packets ultimately die / gets blackholed.

3. Another speculation is: Asymmetric routing. When WAN is down but LAN is up, you are forwarding soem traffic out of LAN but as WAN goes down, the return packets then come up on different interface and creates asymmetric routing.

To narrow down this problem, please capture following interface during three stages:

1. WAN - Down, LAN-UP, is router ID reachable ???

2. WAN-UP, LAN- - Down, is router ID reachable ???

3. WAN Down, LAN Down, is router ID reachable  ???

CLI logs to capture:

    2. show ip wccp
    3. show ip wccp interface detail
    4. show ip wccp service
    5. show ip wccp detail
    6. show ip wccp internal (*)
    7. show running-config
    8. show ip wccp <61 / 62> hash
    9. sh wccp mask tcp-promis
    10. sh tech

Further, as you are using GRE encapsulation for WCCP redirection, the  router uses the router ID IP address as its source IP address. The  router ID IP address is the highest loopback address on the router, or  if the loopback interface is not configured, the router ID IP address is  the highest address of the physical interfaces. The router ID IP  address is used as the source address for packets redirected from the  router to the Cisco WAE, and as a result it is also used as the  destination address for traffic from the Cisco WAE to the router,  Therefore, you must be sure that a route exists from the Cisco WAE to  the router. This is done by configuring a static route on the Cisco WAE  to the router ID IP address. The router ID can be identified with the  command show wccp routers on the Cisco WAE.

As in your case, you have multiple routers, a  static route should be added to each of these router's router ID. The  command to configure such static routes is:

WAE(config)# ip route

Can you please try above and let us know if it works ?

Regards.




  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (2 ratings)
Loading.
Jan Rockstedt Sun, 09/12/2010 - 08:09

Hi again,

We have now tested again and shutdown the WAN interface one of the ISP routers, the TCP traffic din't work after that from the WAN.

There was one router up with right routing table and the ICMP was working.

Then we shutdown the LAN interface on the ISP router and that cleaned up the WCCP some how.

Both are now working again as it should be with redirect.

Could it be an bug?

This is the version we are using:

Router version

#sh ver
Cisco IOS Software, 3800 Software (C3845-SPSERVICESK9-M), Version 12.4(15)T10, RELEASE SOFTWARE (fc3)

Cisco WAAS 4.2.1

#sh vers
Cisco Wide Area Application Services Software (WAAS)
Copyright (c) 1999-2010 by Cisco Systems, Inc.
Cisco Wide Area Application Services (WAAS-UNIVERSAL-K9) Software Release 4.2.1
(build b38 Jun 16 2010)
Version: oe674-4.2.1.38

Here is the router config:

ip wccp 61 redirect-list WCCP
ip wccp 62 redirect-list WCCP

interface GigabitEthernet0/0
description link to PE
bandwidth 9000
no ip address
ip route-cache flow
duplex full
speed 10
media-type rj45
no cdp enable
!
interface GigabitEthernet0/0.22
encapsulation dot1Q 22
ip address -REMOVED- 255.255.255.252
ip wccp 62 redirect in
no cdp enable
service-policy output SE_ASAB_Q8096K_0_0_1M5_1M_99
!
interface GigabitEthernet0/1
no ip address
ip access-group 113 in
duplex full
speed 100
media-type rj45
service-policy output CE_OUT_MARK_0
!
interface GigabitEthernet0/1.2450
description LAN
encapsulation dot1Q 2450
ip address -REMOVED- 255.255.255.224
ip wccp 61 redirect in
no cdp enable
!
interface GigabitEthernet0/1.2459
description Connection to WAE Device For WCCP
encapsulation dot1Q 2459
ip address -REMOVED- 255.255.255.224
ip wccp redirect exclude in
no cdp enable
!
interface GigabitEthernet0/1.2460
encapsulation dot1Q 2460
ip address -REMOVED- 255.255.255.224
ip wccp redirect exclude in
no cdp enable

Correct Answer
Bhavin Yadav Mon, 09/13/2010 - 15:44

Hi Jan,

What is suspect here is WCCP.

When you configure, it allocates buckets if you are using hash assignment. If you are using Mask, it calculates mask depending on your destination / source ip address.

Now, what I understand is - if WAN goes down, some networks are not reachable.

When LAN goes down, WCCP goes down and everything starts running smooth.

Few questions:

1. What happens if LAN goes down but WAN remains up? Does WCCP remain active / usable state?

2. When WAN goes down and LAN remains up, your WCCP is still UP and hence, it continues to forward packets out of same WAN interface but  because that interface is down, packets ultimately die / gets blackholed.

3. Another speculation is: Asymmetric routing. When WAN is down but LAN is up, you are forwarding soem traffic out of LAN but as WAN goes down, the return packets then come up on different interface and creates asymmetric routing.

To narrow down this problem, please capture following interface during three stages:

1. WAN - Down, LAN-UP, is router ID reachable ???

2. WAN-UP, LAN- - Down, is router ID reachable ???

3. WAN Down, LAN Down, is router ID reachable  ???

CLI logs to capture:

    2. show ip wccp
    3. show ip wccp interface detail
    4. show ip wccp service
    5. show ip wccp detail
    6. show ip wccp internal (*)
    7. show running-config
    8. show ip wccp <61 / 62> hash
    9. sh wccp mask tcp-promis
    10. sh tech

Further, as you are using GRE encapsulation for WCCP redirection, the  router uses the router ID IP address as its source IP address. The  router ID IP address is the highest loopback address on the router, or  if the loopback interface is not configured, the router ID IP address is  the highest address of the physical interfaces. The router ID IP  address is used as the source address for packets redirected from the  router to the Cisco WAE, and as a result it is also used as the  destination address for traffic from the Cisco WAE to the router,  Therefore, you must be sure that a route exists from the Cisco WAE to  the router. This is done by configuring a static route on the Cisco WAE  to the router ID IP address. The router ID can be identified with the  command show wccp routers on the Cisco WAE.

As in your case, you have multiple routers, a  static route should be added to each of these router's router ID. The  command to configure such static routes is:

WAE(config)# ip route

Can you please try above and let us know if it works ?

Regards.




Jan Rockstedt Tue, 09/14/2010 - 00:11

Hi,

Is there any email address that I can send the information to?

I don’t want to expose myself on the hole internet.

About the static route in the WAE.

We have configured an B2B interface between the two ISP routers and are using BGP to peer the RIP.

Do we still need the static route, as we only have an default gateway that is point to the ISP router?

Best regards,

Jan Rockstedt

Correct Answer
Bhavin Yadav Tue, 09/14/2010 - 12:13

Hi Jan,

You can open a TAC case and they can assist you to retain confidentiality.

Further, about the static routes , BGP and RIP, I am not sure if I am the right person to answer that. You probably want to put this across Routing forum to get a more specific answer.

From WCCP perspective, WCCP will only change its status if the WCCP goes down by means of non-rechanble router / CE. Otherwise, if the WAN interface goes down, WCCP may still remain alive depending on your configuration because all WCCP cares is redirect traffic to CE and receive it back from CE and then forward it to gateway. If the gateway is not reachable, connectivity breaks but not WCCP. So, you want to make sure to probably add a static route in router to forward the traffic to backup gateway if the WAN link goes down.

Hope this helps.

Regards.

PS: Please mark this as Answered, if this resolves your issue.

Jan Rockstedt Wed, 09/15/2010 - 02:09

Hi again,

Sorry it was an typo RIP should be RIB.

We use only IBGP between the B2B interfaces.

I believe the we have all information to test again,

Thank for all troubleshoot tips.

Jan

mkovar Wed, 09/15/2010 - 11:30

Hello I am from the ISP and wanted to address these issues

2. When WAN goes down and LAN remains up, your WCCP is still UP and hence, it continues to forward packets out of same WAN interface but  because that interface is down, packets ultimately die / gets blackholed.

3. Another speculation is: Asymmetric routing. When WAN is down but LAN is up, you are forwarding soem traffic out of LAN but as WAN goes down, the return packets then come up on different interface and creates asymmetric routing.

On question 2 with WCCP the router would still try to send packets out the wan interface even though its down?   Wouldn't the router be able to tell that routing changed to the source/dest subnets and not blindly send packets to a down interface?   If not then this most likely is what happened.

Here is the WAN interface config WCCP is enabled for inbound redirection but the same for the actual data LAN interface

!
interface GigabitEthernet0/0
description link to PE
bandwidth 9000
no ip address
ip route-cache flow
duplex full
speed 10
media-type rj45
no cdp enable
!
interface GigabitEthernet0/0.22
encapsulation dot1Q 22
ip address **********omit ****** 255.255.255.252
ip wccp 62 redirect in
no cdp enable

and here is the LAN side

!
interface GigabitEthernet0/1
no ip address
ip access-group 113 in
ip route-cache flow
duplex full
speed 100
media-type rj45
service-policy output CE_OUT_MARK_0
!
interface GigabitEthernet0/1.2450
description Customer LAN
encapsulation dot1Q 2450
ip address ********* 255.255.255.224
ip wccp 61 redirect in
no cdp enable
!
interface GigabitEthernet0/1.2459
description Connection to customer-managed WAE Device For WCCP
encapsulation dot1Q 2459
ip address ******** 255.255.255.224
ip wccp redirect exclude in
no cdp enable
!
interface GigabitEthernet0/1.2460
encapsulation dot1Q 2460
ip address ******* 255.255.255.224
ip wccp redirect exclude in
no cdp enable
!

The sister router is configured in much the same way.

On question 3

3. Another speculation is: Asymmetric routing. When WAN is down but LAN is up, you are forwarding soem traffic out of LAN but as WAN goes down, the return packets then come up on different interface and creates asymmetric routing.

Wouldn't Asymetric routing just result in non optimized connections as it would never see the tcp option set for optimization?

We are going to run this same test this weekend and I will look at all these things but it seems as though asymetric routing would result in no optimization but not packet blockage.   Regarding question 2 if wccp remains up and is black holing traffic I can see this as an issue for sure.

One last question also regarding the loopbacks and GRE return.  There are distribute lists that block each router from learning the others loopback when the WAN is down.   Do you think this would matter?    Reason I ask is because on the Asymetric side again lets say a packet comes into router #1 via the lan and gets redirected to the WAE with source ip of the Loopback.   When the Was returns the packet to the router I would think it would not need routing to the #2 routers loopback as the destination at this point would be back to the client/server.   Also when the router forwards to the WAE what ip on the WAE does it use?

Bhavin Yadav Wed, 09/15/2010 - 11:48

Hello,

When WAN interface goes down, WCCP may remain up (depending on the RID) as WCCP interface does not care about the status of all other interfaces. All WCCP cares about is - HIA and ISU packets between Router and Cache Engine (CE) or WAEs.

If the WAN interface goes down, it's router's functionality to change the routing and not WCCP. Depending on your WCCP configuration, it will just keep returning packets to default gateway or GRE tunnel end point, again depending on your WCCP configuration on your router and CE.

About Asymmetric routing, Yes, connections will go in bypass / pass through mode but believe me, Asymmetric routing can cause "n" number of un-expected issues. This is the last thing that you want to see in your network.

Further, if you are running WCCP with multiple router and/or cache engines, it is best to turn off the WCCP completely and turn on after 2 minutes of interval, within 10 seconds on either CEs / Routers simultaneously. This is in order to avoid any bucket corruptions that may have occur due to change in WCCP farm.

If you have any connectivity issues with few subnets / ips and if that starts after CE / Router joining / leaving WCCP farm, flap the WCCP completely. This is the best practice and not MUST. Please also note that this issue will not happen everytime WCCP changes occur. This is noticed in some rare cases.

Hope this helps.

Regards.

Jan Rockstedt Wed, 09/22/2010 - 02:46

Hi again,

Some feedback.

We have now done our test again and everything is ok now, after added two static host routes on both WAE, for the router's loopback interface.

We could see on the ip numbers for the routers in the WAE's with show wccp wide-area-engine

Then I did an traceroute to the two router hosts " Routers seeing this Wide Area Engine" and found that the one was going out the WAN interface and on on the LAN on the router's.

Now both is going the right LAN way.

Thanks for the help.

Jan Rockstedt

Actions

This Discussion