WCCP/WAAS - 7609 Hardware Based ACL issue

Unanswered Question
Jun 21st, 2010

Greetings,

We are trying to get to the bottom of an issue we are seeing, but unfortunately are not sure where to start. We have (2) 7931's in the Main DC and (1) 7931 in the backup datacenter (BDC), and well over 20 remote sites running NM-WAE, OE574 and OE674. We had an issue over the weekend where traffic from several remote sites was redirected to our BDC due to power outage. When this occurred ldap authentication broke for these sites as well as other CIFS traffic for users that were already authenticated.

We have seen this before and each time we have seen this we have noticed that the access-list on the core routers (7609) used for wccp starts matching (meaning the device is using software instead of hardware). The output below shows what we saw last time a site started experiencing issues such as, could not authenticate, could not open files, etc... We removed the site from the ACL and everything started working, of course we were no longer able to accelerate/optimize traffic going to the BDC once it was removed.

We saw this again this weekend. Several sites reported that they could not authenticate, when we investigated they were going to  BDC servers due to a power outage and the ACL's had started incrementing, once again we had to remove them in order for them to be able to authenticate.

At this time we suspect there might have been asymmetric routing occurring during the power outage, but do not have data to back that up at this time. Has anyone see this type of issue before? or can anyone confirm if asymmetric routing could cause this type of behavior.

=================================

Extended IP access list WAAS_WCCP

    10 permit ip 192.168.2.0 0.0.0.255 any

    20 permit ip any 172.25.2.0 0.0.0.255

      ---- cut for brevity ------

     90 permit ip 10.1.64.0 0.0.0.255 any

    100 permit ip any 10.1.64.0 0.0.0.255

    110 permit ip 10.1.74.0 0.0.0.255 any

    120 permit ip any 10.1.74.0 0.0.0.255

    130 permit ip 10.1.130.0 0.0.0.255 any

    140 permit ip any 10.1.130.0 0.0.0.255

    150 permit ip 10.1.213.0 0.0.0.255 any

    160 permit ip any 10.1.213.0 0.0.0.255

    170 permit ip 10.1.236.0 0.0.3.255 any

    180 permit ip any 10.1.236.0 0.0.3.255

    190 permit ip 10.1.24.0 0.0.1.255 any

    200 permit ip any 10.1.24.0 0.0.1.255 (1914211 matches)

    210 permit ip 10.1.48.0 0.0.0.255 any

    220 permit ip any 10.1.48.0 0.0.0.255

===============================================

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Zach Seils Mon, 06/21/2010 - 10:03

Do you see any indication in the WAAS logs that connections are failing due to a redirection loop?  The message in syslog.txt should look something like:

2009 Dec 11 16:08:17 NO-HOSTNAME kernel: %WAAS-SYS-3-900000:1.1.1.1:49114 - 2.2.2.2:22 - opt_syn_rcv: Routing Loop detected -
Packet has our own devid. Packet dropped.

Assuming that WCCP is being handled in software on the the 7609, the counter incrementing in the output you provided would support that traffic isn't being seen symmetrically.  That in and of itself shouldn't cause the connections to fail (they should just be handled as pass-through), so I suspect there may be a redirection loop at your BDC site.

Can you provide a topology diagram of your environment?

For the WCCP in software issue on the 7609, can you provide the following output from IOS:

  • show      version

  • show      ip wccp

  • show ip wccp 61 service

  • show ip wccp 62 service
  • show      ip wccp 61      detail

  • show      ip wccp 62      detail
  • show ip wccp internal (* NOTE: to enable this  command, add "service internal" to the  configuration first)

  • show tcam interface acl in ip (where is the name of each interface with WCCP enabled)
  • show      running-config

Thanks,

Zach

j.shrewsbury Mon, 06/21/2010 - 12:00

Zach,

Thanks for responding. We do indeed see an error in the syslog.txt file showing a routing loop error:

2010 Jun 20 10:59:26 waas-bdc kernel: %WAAS-SYS-3-900000: 192.168.128.134:18
44 - 192.168.210.217:139 - opt_syn_rcv: Routing Loop detected - Packet has our own
devid. Packet dropped.

Unfortunately I cannot post configs/topology/command output, directly to netpro due to internal security restrictions, however I can send them directly to you if you have time to take a look? I would assume from the above that we need to be lookign at the wccp redirect configuration on the router?

Actions

This Discussion