WCCP/WAAS - 7609 Hardware Based ACL issue

j.shrewsbury · ‎06-21-2010

Greetings,

We are trying to get to the bottom of an issue we are seeing, but unfortunately are not sure where to start. We have (2) 7931's in the Main DC and (1) 7931 in the backup datacenter (BDC), and well over 20 remote sites running NM-WAE, OE574 and OE674. We had an issue over the weekend where traffic from several remote sites was redirected to our BDC due to power outage. When this occurred ldap authentication broke for these sites as well as other CIFS traffic for users that were already authenticated.

We have seen this before and each time we have seen this we have noticed that the access-list on the core routers (7609) used for wccp starts matching (meaning the device is using software instead of hardware). The output below shows what we saw last time a site started experiencing issues such as, could not authenticate, could not open files, etc... We removed the site from the ACL and everything started working, of course we were no longer able to accelerate/optimize traffic going to the BDC once it was removed.

We saw this again this weekend. Several sites reported that they could not authenticate, when we investigated they were going to BDC servers due to a power outage and the ACL's had started incrementing, once again we had to remove them in order for them to be able to authenticate.

At this time we suspect there might have been asymmetric routing occurring during the power outage, but do not have data to back that up at this time. Has anyone see this type of issue before? or can anyone confirm if asymmetric routing could cause this type of behavior.

=================================

Extended IP access list WAAS_WCCP

10 permit ip 192.168.2.0 0.0.0.255 any

20 permit ip any 172.25.2.0 0.0.0.255

---- cut for brevity ------

90 permit ip 10.1.64.0 0.0.0.255 any

100 permit ip any 10.1.64.0 0.0.0.255

110 permit ip 10.1.74.0 0.0.0.255 any

120 permit ip any 10.1.74.0 0.0.0.255

130 permit ip 10.1.130.0 0.0.0.255 any

140 permit ip any 10.1.130.0 0.0.0.255

150 permit ip 10.1.213.0 0.0.0.255 any

160 permit ip any 10.1.213.0 0.0.0.255

170 permit ip 10.1.236.0 0.0.3.255 any

180 permit ip any 10.1.236.0 0.0.3.255

190 permit ip 10.1.24.0 0.0.1.255 any

200 permit ip any 10.1.24.0 0.0.1.255 (1914211 matches)

210 permit ip 10.1.48.0 0.0.0.255 any

220 permit ip any 10.1.48.0 0.0.0.255

===============================================

Zach Seils · ‎06-21-2010

Do you see any indication in the WAAS logs that connections are failing due to a redirection loop? The message in syslog.txt should look something like:

2009 Dec 11 16:08:17 NO-HOSTNAME kernel: %WAAS-SYS-3-900000:1.1.1.1:49114 - 2.2.2.2:22 - opt_syn_rcv: Routing Loop detected -
Packet has our own devid. Packet dropped.

Assuming that WCCP is being handled in software on the the 7609, the counter incrementing in the output you provided would support that traffic isn't being seen symmetrically. That in and of itself shouldn't cause the connections to fail (they should just be handled as pass-through), so I suspect there may be a redirection loop at your BDC site.

Can you provide a topology diagram of your environment?

For the WCCP in software issue on the 7609, can you provide the following output from IOS:

show version
show ip wccp
show ip wccp 61 service
show ip wccp 62 service
show ip wccp 61 detail
show ip wccp 62 detail
show ip wccp internal (* NOTE: to enable this command, add "service internal" to the configuration first)
show tcam interface acl in ip (where is the name of each interface with WCCP enabled)
show running-config

Thanks,

Zach

j.shrewsbury · ‎06-21-2010

Zach,

Thanks for responding. We do indeed see an error in the syslog.txt file showing a routing loop error:

2010 Jun 20 10:59:26 waas-bdc kernel: %WAAS-SYS-3-900000: 192.168.128.134:18
44 - 192.168.210.217:139 - opt_syn_rcv: Routing Loop detected - Packet has our own
devid. Packet dropped.

Unfortunately I cannot post configs/topology/command output, directly to netpro due to internal security restrictions, however I can send them directly to you if you have time to take a look? I would assume from the above that we need to be lookign at the wccp redirect configuration on the router?

Zach Seils · ‎06-22-2010

Feel free to email me at seils@cisco.com. The commands I requested are from the router.

Thanks,

Zach

j.shrewsbury · ‎06-22-2010

Thanks Zach, I have sent these along to your email address.