We are trying to get to the bottom of an issue we are seeing, but unfortunately are not sure where to start. We have (2) 7931's in the Main DC and (1) 7931 in the backup datacenter (BDC), and well over 20 remote sites running NM-WAE, OE574 and OE674. We had an issue over the weekend where traffic from several remote sites was redirected to our BDC due to power outage. When this occurred ldap authentication broke for these sites as well as other CIFS traffic for users that were already authenticated.
We have seen this before and each time we have seen this we have noticed that the access-list on the core routers (7609) used for wccp starts matching (meaning the device is using software instead of hardware). The output below shows what we saw last time a site started experiencing issues such as, could not authenticate, could not open files, etc... We removed the site from the ACL and everything started working, of course we were no longer able to accelerate/optimize traffic going to the BDC once it was removed.
We saw this again this weekend. Several sites reported that they could not authenticate, when we investigated they were going to BDC servers due to a power outage and the ACL's had started incrementing, once again we had to remove them in order for them to be able to authenticate.
At this time we suspect there might have been asymmetric routing occurring during the power outage, but do not have data to back that up at this time. Has anyone see this type of issue before? or can anyone confirm if asymmetric routing could cause this type of behavior.
Extended IP access list WAAS_WCCP
10 permit ip 192.168.2.0 0.0.0.255 any
20 permit ip any 172.25.2.0 0.0.0.255
---- cut for brevity ------
90 permit ip 10.1.64.0 0.0.0.255 any
100 permit ip any 10.1.64.0 0.0.0.255
110 permit ip 10.1.74.0 0.0.0.255 any
120 permit ip any 10.1.74.0 0.0.0.255
130 permit ip 10.1.130.0 0.0.0.255 any
140 permit ip any 10.1.130.0 0.0.0.255
150 permit ip 10.1.213.0 0.0.0.255 any
160 permit ip any 10.1.213.0 0.0.0.255
170 permit ip 10.1.236.0 0.0.3.255 any
180 permit ip any 10.1.236.0 0.0.3.255
190 permit ip 10.1.24.0 0.0.1.255 any
200 permit ip any 10.1.24.0 0.0.1.255 (1914211 matches)
Do you see any indication in the WAAS logs that connections are failing due to a redirection loop? The message in syslog.txt should look something like:
2009 Dec 11 16:08:17 NO-HOSTNAME kernel: %WAAS-SYS-3-900000:126.96.36.199:49114 - 188.8.131.52:22 - opt_syn_rcv: Routing Loop detected - Packet has our own devid. Packet dropped.
Assuming that WCCP is being handled in software on the the 7609, the counter incrementing in the output you provided would support that traffic isn't being seen symmetrically. That in and of itself shouldn't cause the connections to fail (they should just be handled as pass-through), so I suspect there may be a redirection loop at your BDC site.
Can you provide a topology diagram of your environment?
For the WCCP in software issue on the 7609, can you provide the following output from IOS:
show ip wccp
show ip wccp 61 service
show ip wccp 62 service
show ip wccp 61 detail
show ip wccp 62 detail
show ip wccp internal (* NOTE: to enable this command, add "service internal" to the configuration first)
show tcam interface acl in ip (where is the name of each interface with WCCP enabled)
Thanks for responding. We do indeed see an error in the syslog.txt file showing a routing loop error:
2010 Jun 20 10:59:26 waas-bdc kernel: %WAAS-SYS-3-900000: 192.168.128.134:18 44 - 192.168.210.217:139 - opt_syn_rcv: Routing Loop detected - Packet has our own devid. Packet dropped.
Unfortunately I cannot post configs/topology/command output, directly to netpro due to internal security restrictions, however I can send them directly to you if you have time to take a look? I would assume from the above that we need to be lookign at the wccp redirect configuration on the router?
Introduction This article will help you understand the steps on how to
download the UCS licenses from the Cisco Systems website and then
installing it on the UCS. The redacted (blue lines) just covers up
certain numbers for privacy please do not take them...
Introduction This article will help you understand and educate the
customer on how to clear their "expired licenses"
(license-graceperiod-expired) from their UCS-M. If a customer just
purchased a license and needs a step by step guide on how to download
==================== VIC FNIC driver does not support Virtual Volumes (
second level LUN ID ) An enhancement request has been created to track
this feature - CSCux64473 UPDATE - 12-14-2016 We made some traction on
the enhancement request - The Fix is in t...