We are trying to get to the bottom of an issue we are seeing, but unfortunately are not sure where to start. We have (2) 7931's in the Main DC and (1) 7931 in the backup datacenter (BDC), and well over 20 remote sites running NM-WAE, OE574 and OE674. We had an issue over the weekend where traffic from several remote sites was redirected to our BDC due to power outage. When this occurred ldap authentication broke for these sites as well as other CIFS traffic for users that were already authenticated.
We have seen this before and each time we have seen this we have noticed that the access-list on the core routers (7609) used for wccp starts matching (meaning the device is using software instead of hardware). The output below shows what we saw last time a site started experiencing issues such as, could not authenticate, could not open files, etc... We removed the site from the ACL and everything started working, of course we were no longer able to accelerate/optimize traffic going to the BDC once it was removed.
We saw this again this weekend. Several sites reported that they could not authenticate, when we investigated they were going to BDC servers due to a power outage and the ACL's had started incrementing, once again we had to remove them in order for them to be able to authenticate.
At this time we suspect there might have been asymmetric routing occurring during the power outage, but do not have data to back that up at this time. Has anyone see this type of issue before? or can anyone confirm if asymmetric routing could cause this type of behavior.
Extended IP access list WAAS_WCCP
10 permit ip 192.168.2.0 0.0.0.255 any
20 permit ip any 172.25.2.0 0.0.0.255
---- cut for brevity ------
90 permit ip 10.1.64.0 0.0.0.255 any
100 permit ip any 10.1.64.0 0.0.0.255
110 permit ip 10.1.74.0 0.0.0.255 any
120 permit ip any 10.1.74.0 0.0.0.255
130 permit ip 10.1.130.0 0.0.0.255 any
140 permit ip any 10.1.130.0 0.0.0.255
150 permit ip 10.1.213.0 0.0.0.255 any
160 permit ip any 10.1.213.0 0.0.0.255
170 permit ip 10.1.236.0 0.0.3.255 any
180 permit ip any 10.1.236.0 0.0.3.255
190 permit ip 10.1.24.0 0.0.1.255 any
200 permit ip any 10.1.24.0 0.0.1.255 (1914211 matches)
210 permit ip 10.1.48.0 0.0.0.255 any
220 permit ip any 10.1.48.0 0.0.0.255