Hey gang. I ran across an unusual issue today I'm trying to figure out. We're in the process of upgrading off of the Aironet 1200 series APs we have on IOS version 12.3 and moving over to Aironet 3600 APs on IOS version 15.3. There are a plethora of reasons why this is a good thing. :)
My upgrade process has included creating a new SSID for our Internal, BYOD, and Guest networks. We're upgrading to WPAv2 with AES and I didn't want issues with the same SSID on different encryptions causing issues as I start upgrading these APs.
The problem that I'm see though is:
Connect to SSID_Original: no issues
Connect to SSID_New (which means they're on a new AP): They can see the first gateway hop, but it won't hop to the ASA5505 I have for our cable modem.
On default router we have:
ip route <business ip address range> <mpls gateway>
ip route 0.0.0.0 0.0.0.0 10.1.1.23 (asa5505) 1 track 23
ip route 0.0.0.0 0.0.0.0 10.100.1.1 (MPLS gateway) 10
Basically all of our business system traffic stays on the MPLS and as long as the cable modem track is up, we're routing anything left over out the cable modem, if it fails, use the MPLS.
If I join a laptop to the old SSID, and test ping 8.8.8.8 I get 100% success. If I join to the new SSID and ping 8.8.8.8 I get 0% success. If I ping 10.100.1.1 (mpls router) I get 100% success. They can connect to the main router and route out the MPLS but all of the 0.0.0.0 traffic out the ASA is failing. This morning when I was trying to test it from my laptop I couldn't connect to the ASA for internet until...I pinged 10.1.1.23 (asa5505). When I tried to ping the ASA from the laptop that wouldn't connect out the ASA the first two packet dropped, and then the pings were successful. So far every computer that has had issues the fix has been just ping the ASA and then everything starts working.
The only difference I can find in the configs is that the interfaces on the old APs have CEF disabled (no ip route-cache). By default on the new APs it looks like CEF is enabled.
I checked the router connected to the MPLS and the FE ports on it have CEF disabled. Is it possible that having CEF enabled on the APs but disabled on the router could cause this issue?
If not, than I think I may be fighting some kind of ARP issue and that the APs are not sure where to switch the traffic to reach the ASA's ip address?
Default route for the clients is to 10.1.1.1 (inside address of main router), the router can ping 10.1.1.23 (asa5505) just fine. But the first few pings from the client to 10.1.1.23 drop, and any traffic from the client routing out the ASA drops until you physically ping the firewall. Watch the first two packets drop and then all the traffic after that is fine.
I've gone line by line through the config, and the only thing that stands out to me is the "no ip route-cache" on the old APs.
Attaching a copy of the config from the new APs. What you are about to see is a true story...only the names and passwords have been changed to protect their security (old TV reference if anyone else remembers).
version 15.3
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec localtime show-timezone
service password-encryption
!
hostname APTest
!
!
logging rate-limit console 9
!
aaa new-model
!
!
aaa authentication login default local
aaa authorization exec default local
!
!
!
!
!
aaa session-id common
clock timezone -0600 -6 0
no ip source-route
no ip cef
!
!
!
!
dot11 syslog
dot11 vlan-name Guest vlan 20
dot11 vlan-name Main vlan 1
dot11 vlan-name MobileDevice vlan 10
!
dot11 ssid Guest
vlan 20
authentication open
authentication key-management wpa version 2
mbssid guest-mode
wpa-psk ascii 7 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
!
dot11 ssid Mobile
vlan 10
authentication open
authentication key-management wpa version 2
mbssid guest-mode
wpa-psk ascii 7 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
!
dot11 ssid WiFi
vlan 1
authentication open
authentication key-management wpa version 2
mbssid guest-mode
wpa-psk ascii 7 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
!
!
dot11 arp-cache
!
no ipv6 cef
!
!
username XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
!
!
bridge irb
!
!
!
interface Dot11Radio0
no ip address
!
encryption vlan 1 mode ciphers aes-ccm
!
encryption vlan 10 mode ciphers aes-ccm
!
encryption vlan 20 mode ciphers aes-ccm
!
ssid Guest
!
ssid Mobile
!
ssid WiFi
!
antenna gain 0
stbc
mbssid
station-role root
!
interface Dot11Radio0.1
encapsulation dot1Q 1 native
bridge-group 1
bridge-group 1 subscriber-loop-control
bridge-group 1 spanning-disabled
bridge-group 1 block-unknown-source
no bridge-group 1 source-learning
no bridge-group 1 unicast-flooding
!
interface Dot11Radio0.10
encapsulation dot1Q 10
bridge-group 10
bridge-group 10 subscriber-loop-control
bridge-group 10 spanning-disabled
bridge-group 10 block-unknown-source
no bridge-group 10 source-learning
no bridge-group 10 unicast-flooding
!
interface Dot11Radio0.20
encapsulation dot1Q 20
ip access-group Guest in
bridge-group 20
bridge-group 20 subscriber-loop-control
bridge-group 20 spanning-disabled
bridge-group 20 block-unknown-source
no bridge-group 20 source-learning
no bridge-group 20 unicast-flooding
!
interface Dot11Radio1
no ip address
!
encryption vlan 1 mode ciphers aes-ccm
!
encryption vlan 10 mode ciphers aes-ccm
!
encryption vlan 20 mode ciphers aes-ccm
!
ssid Guest
!
ssid Mobile
!
ssid WiFi
!
antenna gain 0
peakdetect
dfs band 3 block
stbc
mbssid
speed basic-6.0 9.0 12.0 18.0 24.0 36.0 48.0 54.0 m0. m1. m2. m3. m4. m5. m6. m7. m8. m9. m10. m11. m12. m13. m14. m15. m16. m17. m18. m19. m20. m21. m22. m23.
channel dfs
station-role root
!
interface Dot11Radio1.1
encapsulation dot1Q 1 native
bridge-group 1
bridge-group 1 subscriber-loop-control
bridge-group 1 spanning-disabled
bridge-group 1 block-unknown-source
no bridge-group 1 source-learning
no bridge-group 1 unicast-flooding
!
interface Dot11Radio1.10
encapsulation dot1Q 10
bridge-group 10
bridge-group 10 subscriber-loop-control
bridge-group 10 spanning-disabled
bridge-group 10 block-unknown-source
no bridge-group 10 source-learning
no bridge-group 10 unicast-flooding
!
interface Dot11Radio1.20
encapsulation dot1Q 20
ip access-group Guest in
bridge-group 20
bridge-group 20 subscriber-loop-control
bridge-group 20 spanning-disabled
bridge-group 20 block-unknown-source
no bridge-group 20 source-learning
no bridge-group 20 unicast-flooding
!
interface GigabitEthernet0
no ip address
duplex auto
speed auto
!
interface GigabitEthernet0.1
encapsulation dot1Q 1 native
bridge-group 1
bridge-group 1 spanning-disabled
no bridge-group 1 source-learning
!
interface GigabitEthernet0.10
encapsulation dot1Q 10
bridge-group 10
bridge-group 10 spanning-disabled
no bridge-group 10 source-learning
!
interface GigabitEthernet0.20
encapsulation dot1Q 20
bridge-group 20
bridge-group 20 spanning-disabled
no bridge-group 20 source-learning
!
interface BVI1
mac-address xxxxxxxxx
ip address 10.1.1.20 255.255.255.0
ipv6 address dhcp
ipv6 address autoconfig
ipv6 enable
!
ip default-gateway 10.1.1.1
ip forward-protocol nd
ip http server
ip http authentication aaa
no ip http secure-server
ip http help-path http://www.cisco.com/warp/public/779/smbiz/prodconfig/help/eag
!
ip access-list extended Guest
deny ip any 10.1.0.0 0.0.255.255
deny ip any 192.168.10.0 0.0.0.255
deny ip any 192.168.20.0 0.0.0.255
deny tcp any any eq telnet
deny tcp any any eq 22
permit ip any any
ip access-list extended PermitTelnet
permit tcp 10.1.0.0 0.0.255.255 any eq telnet
!
!
bridge 1 route ip
!
!
!
line con 0
line vty 0 4
access-class PermitTelnet in
transport input all
!
end
