FWSM won't connect after first connection on a specific Vlan

danielfycosta · ‎11-01-2011

Greetings!

I'm using a 6509 Sup720 FWSM VSS Cluster. Failover isn't active on FWSM modules yet, so I'm using the FWSM module on slot 4 of the VSS Switch and I'm using multiple contexts (multiple-vlan-interface enabled), 1 routed and 13 transparent. The routed is FWCTX-BACKEND.

I have a SVI configured for management access in this cluster, which is Vlan 10 (172.20.101.0/24). I have an ACS 1121 appliance on version 5.2.26 which is the TACACS+ AAA device manager (172.20.0.2/25) on Vlan220 whose default gateway is 172.20.0.1 the FWCTX-BACKEND DMZ interface (Vlan220) on FWCTX-BACKEND on the FWSM module.

My aaa rules on the 6500 are as follow:

aaa new-model

aaa authentication fail-message ^CCC*** Failed login. Five consecutive fails will revoke.***^C

aaa authentication login default group tacacs+ local

aaa authentication enable default group tacacs+ enable

aaa authorization exec default group tacacs+ none

aaa accounting exec default stop-only group tacacs+

aaa accounting commands 0 default stop-only group tacacs+

aaa accounting commands 7 default stop-only group tacacs+

aaa accounting commands 15 default stop-only group tacacs+

!

ip tacacs source-interface Vlan10

!

tacacs-server host 172.20.0.2

tacacs-server host 172.20.0.3

tacacs-server timeout 3

tacacs-server directed-request

tacacs-server key <OMITTED>

The route to the ACS is done, from the MSFC through the FWCTX-BACKEND. From the 6500 config prompt I can ping 172.20.101.250 (Intf Vlan10 @FWCTX-BACKEND) but not 172.20.0.2(ACS) - hence my auth problem. But from the FWCTX-BACKEND(172.20.101.250) I can't ping the 6500 (172.20.101.1).

Enabling debug icmp and I get this message:

CORE-01(config)#do ping 172.20.101.250

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 172.20.101.250, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

CORE_CSC-01(config)#

*Nov 1 18:44:24.535 BRT: ICMP: dst (172.20.101.1) host unreachable rcv from 172.20.101.250

*Nov 1 18:44:24.539 BRT: ICMP: echo reply rcvd, src 172.20.101.250, dst 172.20.101.1

*Nov 1 18:44:24.543 BRT: ICMP: dst (172.20.101.1) host unreachable rcv from 172.20.101.250

There is no way that 172.20.101.1 could be unreachble to 172.20.101.250, they are on the same subnet and using the same vlan tag, so no routes are needed, but apparently thats is what the log states.

If I reload the 6500, I can make the first ping from 6500 172.20.101.1 to ACS 172.20.0.2 but as soon as it ends, I can't ping anymore. The same thing happens to my RA access. I've set SSH permission to 6500 at 201.x.x.129, SVI Vlan200, and 201.x.x.132, interface Vlan200 INTERNET FWCTX-BACKEND. I can authenticate a SSH session to the 6500 on ACS only one time after a reboot, every other request to the 6500 are then treated locally, the FWCTX-BACKEND SSH session always authenticate on the ACS and I can't explain why that happens.

--

[6500]

This SVI on the VSS is:

interface Vlan10

ip address 172.20.101.1 255.255.255.0

no ip redirects

no ip proxy-arp

Routing for this SVI is:

ip route 172.20.0.2 255.255.255.255 172.20.101.250

ip route 172.20.0.3 255.255.255.255 172.20.101.250

[FWSM]

@FWCTX-BACKEND

interface Vlan10

nameif MGMT

security-level 60

ip address 172.20.101.250 255.255.255.0 standby 172.20.101.251

interface Vlan200

nameif RA

security-level 1

ip address 201.x.x.132 255.255.255.248 standby 201.x.x.133

!

Interface Vlan220

nameif DMZ

security-level 52

ip address 172.20.0.1 255.255.255.0 standby 172.20.0.4

!

access-list DMZ_IN line 2 extended permit icmp any any

access-list DMZ_IN line 3 extended permit ip host 172.20.0.2 host 10.1.98.64

access-list DMZ_IN line 4 extended permit ip host 172.20.0.3 host 10.1.98.64

access-list DMZ_IN line 5 extended permit ip host 172.20.0.2 any

access-list DMZ_OUT line 1 remark ### ACESSO : GCC-ACS-01 : 172.20.0.2 ###

access-list DMZ_OUT line 2 extended permit tcp any host 172.20.0.2 eq https

access-list DMZ_OUT line 3 extended permit tcp any host 172.20.0.2 eq tacacs

access-list DMZ_OUT line 4 remark ### ACESSO : GCC-ACS-01 : 172.20.0.2 @ad ###

access-list DMZ_OUT line 5 extended permit tcp any host 172.20.0.2 eq 88

access-list DMZ_OUT line 6 extended permit tcp any host 172.20.0.2 eq 445

access-list DMZ_OUT line 7 extended permit tcp any host 172.20.0.2 eq 464

access-list DMZ_OUT line 8 extended permit tcp any host 172.20.0.2 eq 3268

access-list DMZ_OUT line 9 extended permit udp any host 172.20.0.2 eq ntp

access-list DMZ_OUT line 10 extended permit udp any host 172.20.0.2 eq 389

access-list DMZ_OUT line 11 remark ### ACESSO : GCC-ACS-02 : 172.20.0.3 ###

access-list DMZ_OUT line 12 extended permit tcp any host 172.20.0.3 eq https

access-list DMZ_OUT line 13 extended permit tcp any host 172.20.0.3 eq tacacs

access-list DMZ_OUT line 14 remark ### ACESSO : GCC-ACS-02 : 172.20.0.3 @ad ###

access-list DMZ_OUT line 15 extended permit tcp any host 172.20.0.3 eq 88

access-list DMZ_OUT line 16 extended permit tcp any host 172.20.0.3 eq 445

access-list DMZ_OUT line 17 extended permit tcp any host 172.20.0.3 eq 464 6

access-list DMZ_OUT line 18 extended permit tcp any host 172.20.0.3 eq 3268

access-list DMZ_OUT line 19 extended permit udp any host 172.20.0.3 eq ntp

access-list DMZ_OUT line 20 extended permit udp any host 172.20.0.3 eq 389

access-list DMZ_OUT line 21 remark ### ACESSO : GCC-ACS-01 : 172.20.0.2

access-list DMZ_OUT line 22 extended permit tcp host 172.20.0.2 any eq https

access-list DMZ_OUT line 23 extended permit tcp host 172.20.0.2 any eq tacacs

access-list DMZ_OUT line 24 remark ### ACESSO : GCC-ACS-01 : 172.20.0.2 @ad ###

access-list DMZ_OUT line 25 extended permit tcp host 172.20.0.2 any eq 88

access-list DMZ_OUT line 26 extended permit tcp host 172.20.0.2 any eq 445

access-list DMZ_OUT line 27 extended permit tcp host 172.20.0.2 any eq 464

access-list DMZ_OUT line 28 extended permit tcp host 172.20.0.2 any eq 3268

access-list DMZ_OUT line 29 extended permit udp host 172.20.0.2 any eq ntp

access-list DMZ_OUT line 30 extended permit udp host 172.20.0.2 any eq 389

access-list DMZ_OUT line 31 remark ### ACESSO : GCC-ACS-02 : 172.20.0.3

access-list DMZ_OUT line 32 extended permit tcp host 172.20.0.3 any eq https

access-list DMZ_OUT line 33 extended permit tcp host 172.20.0.3 any eq tacacs

access-list DMZ_OUT line 34 remark ### ACESSO : GCC-ACS-02 : 172.20.0.3 @ad ###

access-list DMZ_OUT line 35 extended permit tcp host 172.20.0.3 any eq 88

access-list DMZ_OUT line 36 extended permit tcp host 172.20.0.3 any eq 445

access-list DMZ_OUT line 37 extended permit tcp host 172.20.0.3 any eq 464

access-list DMZ_OUT line 38 extended permit tcp host 172.20.0.3 any eq 3268

access-list DMZ_OUT line 39 extended permit udp host 172.20.0.3 any eq ntp

access-list DMZ_OUT line 40 extended permit udp host 172.20.0.3 any eq 389

access-list DMZ_OUT line 41 extended permit ip host 10.1.98.64 host 172.20.0.2

access-list DMZ_OUT line 42 extended permit ip host 172.20.0.2 host 10.1.98.64

access-list DMZ_OUT line 43 extended permit ip any host 10.1.98.64

access-list MGMT_IN line 1 extended permit ip any any

access-group DMZ_IN in interface DMZ

access-group DMZ_OUT out interface DMZ

access_group MGMT_IN in interface MGMT

--

Resume:

SSH 201.x.x.129 (6500) > aaa credentials @acs: only first connection after a reboot will work

> aaa credentials LOCAL: after that only local credentials will work since it can't connect to ACS through FWSM

SSH 201.x.x.132 (FWSM) > aaa credentials @acs: everytime ok

Can anyone give any help?

Thanks in advance, Dan

Edward Dutra · ‎11-03-2011

Hi Daniel...

Do you have both FWSMs configured even though failover isnt working yet? If so, theres a possibilty that both FWSMs are trying to claim the 172.20.101.250 IP address. If both FWSM are configured with the same config without failover, take one FWSM module offline until you are ready to do failover. Retest your ping again.

Also, use the packet capture commands on the FWSM to find out whats happening with the ICMP packets.

https://supportforums.cisco.com/docs/DOC-17345

Thanks!

danielfycosta · ‎11-03-2011

Hey Edward!

Thx for your reply!!

I'm not using failover, I might've missed to say that the other FWSM module isn't configured, nor have allocated interfaces to it in the VSS MSFC and that happens only to this management vlan/svi, VLAN 10.

I have some issues using capture on FWSM, tried using permit ip any any acls with packet-length 1500 capture with circular-buffer but some traffic isn't captured at all even when connections are ok, my guess, it's because of its virtual interfaces

\o/

Edward Dutra · ‎11-03-2011

If you are using VSS, I would suspect packet capture to work in your version of code.

If you configured capture correctly and you dont see the packet in the capture, than that would mean the packet nevers arrives to the FWSM.

danielfycosta · ‎11-03-2011

That's what I thought but the again, from the 6500 config prompt I actually get echo replys(!) from the FWCTX, with capture enabled as:

access-list CAP permit ip any any

!

capture mgmt access-list CAP interface MGMT packet-length 1500 circular-buffer

But it shows blank and no hit counts. Same happens usind RTMonitor in ASDM (6.2.(2f)) some packets that are permited and routed correctly aren't actually noticed. I don't get any logging for the missing/dropped/denied echo replies from the FWCTX to the 6500 MSFC nor for the successful replies from the 6500 to the FWCTX withh ASDM Debugging logging on.

Edward Dutra · ‎11-03-2011

sho theres no output when doing 'show capture mgmt'? How about just 'show capture'?

danielfycosta · ‎11-03-2011

Nothing man. Counters on zero bytes.