ace transaparent mode-return traffic

Unanswered Question
Sep 11th, 2008

we have fwsm running in routed mode and behind it is the ace. the ace is in bridge mode.we try to connect a sever ove http and see a connection hitting fwsm and then the ace. in ace we can see a return packet from the serverfarm host but then the return packet isnt seen on fwsm. we can ping the server directly hence no routing issues.looks like ace isnt sending the traffic back. below is the config with outputs of show conn from ace and fwsm and show policy of ace

any help will be appreciated

thanks


probe http cer_port

port 9005

interval 5

faildetect 15

passdetect interval 15

receive 2

expect status 200 200

open 2


serverfarm host SFarm3

probe cer_port

rserver ZHC1 9005

inservice

rserver ZHC2 9005

inservice

rserver ZHC3 9005

inservice


sticky http-cookie acecookie sticky-cookie-insert_9005

cookie insert

replicate sticky

serverfarm SFarm3


class-map match-all ACL

2 match access-list FW_Controlled

class-map match-all forms_listener_port_9000

2 match virtual-address 10.7.20.6 tcp eq 9000



policy-map type loadbalance f9000_policy

class class-default

sticky-serverfarm sticky-cookie-insert_9005


policy-map multi-match VIPS

class fot_9000

loadbalance vip inservice

loadbalance policy f9000_policy

loadbalance vip icmp-reply active


ace1-pri/# sh conn


total current connections : 2


conn-id np dir proto vlan source destination state

----------+--+---+-----+----+---------------------+---------------------+------+

15 2 in TCP 720 Mx:2954 10.7.20.6:9000 ESTAB

16 2 out TCP 720 10.7.20.21:9005 Mx:1037 INIT

fw-pri/prod# sh conn


TCP out Mx:2960 in 10.x.x.6:8000 idle 0:00:03 Bytes 1692 FLAGS - UBI



show service-policy


Policy-map : VIPS

Status : ACTIVE

-----------------------------------------

Interface: vlan 7xx 7x1

service-policy: VIPS

class: web_xxxxxxxx_8000

loadbalance:

L7 loadbalance policy: web_xxxxxx0_policy

VIP Route Metric : 77

VIP Route Advertise : DISABLED

VIP ICMP Reply : DISABLED

VIP State: INSERVICE

curr conns : 1 , hit count : 26

dropped conns : 23

client pkt count : 62 , client byte count: 14411

server pkt count : 0 , server byte count: 0



interface vlan 7xx

description interface facing Servers

bridge-group 1

access-group input BPDU

access-group input all

service-policy input VIPS

no shutdown

interface vlan 7xx

description interface facing FWSM

bridge-group 1

access-group input BPDU

access-group input all

service-policy input VIPS

no shutdown


  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 3.5 (2 ratings)
Loading.
Syed Iftekhar Ahmed Thu, 09/11/2008 - 22:13

Since your post doesn't give detail about class fot_9000 (which i sused under service policy VIPS)I am curious if "service-policy VIPS" is really needed under both VLANs and if its causing some looop.


Syed Iftekhar Ahmed

Gilles Dufour Fri, 09/12/2008 - 00:20

the status of the connection on the backend is init. You can also see server pkt count at 0.

So ACE didn't see the response on vlan 720.


Are you sure the packet comes back on the right vlan ???


Implement client nat and if it works you know it was an symetric routing issue.


Gilles.

followurself Fri, 09/12/2008 - 00:53

class-map match-all fot_9000

2 match virtual-address 10.7.20.6 tcp eq 9000


interface bvi 1

ip address 10.7.20.3 255.255.255.0

peer ip address 10.7.20.4 255.255.255.0

no shutdown



Destination Gateway Interface Flags

------------------------------------------------------------------------

0.0.0.0 10.7.20.1 vlan720 S

10.7.20.0/24 0.0.0.0 bvi1 IA



vlan 720 is server side facing vlan and vlan 770 is fwsm side

ip 10.7.20.1 is basically fwsm ip and also the gateway for all servers


Thanks

followurself Fri, 09/12/2008 - 01:09

Friends

config may create confusion,apologies

pls replace 9000 with 8000

thanks


followurself Fri, 09/12/2008 - 01:25

Friends

class: web_listener_port_8000

VIP Address: Port:

10.7.20.6 eq 8000

loadbalance:

L7 loadbalance policy: web_listener_port_8000_policy

VIP Route Metric : 77

VIP Route Advertise : DISABLED

VIP ICMP Reply : DISABLED

VIP State: INSERVICE

curr conns : 1 , hit count : 42

dropped conns : 39

client pkt count : 98 , client byte count: 25323

server pkt count : 0 , server byte count: 0

L7 Loadbalance policy : web_listener_port_8000_policy

class/match : class-default

LB action :

-

hit count : 35

dropped conns : 0

also the drop counts keeps increasing. i m not sure where the packet is going

Gilles Dufour Fri, 09/12/2008 - 01:41

Get a sniffer trace and follow the response.

That's all you can do.

ACE does not see any response and drops the connection marking it as a conn_failure.


Gilles.

followurself Fri, 09/12/2008 - 02:08

Thanks Giles

is there any config issue, service policy in paticular?

the design is standard

msfc--fwsm (routed mode and gateway for all servers) - ace (bridge mode) -- hp enclosure (layer 2 trunk back to msfc)

i can ping the servers directly. will my ping pass thru ace?

can see all the arp in ace for the servers

Gilles Dufour Fri, 09/12/2008 - 02:12

The config looks ok and the ping is not enough to guarantee there is no asymetry.

You will need to get a sniffer trace.


G.

followurself Fri, 09/12/2008 - 02:38

do i need service policy input VIPS on my bridge interfaces looking at the class

when you said client nat can you please help with the config

the servers are teamed with transmit load balancing. this infrastructure is remotely located, i can only sniff the servers as of now


followurself Fri, 09/12/2008 - 03:31

Giles i appreciate if you can help me with the traffic flow please

client---msfc/pfc---fwsm (gateway)---ace (bridgemode)---hp switches---servers

when traffic comes from client it hits msfc, thn to fwsm..fwsm does a static nat for the vip address and sends it to ace. ace receives and sends it to severs..here ace will send it via msfc--to hp--then to severs. all servers default gateway is fwsm. my ip is 10.1.102.232 and server 10.7.20.21, within ace con table we can see a conn intiated by ace with source as 10.1.102.232 and dst 10.7.20.21 ..but when client replies it will send the traffic to fwsm (default gateway) how wil this be intercepted by ace. i guess the arp wd be of ace

Thanks

followurself Fri, 09/12/2008 - 04:44

Giles

I applied client nat on interface 720 (facing servers) and policy only on tht interface and it worked

but i dont want to use the client nat? i m going to sniff the sever

Thanks

followurself Fri, 09/12/2008 - 06:58

i did a capture on fWSM which is the gateway of servers

my design is msfc-fwsm(routed mode)--ace (bridgemode) -hp enclosure where servers are located

looks like the server do respond and they hit their gateway and its not intercepted by ACE

with the above design and no client how can we make it work..pls help




show capture tin detail

14 packets seen, 12 packets captured

1: 13:55:32.1301371320 001e.bed7.5100 000b.fcfe.1b02 0x8100 66: 802.1Q vlan#7

20 P0 10.1.102.232.3012 > 10.7.20.6.8000: S [tcp sum ok] 2354942469:2354942469(0

) win 64512 (DF) (ttl 118, id 38224)

2: 13:55:32.1301371320 000b.fcfe.1b02 001e.bed7.5100 0x8100 64: 802.1Q vlan#7

20 P0 10.7.20.6.8000 > 10.1.102.232.3012: S [tcp sum ok] 1745154344:1745154344(0

) ack 2354942470 win 17408 (ttl 255, id 30819)

3: 13:55:32.1301371340 001e.bed7.5100 000b.fcfe.1b02 0x8100 64: 802.1Q vlan#7

20 P0 10.1.102.232.3012 > 10.7.20.6.8000: . [tcp sum ok] 2354942470:2354942470(0

) ack 1745154345 win 65520 (DF) (ttl 118, id 38226)

4: 13:55:32.1301371340 001e.bed7.5100 000b.fcfe.1b02 0x8100 690: 802.1Q vlan#

720 P0 10.1.102.232.3012 > 10.7.20.6.8000: P 2354942470:2354943102(632) ack 1745

154345 win 65520 (DF) (ttl 118, id 38227)

5: 13:55:32.1301371340 000b.fcfe.1b02 001e.bed7.5100 0x8100 64: 802.1Q vlan#7

20 P0 10.7.20.6.8000 > 10.1.102.232.3012: . [tcp sum ok] 1745154345:1745154345(0

) ack 2354943102 win 17408 (ttl 255, id 30811)

6: 13:55:32.1301371340 001f.296a.860c 001e.bed7.5100 0x8100 64: 802.1Q vlan#7

20 P0 10.7.20.21.8005 > 10.1.102.232.1079: S [tcp sum ok] 2563662798:2563662798(

0) ack 659054809 win 5840 (DF) (ttl 64, id 0)

7: 13:55:35.1301374250 001f.296a.860c 001e.bed7.5100 0x8100 64: 802.1Q vlan#7

20 P0 10.7.20.21.8005 > 10.1.102.232.1079: S [tcp sum ok] 2563662798:2563662798(

0) ack 659054809 win 5840 (DF) (ttl 64, id 0)

8: 13:55:36.1301375140 001f.296a.860c 001e.bed7.5100 0x8100 64: 802.1Q vlan#7

20 P0 10.7.20.21.8005 > 10.1.102.232.1079: S [tcp sum ok] 2563662798:2563662798(

0) ack 659054809 win 5840 (DF) (ttl 64, id 0)

9: 13:55:41.1301380070 001f.296a.860c 001e.bed7.5100 0x8100 64: 802.1Q vlan#7

20 P0 10.7.20.21.8005 > 10.1.102.232.1079: S [tcp sum ok] 2563662798:2563662798(

0) ack 659054809 win 5840 (DF) (ttl 64, id 0)

10: 13:55:42.1301381140 001f.296a.860c 001e.bed7.5100 0x8100 64: 802.1Q vlan#7

20 P0 10.7.20.21.8005 > 10.1.102.232.1079: S [tcp sum ok] 2563662798:2563662798(

0) ack 659054809 win 5840 (DF) (ttl 64, id 0)

11: 13:55:53.1301391720 001f.296a.860c 001e.bed7.5100 0x8100 64: 802.1Q vlan#7

20 P0 10.7.20.21.8005 > 10.1.102.232.1079: S [tcp sum ok] 2563662798:2563662798(

0) ack 659054809 win 5840 (DF) (ttl 64, id 0)

12: 13:55:54.1301393340 001f.296a.860c 001e.bed7.5100 0x8100 64: 802.1Q vlan#7

20 P0 10.7.20.21.8005 > 10.1.102.232.1079: S [tcp sum ok] 2563662798:2563662798(

0) ack 659054809 win 5840 (DF) (ttl 64, id 0)

followurself Fri, 09/12/2008 - 07:03

correction in the statement

with the above design and no client NAT how can we make it work..pls help

followurself Fri, 09/12/2008 - 07:03

correction in the statement

with the above design and no client NAT how can we make it work..pls help

followurself Fri, 09/12/2008 - 08:34

Syed/Giles

any suggestions with this design where i dnt have to do nat.

even if put say this way

msfc-ace(bridge)-fwsm (routed and gateway)--servers

seems the prblm will be still there..pls suggest

Syed Iftekhar Ahmed Fri, 09/12/2008 - 10:52

Which Vlans are allowed on the trunk between the HP Blade & CAT.


If the Vlan where FWSM belongs is included then servers ARP will be replied by FWSM interface directly and SErvers will bypass ACE.


You can also try hookin up a laptop to a CAT port and assign it the ACE VLAN and see if you face the same issue. If it works then somehow Servers are bypassing ACE.


Syed


Syed

followurself Tue, 09/16/2008 - 09:21

Thanks Syed for the response

since fwsm is the gateway of the servers then on the trunk back to catalyst, tht vlan is allowed or else how will they reach thier gateway?

also i am not sure the importance and traffic hitting the other vlan on ace which is part of bridge group (770 vlan)

this is what my archiectecture is

msfc-vlan 760-fwsm outside(routed mode),fwsm inside inerface (vlan720) ---ace (bridge mode) vlan 770 and 720 where 720 is facing servers. bvi interface is having the ip in vlan 720. i am not sure what kind of traffic will hit 770.because fwsm inside is 720 and is sharing the subnet with ace.

can you please suggest on why vlan 720 shouldnt be allowed on trunk interface between hp and cat

thanks


Syed Iftekhar Ahmed Tue, 09/16/2008 - 10:54

Servers should reach the Gateway through ACE.


If the FWSM is accessible to servers directly and Servers default gateway is FWSM (which it should be as you are using bridge mode) then when Server sends ARP request for the GAteway (FWSM) ip address then FWSM will respond back directly and Servers will send the response back to FWSM directly - bypassing ACE (hence breaking the connections).


In normal setup the server's arp entry for Default gateway (FWSM IP) should show the MAc address of the ACE interface.


Could you check your servers and see what do you have listed as MAC add for Gateway IP?


Syed




followurself Wed, 09/17/2008 - 02:25

does that mean my vlan assigment should be like below


msfc-v760-fwsm outside(client facing)

then -v770 with fwsm inside (server facing) and ace outside client facing in vlan 770

then vlan 720 ace inside server facing


with fwsm inside vlan 770 and ip address 10.7.20.1

vlan of ace 720 with ip address 10.7.20.3

and all servers in vlan 720 with ip 10.7.20.x


can you please suggest what vlans fwsm will have and facing which side and ace vlans facing which side



Syed Iftekhar Ahmed Wed, 09/17/2008 - 04:39

You are almost there. Just one clarification, In bridge mode (ACE) you do not assign IPs on the interfaces, YOu just assign a BVI ip address which is only used by management traffic originated from ACE (for e,g Syslogs, snmp....) . Loadbalanced traffic / Traffic passing through ACE doesnt use this address. So the 10.7.20.3 ip on ACE will be the BVI ip address. It will not be assigned to vlan 720 interface.


In summary your setup will be


(VLAN 760)(IP subnet diff than 10.7.20.x)Outside-FWSM-Inside(Vlan770)(10.7.20.1)---->(VLAN 770)ClientSide-ACE(BVI IP 10.7.20.3)-ServerSide(VLAN 720)-->Servers in VLAN 720 with IP addresses in 10.7.20.x range.


If you have redundant pair of ACE/FWSM in place then have to keep few things in mind when you are configuring ACE in bridge mode.


1. Create a ethertype ACL to allow BPDUs through ACE to make sure you have a single STP instance for both VLAN 720 & 770.


2. Disable Globally configured Loopguard & BPDU guard on the CAT6K.


3. Your servers should have FWSM (10.7.20.1) as the default gateway.




HTH

Syed Iftekhar Ahmed

followurself Wed, 09/17/2008 - 05:30

Thanks for the clarification, will need further when i shall test layer 7 policies and stickness

i will need to change the config and then test.

cisco docs confuses

followurself Wed, 09/17/2008 - 02:27

btw i m using fwsm in routed mode --followed by ace in bridge mode

so its msfc-fwsm(routed) then --ace (bridge)--servers

followurself Fri, 09/12/2008 - 07:03

correction in the statement

with the above design and no client NAT how can we make it work..pls help

followurself Fri, 09/12/2008 - 07:03

correction in the statement

with the above design and no client NAT how can we make it work..pls help

Actions

This Discussion