CSS 11503 Destination NAT - can only enable one service

[kiplandiles]

I have three web servers configured as six services. Three are for MOSS (Microsoft Office Sharepoint Server) and three are for SSRS (SQL Server Reporting Services 2006 in integration mode).

THE PROBLEM:

When more than one MOSS service is active I can no longer connect to the SSRS services.

This is a trunked Configuration:

interface 1/1

trunk

redundancy-phy

vlan 1

default-vlan

vlan 100

vlan 101

vlan 103

interface 3/16

bridge vlan 4000

circuit VLAN100

redundancy

ip address 192.168.100.xx0 255.255.255.0

circuit VLAN103

redundancy

ip address 192.168.103.xx0 255.255.255.0

circuit VLAN4000

ip address 1.x.x.2 255.255.255.252

redundancy-protocol

circuit VLAN101

redundancy

ip address 192.168.101.xx0 255.255.255.0

service MOSSWeb01

ip address 192.168.103.xx1

keepalive port 80

keepalive type tcp

active

service MOSSWeb02

ip address 192.168.103.xx2

keepalive port 80

keepalive type tcp

active

service MOSSWeb03

ip address 192.168.103.xx3

keepalive port 80

keepalive type tcp

active

service SSRSWeb01

ip address 192.168.103.xx1

active

service SSRSWeb02

ip address 192.168.103.xx2

active

service SSRSWeb03

ip address 192.168.103.xx3

active

owner MOSS

content MOSS

vip address 192.168.100.xx1

vip-ping-response local-remote

add service MOSSWeb01

add service MOSSWeb02

add service MOSSWeb03

active

owner SSRS

content REPORTSERVER

vip address 192.168.100.xx2

add service SSRSWeb01

add service SSRSWeb02

add service SSRSWeb03

vip-ping-response local-remote

active

group MOSS2007-DSTNAT

vip address 192.168.100.xx1

add destination service MOSSWeb01

add destination service MOSSWeb02

add destination service MOSSWeb03

active

group SSRS2005-DSTNAT

vip address 192.168.100.xx2

add destination service SSRSWeb01

add destination service SSRSWeb02

add destination service SSRSWeb03

active

NOTES:

All (3) real servers have a default route to 192.168.103.xx0 which insures traffic passing through the CSS (so I don't understand why I still need a destination service group).

When MOSS accesses SSRS it does so via http://SSRS2005/reportserver. This is configured in DNS as 192.168.100.xx2. I would think that this would also insure traffic through the CSS but I still had to configure a destination service for these.

All clients connect to the MOSS services via one VIP (192.168.100.xx1) and the MOSS services connect to the SSRS services via a 2nd VIP (192.168.100.xx2). MOSS also connects to itself for indexing content and a variety of other services (I had originally tried separating the MOSS content rules using layer 5 matching on Host Headers. This seemed to cause issues with access to ports 139 and 445 for UNC access to document libraries so I simplified the MOSS content rule back to layer 3).

I have setup two distinct groups and have used destination NAT so that the servers can communicate to each other.

When using Wireshark on the servers to run packet traces and all services are up I do not even see any packets destined for the SSRS services leading me to believe that they are dropped by the CSS (however, I don't see them using show flows on the CSS either).

Can anyone here shed some light on the correct way to configure the CSS in such a scenario?

Thanks in advance.

[didyap]

This link may help :

http://www.cisco.com/en/US/docs/app_ntwk_services/data_center_app_services/css11500series/v7.20/configuration/basic/guide/ContRule.html

[kiplandiles]

Thanks. I actually have downloaded all the guides and have read through this chapter several times. I will continue to study this, though, since the answer is likely to be there somewhere.

[opc.perera]

please post the output for the following commands

show rule MOSS MOSS

show rule SSRS REPORTSERVER.

show keepalive AUTO_MOSSWeb01/2/3

show keepalive AUTO_SSRSWeb01/2/3

You are using the same IP address for two service names with no distinct port numbers for them.

[opc.perera]

Also please list if you have a Layer 2 network between CSS and the Servers?

[kiplandiles]

I have attached the output of the SHOW commands you requested.

>>You are using the same IP address for two service names with no distinct port numbers for them.

They are serviced by two distint VIPs. They are physically the same servers (3 servers, 6 services).

>>Also please list if you have a Layer 2 network between CSS and the Servers?

This is a trunked configuration but the three servers are on a layer2 switch trunked to the CSS at VLAN 103 (192.168.103.x). The front end of the CSS is trunked to VLAN 100 (192.168.100.x). The VIPs are all assigned to 192.168.100.x and the default route for all servers is 192.168.103.246 which is the CSS (same IP I access the admin interface on).

Logically traffic comes in on 192.168.100.x -> 192.168.103.x and returns the opposite way 192.168.103.x -> 192.168.100.x but service-initated traffic (MOSS to SSRS for instance) I assume does not pass through the CSS because the CSS is the default route. I'm not exactly sure how the CSS handles this service-initated traffic but I do know that I must have the Destination Service Groups defined to even get traffic from server to server.

When MOSS (VIP 192.168.100.162) connects to SSRS it is via the SSRS VIP (192.168.100.161). I don't expect this traffic is going back out through the CSS and back in again.

If I log into any of the three servers I can get to the SSRS endpoint (at 192.168.100.161) interactively using a web browser so I know that the communications is working.

[opc.perera]

All seems normal except for you have taken two services out of action/service.

Please try the following :

1. Configure one port on the switch to which the servers are connected as a monitor port and monitor VLAN 103 and connect PC/Laptop with sniffer/wireshark running.

2. Connect another PC/Laptop with 192.168.103.xx6(or any unused IP) as its IP address and try and connect

(a). by telneting to 192.168.103.001/2/3 on the port(s) MOSS and REPORTSERVER listening to and list the results. (save the captured sniffer file)

(b). by telneting to 192.168.100.xx1/xx2 on the same ports used in (a) and register the results. (save the captured sniffer file separately)

If you are able to telnet successfully in (a) but NOT in the case of (b), the server response short circuits the CCS as the MAC switches internally.

One way to get around with this is to establish pVLAN. Configure three ports as isolated pVLANs.

[kiplandiles]

I have two MOSS services down because MOSS can't get to SSRS if more than one MOSSservice is active. That's the crux of the biscuit.

I had hoped to avoid the whole packet sniffing activity but it looks like I may need to capture more information. I don't really want to change the VLAN configuration since this CSS is managed by our network team and there are other services configured on the CSS that I have not indicated.

I appreciate your advice, so far. I will actually have some downtime this coming weekend where I can try some additional configuration options after prime time from home.

One thing that may not be apparent in this whole discussion is that all of the sites on both MOSS and SSRS use HOST Headers for HTTP. That's what keeps them separated. I had tried using layer 5 content rules but had the same issue plus other issues with non-HTTP traffic. I also did not care for the fact that the CSS actually spoofs the responses when using layer 5. There is a lot of NTLM Challenge/Response traffic for Windows Integrated Authentication and Negotiated Kerberos. The bottom line is that even without Layer 5 content rules the Host Headers do get passed to IIS and the sites are selected properly based on that header. The exception is that Host Headers are no longer required for SSRS since it is the default website on port 80 (besides - setting up host headers for SSRS in MOSS integration mode has it's own set of issues). Still, the host headers are sent to SSRS SOAP Endpoints and there are no issues connecting to any of the three SSRS services from any of the three MOSS servers interactively. The issue is when a client outside of these VLANs makes a request for a report.

client->MOSS->SSRS->MOSS->client

Be aware too that both MOSS and SSRS are making connections back through the CSS to their respective databases for each request.