ASA 5505 PAT with multiple public IP addresses is non-obvious

[Giblet535]

Hi,

I have an ASA 5505 w/ Security Plus.

I have a block of 5 consecutive IP addresses from my ISP.

I want to put 5 sets of services, located within a single dmz, port-forwarded to specific external IP addresses (eg, mail.mybiz.us, www.mybiz.us, etc).

I currently have vlan2 assigned to 'outside', identified by the lowest ISP IP address with a mask to accept inbound traffic from all of them.

I set up static NAT rules for each service of each server in the dmz, translating the dmz-addy of each server to its respective external IP address. I added ACLs for those services (any -> external IP address).

But nmap'ing each of the IPs (to scan for open ports) shows that there is no service exposed on any IP!

Packet trace from one of the external IPs to its NAT'ed dmz host indicates that the dmz's implicit IP blocking rule prevents it from getting to the server, even though I have an explicit ACL for that service to the external IP and the on the dmz incoming interface.

a) Should I be trunking five vlans to those five external IPs and NATing to the vlan interfaces instead?

b) How can I get around the implicit IP-DENY rule on the dmz, since explicit ACLs don't appear to override it?

[cadet alain]

The ACL permitting traffic to natted public IP must be inbound on outside interface. Is it the case?

[Giblet535]

Yes it is.

The config is the same as the Getting Started Guide's typical outside-dmz-inside example, as well as many examples on this excellent forum.

The ACL permits tcp/smtp from Any to outside/if-ip. The NAT translates the dmz server's real ip to the if-ip and forwards tcp port 25.

The server has port 25 open.

And yet port 25 is not open on the if-ip. Packet trace says it's being blocked by the implicit ip-deny rule on the dmz.

I'll post a scrubbed 'show running' later today.

------------------------

Just as importantly, I'm wondering if this device handles the ISP setup in the best way:

5 IPs 3.3.3.12-3.3.3.16, netmask 255.255.255.248, gateway 3.3.3.11

Vlan2 (outside) has IP 3.3.3.13 (I didn't set that up, it's an heirloom).

I'm NATing ports to 3.3.3.13 and 3.3.3.14.

Should I trunk five vlans to that switch port instead?

[cadet alain]

DMZ is 1 subnet so 1 VLAN so no need for trunking and anyway your ISP is not trunking with you, your outside interface is a routed interface not connected to a switch.

Regards.

[Jon Marshall]

Just as importantly, I'm wondering if this device handles the ISP setup in the best way:

5 IPs 3.3.3.12-3.3.3.16, netmask 255.255.255.248, gateway 3.3.3.11

Vlan2 (outside) has IP 3.3.3.13 (I didn't set that up, it's an heirloom).

I'm NATing ports to 3.3.3.13 and 3.3.3.14.

Should I trunk five vlans to that switch port instead?

As long as your ISP is routing those addresses to your ASA then you only need one IP on the outside interface.

Jon

[Giblet535]

It was an acl.

Servers in the dmz don't seem to be able to access the dmz's interface ip w/o an explicit rule to allow this.

Example:

dmz ip = 1.2.3.1 255.255.255.255

server = 1.2.3.2 255.255.255.0 gateway 1.2.3.1

That won't work unless an explicit acl allows dmz<->dmz traffic.

Since this doesn't appear in the forums or examples, I have to assume it's a glitch in my firmware revision. But, that fixed everything... Go figure.

[Giblet535]

OK, the symptom has changed, but it's far from resolved.

Here's the scenario:

A block of 5 external IPs, 111.112.113.82 - 111.112.113.86 mask 255.255.255.248, assigned to outside vlan (111.112.113.82/29).

Two servers, 192.168.6.2 and 192.168.6.3, residing on the dmz vlan (192.168.6.1).

192.168.6.2 presents Email-Ext services (actually a subset) to the .83 outside interface.

192.168.6.3 presents Email-Ext services (a superset) to the .84 outside interface.

The .84 services are there in a portscan.

The .83 services are not.

It seems to me that they're set up identically, yet one works and one does not. Portscans from the dmz indicate both servers are listening.

Here's the scrubbed config:

: Saved
:
ASA Version 7.2(2) 
!
hostname CZ-firewall
domain-name CZ.local
enable password xxx encrypted
names
!
interface Vlan1
 nameif inside
 security-level 100
 ip address 192.168.3.1 255.255.255.0 
!
interface Vlan2
 nameif outside
 security-level 0
 ip address 111.112.113.82 255.255.255.248 
!
interface Vlan3
 nameif dmz
 security-level 50
 ip address 192.168.6.1 255.255.255.0 
!
interface Ethernet0/0
 switchport access vlan 2
 speed 10
 duplex full
!
interface Ethernet0/1
!
interface Ethernet0/2
 switchport access vlan 3
!
interface Ethernet0/3
!
interface Ethernet0/4
!
interface Ethernet0/5
!
interface Ethernet0/6
!
interface Ethernet0/7
!
passwd xxx encrypted
ftp mode passive
clock timezone EST -5
clock summer-time EDT recurring
dns domain-lookup dmz
dns server-group Default
 name-server 8.8.8.8
 name-server 8.8.4.4
dns server-group DefaultDNS
 domain-name crapzapper.local
object-group service Email-ext tcp
 description External email services
 port-object range smtp smtp
 port-object range https https
 port-object range 465 465
 port-object range 563 563
 port-object range 587 587
 port-object range ldaps ldaps
 port-object range www www
 port-object range 993 993
 port-object range 995 995
object-group service Email-int tcp
 description Internal email services
 port-object range pop3 pop3
 port-object range nntp nntp
 port-object range imap4 imap4
 port-object range ssh ssh
 port-object range smtp smtp
 port-object range ldap ldap
 port-object range https https
 port-object range 465 465
 port-object range 563 563
 port-object range 587 587
 port-object range ldaps ldaps
 port-object range www www
 port-object range 993 993
 port-object range 995 995
object-group service VPN tcp
 port-object range ssh ssh
 port-object range telnet telnet
 port-object range https https
access-list dmz_access_in remark For DNS, etc
access-list dmz_access_in extended permit udp any any 
access-list dmz_access_in remark So that DMZ hosts can access the internet
access-list dmz_access_in extended permit tcp any any 
access-list dmz_access_in remark So that PING works
access-list dmz_access_in extended permit icmp any any 
access-list dmz_access_in extended permit ip any any 
access-list CZ standard permit 192.168.3.0 255.255.255.0 
access-list outside_access_in extended permit icmp any any 
access-list outside_access_in extended permit tcp any host 111.112.113.82 object-group VPN 
access-list outside_access_in extended permit udp any host 111.112.113.82 
access-list outside_access_in extended permit tcp any host 111.112.113.83 object-group Email-ext 
access-list outside_access_in extended permit udp any host 111.112.113.83 
access-list outside_access_in extended permit tcp any host 111.112.113.84 object-group Email-ext 
access-list outside_access_in remark SSH access from Pat's house to the email server
access-list outside_access_in extended permit tcp host 14.15.16.8 host 111.112.113.84 eq ssh 
access-list outside_access_in remark Mail Admin access from Pat's house to the email server
access-list outside_access_in extended permit tcp host 14.15.16.8 host 111.112.113.84 eq 4040 
access-list outside_access_in extended permit udp any host 111.112.113.84 
access-list outside_access_in extended permit tcp any host 111.112.113.85 object-group Email-ext 
access-list outside_access_in extended permit udp any host 111.112.113.85 
access-list outside_access_in extended permit tcp any host 111.112.113.86 object-group Email-ext 
access-list outside_access_in extended permit udp any host 111.112.113.86 
access-list outside_access_in extended permit tcp host 111.112.113.83 host 192.168.6.2 object-group Email-ext 
access-list outside_access_in extended permit tcp host 111.112.113.84 host 192.168.6.3 object-group Email-ext 
access-list outside_access_in extended permit tcp host 111.112.113.85 host 192.168.6.2 object-group Email-ext inactive 
access-list outside_access_in extended permit tcp host 111.112.113.86 interface dmz object-group Email-ext inactive 
access-list outside_access_in extended permit udp interface dmz host 192.168.6.3 
access-list outside_access_in extended permit udp interface dmz host 192.168.6.2 
access-list restrict-outbound-from-lan extended permit udp any any 
access-list restrict-outbound-from-lan extended permit esp any any 
access-list restrict-outbound-from-lan extended permit tcp any any 
access-list restrict-outbound-from-lan extended permit icmp any any 
access-list restrict-outbound-from-lan extended permit gre any any 
access-list restrict-outbound-from-lan extended permit tcp host 192.168.6.3 192.168.3.0 255.255.255.0 object-group Email-ext 
pager lines 24
logging enable
logging emblem
logging asdm notifications
logging mail notifications
logging host dmz 192.168.6.3 format emblem
logging class email mail emergencies 
mtu inside 1500
mtu outside 1500
mtu dmz 1500
ip local pool RemoteAccess 192.168.3.41-192.168.3.50 mask 255.255.255.0
no failover
monitor-interface inside
monitor-interface outside
monitor-interface dmz
icmp unreachable rate-limit 1 burst-size 1
asdm image disk0:/asdm-522.bin
no asdm history enable
arp timeout 14400
global (inside) 1 interface
global (outside) 1 interface
global (dmz) 1 interface
nat (inside) 1 192.168.6.3 255.255.255.255 dns
nat (inside) 1 192.168.3.0 255.255.255.0
nat (dmz) 1 192.168.6.0 255.255.255.0 dns
static (dmz,outside) tcp 111.112.113.83 smtp 192.168.6.2 smtp netmask 255.255.255.255  dns 
static (dmz,outside) tcp 111.112.113.83 www 192.168.6.2 www netmask 255.255.255.255 
static (dmz,outside) tcp 111.112.113.83 https 192.168.6.2 https netmask 255.255.255.255  dns 
static (dmz,outside) tcp 111.112.113.83 587 192.168.6.2 587 netmask 255.255.255.255  dns 
static (dmz,outside) tcp 111.112.113.83 993 192.168.6.2 993 netmask 255.255.255.255  dns 
static (dmz,outside) tcp 111.112.113.83 995 192.168.6.2 995 netmask 255.255.255.255  dns 
static (dmz,outside) tcp 111.112.113.84 465 192.168.6.3 465 netmask 255.255.255.255  dns 
static (dmz,outside) tcp 111.112.113.84 smtp 192.168.6.3 smtp netmask 255.255.255.255  dns 
static (dmz,outside) tcp 111.112.113.84 995 192.168.6.3 995 netmask 255.255.255.255  dns 
static (dmz,outside) tcp 111.112.113.84 993 192.168.6.3 993 netmask 255.255.255.255  dns 
static (dmz,outside) tcp 111.112.113.84 563 192.168.6.3 563 netmask 255.255.255.255  dns 
static (dmz,outside) tcp 111.112.113.84 ldaps 192.168.6.3 ldaps netmask 255.255.255.255  dns 
static (dmz,outside) tcp 111.112.113.84 https 192.168.6.3 https netmask 255.255.255.255  dns 
static (dmz,outside) tcp 111.112.113.84 www 192.168.6.3 www netmask 255.255.255.255  dns 
static (dmz,outside) tcp 111.112.113.84 ssh 192.168.6.3 ssh netmask 255.255.255.255  dns 
static (dmz,outside) tcp 111.112.113.84 4040 192.168.6.3 4040 netmask 255.255.255.255  dns 
static (dmz,inside) 111.112.113.83 192.168.6.2 netmask 255.255.255.255 dns 
static (dmz,inside) 111.112.113.84 192.168.6.3 netmask 255.255.255.255 dns 
access-group restrict-outbound-from-lan in interface inside
access-group outside_access_in in interface outside
access-group dmz_access_in in interface dmz
route outside 0.0.0.0 0.0.0.0 111.112.113.81 1
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout uauth 0:05:00 absolute
eou clientless password xxx
group-policy CZREMOTE internal
group-policy CZREMOTE attributes
 dns-server value 192.168.3.3
 vpn-filter none
 vpn-tunnel-protocol IPSec 
 ipsec-udp enable
 split-tunnel-policy tunnelspecified
 split-tunnel-network-list value CZ
 default-domain value crapzapper.local
username Bob password xxx encrypted privilege 0
username Bob attributes
 vpn-group-policy CZREMOTE
http server enable
http 192.168.3.0 255.255.255.0 inside
http 14.15.16.8 255.255.255.255 outside
http redirect inside 80
http redirect dmz 80
no snmp-server location
no snmp-server contact
snmp-server enable traps snmp authentication linkup linkdown coldstart
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac 
crypto dynamic-map outside_dyn_map 20 set pfs 
crypto dynamic-map outside_dyn_map 20 set transform-set ESP-3DES-SHA
crypto map outside_map 65535 ipsec-isakmp dynamic outside_dyn_map
crypto map outside_map interface outside
crypto isakmp enable outside
crypto isakmp policy 10
 authentication pre-share
 encryption 3des
 hash sha
 group 2
 lifetime 86400
tunnel-group CZREMOTE type ipsec-ra
tunnel-group CZREMOTE general-attributes
 address-pool RemoteAccess
 default-group-policy CZREMOTE
tunnel-group CZREMOTE ipsec-attributes
 pre-shared-key *
vpn-sessiondb max-session-limit 5
telnet 192.168.3.0 255.255.255.0 inside
telnet 14.15.16.8 255.255.255.255 outside
telnet timeout 5
ssh 192.168.3.0 255.255.255.0 inside
ssh 14.15.16.8 255.255.255.255 outside
ssh timeout 5
console timeout 0
dhcpd auto_config outside
!

!
!
!
policy-map type inspect dns preset_dns_map
 parameters
  message-length maximum 512
!
ntp server 66.96.30.35 source outside prefer
prompt hostname context 
Cryptochecksum:002146128cb07b0965c57973bc8889e3
: end
asdm image disk0:/asdm-522.bin
no asdm history enable

(Yes, I'm aware that this is not an ideal config. I'm granting wide-open access in some cases. I can clean that up once I figure out what's broken.)

Why are the .84 services available while the .83 services are not?

[cadet alain]

Hi Pat,

Servers in the dmz don't seem to be able to access the dmz's interface ip w/o an explicit rule to allow this

You couldn't ping your dmz interface from your servers in the dmz, that's right? that's normal behaviour to enable pinging on asa interface you must use

icmp command, no need to use an ACL.

For your actual problem:

Have you tried  packet-tracer command? you'll see if the problems lies in NAT or ACL or maybe something else.

Regards.

[Giblet535]

Hi,

Yes, I have used packet tracer.

Packets are allowed outsideif->dmzhost and dmzhost->outsideif. According to packet tracer, it works great.

111.112.113.83 <---> 192.168.6.2 appears like straight wire on the allowed ports (according to packet tracer), yet the .83 ports are all closed (according to the outside world).

Syslog indicates no problems, acl denials, or nat failures.

++

Here's a weird behavior, maybe related?:

I can't access any of the exposed services on the working PAT config from the dmz or inside LANs. (no denials or nat errors in syslog)

Of course, that .83 address appears dead from the dmz and from outside, but it's extremely weird that I can't access services at the outside interface from the dmz or inside. Connections just time out. The rest of the internet is accessible from either the dmz or inside...

Attempting a portscan of the working outside if from the dmz indicates a down host. Syslog shows the non-acl port scans being denied, but that's all. From a completely different outside network, that .84 address is doing exactly what I expect. It just looks like a locked-down email server.

[cadet alain]

From inside or dmz you can't access this interface but to me it seems normal behaviour for me.

What is not normal is you can't access the services of your server  from outside to natted public address.

You tried with packet -tracert to communicate with this internal machine? and with the other server?

Regards.

[Giblet535]

Sorry cadetalain, but I can't imagine how not being able to access your internet-facing website from inside your network can be normal unless your inside users are blocked from accessing the entire internet. Our users aren't constrained but our website is somehow blocked.

• Packet tracer indicates that outside<-->dmz work in both directions for the PATed ports.
• I get no translate failures in syslog.
• I get no ACL denials in syslog.
• The server whose ports are forwarded are listening on all of the expected ports.
• The NATed address appears dead.
• Another server and adjacent address work fine....other than the issue where inside users can't access that address' services.
• I even began to suspect that my ISP was wrong about the IP block addresses. The block is correct.

Our web site and our email services are the only internet services that my inside users can't access. Everyone else in the world can see those services.

That's not normal. The real address 192.168.6.3 should be translated to 111.112.113.84 in this config. There are some indications that this is so. The documentation accounts for all of the behavior on the .84 interface (other than inside users being blocked from that one address). The .83 network is configured the same way, but doesn't work from anywhere. That's a Windows box though; so there could be additional problems froms Windows and nobody would be surprised.

I see nothing in the config to account for the current behavior. I've tried what makes sense, and more than a few things that don't make sense, and I'm about ready to sentence this ASA to 20 years in that scary box at the back of the supply closet. That'll teach it some respect.

[cadet alain]

Hi Pat,

static (dmz,inside) 111.112.113.83 192.168.6.2 netmask 255.255.255.255 dns
static (dmz,inside) 111.112.113.84 192.168.6.3 netmask 255.255.255.255 dns

Is it normal? because 111.112.113.83 are outside addresses no?

regards.