Weird behaviour

Unanswered Question
May 8th, 2010
User Badges:

Hello


I'm having a weird problem with a Cisco 878 router.

The problem is that the router is filtering ports randomly used for NAT. For example if i use nmap to scan open ports

of my router it shows me this:



[[email protected] init.d]# nmap x.x.x.x


Starting Nmap 4.76 ( http://nmap.org ) at 2010-05-08 15:37 UYT

Interesting ports on foobar.com (x.x.x.x):

Not shown: 993 closed ports

PORT    STATE SERVICE

22/tcp  open  ssh

23/tcp  open  telnet

25/tcp  open  smtp

80/tcp  open  http

110/tcp open  pop3

135/tcp open  msrpc

443/tcp open  https


Nmap done: 1 IP address (1 host up) scanned in 177.08 seconds

[[email protected] init.d]#

The weird thing is that at random times if i use nmap again it shows me that most of the ports are filtered!
Starting Nmap 4.76 ( http://nmap.org ) at 2010-05-08 14:37 UYT
Interesting ports on foobar.com (x.x.x.x):
Not shown: 993 closed ports
PORT    STATE    SERVICE
22/tcp  open     ssh
23/tcp  open     telnet
25/tcp  filtered smtp
80/tcp  filtered http
110/tcp filtered pop3
135/tcp filtered msrpc
443/tcp filtered https
Nmap done: 1 IP address (1 host up) scanned in 11.26 seconds
Does anyone has a clue of what is going on? Its a big problem because i use ports 25 ans 110 por accessing an internal exchange server
and sometimes client connect to it and sometimes dont.Below i attach my router's configuration
Thanks!
Current configuration : 4621 bytes
!
version 12.4
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname Cisco878
!
boot-start-marker
boot-end-marker
!
logging message-counter syslog
logging buffered 51200
logging console critical
enable secret 5 xxxxxxxxxxxxxxxx
enable password xxxxxx
!
no aaa new-model
clock timezone PCTime 1
clock summer-time PCTime date Mar 30 2003 2:00 Oct 26 2003 3:00
!
crypto pki trustpoint TP-self-signed-577650748
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-577650748
revocation-check none
rsakeypair TP-self-signed-577650748
!
!
crypto pki certificate chain TP-self-signed-577650748
certificate self-signed 01
  3082023E 308201A7 A0030201 02020101 300D0609 2A864886 F70D0101 04050030
  30312E30 2C060355 04031325 494F532D 53656C66 2D536967 6E65642D 43657274
  69666963 6174652D 35373736 35303734 38301E17 0D313030 35303831 37353935
  365A170D 32303031 30313030 30303030 5A303031 2E302C06 03550403 1325494F
  532D5365 6C662D53 69676E65 642D4365 72746966 69636174 652D3537 37363530
  37343830 819F300D 06092A86 4886F70D 01010105 0003818D 00308189 02818100
  EDB74A46 7D50F663 5D80CEA8 697DB6F3 2797C8A4 DA3D1110 7D045FCC 48418C56
  6F4DD64D E665FD03 A36F5A6E 5515D20D C9559433 E327DE2D 4D406322 1466DE95
  252C1629 025E826C 019837A1 72A6AC40 1AD71B07 1F7F85D4 62BE757B 77557904
  FB191757 1B2CE2B1 5E2785C7 654D6487 A75330B7 7A3F75F6 62B284A6 E997FC0D
  02030100 01A36830 66300F06 03551D13 0101FF04 05300301 01FF3013 0603551D
  11040C30 0A820843 6973636F 38373830 1F060355 1D230418 30168014 8217C557
  29C7F74E AE522995 8B21699E FD507FD6 301D0603 551D0E04 16041482 17C55729
  C7F74EAE 5229958B 21699EFD 507FD630 0D06092A 864886F7 0D010104 05000381
  8100A0C4 AA28A09C 09FE78C6 E53F38DD C57ADB76 982F0FE2 49A6011E D913A47C
  5CBEF602 9D655082 865F91BF 1D569F68 4D7850F2 A4A8B6A5 AA0849B8 29BB57EF
  D76D516C 323B0BD0 EF1A0C7D 7377D689 37F6E996 76390AA4 48DDB687 80B4D579
  584BB16E DAB88C53 DD2F4BF6 2266BB26 E7AE6B26 B7F7D7E0 68A33FB9 B24CE77D 1D13
  quit
dot11 syslog
ip source-route
!
!
!
!
ip cef
no ip bootp server
no ipv6 cef
!
multilink bundle-name authenticated
!
!
!
username mastercisco privilege 15 secret 5 xxxxxxxxxxxxxxxx
!
!
!
archive
log config
  hidekeys
!
!
controller DSL 0
mode atm
dsl-mode shdsl symmetric annex B
!
!
!
!
interface Null0
no ip unreachables
!
interface BRI0
no ip address
no ip redirects
no ip unreachables
no ip proxy-arp
encapsulation hdlc
shutdown
!
interface ATM0
no ip address
no ip unreachables
no ip proxy-arp
ip virtual-reassembly
no atm ilmi-keepalive
pvc 8/35
  encapsulation aal5mux ppp dialer
  dialer pool-member 1
!
!
interface FastEthernet0
!
interface FastEthernet1
!
interface FastEthernet2
switchport access vlan 2
!
interface FastEthernet3
switchport access vlan 2
!
interface Vlan1
description $FW_INSIDE$
ip address 192.168.0.12 255.255.255.0
no ip unreachables
no ip proxy-arp
ip nat inside
ip virtual-reassembly
!
interface Vlan2
description $FW_INSIDE$
ip address 192.168.1.12 255.255.255.0
no ip unreachables
no ip proxy-arp
ip nat inside
ip virtual-reassembly
!
interface Dialer0
description $FW_OUTSIDE$
ip address negotiated
no ip unreachables
no ip proxy-arp
ip nat outside
ip virtual-reassembly
encapsulation ppp
dialer pool 1
dialer-group 1
ppp authentication chap callin
ppp chap password 0 xxxxx
ppp ipcp route default
ppp ipcp address accept
!
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 Dialer0
ip http server
ip http authentication local
ip http secure-server
!
!
ip nat pool pool1 192.168.0.0 192.168.1.254 netmask 255.255.0.0
ip nat inside source list 1 interface Dialer0 overload
ip nat inside source list 2 interface Dialer0 overload
ip nat inside source static tcp 192.168.0.5 8060 interface Dialer0 8060
ip nat inside source static tcp 192.168.0.13 135 interface Dialer0 135
ip nat inside source static tcp 192.168.0.13 80 interface Dialer0 80
ip nat inside source static tcp 192.168.1.51 4050 interface Dialer0 4050
ip nat inside source static tcp 192.168.0.13 443 interface Dialer0 443
ip nat inside source static tcp 192.168.0.13 25 interface Dialer0 25
ip nat inside source static tcp 192.168.0.13 110 interface Dialer0 110
!
logging trap debugging
access-list 2 permit 192.168.1.0 0.0.0.255
dialer-list 1 protocol ip permit
no cdp run
!
!
!
!
!
control-plane
!        
banner login ^CWelcome to AIX 5.3^C
!
line con 0
no modem enable
line aux 0
line vty 0 4
password cisco
login
!
scheduler max-task-time 5000
end
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
Peter Paluch Sat, 05/08/2010 - 12:16
User Badges:
  • Cisco Employee,

Hi Oskar,


I am not quite sure neither what is going on, but these are my suggestions:


  1. You've got two ip nat inside source commands in your config, one is referencing ACL 1 which does not exist, the second is referencing ACL 2. The one referencing the nonexistent ACL 1 should be removed.
  2. If combining static NAT entries with dynamic NAT/PAT, the ACL used for the dynamic NAT/PAT should explicitely exclude the IP addresses/ports that are already used for static NAT. In your case, there is the 192.168.1.51 address being used both for static NAT and also permitted by the ACL 2 in the dynamic NAT. You should probably exclude that one combination of IP, transport protocol and port in the ACL, but in order to specify transport protocol and port, you will need to use an extended ACL instead of the standard ACL 2.
  3. How long does it usually take until the filtered port reverts back to open state? If that state takes longer, can you have a look at the show ip nat translation output to see if there is an appropriate translation entry created on your router? Also, are you absolutely sure that it is the router that is dropping your packets? Isn't it possible that it is in fact the internal server that drops your packets for some reason?
  4. Do you have a stable IP address on the Dialer0 interface? I see it is negotiated but it is the same all the time? That address changes could account for connectivity issues.


Best regards,

Peter

strutter79 Sat, 05/08/2010 - 12:32
User Badges:

Hi peter

thanks for your quick answer.


The ip address on dialer0 interface is stable. it uses all the time a static IP.

The ports change their open/filtered state at random, i haven't been able to measure the time it elapses

between the changes so i think it's random.

the router is dropping the packets because when i do a telnet exchange_server 25 from the router console,it works fine

always.LAN communication between the exchange server and the rest of the network is fine.

I have also upgraded the IOS software from a 2006 to a 2009 release. Could it be that the router is buggy?

how can i disable dynamic NAT?


Greetings


Oskar

Peter Paluch Sat, 05/08/2010 - 12:48
User Badges:
  • Cisco Employee,

Hi Oskar,


What is the current version of your IOS? I personally believe that installing a recent version can be only helpful. A bug in the IOS is surely possible but I would rather explore other options before assuming that the IOS itself is the cause. The router itself (the hardware) probably should not cause the problems you are experiencing.


Regarding your question "how can i disable dynamic NAT?" - I am not sure what exactly you are asking about. If you are asking about removing the superfluous command I have indicated in my previous post, you can make that by entering the global configuration mode and issuing the command


no ip nat inside source list 1 interface Dialer0 overload


By the way, I recommend very strongly removing that one. I am not sure how the NAT code behaves if the ACL does not exist.


Is it also possible that your ISP may be limiting the number of concurrent connections onto your public IP address?


Best regards,

Peter

strutter79 Sat, 05/08/2010 - 12:59
User Badges:

Ok peter this is my IOS version:



Cisco IOS Software, C870 Software (C870-ADVIPSERVICESK9-M), Version 12.4(24)T, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2009 by Cisco Systems, Inc.

Compiled Thu 26-Feb-09 07:56 by prod_rel_team


ROM: System Bootstrap, Version 12.3(8r)YI4, RELEASE SOFTWARE


System returned to ROM by reload

System image file is "flash:c870-advipservicesk9-mz.124-24.T.bin"

Last reload reason: Reload Command

I have already removed the line about NAT without ACL.
My question about disabling dynamic NAT was because i would like to only use static NAT, keeping all
the static nat statements and removing the other  ip nat inside source list 2 interface Dialer0 overload
would do the trick?
i have already created a bash script that uses nmap each minute to scan the NAT ports of my router from the outside.
it then logs the result to a file. I will see if it keeps filtering the ports or not, and if it does, i will see how often it changes
the open/filtered port state.
I will inform you later
thanks
Oskar.
Peter Paluch Sat, 05/08/2010 - 13:05
User Badges:
  • Cisco Employee,

Hi Oskar,


Yes, removing the second ip nat inside source list 2 interface Dialer0 overload command will completely disable the dynamic PAT that is running for your internal networks specified by the ACL 2. The only ip nat inside source commands that should remain in your configuration should be those specifying the static mappings. Of course, the ip nat inside/outside commands must remain in place, too.


Please bear in mind that the internal networks will lose their connectivity with the internet if you implement this change - I am not sure if this is what you want.


Best regards,

Peter

strutter79 Sat, 05/08/2010 - 16:56
User Badges:

Peter i dont understand why removing the line ip nat inside source list 2 interface Dialer0 overload

would deprived my internal network of getting access to the internet. Is it not enough

to have the internal static mappings so those machines could access the internet by themselves?

Could you explain it to me?


thanks


oskar.

Peter Paluch Sun, 05/09/2010 - 01:00
User Badges:
  • Cisco Employee,

Hello Oskar,


I am sorry - I have perhaps not been quite clear on the subject.


What I wanted to say that if you remove the ip nat inside source list 2 interface Dialer0 overload command, the only NAT entries that will remain in place will be the static NAT/PAT entries. The machines and ports defined with the static NAT/PAT entries will continue to be reachable from the internet.


However, each internal machine for which there is currently no static NAT entry will go to internet untranslated, i.e. with its private IP address. For example, you do not have any translation defined for the IP address 192.168.1.222. With the ip nat inside source list 2 int Di0 overload command, this command "catched" this IP address (thanks to the ACL 2) and overwrote it to the IP address on the Di0 interface. But if you remove that command, there is no other translation entry prepared for that IP address - the dynamic translation has been removed and no static translation entry is created for the 192.168.1.222. That's why this IP will go to the internet untranslated and replies will not be able to return back.


If the inside network contains only the servers you have already covered in your static NAT entries and if these servers do not make any outgoing connections on their own behalf then the static NAT entries are sufficient. However, if there are also clients in the internal network, or if the servers do also create outbound connections, you will need to have a dynamic NAT/PAT in place.


Best regards,

Peter

strutter79 Sun, 05/09/2010 - 07:44
User Badges:

Hi Peter!


I found the error! The cisco router always worked fine. I modified the router's configuration to open port 22 for a linux internal machine and i ran nmap again. It showed me that the port 22 was filtered(among those of the exchange server). I was lucky to noticed that this linux machine had as a default gateway another router that i have(192.168.0.11).

I changed the linux machine's default gateway to use the cisco 878 router's IP(192.168.012). I ran nmap again: all ports were shown as open!

Still this didn't explain why the exchange server had their ports closed at random by the cisco 878 router. This server had defined as default gateway the IP of the cisco 878 router. So why shouldn't it work? then i found the problem, i enter using rdp to the exchange server and executed

the ipconfig command: this server has 2 ethernet adapters, one with ip 192.168.0.3 and gateway 192.168.0.11(the other router) and the other

with ip 192.168.0.13(this ip used for NAT) and default gateway 192.168.0.12(cisco 878 router). When i executed the command ipconfig it showed me the gw of the second ethernet adapter as non existant, even though it had it on the configuration under properties of the adapter. That was what made me realize that the cisco router was filtering the exchange NAT ports when the gateway associated with the adapter dissapeared.

then i found this article about this common problem of using two default gateways on windows 2003 servers under the same network:


http://support.microsoft.com/kb/159168


This combination was troublesome. The exchange server was losing the secondary default gateway at random, and when that was going on

the cisco router would block the NAT ports for exchange. Does this make sense to you?

Anyways thanks for your help!


Greetings


Oskar.

Peter Paluch Sun, 05/09/2010 - 08:37
User Badges:
  • Cisco Employee,

Hello Oskar,


I am glad you got it running. What I think about your issue is that the 878 was actually not blocking any ports but simply when the Windows decided to use a different gateway (i.e. when it lost the 192.168.0.12 gateway), the replies were sent through the 192.168.0.11 gateway, resulting in their loss or improper NATting.


Well, I suggested in my original post to verify whether it can be some other device in your network actually dropping the packets But you've got it working, finally, and I am happy about that.


Can you perhaps repost the entire configuration once again? We've made some changes to it and I would like to verify that the current version does not contain any outstanding issues.


Best regards,

Peter

strutter79 Sun, 05/09/2010 - 09:48
User Badges:

Hi peter


This is the current configuration of the router:


###################################################################################


Current configuration : 4682 bytes

!

version 12.4

no service pad

service timestamps debug datetime msec

service timestamps log datetime msec

no service password-encryption

!

hostname Cisco878

!

boot-start-marker

boot-end-marker

!

logging message-counter syslog

logging buffered 51200

logging console critical

enable secret 5 xxxxxxxxxxxxxx

enable password xxxxxxxx

!

no aaa new-model

clock timezone PCTime 1

clock summer-time PCTime date Mar 30 2003 2:00 Oct 26 2003 3:00

!

crypto pki trustpoint TP-self-signed-577650748

enrollment selfsigned

subject-name cn=IOS-Self-Signed-Certificate-577650748

revocation-check none

rsakeypair TP-self-signed-577650748

!

!

crypto pki certificate chain TP-self-signed-577650748

certificate self-signed 01

  3082023E 308201A7 A0030201 02020101 300D0609 2A864886 F70D0101 04050030

  30312E30 2C060355 04031325 494F532D 53656C66 2D536967 6E65642D 43657274

  69666963 6174652D 35373736 35303734 38301E17 0D313030 35303831 37353935

  365A170D 32303031 30313030 30303030 5A303031 2E302C06 03550403 1325494F

  532D5365 6C662D53 69676E65 642D4365 72746966 69636174 652D3537 37363530

  37343830 819F300D 06092A86 4886F70D 01010105 0003818D 00308189 02818100

  EDB74A46 7D50F663 5D80CEA8 697DB6F3 2797C8A4 DA3D1110 7D045FCC 48418C56

  6F4DD64D E665FD03 A36F5A6E 5515D20D C9559433 E327DE2D 4D406322 1466DE95

  252C1629 025E826C 019837A1 72A6AC40 1AD71B07 1F7F85D4 62BE757B 77557904

  FB191757 1B2CE2B1 5E2785C7 654D6487 A75330B7 7A3F75F6 62B284A6 E997FC0D

  02030100 01A36830 66300F06 03551D13 0101FF04 05300301 01FF3013 0603551D

  11040C30 0A820843 6973636F 38373830 1F060355 1D230418 30168014 8217C557

  29C7F74E AE522995 8B21699E FD507FD6 301D0603 551D0E04 16041482 17C55729

  C7F74EAE 5229958B 21699EFD 507FD630 0D06092A 864886F7 0D010104 05000381

  8100A0C4 AA28A09C 09FE78C6 E53F38DD C57ADB76 982F0FE2 49A6011E D913A47C

  5CBEF602 9D655082 865F91BF 1D569F68 4D7850F2 A4A8B6A5 AA0849B8 29BB57EF

  D76D516C 323B0BD0 EF1A0C7D 7377D689 37F6E996 76390AA4 48DDB687 80B4D579

  584BB16E DAB88C53 DD2F4BF6 2266BB26 E7AE6B26 B7F7D7E0 68A33FB9 B24CE77D 1D13

  quit

dot11 syslog

ip source-route

!

!

!

!

ip cef

no ip bootp server

no ipv6 cef

!

multilink bundle-name authenticated

!

!

!

username mastercisco privilege 15 secret 5 xxxxxxxxxxxxxx

!

!

!

archive

log config

  hidekeys

!

!

controller DSL 0

mode atm

dsl-mode shdsl symmetric annex B

!

!

!

!

interface Null0

no ip unreachables

!

interface BRI0

no ip address

no ip redirects

no ip unreachables

no ip proxy-arp

encapsulation hdlc

shutdown

!

interface ATM0

no ip address

no ip unreachables

no ip proxy-arp

ip virtual-reassembly

no atm ilmi-keepalive

pvc 8/35

  encapsulation aal5mux ppp dialer

  dialer pool-member 1

!

!

interface FastEthernet0

!

interface FastEthernet1

!

interface FastEthernet2

switchport access vlan 2

!

interface FastEthernet3

switchport access vlan 2

!

interface Vlan1

description $FW_INSIDE$

ip address 192.168.0.12 255.255.255.0

no ip unreachables

no ip proxy-arp

ip nat inside

ip virtual-reassembly

!

interface Vlan2

description $FW_INSIDE$

ip address 192.168.1.12 255.255.255.0

no ip unreachables

no ip proxy-arp

ip nat inside

ip virtual-reassembly

!

interface Dialer0

description $FW_OUTSIDE$

ip address negotiated

no ip unreachables

no ip proxy-arp

ip nat outside

ip virtual-reassembly

encapsulation ppp

dialer pool 1

dialer-group 1

ppp authentication chap callin

ppp chap hostname [email protected]

ppp chap password 0 xxxxxxx

ppp ipcp route default

ppp ipcp address accept

!

ip forward-protocol nd

ip route 0.0.0.0 0.0.0.0 Dialer0

ip http server

ip http authentication local

ip http secure-server

!

!

ip nat pool pool1 192.168.0.0 192.168.1.254 netmask 255.255.0.0

ip nat inside source static tcp 192.168.0.5 8060 interface Dialer0 8060

ip nat inside source static tcp 192.168.0.13 135 interface Dialer0 135

ip nat inside source static tcp 192.168.0.13 80 interface Dialer0 80

ip nat inside source static tcp 192.168.1.51 4050 interface Dialer0 4050

ip nat inside source static tcp 192.168.0.13 443 interface Dialer0 443

ip nat inside source static tcp 192.168.0.13 110 interface Dialer0 110

ip nat inside source static tcp 192.168.0.150 22 interface Dialer0 22

ip nat inside source static tcp 192.168.0.13 25 interface Dialer0 25

ip nat inside source static tcp 192.168.0.150 109 interface Dialer0 109

ip nat inside source static tcp 192.168.0.150 111 interface Dialer0 111

!

logging trap debugging

dialer-list 1 protocol ip permit

no cdp run


!

!

!

!

!

control-plane

!

banner login ^CWelcome to AIX 5.3^C

!

line con 0

no modem enable

line aux 0

line vty 0 4

password cisco

login

!

scheduler max-task-time 5000

end  


#############################################################################


Do you notice anything wrong?


Greetings


Oskar.

Peter Paluch Sun, 05/09/2010 - 11:31
User Badges:
  • Cisco Employee,

Hi Oskar,


Only two minor issues, nothing serious:


  1. You have an unusued NAT pool pool1 in your configuration (ip nat pool pool1 192.168.0.0 192.168.1.254 netmask 255.255.0.0). It can be removed.
  2. Your router is configured to accept source-routed packets (packets whose route is explicitly defined in their headers). While this can be, in some cases, a good troubleshooting tool, in general it is considered a security threat. Disable the source routing using the command no ip source-route


It looks otherwise good.


Best regards,

Peter

strutter79 Tue, 05/11/2010 - 21:11
User Badges:

Hi peter


thanks for your recommendations.I will make those changes for my Cisco router.

It's great to have found the problem that was giving me headaches!!

Greetings


Oskar.

Actions

This Discussion

Related Content