Dissapearing NAT for 3725 to FIOS, loss of FTP service help

jmasters · ‎01-19-2010

We recently had Business FIOS installed as a replacement for our company's DSL service, and I've encountered a strange problem as I've been migrating our Internet services over to it. Internet access for client computers via overloaded PAT pool works fine under all circumstances. The 1st service I moved was a webcam for the milling department, which works fine. The next was our SMTP server, which also works fine. The one I'm having issue with is our FTP server. It is a Windows Server 2003 machine running IPswitch WS_FTP. It is configured on the router with a static NAT to one of our assigned IP addresses from FIOS. In a nutshell, it will work properly for 4-6 hours, and then randomly stop working completely. The server is up and available, but it is unreachable on the Internet, and it loses its Internet connection outbound as well. Traces/debugs show the outbound traffic passing through the router, but no inbound. The ARP entry in the ARP table for this specific static NAT is lost (although the other ones remain - SMTP and mill camera). Traces on the Internet side go off to nowhere.

The only way to resolve the issue is to remove the static nat entry via 'no ip nat inside source static...." and then re-add it. At that point, it immediatly starts working again, and will randomly stop 4-6 hours later.

I have tried putting a static ARP entry in the config for this host - the result is the same as above except (ODDLY ENOUGH) the static ARP entry is still in the ARP table. No Internet access for the server, unreachable on the Internet as well.

I have tried changing the public IP address of the server as well, but the problem follows. As a test, I set up another host on the original IP address, and it works without any problem - so it has to be a combination problem with the FTP server as well.

I've searched and found some issues with ARP and FIOS, but I don't understand why the anonomous ARP request from the FIOS side doesn't break all of the static NAT's I have.

The relavant portions of the config are below - the FTP server is internal 191.168.1.26, external 74.106.12.12

Thanks for the look.

##########################################################

!
boot-start-marker
boot system flash c3725-adventerprisek9-mz.124-15.T11.bin
boot system flash c3725-adventerprisek9-mz.124-2.T1.bin
boot-end-marker
!
logging buffered 4096
!
no aaa new-model
clock timezone UTC -5
clock summer-time UTC recurring
no network-clock-participate slot 1
!
!
no ip cef
!
!
!
!
ip inspect udp idle-time 300
ip inspect dns-timeout 7
ip inspect tcp idle-time 1800
ip inspect tcp finwait-time 10
ip inspect tcp synwait-time 60
ip inspect tcp max-incomplete host 100 block-time 2
ip inspect name fw_url http urlfilter
ip inspect name fw_1 ftp timeout 1800
ip inspect name fw_1 smtp timeout 1800
ip inspect name fw_1 tcp timeout 1800
ip inspect name fw_1 udp timeout 1800
ip inspect name fw_1 http timeout 1800
ip urlfilter max-request 3000
ip urlfilter allow-mode on
ip urlfilter urlf-server-log
ip urlfilter server vendor websense 192.168.1.9 timeout 2
ip accounting-threshold 256
!
multilink bundle-name authenticated
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
controller DSL 0/0
shutdown
line-term cpe
!
ip tcp synwait-time 10
!
!
!
!
interface FastEthernet0/0
description ChoiceOne DSL Internet
bandwidth 1000
ip address 64.179.109.145 255.255.255.128
ip access-group 121 in
ip nat outside
ip inspect fw_url out
no ip virtual-reassembly
speed 10
half-duplex
no cdp enable
!
interface FastEthernet0/1
description local network
ip address 192.168.1.1 255.255.248.0
ip nat inside
no ip virtual-reassembly
ip policy route-map bal_1
duplex auto
speed auto
no cdp enable
!
interface FastEthernet1/0
description DMZ network
ip address 192.168.50.1 255.255.255.0
ip nat inside
no ip virtual-reassembly
ip policy route-map dmzctrl
duplex auto
speed auto
no cdp enable
!
interface FastEthernet1/1
description FIOS Internet connection
bandwidth 25000
ip address 74.106.12.2 255.255.255.0
ip access-group 140 in
ip nat outside
ip inspect fw_1 out
no ip virtual-reassembly
ip route-cache flow
speed 100
full-duplex
no cdp enable
!
no ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 74.106.12.1
ip route 192.168.254.0 255.255.255.0 192.168.1.2
!
ip flow-cache timeout active 1
ip flow-export source FastEthernet0/1
ip flow-export version 5
ip flow-export destination 192.168.1.9 2048
ip flow-export destination 192.168.1.53 9996
!
ip http server
ip http authentication local
ip http secure-server
ip nat translation timeout 3600
ip nat translation tcp-timeout 300
ip nat translation udp-timeout 10
ip nat translation finrst-timeout 30
ip nat translation dns-timeout 10
ip nat pool fiospool3 74.106.12.3 74.106.12.3 netmask 255.255.255.0
ip nat inside source route-map fiosnat pool fiospool3 overload
ip nat inside source static 192.168.1.75 64.179.109.147
ip nat inside source static 192.168.1.73 64.179.109.192
ip nat inside source static 192.168.1.6 64.179.109.193
ip nat inside source static 192.168.50.10 64.179.109.194
ip nat inside source static 192.168.50.2 64.179.109.195
ip nat inside source static 192.168.1.202 74.106.12.4
ip nat inside source static 192.168.1.59 74.106.12.5
ip nat inside source static 192.168.1.60 74.106.12.6
ip nat inside source static 192.168.1.26 74.106.12.12
!
logging history size 250
logging history debugging
logging trap debugging
logging 192.168.1.202
access-list 120 permit ip 192.168.0.0 0.0.7.255 any
access-list 121 (list not shown)

access-list 140 permit tcp any host 74.106.12.12 eq www
access-list 140 permit tcp any host 74.106.12.12 eq ftp
access-list 140 permit tcp any host 74.106.12.12 eq ftp-data established
access-list 140 permit tcp any host 74.106.12.12 gt 1024
access-list 140 permit icmp any host 74.106.12.12 echo
access-list 140 permit icmp any host 74.106.12.12 echo-reply
access-list 140 permit tcp any host 74.106.12.5 eq www
access-list 140 permit tcp any host 74.106.12.6 eq smtp
access-list 177 permit ip any host 74.106.12.4
access-list 177 permit ip host 74.106.12.4 any
access-list 177 permit ip host 192.168.1.26 any
access-list 177 permit ip any host 192.168.1.26
access-list 180 permit ip host 192.168.1.6 any
access-list 183 permit ip host 192.168.1.60 any
access-list 184 permit ip host 192.168.1.73 any
access-list 185 permit ip host 192.168.1.75 any
access-list 186 permit ip host 192.168.50.2 any
access-list 187 permit ip host 192.168.50.10 any
no cdp run
!
!
!
route-map bal_1 permit 15
match ip address 180 184 185
set ip next-hop 64.179.109.129
!
route-map fiosnat permit 10
match ip address 120
!
route-map dmzctrl permit 10
match ip address 187 186
set ip next-hop 64.179.109.129
!
!
!
!
control-plane
!

jmasters · ‎01-20-2010

As an update - I'm running some monitoring/diag programs and can now say that the problem happens almost exactly every 6 hours.

rmatthewcole · ‎04-23-2010

Joel,

I attempted to install FiOS service 2 years ago and ran into a problem that sounds exactly like this .If you are running multiple IP addresses with a Cisco perimeter device and a Alcatel head end your bound for trouble.

The root cause is the Alcatel equipment does not source the arp requests with the real IP address. Essentially there is no static routing on the Verizon device that would point to your perimeter router or firewall, it is dynamically built by arp requests from the Verizon network. The problem is that the source IP for the arp is 0.0.0.0 which Cisco equipment will ignore as invalid. Routing for those static IP addresses will be removed from their routing table if they do not have a response every 6 hours, which is when the cache expires. I was able to restore service on a temporary basis by changing the primary IP on the external firewall to each of the external IPs in my assigned block. this would force the Cisco equipment to for a gratuitous arp which also brings the circuit up. I have also heard of others working around this problem running perl scripts to force an arp back to the head end ip for those addresses on a timer.

People that use the Actiontec router don’t have this problem because the Actiontec routers that Verizon typically uses don’t follow RFC standards and respond to the arp anyway.

Hope this helps,

Matt

jmasters · ‎04-23-2010

Matt - thank you so much for your great reply, especially since this has been out there for a while on the forum. I've been running a script on my laptop that is constantly checking (and resetting) the NAT when necessary for the past 3 months. What you've written makes sense and at least I know I'm not crazy for what I was seeing and thinking that it wasn't a config issue on my router. I'm thinking about putting the Actiontec router inbetween my 3725 and the FIOS premise equiment, as I have several more external IP's that I've been waiting to migrate until this issue is resolved. I've also got my TAC access straightened out and will open a ticket with Cisco to see if they have any other advice.

Joel

rmatthewcole · ‎04-27-2010

I fought this issue for a while and gave up, do to some recent events it looks like we are going to have start looking at this again. We had the FiOS line re-installed today for testing. If you get anything usefull I would appreciate it if you let me know.

ChopSueyAR · ‎05-05-2010

"I was able to restore service on a temporary basis by changing the primary IP on the external firewall to each of the external IPs in my assigned block. this would force the Cisco equipment to for a gratuitous arp which also brings the circuit up."

Matt, can you go into a bit more detail about this?

Also, I have heard several "stories" of the Verizon ONT (glorified media converter) having a default limit of 1 MAC address. Any experience with that?

I'm told Verizon can increase the number of MAC addresses on the device with a phone call.

Trying to get an 1812 router up and running on Fios w/ a static IP.

rmatthewcole · ‎05-05-2010

Sure..The only way I was able to get the static routing restored was to change the external IP address on the router. My config had something like:

int f0/0

ip address 1.2.3.1 255.255.255.0

ip address 1.2.3.2 255.255.255.0 secondary

ip address 1.2.3.3 255.255.255.0 secondary

ip address 1.2.3.4 255.255.255.0 secondary

ip address 1.2.3.5 255.255.255.0 secondary

By changing the primary address to each of the secondary interfaces it would force a gratuitous arp. This was detected by the Alcatel equipment and would restore routing for the additional ip addresses. unfortunately I would have to do this every 6 hours however or the non-primary IPs would stop getting traffic. If you were running a single static IP address none of this should be an issue, the primary IP worked fine.

As far as the MAC question, yes that is correct. There is a MAC limit on the ONT to prevent customers without multiple IP addresses from just plugging in additional PC's. By default the ONT will only allow 1 MAC, and if you change it you must either call Verizon or reset the ONT to clear the lock. If you are going to run multiple devices directly connected to the ONT you will need to ensure that this limit is set correctly. If you are running everything behind the 1812 you may be ok.

One other thing you may want to look at as well. What speed service are you using? One of the other discoveries I made in this effort is many of the cisco routers used with this service actually aren't capable of pushing the bandwidth that FiOS can provide. This was a bit of a shock to me when I first discovered it. According to the Cisco performance documentation, that router can only handle about 35Mbps throughput. If you are running more than that with upstream and downstream traffic, you may have issues with overruns on the interface. The link to the performance document is listed below.

(http://www.cisco.com/web/partners/downloads/765/tools/quickreference/routerperformance.pdf)