ASA 5525X creating inside to inside connections

[daniel.bales]

Hi

I have a ASA 5525-X Running the following software version

Cisco Adaptive Security Appliance Software Version 9.1(2) 
Device Manager Version 7.1(3)

For some reason randomly, usually at 1-2 week intervals it seems to create connections incorrectly that are from the Inside interface to the Inside when it should be from Inside to DMZ. As you can see below from the show connection output

UDP DMZ1  10.20.241.62:8812 Inside  10.150.241.26:8812, idle 0:00:00, bytes 4414422, flags - 
UDP DMZ1  10.20.241.62:8812 Inside  10.8.244.62:8812, idle 0:00:00, bytes 20808060, flags - 
UDP DMZ1  10.20.241.62:8812 Inside  10.82.241.62:8812, idle 0:00:00, bytes 4416360, flags - 
UDP DMZ1  10.20.241.62:8812 Inside  10.80.241.62:8812, idle 0:00:00, bytes 17509840, flags - 
UDP DMZ1  10.20.241.62:8812 Inside  10.8.244.62:15086, idle 430:02:06, bytes 195940, flags - 
UDP Inside  10.8.244.62:8812 Inside  10.20.241.62:8812, idle 0:00:00, bytes 531732092, flags - 
UDP Inside  10.150.241.26:8812 Inside  10.20.241.62:8812, idle 0:00:01, bytes 1515573, flags - 
UDP Inside  10.80.241.62:8812 Inside  10.20.241.62:8812, idle 0:00:00, bytes 551717703, flags - 
UDP Inside  10.82.241.62:8812 Inside  10.20.241.62:8812, idle 0:00:05, bytes 1517283, flags - 

At the top are the correct connections, but the ones beneath get created somehow and all traffic for the proper connection fails until I clear the connections manually then traffic resumes.

When it first occurred I had a ticket open with TAC and they told me that the ASA will do this if the DMZ connection goes down the way the routing table is setup means the ASA will believe the traffic should go via the Inside interface instead and create these connections. However I have not seen any issues with my switch that would point to the interfaces going down at all

Does anyone have any ideas? Its a tricky issue and a right pain in the butt at the moment

[Vibhor Amrodia]

Do you have an overlapping subnet on the ASA device ? I think this has to do with the Routes and NAT configured on the ASA device.

Also , you must be having the same-security-traffic permit intra-interface on the ASA device.

Can you share the show run from the ASA device ?

Thanks and Regards,

Vibhor Amrodia

[Marius Gunnerud]

Would you be able to post your ASA's running-config?

It would almost seem, as the TAC engineer point out, that your ASA has a route to the 10.0.0.0/8 network point to the inside interface and then more specific routes pointing out the DMZ (or is the DMZ IP directly connected)? 

Also, when the issue happens again run a quick packet tracer for the two IPs before you clear the connections to see what the output interface ends up being.

--

Please remember to select a correct answer and rate helpful posts

[daniel.bales]

Thanks for the responses guys!

I Have attached my running config, I removed external IPs, Objects/Object groups and my ACLs just for security reasons hopefully you don't need those :)

You are right when you say I have overlapping subnets/routes, I have a 10.0.0.0/8 and then the connected route to 10.20.241.0/24. 

If I add more specific routes do you think it will resolve it? My thought is that if i add specific routes and the same issue happens won't the FW just think the traffic is outside as that is the default route? 

I do have same-security-traffic permit intra-interface enabled, not sure if i require it I may be able to remove it but I believe I had issues with NAT when I didn't have it enabled

Again thanks for your help! Hopefully I can determine a way to fix it so that it doesn't happen going forward, I believe it is something config related as my other firewalls in the region don't have this issue and this one is quite new only a few months old so there must be something in there that I have not done correctly.

[Vibhor Amrodia · Accepted solution]

Hi Daniel,

Thank you for sharing the outputs.

Can you make this change on the ASA device:-

timeout floating-conn 0:00:10

Clear the connections which are incorrect and see if they are created again or not ?

I think you might want to revisit the routes configured on the ASA device as well:-

route Inside 10.0.0.0 255.255.0.0 10.20.240.1 1

route Inside 10.20.0.0 255.255.0.0 10.20.240.1 1

PS:- I was not able to check the NAT statements as the objects were missing in the configuration attached.

Reference:-

http://www.cisco.com/c/en/us/td/docs/security/asa/asa-command-reference/T-Z/cmdref4/t1.html#pgfId-1569874

[Marius Gunnerud]

Actually the routes are fine, though, the 10.20.0.0 255.255.0.0 route will cause traffic for the DMZ to be routed back out the inside interface if the DMZ interface fails. 

Though if you ask me it is always best to have as specific routes as possible.  Of course if you are constantly adding new subnets to your network then using broad subnets as you have will make administration easier.

As Vibhor has mentioned, we would need to see your network objects to be able to check your NAT statements.

If I add more specific routes do you think it will resolve it? My thought is that if i add specific routes and the same issue happens won't the FW just think the traffic is outside as that is the default route? 

If you add a more specific route for each network then it will resolve the inside to inside connection issue you are having.  While the DMZ1 interface is up and running it sees the route as directly connected and this will take precedence over any other route.  When the interface fails the ASA no longer sees the 10.20.241.0 network directly connected so it looks to the routing table and there it sees it has a route to 10.20.0.0/16 which includes the 10.20.241.0 network and will therefore send the traffic out the inside interface in an attempt to reach that subnet.

Now, if you add more specific routes you will solve this issue. But you are correct that if the DMZ1 interface fails the default route will be used.  However, if you upgrade your ASA version to 9.2(1) or higher you can use the null0 interface for routing.  That if/when the DMZ1 interface fails all routing to the 10.20.241.0 network will be dropped.  Keep in mind that this will only work if the network is directly connected to the ASA (as it seems to be in your case).

--

Please remember to select a correct answer and rate helpful posts

[daniel.bales]

Thanks for the responses guys

I tried to change the timeout floating-conn setting but got the following - 

ERROR: % Time should be >= 0:0:30

Should I just set it to 0:0:30 then?

Also attached my Objects and NAT statements, let me know if you find anything else in that

I will revise the routes to improve it and look at upgrading the ASA Software during our next maintenance window as well

Thanks again!

[Vibhor Amrodia]

Hi Daniel,

Yeah , you can set it to:-

timeout floating-conn 0:00:30

Thanks and Regards,

Vibhor Amrodia

[daniel.bales]

Thanks Vibhor, I've set that command and read up on it. Sounds like it may help my issue

So far its been there for a week and the issue has not reoccurred, I am not waving the flag yet but it looks promising. If it goes a couple more weeks without issue then i'll say its a success!

Thanks again!

[Jan]

I think i am seeing similar "features" in https://supportforums.cisco.com/discussion/12319041/asa-session-setup-through-wrong-interface-session-stays-there

In my case i had the inside interface "downed" as i rebooted the internal switch.
during that time new sessions are setup using the default route (pointing to outside) - this ist something i understand.. but I would expect/hope/.. that these session get re-established/invalidated as soon as i get a "better/different" route for the session.

e.g. invalidate all existing sessions, that are using a wrong interface after a routing table change.

On other firewall products i have seen similar issues, here i had null-routed the internal networks using a higher metric, which stops sessions from being created when the interface is down.
Not sure whether ASA works the same.

It's only UDP sessions, as they don't care about replies. TCP sessions will probably die on their own.