Re: inspect sip dropping "sip request: MESSAGE" packets

Dennis Goh · ‎03-18-2011

Hi,

I have 2 ASA 5510 firewalls at 2 different sites. Both running on version 8.0.4. Users are using an Instant Messaging type of application provided by a local telco here which is able to send and receive SMS using SIP (from the packet capture that I've done).

When users use the IM in site A, they are able to send and receive text messages via the IM from behind the firewall. However, when the users are in site B, users are able to send out text messages but not able to receive them.

I noticed that when I remove "inspect sip" from site-B's global policy map, users from site-B can successfully receive text messages. I have confirmed that it is the firewall that drops the packets as I have captured the inside and outside interfaces of site-B's ASA and I can see the incoming sip "request: MESSAGE" packet on the outside interface but I do not see the packet exiting the inside interface.

I have cross check both firewall configurations, and I do not see anything suspicious commands relating to sip that might cause this issue. Is there any command to troubleshoot why the sip inspection is dropping the sip packets on site-B?

Thanks in advance,

Dennis

Shrikant Sundaresh · ‎03-20-2011

Hi Dennis,

Could you perhaps collect the output of "debug sip" for a test sms on the Site-B firewall (with inspect sip enabled); and paste it here.

[Please make sure that there is very little Sip traffic going through the network, otherewise the debugs might overwhelm the ASA]

The debugs might help in determining why it is dropping the packets from Site-A.

Dennis Goh · ‎03-21-2011

Hi Shrikant,

Please find the debug sip and the sip packet pcap in the attached. I have filtered out the unnecessary and those in the text file should be the part that is causing the issue.Please let me know if the parts that I've filtered out were wrong.

Looking at the debug messages, it seems that the firewall is dropping the sip packet because the 'TEL URI' field in the SIP packet is invalid. If this is true, why is it that the site-A's firewall is not dropping the packets as in site-B? Thanks in advance for your help!

Regards,

Dennis Goh

Shrikant Sundaresh · ‎03-21-2011

Hi Dennis,

There is a bug, where the ASA drops SIP packets if the display name contains "tel". However this bug was fixed before 8.0.4.

Can you please confirm whether both the ASAs are running 8.0.4, and whether both have "inspect sip" enabled on them, when it works from one site, but not from the other?

Also if the above is correct, then can you also attach debug sip and packet captures of a working scenario? (from the other ASA).

-Shrikant

Dennis Goh · ‎03-21-2011

Hi Shrikant,

Site-A firewall:

sh run
: Saved
:
ASA Version 8.0(4)
!
hostname KL-5520-OL-FW01

Site-B firewall:

sh run
: Saved
:
ASA Version 8.0(4)
!
hostname rmt

As shown above, both firewalls are running the same firmware. And it is confirmed that one firewall has no problems with receiving SMSes from outside (site-A) and another does (site-B). I'll try to get the debug tonight when the SIP traffic are lesser on site-A. Will get back to you once I've obtained it. Thanks in advance!

Regards,

Dennis Goh

lcaruso · ‎03-21-2011

Hi,

Just wondering if there's an overriding reason not to upgrade the ASA code to 8.24?

There's a number of fixes since 8.0x and some include SIP. This can be seen with a cursory inspection of the relase notes found here http://www.cisco.com/en/US/docs/security/asa/asa82/release/notes/asarn82.html#wp372893

For example this one:

CSCsx57142 SIP Inspection Doesn't NAT Call-info field in SIP Notify message

or this

CSCti38496 ASA SIP inspection does not rewrite with interface pat

There are frequently other benefits from upgrading from the .0 version of any code. You might spend time debugging and hunting this down successfully only to find the solution is to upgrade or you might consider upgrading now and seeing if this is resolved.

Here's more SIP fixed since 8.0x...

CSCsj40174 SIP CRLF keepalives stall TCP-based SIP connections

CSCsl04124 SIP does not support 'early RTCP'

CSCso65967 SIP builds many secondary conns with register msg but no registrar

CSCsx73295 CSF-MOC clients can not register with OCS with ASA SIP-INSPECT

CSCsy81426 Sip inspection is dropping ftp secondary connection on port 5060

CSCta58656 SIP: Filtering by calling/called party should apply to ALL SIP messages

CSCtb23281 ASA: SIP inspect not opening pinhole for contact header of SIP 183 msg

CSCtc03451 TCP SIP Call Dropped When Resuming from Hold Due to Incorrect Timeout

CSCtc23007 Sip inspection drops 200 OK packet with early RTP/RTCP

CSCtc30413 Traceback with SIP pinhole replication Thread Name: Dispatch Unit

CSCtc96018 ASA watchdog when inspecting malformed SIP traffic

CSCtd36422 TCP proxy in SIP inspection causing 1550 block deplete temporarily

CSCte81368 Sip inspection fails to nat embedded media port

There's more than this, but you can see my point. Upgrading is likely to fix your problem and several others.

Dennis Goh · ‎03-23-2011

Hi Icaruso,

Thank you for your suggestion. My thinking is that if everything is running smoothly, I should minimize change unless purely necessary. If both of the boxes were using different IOS versions, I would have suspected the IOS problem. However this is not the case. If it was a bug, it would have affected both boxes.

My configurations are similar in terms of things related to SIP. That is why I've come to this forum for help I was hoping that forumers has experience in this issue and may guide me on how to resolve this issue. For all I know, maybe the Site-A box has a configuration that might have been applied unknownly that works as a workaround for this issue.

Nevertheless, upgrading the IOS would be my last resort option

Thanks again!

Regards,

Dennis Goh

Dennis Goh · ‎03-26-2011

Hi Shrikant,

My customer just realized that their Internet traffic in Site-A does not pass through the ASA firewall like how I explained it to be.

Like you mentioned, I might be hitting a bug in the software version that my Site-B ASA firewall is using. I've run through the Cisco bug database, however I can't seem to find any bugs related to the TEL error issue. How sure are we that we are hitting a bug? I don't want to be introducing downtime to my customer's network by upgrading the firmware and that itself did not help. Thanks

Regards,

Dennis Goh

lcaruso · ‎03-26-2011

Hi,

I see the facts have changed with regards to both ASAs not handling the same traffic.

I think you are splitting hairs when it comes to your bug search. Did you count up all of the SIP inspection related bugs fixed since 8.0x ? Do you suppose there's a reason hidden in those fixes that will address your issue even though it's not explicity spelled out? Is it possible there's an interaction between some of these SIP inspection flaws that might be related to your issue? How about a non-SIP flaw in the code that is indirectly affecting the SIP packets? Have you ever coded for living--do you know about the combinatorial explosion of complexty that exists in finite state machine execution tree that makes it impossible for any software company to release code that is bug free?

I had an 8.0x ASA spontaneously rebooting on a client recently. The fix? Upgrade the code to 8.2(4). Hasn't happened since.

Have you ever upgraded an ASA--do you know how much downtime is involved?

I did five last week and the prep time can be about 30-45 minutes and the downtime is the time it takes to reboot unless you have issues with the upgrade. The only issues I have seen in going fro 8.0x to 8.2(4) is on ASAs where the snmp process is running. I always turn it off. You will want to be onsite if at all possible or be prepared to visit the site if a problem occurs. You can prep your upgrade so two images are available to boot allowing you to back out to an earlier image if need be. I've never had to do that, but it is possible.

Let us know how it all turns out.

Dennis Goh · ‎03-26-2011

H lcaruso,

Yup indeed its a headache searching for bugs related to SIP on the 8.0.4. There are so many!

I know it is most probably a software bug. However, from the management point of view (non-technical), if there is no proof, they won't allow downtime, even if it's for a second.

The ASA runs in HA, I can propose to the management that upgrading would not incur downtime at all, however, I would still have to highlight the pros and cons to upgrading. And that time, what would I tell the upper management?
Pros: most probably might solve the sip issue.

Cons: might introduce bugs found in 8.2.4 but not found in 8.0.4.

I can tell the upper management what you said about bug database explicitly spelling out the bug we are hitting, but as you know, non-technical people are sometimes very hard to get through to.

Anyway, its not my responsibility to solve this SIP issue, thus this is why I did not open up a TAC case. This was just an attempt to help my customer. I was hoping some people here in the forum can point me in the right direction in terms of finding out the root cause of this issue (because I have little experience in SIP) be it configuration issues or bug issues.

Thanks for your insight though. Really appreciate it. Thanks!

Regards,

Dennis Goh

lcaruso · ‎03-26-2011

Of course it is a good approach to specifically search out this exact issue and its fix, if it is known.

If you think it's possibly a configuration issue, post the sanitized config.

If management won't allow it because it's not currently a known issue, ask them to consider the possibilities...

it was a known bug and was supposedly fixed but seems to have re-appeared in 8.0x

it is a new bug in 8.0x and you discoverd it; there may or may not be a fix in subsequent releases

it is a bug related to other code issues in 8.0x that have since been fixed

etc

In general, the idea behind major release numbers (7.0, 8.0) and minor release numbers (.0, .2) is significant code changes are made in major releases and revisions and fixes are made in minor releases. So maybe management can understand ever since 8.0 was released, bug fixes have continued to be made while only minor changes have been made in functionality (obviously not true in going from 8.2 to 8.3, but this is generally true and I'm not recommending you make that jump). The general trend is the software gets more reliable the further it progresses from the .0 release. If you look at the known caveats in 8.2(4) you will see the list has gotten smaller.

Does management have any experience with the software industry? Do they realize this is the only industry that can sell you a defective product and then ask you to determine what is wrong with it and get back to them with the problems so they can fix it later? Reason: combinatorial explosion of possible states the software can take on, especially when considering it is impossible to predict the software's behavior in different environments with all possible inputs and data therein. Just because the bug isn't listed doesn't mean it's not there and cannot be resolved by upgrading.

Why not print out the list of bugs fixed since 8.0x. It is quite long. Show that to managment and ask them if it seems worth doing.

I guess it boils down how much longer do they want to deal with this issue. If this is an HA site, why can't you upgrade one while the other remains in service and visa versa. Then there is no downtime.

I didn't mean to imply there is no risk in upgrading. It has to be carefully planned and exectued.

Shrikant Sundaresh · ‎03-27-2011

Hi Dennis,

Sorry for the delayed reply, but I was away for a few days.

So concluding from your previous update, the IM traffic passes only through Site B ASA right?

And removing "inspect sip" solves the issue.

From the captures i see that there are two packets:

183.78.0.68:5060 to 10.123.3.71:2450 Invite

10.123.3.71:2450 to 183.78.0.68:5060 Response

Since the destination ip is private, i assume the captures were taken on inside interface.

"inspect sip" rewrites the NAT ip inside the packet.

Since it was disabled for the capture, i see that within the packet, the destination is a public ip of: 203.158.29.66

Is this the public ip of 10.123.3.71 on Site B??

Also, is 183.78.0.68 the IM application server?

There are some applications which allow you to define whether they should see the public ip or the private ip. (ex: Polycom PVX)

Could it be that the packet is reaching with the private ip while it is expecting the Public IP? (when inspect sip is enabled)

Another thing is that, "inspect sip" requires that invites use the port 5060. I am not sure of this, but in my opinion, the destination should be 5060 and not the source. I shall confirm this as soon as possible. If the ASA is dropping the packet then this could be another reason.

Could you also please confirm that packets are being dropped because of inspect sip, by checking the output of "show service-policy". You would see the drop counters increasing beside "inspect sip" for every IM that doesnot go through.

I shall get back to you on the inspect sip and port 5060 thing, asap.

-Shrikant

Shrikant Sundaresh · ‎03-28-2011

Hi Dennis,

The invite message should have a destination port of 5060, as only then does ASA identify it as SIP traffic.

However, I think the first message in the capture is not the invite packet. I think it is a reply to the invite or something like that.

Could you please confirm that ASA is dropping by monitoring the output of "show service-policy" (as described in the previous post), and also, if possible, please provide a capture with the complete stream of packets.

-Shrikant

Dennis Goh · ‎03-30-2011

Hi Shrikant,

I've attached the pcap files in the attachments. FYI, the packets were sniffed from the inside and outside ports of the ASA by SPANning the ports on th switch where the ASA connects to.

I am almost certain that the "inspect sip" drops the packets due to the reason that whenever the function is switched ON, text messages cannot come in.

The customer has confirmed with me that Site-A does not go through an ASA to the internet. That is why it is not experiencing any issues. And yup, you are right, the external IP of Site-B is 203.158.29.66 and I would assume the IM server is at 183.78.0.68 after analyzing the packet captures.

In regards to the incoming packets having a source port of 5060 instead of destination, I think it's because the IM client has registered with the server using a random port as a source and 5060 as destination port. Thus, when the IM server replies, it'll reply back with the same port as it's source (5060) and destination port as the one the client used. I'm not sure whether this is a normal behaviour, but this is through my observations. Please correct me if I'm wrong

Thanks for your help again. Really appreciate it. Thanks!

Regards,

Dennis Goh

Dennis Goh · ‎04-10-2011

Hi All,

Just an update.

I've installed the IM-type application in my laptop and try it out through another ASA firewall running version 8.2.2. I seem to stumble upon the same problem. Please view the debug SIP below for the message. Notice that it is the same type of message received on the firewall with 8.0.4. Thanks

SIP::MESSAGE received from outside:183.78.0.68/5060 to inside:10.24.32.197/2450
    Found port 29865
    Found port 5060
Via Port 5060
    Found port 5060
SIP::Found User-Agent
SIP::regex engine has reached end of packet
SIP::Found CSeq 5985 MESSAGE
SIP::Found valid TEL URI: 0162380211
SIP::Found From addr "0162380211" (10)
SIP:: ERROR: local subscriber is missing phone-context in TEL URI
SIP:: ERROR: invalid TEL URI
SIP:: Mandatory field To address is missing
SIP::Parse Message failed!
SIP::MESSAGE received from outside:183.78.0.68/5060 to inside:10.24.32.197/2450
    Found port 29865
    Found port 5060
Via Port 5060
    Found port 5060
SIP::Found User-Agent
SIP::regex engine has reached end of packet
SIP::Found CSeq 5985 MESSAGE
SIP::Found valid TEL URI: 0162380211
SIP::Found From addr "0162380211" (10)
SIP:: ERROR: local subscriber is missing phone-context in TEL URI
SIP:: ERROR: invalid TEL URI
SIP:: Mandatory field To address is missing
SIP::Parse Message failed!
SIP::MESSAGE received from outside:183.78.0.68/5060 to inside:10.24.32.197/2450
    Found port 29865
    Found port 5060
Via Port 5060
    Found port 5060
SIP::Found User-Agent
SIP::regex engine has reached end of packet
SIP::Found CSeq 5985 MESSAGE
SIP::Found valid TEL URI: 0162380211
SIP::Found From addr "0162380211" (10)
SIP:: ERROR: local subscriber is missing phone-context in TEL URI
SIP:: ERROR: invalid TEL URI
SIP:: Mandatory field To address is missing
SIP::Parse Message failed!
SIP::MESSAGE received from outside:183.78.0.68/5060 to inside:10.24.32.197/2450
    Found port 29865
    Found port 5060
Via Port 5060
    Found port 5060
SIP::Found User-Agent
SIP::regex engine has reached end of packet
SIP::Found CSeq 5985 MESSAGE
SIP::Found valid TEL URI: 0162380211
SIP::Found From addr "0162380211" (10)
SIP:: ERROR: local subscriber is missing phone-context in TEL URI
SIP:: ERROR: invalid TEL URI
SIP:: Mandatory field To address is missing
SIP::Parse Message failed!
SIP::MESSAGE received from outside:183.78.0.68/5060 to inside:10.24.32.197/2450
    Found port 29865
    Found port 5060
Via Port 5060
    Found port 5060
SIP::Found User-Agent
SIP::regex engine has reached end of packet
SIP::Found CSeq 5985 MESSAGE
SIP::Found valid TEL URI: 0162380211
SIP::Found From addr "0162380211" (10)
SIP:: ERROR: local subscriber is missing phone-context in TEL URI
SIP:: ERROR: invalid TEL URI
SIP:: Mandatory field To address is missing
SIP::Parse Message failed!
SIP::MESSAGE received from outside:183.78.0.68/5060 to inside:10.24.32.197/2450
    Found port 29865
    Found port 5060
Via Port 5060
    Found port 5060
SIP::Found User-Agent
SIP::regex engine has reached end of packet
SIP::Found CSeq 5985 MESSAGE
SIP::Found valid TEL URI: 0162380211
SIP::Found From addr "0162380211" (10)
SIP:: ERROR: local subscriber is missing phone-context in TEL URI
SIP:: ERROR: invalid TEL URI
SIP:: Mandatory field To address is missing
SIP::Parse Message failed!
SIP::REGISTER received from inside:10.24.32.197/2450 to outside:183.78.0.68/5060
SIP::regex engine has reached end of packet
SIP::Found CSeq 58 REGISTER
SIP::Found URI in request line "sip:yes.my" (10)
SIP::Found valid SIP URI: sip:cheeyong@yes.my
SIP::Found From addr "sip:cheeyong@yes.my" (19)
SIP::Found From addr tag "ICF_4020116016-3162" (19)
SIP::Found valid SIP URI: sip:cheeyong@yes.my
SIP::Found To addr "sip:cheeyong@yes.my" (19)
SIP::Found Via branch "z9hG4bK3242059720-3238" (22)
SIP::Found Via addr "SIP/2.0/UDP 10.24.32.197:2450;branch=z9hG4bK3242059720-3238;rport" (65)
SIP::Found Max-Forwards 70
SIP::Found Call-ID 4020116016-3160@10.24.32.197 (28)
SIP::Found Expires, 3600 seconds
SIP::Found valid SIP URI: sip:cheeyong@203.158.25.139:29865
SIP::Found Contact sip:cheeyong@203.158.25.139:29865
SIP::Found Content-length 0
    Found port 2450
Via Port 2450
    Found port 5060
SIP::Found User-Agent
SIP::Found Expires, 3600 seconds
    Found port 29865
SIP::Found Expires, 3600 seconds
Created SIP Transaction for inside:10.24.32.197/2450 to outside:183.78.0.68/5060
    Call-ID: 4020116016-3160@10.24.32.197 (28)
    CSeq: 58 REGISTER
    Branch: z9hG4bK3242059720-3238
SIP::Updating xlate timeout for 203.158.25.139/29865 to 1:00:00
SIP:: Forward 954 bytes, total 954
SIP::200 received from outside:183.78.0.68/5060 to inside:10.24.32.197/2450
    Found port 29865
Via Port 29865
    Found port 29865
SIP::Found Expires, 1874 seconds
    Found port 5060
SIP::regex engine has reached end of packet
SIP::Found CSeq 58 REGISTER
SIP::Found valid SIP URI: sip:cheeyong@yes.my
SIP::Found From addr "sip:cheeyong@yes.my" (19)
SIP::Found From addr tag "ICF_4020116016-3162" (19)
SIP::Found valid SIP URI: sip:cheeyong@yes.my
SIP::Found To addr "sip:cheeyong@yes.my" (19)
SIP::Found To addr tag "aprqf8ejnt1-7qi85f00000k3" (25)
SIP::Found Via branch "z9hG4bK3242059720-3238" (22)
SIP::Found Via addr "SIP/2.0/UDP 10.24.32.197:2450;received=203.158.25.139;branch=z9hG4bK3242059720-3238;rport=29865" (95)
SIP::Found Call-ID 4020116016-3160@10.24.32.197 (28)
SIP::Found valid SIP URI: sip:cheeyong@10.24.32.197:2450
SIP::Found Contact sip:cheeyong@10.24.32.197:2450
Cloned SIP session for outside:183.78.0.68/5060 to inside:10.24.32.197/2450, 5 total
    From: sip:cheeyong@yes.my (19);tag=ICF_4020116016-3162 (19)
    To: sip:cheeyong@yes.my (19);tag=aprqf8ejnt1-7qi85f00000k3 (25)
    Call-ID: 4020116016-3160@10.24.32.197 (28)
Cloned SIP Transaction for outside:183.78.0.68/5060 to inside:10.24.32.197/2450
    Call-ID: 4020116016-3160@10.24.32.197 (28)
    CSeq: 58 REGISTER
    Branch: z9hG4bK3242059720-3238
Deleted SIP Transaction
    Call-ID: 4020116016-3160@10.24.32.197 (28)
    CSeq: 58 REGISTER
    Branch: z9hG4bK3242059720-3238
SIP::Updating xlate timeout for 203.158.25.139/29865 to 0:31:14
SIP:: Forward 516 bytes, total 516
Deleted SIP Transaction
    Call-ID: 4020116016-3160@10.24.32.197 (28)
    CSeq: 55 REGISTER
    Branch: z9hG4bK3062014720-3235
SIP::Timeout, deleting transaction
SIP::Timeout, deleting session
SIP::Deleting session for 10.24.32.197 to 183.78.0.68, 4 total
    From: sip:cheeyong@yes.my (19);tag=ICF_4020116016-3162 (19)
    To: sip:cheeyong@yes.my (19);tag=aprqf8ejnt1-7qi85f00000c3 (25)
    Call-ID: 4020116016-3160@10.24.32.197 (28)
SIP::MESSAGE received from outside:183.78.0.68/5060 to inside:10.24.32.197/2450
    Found port 29865
    Found port 5060
Via Port 5060
    Found port 5060
SIP::Found User-Agent
SIP::regex engine has reached end of packet
SIP::Found CSeq 5985 MESSAGE
SIP::Found valid TEL URI: 0162380211
SIP::Found From addr "0162380211" (10)
SIP:: ERROR: local subscriber is missing phone-context in TEL URI
SIP:: ERROR: invalid TEL URI
SIP:: Mandatory field To address is missing
SIP::Parse Message failed!
SIP::MESSAGE received from outside:183.78.0.68/5060 to inside:10.24.32.197/2450
    Found port 29865
    Found port 5060
Via Port 5060
    Found port 5060
SIP::Found User-Agent
SIP::regex engine has reached end of packet
SIP::Found CSeq 5985 MESSAGE
SIP::Found valid TEL URI: 0162380211
SIP::Found From addr "0162380211" (10)
SIP:: ERROR: local subscriber is missing phone-context in TEL URI
SIP:: ERROR: invalid TEL URI
SIP:: Mandatory field To address is missing
SIP::Parse Message failed!

Regards,

Dennis Goh