AsyncOS 5.5 - errno 60 in mail logs

[beneckij_ironport]

We upgraded our IronPorts to AsyncOS 5.5 two weeks ago and over the past week, an issue has begun to manifest itself. We are wondering if anyone else has seen this.

When sending to some domains (at this point about 20 or so) we receive error responses (soft bounces) and eventually hard bounces. The errors we are seeing in the mail logs are:

Thu Nov 15 11:43:57 2007 Info: Delayed: DCID 11509176 MID 43979667 to RID 0 - 4.4.0 - Other network problem ('000', ['[Errno 60] Operation timed out']) []

and less often:

Thu Nov 15 11:33:23 2007 Info: Delayed: DCID 11508361 MID 44031268 to RID 0 - 4.4.0 - Other network problem ('000', ['[Errno 54] Connection reset by peer']) []


The systems on the other side appear to be all fronted by PIX Firewalls and do not have the SMTP Fixup command enabled. (If you telnet to it you get 220 ************************) Only very small manual telnet emails seem to make it through. The MTU packet size by default on their end is 1380. They lowered it and more emails made it through, but not all. The MTU packet size on our firewalls in 1500 and adjusting the packet size on the IronPorts has not helped the issue.

Has anyone else seen this, or better yet, fixed this issue? If you look at delivery status and see domains that are up and have a lot of softbounces, this may be a sign that this is happening.

Any and all help would be appreciated.

[jaigill]

It seems that path MTU discovery may be failing on some router/network device in your company. I would recommend checking the MTU sizes configured on all network devices involved in email routing within your organisation.

[beneckij_ironport]

It seems that path MTU discovery may be failing on some router/network device in your company. I would recommend checking the MTU sizes configured on all network devices involved in email routing within your organisation.

It is definitely an incompatibility with the PIX Firewall. We have called two of the Email admins at the domains we are having problems sending to and they turned off their PIX firewalls and emails started routing correctly. Has anyone else had issues sending to domains with PIX firewalls and the SMTP Fixup command on since upgrading to 5.5?

Thanks

[beneckij_ironport]

It seems that path MTU discovery may be failing on some router/network device in your company. I would recommend checking the MTU sizes configured on all network devices involved in email routing within your organisation.

We've had this checked and it is fine across the entire network. We are running multiple systems with mirrored configurations on the east and west coast and everything is correct. We've had this confirmed by our firewall team and network services.

[Erich_ironport]

I am guessing you may have implemented DKIM with the 5.5 upgrade.

It looks like older PIX Firewalls have a bug:
CSCsi01498 - ESMTP inspect cannot handle content-type string in DKIM headers

The problem is that the recieving side needs to upgrade their PIX Firewalls to Software Version 7.2(3). It is not something we can fix on the IronPort side, it also is not specific to IronPort, all DKIM signed emails from, Google and the like would mostlikely have the same issue to these recipients.

Link to Cisco PIX release notes below which list this specific bug. Search for "DKIM" to find it.

http://www.cisco.com/en/US/docs/security/pix/pix72/release/notes/pixrn723.html

Hope this helps.

- Erich

[tminchin_ironport]

This is annoying as we have been DKIM signing for ages (years) without having this problem. It has only recently appeared in the last couple of months.

Given that Ironport is owned by Cisco - hopefully Cisco+Ironport will be doing some proactive about the issue.

Would it be possible for Ironport AsyncOS to detect [broken] PIX smtp fixup (ie the banner of ************s) and not do DKIM signing?

[ppeterson_ironport]

Are you sure you've had DKIM signing for years? IronPort has supported DK (domain keys) since 2005 but DKIM support came only with AsyncOS 5.5 recently. Although the DK to DKIM changes are small they are significant.

It's not possible for us to selectively sign with DKIM based on detection of PIX SMTP fixup. We currently do the signing of the message and header insertion before connecting to the remote MTA. The signature on a very large message can take a fair bit of CPU. While technically possible to rearchitect the email pipeline to wait for the SMTP banner to DKIM sign the message it would be a major change to the product.

We have a dialog with Cisco PIX folks and will continue to encourage folks to upgrade or turn off this "feature' where appropriate.

pat

[tminchin_ironport]

We turned it on as soon as it was a feature in AsyncOS - around 2005 - despite it being half implemented (ala AsyncOS's support of SMTP-TLS).

However, it is pretty much finished [for everyone on the internet] for the next 12+ months unless Cisco+Ironport fix the PIX smtp fixup issue. Too many people who have no idea and also have PIXes with smtp fixup - even though it is seriously broken for esmtp.

The solution (or destruction) of DKIM rests 100% in Cisco+Ironport hands now.

[Doc_ironport]

We turned it on as soon as it was a feature in AsyncOS - around 2005 - despite it being half implemented (ala AsyncOS's support of SMTP-TLS).

It's not that it was only half implemented - it's that there are two different protocols at play here.

The first is DomainKeys, which IronPort has supported for some time. This is an early standard designed largely by Yahoo. DomainKeys is covered in Historic RFC 4870.

The second is DomainKeys Identified Mail, or DKIM, which is a newer standard based on a combination of DomainKeys and Cisco's Identified Internet Mail. DKIM is covered in the Standards Track RFC 4871, and was implemented in AsyncOS 5.5.

However, it is pretty much finished [for everyone on the internet] for the next 12+ months unless Cisco+Ironport fix the PIX smtp fixup issue.

It's been fixed since August in PIX code 7.2(3). The problem is of course that it's the recipients end that needs this upgrade (or to disable MailGuard), not the end doing the DKIM signing, which obviously make it more difficult to resolve.