Were getting our third IronPort unit due to bad quality

keithsauer507 · ‎11-04-2009

We ordered an IronPort C160 with three years of service which ran us over $9000. Expecting the best from Cisco we thought how could this go wrong?

Well our first IronPort within the first week of ownership complained about a DEGRADED RAID array. Opening up a ticket, we received a new SATA hard drive. They said it's hot swappable, just install it while running. Fine OK.

Well that IronPort started freezing up and not passing mail. UNACCEPTABLE in a place of business. After a few reboots it kernel panic'd on bootup (as seen by a serial connection to a laptop running PuTTY. So for S&G I put the old hard drive back in (which Cisco said was bad). It booted up fine and worked the rest of the weekend. However come Monday morning it was froze up again and would not boot.

So our second IronPort was put into place October 20th. It's a brand new unit and it has been working fine ever since. Well, until today that is. I get yet another e-mail, and verified by logs show that one of the disks dropped off the array again. Yet again another DEGRADED raid array. Supports answer? They are going to send us another IronPort C160 unit.

So this is the THIRD IronPort unit and a lot of hassle. Were not doing anything wrong to the device. We get it, update it to 6.5.3 as recommended by the system update tool, upload our configuration file back to it and were up and running. It's in a cool temperature controlled data center, in the same rack as an EMC SAN and 3 Dell R710 servers running VMWare. None of that equipment is failing and it's all on the same PDU along with the same temperatures.

What is with the build quality of these things lately? I see they look like DELL servers. We have 3 R710's which are flawless. Our entire end user PC base is DELL. While we suffer the occasional dell power supply problem, they seem pretty stable machines.

Any other reliability issues anyone wants to share about their Ironport experience?

Thanks!

karlyoun · ‎11-05-2009

It sounds like you had a run of bad luck there. Hardware failures do happen, but most of our customers will never have to do an RMA, let alone 3 in a month.

I'm surprised that you were told that the drives are hot swappable. The C160 should be powered down before changing the drives. If it's OK with you, I can have one of our Support Managers follow up with you to see what went wrong.

_________________
Karl Young
Product Support Engineer
Email Security
Cisco IronPort Customer Support

We ordered an IronPort C160 with three years of service which ran us over $9000. Expecting the best from Cisco we thought how could this go wrong?

Well our first IronPort within the first week of ownership complained about a DEGRADED RAID array. Opening up a ticket, we received a new SATA hard drive. They said it's hot swappable, just install it while running. Fine OK.

Well that IronPort started freezing up and not passing mail. UNACCEPTABLE in a place of business. After a few reboots it kernel panic'd on bootup (as seen by a serial connection to a laptop running PuTTY. So for S&G I put the old hard drive back in (which Cisco said was bad). It booted up fine and worked the rest of the weekend. However come Monday morning it was froze up again and would not boot.

So our second IronPort was put into place October 20th. It's a brand new unit and it has been working fine ever since. Well, until today that is. I get yet another e-mail, and verified by logs show that one of the disks dropped off the array again. Yet again another DEGRADED raid array. Supports answer? They are going to send us another IronPort C160 unit.

So this is the THIRD IronPort unit and a lot of hassle. Were not doing anything wrong to the device. We get it, update it to 6.5.3 as recommended by the system update tool, upload our configuration file back to it and were up and running. It's in a cool temperature controlled data center, in the same rack as an EMC SAN and 3 Dell R710 servers running VMWare. None of that equipment is failing and it's all on the same PDU along with the same temperatures.

What is with the build quality of these things lately? I see they look like DELL servers. We have 3 R710's which are flawless. Our entire end user PC base is DELL. While we suffer the occasional dell power supply problem, they seem pretty stable machines.

Any other reliability issues anyone wants to share about their Ironport experience?

Thanks!

mychrislo_ironport · ‎11-05-2009

Perhaps it should go like this......

Except the C160 model (or C1xx series). All other models are hot-swappable harddrive.

I was shocked when i first read the message.

keithsauer507 · ‎11-05-2009

Well I have the new one in. The RMA process is very straightforward, simple and they are on the ball in terms of getting a unit out so fast. The real issue is the inconvenience of getting the new unit, staging it, scheduling time to swap it out, reinstalling config, testing it, and loosing the quarantines.

Do you know if there's any way to save the quarantine files or mail logs? We already had one call today asking about an email, and the logs are useful to see if an e-mail was sent or not.

Also the statistics ie) history goes away. Not a huge deal, it will rebuild itself of course, but all part of the hassle.

If you want to reach out you can, I'll shot you a PM with contact info. I'm curious if there's a known issue with a round of bad hard drives or raid controllers, motherboards, etc... Maybe there's a way to backtrack and investigate all of the RMA's done in the past 6 months and see if there's any common denominator to the problem. Heck maybe you'll find all of the RMA's are on AsyncOS 6.3... Perhaps it's a quirk in the 6.3 update? I'm putting it on regardless, it's published and recommended, so I should be able to install it.

Interesting this time they sent a new unit right away. First RMA a few weeks ago, they sent just a hard drive. A Dell branded Seagate SATA if I recall. I guess the policy right now has changed to just send the whole unit out.

keithsauer507 · ‎11-05-2009

I rebooted our supposed 'bad' ironport because i quickly swapped the IP's between the old and the new unit that we've received today.

On the 'bad' IronPort it now says RAID Status: REBUILDING 1% completed

I don't think the hard drive's in these things are bad. For whatever reason, a drive drops off the controller. Upon reboot the drive is back alive and kicking. The RAID of course rebuilds because of the data that's changed since last reboot.

Oh well, I already got my RMA. Just something for you guys to look into.

mychrislo_ironport · ‎11-05-2009

I want to share "disk" experience on ironport.

I had disk failed in C650
==> yes, I saw "bad" drive can rebuild and is reuseable upon retry.

We did upgrade from C650 -> C660. The new unit developed a failed drive within the first week.

But I guess it is the new drive technology....I also saw this "fresh" drive failed upon delivery with IBM...SUN too. SATA/SAS...

efoster_ironport · ‎11-05-2009

Hello,

Either myself or the US East Coast support manager will be calling you about this. I apologize that the information you were provided about C160 drives being hot swappable, which is inaccurate. We have recently removed non-hot swappable drives from our RMA parts list.

I have gone ahead and requested failure analysis for you on the C160s that were replaced under tickets 509904 and 516170. You will receive a failure analysis report as soon as these devices have been received and testing completed.

Don't hesitate to contact support or escalate a support ticket if you experience any additional issues on your C160s.

Best regards,

Erin Foster
CICS Support Manager

karlyoun · ‎11-06-2009

I've had the article about replacing hard drives updated. It now makes it very clear that hot swapping is only available in 2U appliances.

_________________
Karl Young
Product Support Engineer
Email Security
Cisco IronPort Customer Support

Perhaps it should go like this......

Except the C160 model (or C1xx series). All other models are hot-swappable harddrive.

I was shocked when i first read the message.

keithsauer507 · ‎11-06-2009

Thank you for the response. I did get the voice mail and it's great to hear from you.

I do want to make note, that our IronPort C160 that is to be RMA'd, now shows a "READY" status on the RAID array. Once I put the new IronPort in and configured it, I rebooted both units when I swapped their IP addresses. So the old one upon booting up, found both SATA hard drives and rebuilt the array. So it might appear to the team that the IronPort is fully functional. It could take a few days, or even weeks to see the hard drive drop off again.

I'm going to remove our configuration for security purposes.

I'm wondering if the tech support team should of simply told me to reboot the unit? Upon reboot, the raid array rebuilt itself and 'looks' good to the end user.