IPS 4260 and hardware bypass

Unanswered Question
May 29th, 2008
User Badges:

Last night we were testing our 4260 hardware bypass and noticed there was a 30 second to 75 second delay before the NIC's started passing traffic if the power cord was unplugged from the IPS. Is their supposed to be this long of a delay? I was under the assumtption that hardware bypass should take less then a second to pass traffic? 30 -75 seonds seems like an aweful long time to have a network not passing traffic. Does anyone have any exerpience with this? Is this normal?

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 5 (3 ratings)
Loading.
attmidsteam Thu, 05/29/2008 - 06:24
User Badges:
  • Silver, 250 points or more

I would recommend investigating a switch bypass solution using BPDU. We've got our bypass latency down to 0.3 seconds.

marcabal Thu, 05/29/2008 - 06:41
User Badges:
  • Cisco Employee,

What type of devices are the 2 interfaces of the IPS-4260 plugged in to?


When power is removed from the sensor, the links are immediately dropped on the interfaces. While power is on, the sensor itself links to the 2 other devices. But when power is removed the links are dropped and the NIC simulates a wire and now the 2 devices link directly to each other.


Any delay in passing of packets is most likely a delay within one of those 2 other devices.


The most common delay is spanning-tree. If one of the 2 devices is a switch, then the default for most ports is to participate in spanning-tree. So before sending the standard packets through the interface it will first send through spanning-tree packets in order to map the local network and prevent packet loops. This generally takes about 40 seconds. Once spanning-tree is run, then the switch will start passing through packets.


THe spanning-tree time cna be removed by setting spanning-tree portfast on the switch interface. This, however, should only be used if you can guarantee no possible spanning-tree loops. It is recommended this only be done for end hosts, and not for networking devices like the sensor. Because if there is a loop it won't be detected and could cause serious network issues.


Ways to see if this is what you are seeing.

Power sensor off

Look at both devices to determine when the link is established.

Watch spanning-tree and determine how long it takes for the ports to be put into a Forwarding state (packets don't get passed until it goes to a Forwarding state).

Determine when packets start making it through.


If the link goes down and back up within a couple of seconds. But takes 40 seconds to get into Forwarding state and start passsing traffic then spanning-tree on the switch is the cause for the delay.

The spanning-tree delay is not unique to hardware bypass. You could remove the sensor and plug the devices directly to each other. If you unplug that cable, and then plug it back in you would still see the same delay before packets start being sent through.


If instead you are seeing 30-75 seconds before it establishes a link, then this could be an issue Cisco is not aware of. Do the same test with just a wire and no sensor to see if the problem is unique to a hardware bypass sensor or if the same delay is also seen with just a wire.


rhermes Thu, 05/29/2008 - 07:51
User Badges:
  • Gold, 750 points or more

The delay you are seeing is the time it takes for the switch to change over from MDI to MDI/X. The interfaces on the 4260, when under power are both set to MDI (host side) and the switch has both ports set to MDI/X. When the 4260 looses power, a hardware relay kicks in connecting both ports together. The switch now has to detect a loss of signal and go throught it's auto configuration process to change one port from MDI/X to MDI.

http://en.wikipedia.org/wiki/Medium_dependent_interface

It seems that Cisco could have rolled over the pairs on their hardware bypass, but that didn't happen.

As a work around, it has been suggested in this forum to use spanning tree to fail around the sensor with a higher STP cost patch cable (thanks folks). If you go that route you'll want to play with some of the STP parmeters to reduce the failover time. Stock STP runs at about 33 seconds to switch over.

niall-wilkins Fri, 05/30/2008 - 04:32
User Badges:

Regarding the above:

Did you mean that when the sensor is powered up its interface is set MDI and the port on the switch is set to MDI as well. When the Sensor loses power the interface is set to MDI/X and then the switch has to fo through auto configuration to change the port from MDI to MDI/X to match the sensors interface?

niall-wilkins Fri, 05/30/2008 - 07:15
User Badges:

Just to add I forgot to mentioned that the IPs is plugged into a Cisco Catalyst Express 500 Series Switch. Could this be the issue?

rhermes Fri, 05/30/2008 - 09:42
User Badges:
  • Gold, 750 points or more

The sensor interfaces are always MDI when powered up. The switch interfaces are autosensing, so when the sensor has power the switch interfaces autosense and set themselves to MDI/X. When the sensor looses power (and configured for hardware bypass) a relay closes in the sensor and the two interfaces are connected together via wire (no active components). Now we have the two switch interfaces essentionally connected together by nothing more than a striaght through cable. Since MID/X doesn't communicate with MID/X, one of the switch interfaces needs to auto configure itself to MDI before traffic will pass. (Cisco COULD have made the bypass connection a roll-over, but didn't)

In looking through my notes I found that the MDI/X reconfiguration in a 2960 switch takes 3-4 seconds, so I believe that the majority of your delay is due spanning-tree portfast not enabled as marcabal mentioned.

cashqoo Mon, 06/02/2008 - 17:32
User Badges:

maybe you try to see the logs for information. You may want to look out for auto neg errors.

niall-wilkins Tue, 06/03/2008 - 03:26
User Badges:

Configuring spanning-tree portfast on the switch seems to have resolved the issue. We are down to less then a second for hardware bypass to kick in.

Thanks

Actions

This Discussion