This article is the first in what we hope will be a series of interesting problems where we solicit customer input to solve elusive issues. The first 10 customers who can contribute a root cause explanation will receive a prize as a token of our gratitude.
A few years back, we discovered a bug in the async drivers used in Cisco IOS.
In a nutshell, when one of the modem control or flow control input leads changed state, the TTY line would be put on an internal queue for later processing, and this could happen multiple times. If it happened rapidly enough, the TTY could use up all of the buffer elements allocated in the router - resulting in a SYS-3-NOELEMENT error message, among other things.
Eventually, we were able to diagnose, reproduce, and fix the software in the several different async drivers (one for each type of async controller) used on the multitude of IOS platforms. The most common bug fix was CSCsx68596, but the bug id depends on the exact hardware in use.
Now, this is an interesting problem because there really isn't a valid reason for the input leads to change state rapidly enough to exhaust all of the buffer elements in a system. There are usually hundreds available, and the background process that processes TTY events runs once a second. In order to run out of elements, the input leads have to change state very rapidly, which should be quite rare.
What's interesting is we have seen a surprising number of cases exhibiting this bug, but rarely have customers wanted to track down the connected device causing the problem. In many cases, people refer to a "bad cable", but it isn't obvious what sort of cable could cause this.
We have a few known causes. In some cases, an apparently defectiveUSB-> async adapter has been isolated. In others, a long async cable, left unconnected to anything, could cause the issue, clearly acting as an antenna. Finally, we know of one Cisco device which, left in rommon, would behave in such a way as to cause the element exhaustion on a Cisco console server.
Have you experienced this problem? If so, send an email to email@example.com and let us know what the connected device was in your case. We'll publish a summary of the results.