Cisco Errors

Unanswered Question
Jul 17th, 2009

Hello,

I have a point to point network with 4 2620 routers at the hub sites and 1 3640 router at the main site. all of the sudden all of my routers are rebooting automatically with the following error when i do a sh ver:

1 - 2620: WASHINGTON uptime is 7 hours, 4 minutes

System returned to ROM by error - a SegV exception, PC 0x8079D7C8

System image file is "flash:c2600-is-mz.121-5.T10.bin"

2 - 2620: MORRIS_PLAINS uptime is 7 hours, 3 minutes

System returned to ROM by error - a SegV exception, PC 0x81017310

System image file is "flash:c2600-is-mz.121-5.T10.bin"

my other routers i cannot telnet to but they are running and i do know that the recieve the same error but with a different PC number. all are running the same ios as well.

Can anyone please help me with analyizing my problem i am having? these are old routers and i cannot get smartnet for them becuase they are out of date out of warranty and end-of life. these are installed at a nonprofit credit union so before we make the jump into purchasing new equipment they would like me to try and resolve the issue first.

attached is the output of sh tech-support for the 2 routers that i can telnet to:

I have this problem too.
0 votes
  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 4 (1 ratings)
Loading.
servantude Fri, 07/17/2009 - 07:37

SegV Exceptions are usually a software issue. Since all your routers are using the same IOS, I'm guessing that some new condition is occurring which is affecting them all. You should check for a new release of your IOS, but since you don't have smartnet, you may not be able to do that.

sgoldschmidt Fri, 07/17/2009 - 08:16

there hasn't been any changes done to the system. they've been running the same config for years with no major changes that i know of. the only changes i made recently what changing the logging to a new server but i removed this from the config and i'm still getting the error.

i tried to get a new ios but since my hw is out of service/out of warranty/end of life, cisco will not sell me smartnet for my hardware.

jimmysands73_2 Fri, 07/17/2009 - 14:56

Here is article about it found on Cisco's site w/some troubleshooting info:

SegV Exceptions

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a0080189ddb.shtml

When you say "no major changes", what changes at all have been done, if any?

This one person on this site solved it by nailing down a NAT issue...

http://www.velocityreviews.com/forums/t31084-segv-exception-pc-0x809337d0.html

Good luck

Jimmy

sgoldschmidt Sun, 07/19/2009 - 18:20

the "minor change" that i added just recently was "logging 10.43.144.95" that's it. nothting else. so i thought maybe this might have been the problem but it wasn't.

i saw both of the links you provided but none seem to help me.

i cannot access the tool, because everytime i go to access the tool i get a 403 webpage error stating that the page has moved. and does not tell me the new link or i cannot access it becuase i do not have smartnet with valid support becuase cisco won't sell me support for my routers.

i do not have a nat issue because i am not nat-ing anything through my routers. all nat-ting is done when my routers forward all network traffic to my firewall.

pompeychimes Sun, 07/19/2009 - 07:06

Sounds like a change, however minor, was made and it triggered a bug in the code.

Can you roll back the config?

Otherwise, spend the 100 bucks and get coverage on at least one of the routers.

pcameron Mon, 07/20/2009 - 06:22

Found Cisco bug "CSCdt09053 - Align in local_snmp_engine after questioning qos mib" that has similar symptoms. According to the bug details, questioning the QoS MIB several times under v2 will provoke ALIGN tracebacks that after several appearences will reload the router. A possible workaround is to use mib v1.

If you are feeling brave, you could try to see if the snmp query is the cause, for example run a snmpwalk from a NMS server like this -

snmpwalk -v2c 10.1.1.1 public .1.3.6.1.4.1.9.9.166

Since this is happening on several routers it looks like the same trigger hence a management app could be the possible cause. See if you can correlate router crashes with NMS activity.

This bug was fixed in early 12.2 mainline code. Would suggest you upgrade code on routers to the latest 12.2 mainline release. The code on these box is almost 8 years old now, lot of bugs been fixed since then.

sgoldschmidt Tue, 07/21/2009 - 08:26

Thank you, you were correct in that it was the snmp querying an cpu mib that was causing my issues due to the fact that i have an older IOS.

would you happen to know how i could see what ios i can upgrade my rouers to?

pcameron Tue, 07/21/2009 - 14:17

providing you have a Cisco.Com login, you can go the Support/Software

The latest 12.2 code for the 2600 is 12.2.46a. According to the notes, the IP Plus feature set that you are running needs 28 meg of DRAM and 16 mag of flash. You have 40 meg of DRAM. Chances are this would run OK if the routers aren't holding any large routing tables or doing any other complex functions.

Otherwise you may need to consider a memory upgrade (which is pretty cheap for the 2600's)

IP PLUS

c2600-is-mz.122-46a.bin

Release Date: 01/Aug/2007

Size: 10085.63 KB (10327680 bytes)

Minimum Memory: DRAM:48 MB Flash:16 MB

download details are here -

http://tools.cisco.com/support/downloads/go/ImageList.x?relVer=12.2.46a&mdfid=268437951&sftType=IOS+Software&optPlat=&nodecount=23&edesignator=GD&modelName=Cisco+2620+Multiservice+Platform&treeMdfId=268437899&modifmdfid=null&imname=&treeName=Routers&hybrid=Y&imst=N

Actions

This Discussion