Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
New Member

Cisco 2600 running config changed by itself - turned off routing

Hello,

I'm looking for help from the experts here to confirm some facts:

We have a 2600 (yes, still running) at our Edge interfacing the ISP. The last couple days, overnight, the router would just stop routing. All links were up, router was accessible. Restarting the router brought things back.

After not seeing anything wrong in the config, we switched to a backup 2600 last night. Again, overnight, the same thing happened. This time, I found the running config had been changed! The default route had been deleted and "no ip routing" had been set.

Can anyone confirm that IOS cannot change this config on its own? Is there any technical reason this config would have changed without user intervention? If not, we may have been hacked. Telnet is not accessible from the Internet on either FE interface, but may have been accessed via a guest network. Thanks for any insight.

1 ACCEPTED SOLUTION

Accepted Solutions
New Member

Thank you Rick. Taking a

Thank you Rick. Taking a closer look at the routers that had been affected I found that their community strings with RW were either not locked by an ACL or were locked by an ACL but the ACL was not configured.

I believe not having the SNMP community string RW locked down is the cause in our case.

 

When setting RO (readonly) or RW (Read and write) community strings make sure they are locked with an ACL. Also, make sure you have the ACL configured and permiting only the snmp server monitoring the router.

If you don't lock RO someone could read your current configuration through SNMP.

If you don't lock RW someone could read and write the configuration on the router through SNMP.

 

Example:

access-list 4 permit (IP of your SNMP server)

snmp-server community private RW 4  

 

(config)#snmp-server community private RW ?

  <1-99>       Std IP accesslist allowing access with this community string
  <1300-1999>  Expanded IP accesslist allowing access with this community string
  WORD         Access-list name
  ipv6         Specify IPv6 Named Access-List
  <cr>

20 REPLIES
Silver

Hi dclark005, Well that´s a

Hi dclark005

Well that´s a weird behavihor , next time this happen I would run a "Show version" and check the last reload reason , also enable logs and point them to a log server (it could be a simple PC running a software for example kiwi syslog server). 

 With that information we can have and idea of what´s going on . 

 

Regards ,

Hall of Fame Super Gold

Show version may provide

Show version may provide helpful information. Also if the router has learned authoritative time (especially if it is running NTP) then show run will have the last time the config was changed (assuming that the config change was after the last reboot.

 

It might also be useful to enable accounting for level 15 commands which will clearly indicate in someone has issues the config t command.

 

HTH

 

Rick

New Member

dclark005,We started having

dclark005,

We started having this same problem yesterday also.  We were running a 2621 router and switched to a 2811 and have the same problem.  Did you figure out what is happening?

Ross

New Member

Thank you Rick and rvarelac

Thank you Rick and rvarelac for the replies.

It happened again two hours ago. I have to be on site on console to gain access. I will run your suggestions and post what I find. It sounds like you are confirming what I thought - the config cannot be changed without a user changing it - IOS won't change it on me in response to an event or a trigger, correct?

 

rskov - That's crazy you're having the same problem. No luck yet. My first thought was a security breach. I changed the enable password yesterday with no luck. Like you, we tried on two separate 2600 routers. Perhaps there's a security hole in the firmware? What version of IOS are you running? I'll post mine when on-site. Also, looking for similarities - who is your ISP?

New Member

We use Enventis for our ISP.

We use Enventis for our ISP.   

The IOS on the 2811 we are using now is 12.4(2)T4.

We noticed that 2 commands were added to the running config.

no ip routing

no ip cef

The startup config is not changed.  We don't understand how the running config is being changed.  Since the startup config isn't changed a reload fixes the problem.

 

 

 

Hall of Fame Super Gold

One time I observed IOS

One time I observed IOS dynamically put an interface into the admin down state in response to a network event. Up to that moment I was certain that the only way an interface got to admin down was when a person configured it. Since then I have been extremely cautious about saying that IOS would never do something (like change a default route).

 

While I can not state with certainty that IOS did not make the changes, I am certainly thinking that it is much more likely to be a person who is doing this. I continue to think that looking for timestamps of the last config change, time of last reboot, and logging of level 15 commands would be effective steps.

 

It also occurs to me that if you have not already done it, that changing passwords on the router, especially the enable password, would be a good idea.

 

I would think that you might also want to evaluate the possibilities of remote access to the router and perhaps to put restrictions on remote access.

- you might require SSH instead of telnet since SSH will require a user name and password (and SSH is more secure than telnet anyway).

- you might configure an access list and apply it to the vty using access-class to restrict remote access to the router.

- do you permit SNMP read/write access to the router? That is another avenue through which the change could be made. Perhaps you want to restrict where SNMP will be accepted from.

 

HTH

 

Rick

New Member

Ross,That exact behavior is

Ross,

That exact behavior is what we're experiencing. Startup config isn't touched, and run config gets no ip def and no ip routing added to it. A reload or restart fixes it.

Richard,

These are good suggestions; thank you. One of the first things I tried was changing the enable password. Just now, I logged back into the tty and tried to enable, using the new password (which I had used successfully last night), and the password has reverted to the previous password! This makes me even more suspicious. Is it possible an entire ghost config is being loaded somehow, either by IOS or a person? I will most definitely check SNMP...that's a good thought.

 

Ross, have you been able to make any more progress on tracking this down?

 

My "sh ver" is below:

OSBA# sh ver
Cisco Internetwork Operating System Software
IOS (tm) C2600 Software (C2600-IPBASE-M), Version 12.3(6c), RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2004 by cisco Systems, Inc.
Compiled Tue 20-Jul-04 05:24 by kellythw
Image text-base: 0x80008098, data-base: 0x80ED06CC

ROM: System Bootstrap, Version 12.2(8r) [cmong 8r], RELEASE SOFTWARE (fc1)

OSBA uptime is 1 day, 4 hours, 21 minutes
System returned to ROM by reload
System image file is "flash:c2600-ipbase-mz.123-6c.bin"

cisco 2621XM (MPC860P) processor (revision 0x400) with 126976K/4096K bytes of memory.
Processor board ID JMX0849L0TE (4186345228)
M860 processor: part number 5, mask 2
Bridging software.
X.25 software, Version 3.0.0.
2 FastEthernet/IEEE 802.3 interface(s)
1 Serial network interface(s)
32K bytes of non-volatile configuration memory.
32768K bytes of processor board System flash (Read/Write)

Configuration register is 0x2102

 

Thanks!

 

New Member

Ross, Curious - do you have

Ross,

 

Curious - do you have "ip http server" in your config file? I just realized this is on, and upon testing, I can access the router web server from the outside world. Using admin/<<enable pw>> I can run any config command I want via the web GUI, and any changes do NOT show up in the syslog.

I'm going to disable and see if this fixes the problem.

New Member

Ross, I've done a couple

Ross,

 

I've done a couple things today to try to troubleshoot. If I can't figure it out on this next outage, I'm simply going to replace the unit, seeing how this one is at least 15 years old and EOL.

 

1. I power-cycled the router so sh ver shows a power-on event. If, after the next outage, it shows it restarted due to "reload", this is a clear indication someone on a vty reloaded the config (according to Cisco docs).

2. I enabled debug logging and a syslog server to log all events.

 

Hopefully this will shed some light on the situation. I also disabled snmp community private RW per Richard's suggestion. 

Rick, I ran out of time to setup Accounting - hopefully the logs show some info. I'm still perplexed about the enable password reverting....

 

Ross, if you find out anything else, will you post?

 

Thanks, guys.

Hall of Fame Super Gold

The comment about http server

The comment about http server/http secure server is a helpful reminder about another vehicle for remote access to the router. Along with SNMP I would either disable these, or (if you actually use them) make sure that they are doing authentication and that the passwords have been changed.

 

While there is often some value in replacing equipment that is as old as this router, I would caution you that what is happening probably has little to do with the age of the router. And if you replace this old router with a newer router that is configured the same, then whatever is allowing someone to access your old router will probably allow them to access the new router.

 

HTH

 

Rick

New Member

Switching from a 2621 to a

Switching from a 2621 to a 2811 did not solve the problem. We ended up running the command auto secure to harden the 2811 router we are using now.  I answered no to the firewall part as that seemed to slow the router down considerably.  Here is an article on the auto secure command.  The auto secure command changes all your passwords and sets up accounting for you, it also runs a bunch of other commands to disable unneeded features.

http://www.firewall.cx/cisco-technical-knowledgebase/cisco-routers/865-how-to-secure-your-cisco-router-using-cisco-autosecure-feature.html

Our router has been up for almost 24 hours now with out any outages and/or config changes.

New Member

Thanks, Ross. I will attempt

Thanks, Ross. I will attempt to run this as well. After disabling the http server interface, I haven't seen any more issues either.

New Member

http and https were both

http and https were both disabled on our router.

New Member

  Hi, pls check the console

 

 Hi,

 pls check the console and telnet password that you have given to this 2600 series Router, are you sure that passwords are complex ?

 Also pls check which version of IOS are you using, make sure that the router has new version of IOS.

 IOS is also a software for network device and if it is not updated, then their are chances that any hacker can hack the device.

 Thanks

New Member

We have ran into this same

We have ran into this same exact issue. 

No change to the running-config since it was last saved to the startup-config. 

No ip routing & no ip cef are in the running config but are not in the startup-config.

Ran into several different routers with different IOS versions that had the same issue occur. 

Several different 1721's  for different customers running different IOS versions.

Even ran into it with a Cisco 2951 running (C2951-UNI​VERSALK9-M), Version 15.1(3)T1 

So it doesn't seem to be specific to version or type of router. 

Not an issue with the ip http server and/or ip http secure server getting left on in the config. The routers we ran into the issue with had them disabled. 

Because the startup-config was never touched when the router is reloaded it pulls the startup-config and works like it was before those two commands showed up in the running config.

Could it possibly be an SNMP type hack?

Hall of Fame Super Gold

As I suggested in a previous

As I suggested in a previous response in this thread, yes SNMP is certainly one of the ways that changes like this could be made.

 

HTH

 

Rick

New Member

Thank you Rick. Taking a

Thank you Rick. Taking a closer look at the routers that had been affected I found that their community strings with RW were either not locked by an ACL or were locked by an ACL but the ACL was not configured.

I believe not having the SNMP community string RW locked down is the cause in our case.

 

When setting RO (readonly) or RW (Read and write) community strings make sure they are locked with an ACL. Also, make sure you have the ACL configured and permiting only the snmp server monitoring the router.

If you don't lock RO someone could read your current configuration through SNMP.

If you don't lock RW someone could read and write the configuration on the router through SNMP.

 

Example:

access-list 4 permit (IP of your SNMP server)

snmp-server community private RW 4  

 

(config)#snmp-server community private RW ?

  <1-99>       Std IP accesslist allowing access with this community string
  <1300-1999>  Expanded IP accesslist allowing access with this community string
  WORD         Access-list name
  ipv6         Specify IPv6 Named Access-List
  <cr>

Hi everyone!we are also

Hi everyone!

we are also experiencing exactly the same issue at our office, can cisco support gusy find out a way to get rid of this?  because, i'm pretty sure this is not an attack (after reading all these posts from peoples trying to solve it) since it happens excatly the same way on different routers with different configurations, also if was trying to take down a network i would not just set "no ip routing on the running-config" at least i would write it, or do something even worse, like wipe out the whole config. I hope you'll work it out.

Thanks!

Cisco Employee

Timmy has the right answer. 

Timmy has the right answer.  TAC has confirmed that this behavior is seen with routers configured with easy to guess SNMP strings (like "private" and "public) and RW access. 

 

Please see http://www.cisco.com/c/en/us/support/docs/ip/access-lists/13608-21.html#anc50 for tips on how to harden SNMP.

 

The ipForwarding MIB allows users to disable routing and forwarding with an snmpset.

New Member

In our case, in addition to

In our case, in addition to http Server being enabled, we also had SNMP private RW wide open. In the process to resolution this was removed. We haven't had a problem since.

620
Views
15
Helpful
20
Replies
CreatePlease to create content