Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements
Webcast-Catalyst9k
New Member

Traffic flapping in regular intervals

traffic flapping.pngWe are telecom provider. Edge router is Cisco 7604 with SIP 200 cards

Core Router is cisco 12406. We have a MPLS network. router were connected using

STM1 link. Link is terminated by POS interface of those routers. the issue is that every day traffic flapp in regular intervals (15minuts) from 8.35am to 5pm.

please advice is that virus issue or link issue or router issue.

Anyway no errors in links and router interfaces.

In every flapping time, few packets drops and delays go over 1000ms. but normal delay is 2ms when pingging between routers.

Images is attached .

Please help us to resolve this

Everyone's tags (3)
18 REPLIES
Hall of Fame Super Silver

Re: Traffic flapping in regular intervals

Hello Tharaka,

first of all you have to find out if what you see in SNMP derived graphs is real or not.

if routing protocol are stable ( show ip ospf neig det or show isis neigh detail provide the uptime) the problem may not be real.

What kind of counters are used to build the graph?

the standard 32 bit counters ifincoctets ifoutoctets are not able to deal with speeds like STM-1 or greater.

At full speed an STM-1 causes wrapping of the standard byte counters in less then 4 minutes. (229 seconds)

(2^32 is roughly 4 billions = 4294967296    150 Mbps is 18750000 Bps byte per second  divide the first by the second and you get 229 seconds.)

The problem is called wrapping of counters.every N minutes the counter goes in overflow and the calculation of speed rate is made by making the difference between last two reading of octets variable. So every N minutes depending on average bit rate the counter make an hop that can appear in the graph as a minimum.

A solution to this problem is the use of 64 bit counters that contain HC in the prefix they are called

ifHCinoctets and ifHCoutoctets these are able to represent 2^64 range of values.

You should verify if the C7600 supports the HC version of the counters, if so the solution is to poll these other MIB variables.

C12000 should support the HC counters it was supported on first version of C12000 in 1999.

Hope to help

Giuseppe

New Member

Traffic flapping in regular intervals

Thanks for your comments.

But we check SNMP using MRTG and PRTG.further we notice that every flap, ping delays go beyond 1000ms with few packet drops. otherwice it is 2ms. IS-IS protocol logs , hols time expired and new ajacency created/!!!!.

Our customers also expericancing this traffic pattern and actucally they are suffering. So traffic delay is expericncing by us.

please comment.

Hall of Fame Super Silver

Traffic flapping in regular intervals

Hello Tharaka,

the problem is real if ISIS flaps.

I agree I had missed you had mentioned the increased delay

I would say that the problem may be  probably in the SDH world and it is probably related to clocking or the way the VC are carried in the SDH hierarchy.

Are you using default timers for ISIS including for max-age?

Hope to help

Giuseppe

New Member

Traffic flapping in regular intervals

Hello, Thanks for support.

We increased halding time upto 120s. Now IS IS log is not over there. But still issue is exist.

How change cloking in SDH. is it in router or have to change those with STM receiver and all.

Please respond.

Thanks

Hall of Fame Super Silver

Re: Traffic flapping in regular intervals

Hello Tharaka,

may you post ( if supported)  (where posx/y/z is your pos interface)

a

show controller pos x/y/z

+

show interface pos  x/y/z

to see if there are errors at SDH level like severly errored seconds (SES) and so on

Hope to help

Giuseppe

New Member

Re: Traffic flapping in regular intervals

Hello,

Please refer to attachment.both sh cont is attached.

Thanks you very much

Cisco 12406 core Router

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

POS2/2/2

SECTION

  LOF = 0          LOS    = 0                            BIP(B1) = 1096

LINE

  AIS = 2425       RDI    = 57         FEBE = 4296782    BIP(B2) = 37334688

PATH

  AIS = 2376       RDI    = 2323       FEBE = 1814026    BIP(B3) = 11892092

  LOP = 2530       NEWPTR = 0          PSE  = 0          NSE     = 0

Active Defects: None

Active Alarms:  None

Alarm reporting enabled for: SF SLOS SLOF B1-TCA B2-TCA PLOP B3-TCA

Framing: SDH

APS

  COAPS = 6677       PSBF = 3554     

  State: PSBF_state = False

  ais_shut = TRUE

  Rx(K1/K2): 00/00  S1S0 = 02, C2 = 16

  Remote aps status (none); Reflected local aps status (none)

CLOCK RECOVERY

  RDOOL = 0        

  State: RDOOL_state = False

PATH TRACE BUFFER : STABLE

  Remote hostname : GPAS-MEDG-004          

  Remote interface: POS3/2/0     

  Remote IP addr  : 10.13.1.2      

  Remote Rx(K1/K2): 00/00  Tx(K1/K2): 00/00

BER thresholds:  SF = 10e-3  SD = 10e-6

TCA thresholds:  B1 = 10e-6  B2 = 10e-6  B3 = 10e-6

Optical Power Monitoring

Module temperature    =  26.968 C

Transmit Bias Current =  3548 uAmp

Tx Output Power       =  -10 dBm  (750 in units of 0.1 uW)

Rx Output Power       =  -11 dBm (711 in units of 0.1 uW)

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Edge router -cisco 7604

Hall of Fame Super Silver

Traffic flapping in regular intervals

Hello Tharaka,

there are errors on the C12000 side.

for BIP errors explanation have a look at

http://www.cisco.com/en/US/tech/tk482/tk607/technologies_tech_note09186a0080094a79.shtml

about clocking see

http://www.cisco.com/en/US/tech/tk482/tk607/technologies_tech_note09186a0080094bb9.shtml

if the two routers are connected back to back one of the two devices has to act as DCE providing clock

one side needs the command

clock source internal

the other side should be in clock source line

Hope to help

Giuseppe

New Member

Traffic flapping in regular intervals

Hello,

Can we change one side in to INTERNAL when the router is sending traffic and in productions state.

Are there any risk my friend.

Prageeth

Hall of Fame Super Silver

Traffic flapping in regular intervals

Hello Tharaka,

can you confirm that the two routers are connected in back to back without any SDH device in the middle?

if so one device needs to be configured with clock internal. The impact should be limited you can ask for a maintenance time window for safety but it should be less of what happens every 15 minutes during daytime.

The interface may flap up/down this should be the maximum impact but it shouldn't.

Hope to help

Giuseppe

New Member

Traffic flapping in regular intervals

Dear Giuseppe,

Thanks for your help.

No SDH device in middle.

in this case do we need to setup Interal - Line mode.

Thanks

Prageeth

Hall of Fame Super Silver

Re: Traffic flapping in regular intervals

Hello Prageeth,

one side must be with clock internal, according to documentation both is allowed.

You need to avoid to have both devices configured with clock source line because it causes clocking problems in back to back scenario.

The side with clock internal will become the clock reference for the link this is the result.

Change one side to internal in a maintenance window and wait some time to see if your troubles are fixed.

Hope to help

Giuseppe

New Member

Re: Traffic flapping in regular intervals

Hello Giuseppe,

We have done the changes as you describe. But still flapping is there. Does it need shutdown and no shutdown to apply the clock setting.

Waiting your feedback.

Thanks

Prageeth

Hall of Fame Super Silver

Traffic flapping in regular intervals

Hello Prageeth,

please post the configuration of the POS interfaces both sides and

show interface pos

show controller pos

There may be something else to tune like payload scrambling

Hope to help

Giuseppe

New Member

Re: Traffic flapping in regular intervals

Hello Giuseppe,

hearwith attached sh int pos .. please see it!!

My friend, 1st i change clock to Internal at Cisco 7604 side. issue is exist. but no any alarms generated.

then 2nd  i set to original and change clock to Internal on Cisco 12406 side. then issue is exist and many alarms were generated (transmmission alarms). and traffic goes down for a 5 minute time.So we set all to original state.

But 1st case we didnt reset port (shutdown and up).

Thanks for your help

Prageeth

Hall of Fame Super Silver

Traffic flapping in regular intervals

Hello Prageeth,

the original state is a wrong configuration as you can read in the document about clocking on POS interfaces.

I would suggest to put clock internal on C7604 side and to leave it a few days.

In order to understand if the C12000 side is faulty or is a fiber problem you need a maintenance time window in which you perform loopback tests ( this is totally intrusive as  stops user traffic) by configuring loopback mode on C7604 side towards line.

The objective of the test should be to verify if the state of interface on C12000 side reach the up/up looped or up/down looped state.

Hope to help

Giuseppe

New Member

Traffic flapping in regular intervals

Dear Friend,

Ok. we will set it like that and let you know the status soon.

Thanks

Prageeth

New Member

Traffic flapping in regular intervals

Dear Friend,

Still flapping is occaring.this is the ping over this link. regular interval traffic delay and drop can be shows every 15 minut intervals.

Normal delay time = 2s

delay time When issue is happening = more than 1000ms

Reply from 10.13.0.9: bytes=32 time=3ms TTL=251

Reply from 10.13.0.9: bytes=32 time=2ms TTL=251

Reply from 10.13.0.9: bytes=32 time=2ms TTL=251

Reply from 10.13.0.9: bytes=32 time=2ms TTL=251

Reply from 10.13.0.9: bytes=32 time=3ms TTL=251

Reply from 10.13.0.9: bytes=32 time=2ms TTL=251

Reply from 10.13.0.9: bytes=32 time=2ms TTL=251

Reply from 10.13.0.9: bytes=32 time=3ms TTL=251

Reply from 10.13.0.9: bytes=32 time=211ms TTL=251

Reply from 10.13.0.9: bytes=32 time=135ms TTL=251

Reply from 10.13.0.9: bytes=32 time=435ms TTL=251

Reply from 10.13.0.9: bytes=32 time=382ms TTL=251

Reply from 10.13.0.9: bytes=32 time=108ms TTL=251

Reply from 10.13.0.9: bytes=32 time=286ms TTL=251

Reply from 10.13.0.9: bytes=32 time=282ms TTL=251

Reply from 10.13.0.9: bytes=32 time=181ms TTL=251

Reply from 10.13.0.9: bytes=32 time=87ms TTL=251

Reply from 10.13.0.9: bytes=32 time=189ms TTL=251

Reply from 10.13.0.9: bytes=32 time=146ms TTL=251

Reply from 10.13.0.9: bytes=32 time=57ms TTL=251

Reply from 10.13.0.9: bytes=32 time=366ms TTL=252

Reply from 10.13.0.9: bytes=32 time=414ms TTL=251

Reply from 10.13.0.9: bytes=32 time=807ms TTL=251

Reply from 10.13.0.9: bytes=32 time=247ms TTL=251

Reply from 10.13.0.9: bytes=32 time=74ms TTL=251

Reply from 10.13.0.9: bytes=32 time=19ms TTL=251

Reply from 10.13.0.9: bytes=32 time=420ms TTL=251

Reply from 10.13.0.9: bytes=32 time=378ms TTL=251

Reply from 10.13.0.9: bytes=32 time=289ms TTL=251

Reply from 10.13.0.9: bytes=32 time=262ms TTL=251

Reply from 10.13.0.9: bytes=32 time=1005ms TTL=251

Reply from 10.13.0.9: bytes=32 time=3ms TTL=251

Request timed out.

Reply from 10.13.0.9: bytes=32 time=2ms TTL=252

Reply from 10.13.0.9: bytes=32 time=2ms TTL=251

Reply from 10.13.0.9: bytes=32 time=2ms TTL=251

Reply from 10.13.0.9: bytes=32 time=2ms TTL=251

Reply from 10.13.0.9: bytes=32 time=2ms TTL=251

Reply from 10.13.0.9: bytes=32 time=2ms TTL=251

Reply from 10.13.0.9: bytes=32 time=2ms TTL=251

Please advice

Prageeth

Hall of Fame Super Silver

Traffic flapping in regular intervals

Hello Prageth,

if you have a free STM-1 port on the C12000 side I would try to move the link to this different port during a maintenance time window.

You can configure the port as the one in use without causing errors before of the maintenance time window.

Hope to help

Giuseppe

1123
Views
0
Helpful
18
Replies
CreatePlease to create content