Intermittent connectivity issues with link to Datacenter

Unanswered Question
Nov 14th, 2007
User Badges:

I'm looking for suggestions on troubleshooting a link to our datacenter that has intermittent connectivity and speed issues. The link to the datacenter is approximately 30 miles in distance so even on a good day there is 3-4ms of latency. At this point I'm not too familiar with the setup as I've inherited this from a previous employee without much documentation. I know at this point the link is connected between a 2970 and a 3750. Both interfaces are set as auto and are showing fd 1000mbs. The link is a layer 2 100 mb and somewhere in between goes through our providers switchgear. I only figured this out due to some vlan issues. I'm not sure if this is an issue but the 2970 shows 12056 big buffer misses for today. This is interesting as hardly any of the other switches show any buffer misses. The site has approx. 150 machines accessing the datacenter over the 100mb line ranging from email, terminal services, web, etc... I would appreciate if anyone could help in troubleshooting this issue. I can post configs or provide any other info if needed. I appreciate the help.

  • 1
  • 2
  • 3
  • 4
  • 5
Overall Rating: 0 (0 ratings)
Loading.
rajatsetia Wed, 11/14/2007 - 08:28
User Badges:
  • Bronze, 100 points or more

hi


i have not faced this issue , so cannt tell anything from my expereince of troubleshooting but you can read the below doc for better understanding of the issue and also has some troubleshooting guidelines.


http://www.cisco.com/warp/public/650/41.html


HTH


rgds

rajat

Kevin Dorrell Wed, 11/14/2007 - 08:31
User Badges:
  • Green, 3000 points or more

Just a few thoughts: your 1 Gbit links are to the provider, and not directly to each other. The provider is providing 100 Mbps, so there is a bottleneck. Do you have any traffic shaping going into the provider to stop it exceeding 100 Mbps? Otherwise the provider may be dropping packets.


Does the provider give you any flow control at the interface? If so, do you have any flow-control stats from the links?


I'm not sure what the big buffer misses are about, but it sounds like a memory problem on the 2970. Does the show mem give you any clues? How long has the 2970 been up?


That's just a few starters.


Kevin Dorrell

Luxembourg


jstewart33 Wed, 11/14/2007 - 09:34
User Badges:

Yes 1 Gbit links are to provider. I just got off the phone asking for configuration best practices, which they had none. They did say configure for 1000 full on both sides, but they limit in the middle to a 100. There is no flow-control running on the two interfaces. Here is the show mem from the 2970


Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)

Processor EAFC3C 106234820 12102616 94132204 92500364 90139832

I/O 7400000 12574720 7055280 5519440 5266384 5510552


The last boot was August 21,2007 on the 2970 also. Thanks for the ideas!

glen.grant Wed, 11/14/2007 - 09:44
User Badges:
  • Purple, 4500 points or more

when you are having a problem take a look at the links with a show int command and see how much traffic is actaully passing at that time and see if it is bumping up against that 100 meg ceiling . If not then your problem is elsewhere . Check the logs for any anomalies.

rajatsetia Wed, 11/14/2007 - 15:16
User Badges:
  • Bronze, 100 points or more

hi glen


In the first post jstewart has mentioned abt some buffer issue , i never faced this issue and cudnt really understand the impact of this buffer miss problem.


but i have posted the link which address this issue and it doesnt talk abt any problem at physical or data layer but does have mention of tweaking the buffer size.


but you and kevin have not touched this part, i was wandering if this buffer miss part does have any link to the intermittent connectivtiy problem.


need your expert opinion here..

glen.grant Wed, 11/14/2007 - 19:22
User Badges:
  • Purple, 4500 points or more

Usually these days you shouldn't have to mess with adjusting any buffer sizes . If its overrunnin the buffer then maybe it is handling too much traffic .

rajatsetia Thu, 11/15/2007 - 02:37
User Badges:
  • Bronze, 100 points or more

hmm, understood....


point noted,Thanks


rgds

rajat


jstewart33 Thu, 11/15/2007 - 10:00
User Badges:

Just for kicks here is the 2970 show buffers command.


#show buffers

Buffer elements:

481 in free list (500 max allowed)

173158089 hits, 0 misses, 0 created


Public buffer pools:

Small buffers, 104 bytes (total 34, permanent 25, peak 118 @ 7w0d):

33 in free list (20 min, 60 max allowed)

80612124 hits, 3820 misses, 11441 trims, 11450 created

0 failures (0 no memory)

Middle buffers, 600 bytes (total 15, permanent 15, peak 42 @ 7w0d):

13 in free list (10 min, 30 max allowed)

706662 hits, 57 misses, 171 trims, 171 created

0 failures (0 no memory)

Big buffers, 1536 bytes (total 18, permanent 5, peak 29 @ 7w0d):

4 in free list (5 min, 10 max allowed)

23149403 hits, 2013109 misses, 6070076 trims, 6070089 created

0 failures (0 no memory)

VeryBig buffers, 4520 bytes (total 11, permanent 0, peak 13 @ 7w0d):

1 in free list (0 min, 10 max allowed)

199246 hits, 14 misses, 782 trims, 793 created

0 failures (0 no memory)

Large buffers, 5024 bytes (total 1, permanent 0, peak 2 @ 7w0d):

1 in free list (0 min, 5 max allowed)

12999 hits, 1 misses, 766 trims, 767 created

0 failures (0 no memory)

Huge buffers, 18024 bytes (total 1, permanent 0, peak 14 @ 7w0d):

1 in free list (0 min, 2 max allowed)

344211 hits, 80432 misses, 161628 trims, 161629 created

0 failures (0 no memory)


Interface buffer pools:

Supervisor MIC Fallback pool buffers, 2040 bytes (total 904, permanent 904):

899 in free list (0 min, 904 max allowed)

2401242 hits, 0 misses

supervisor_cpuq_0_pool buffers, 2040 bytes (total 1200, permanent 1200):

700 in free list (0 min, 1200 max allowed)

20822801 hits, 0 misses

supervisor_cpuq_1_pool buffers, 2040 bytes (total 128, permanent 128):

1 in free list (0 min, 128 max allowed)

3810148 hits, 31166 fallbacks

supervisor_cpuq_4_pool buffers, 2040 bytes (total 128, permanent 128):

1 in free list (0 min, 128 max allowed)

1496351 hits, 11743 fallbacks

supervisor_cpuq_15_pool buffers, 2040 bytes (total 4, permanent 4):

0 in free list (0 min, 4 max allowed)

147986465 hits, 147986461 misses

supervisor_cpuq_10_pool buffers, 2040 bytes (total 64, permanent 64):

1 in free list (0 min, 64 max allowed)

472320 hits, 11233 fallbacks

supervisor_cpuq_12_pool buffers, 2040 bytes (total 96, permanent 96):

0 in free list (0 min, 96 max allowed)

96 hits, 0 misses

supervisor_cpuq_6_pool buffers, 2040 bytes (total 128, permanent 128):

0 in free list (0 min, 128 max allowed)

2318 hits, 2189 misses

supervisor_cpuq_3_pool buffers, 2040 bytes (total 128, permanent 128):

2 in free list (0 min, 128 max allowed)

24138294 hits, 2347048 fallbacks

supervisor_cpuq_7_pool buffers, 2040 bytes (total 192, permanent 192):

63 in free list (0 min, 192 max allowed)

3997377 hits, 0 misses

supervisor_cpuq_8_pool buffers, 2040 bytes (total 64, permanent 64):

0 in free list (0 min, 64 max allowed)

59592906 hits, 58464492 misses


I find it real interesting that the 3750 shows 0 misses. Does creating buffers cause performance issues? If so this may be a definite issue in this case. Thanks!

Actions

This Discussion