Weird delays when BPDU's show up on an access port

[Mike Elliott]

I am troubleshooting a client's network and I am noticing something very strange.

I am running Wireshark on a desktop that is experiencing a delay in an application.  When I view the wireshark logs, I've noticed an interesting pattern.

Every time a BPDU comes down the link, there is anywhere from 2 to 10 seconds delay in application data on the port.  EIGRP hello's aren't affected by this but all other traffic is.  If multiple BPDU's arrive on the link within 2 seconds of each other, the delay increases significantly.  This causes our application server to retransmit packets.  I have a 10% retransmit rate from the app server.

The network behaviour is VERY consistent, however the end user experience isn't always consistent.

I wouldn't have noticed it but when I did some IO graphs of the data.  I noticed some regular flat lines in the display of packets on the graph.  These flat lines occur even in the middle of a stream from our app server to the offending workstation(s).  When I check to see what kind of packet was at the beginning of every flat line, it was at least one BPDU.  None of the BPDU's have any topology changes in them. 

I also sniffed at another location they have, across a wan link (DS3).  This network does not exhibit the same delay.  Packets do not get delayed at all.  The app server is on the other side of the DS3 from the offending workstations.  Utilization on the link does not seem to be an issue (3% avg utilization over the period in question).  I have checked MTU etc across the link and it seems fine.

I do not have detailed topology information, nor do I have switch/router configs of this network.  I do know that they are running all Cisco gear and the BPDU's are all from a Cisco Switch. I am basing all my findings on the sniffer traces that I've been supplied.  I have the ability to sniff from the server, the offending workstation and a spanned port of the workstation.

So my questions are these:

1) Is this expected behaviour?

2) If it's not expected, what are the likely causes (I am suspecting inefficient vlan design).  I understand it will be sheer speculation due to the lack of configs

3) What questions should I ask the client regarding their network to help further isolate the issue?

Thanks

Mike

[lgijssel]

What type of spanning tree are you running? (Check BPDU type)

You are sniffing bpdu's on an access port?

They should arrive at a regular interval, equal to hello_time.

You may see two bpdu's on a port, one from the data vlan and eventually another from the voice vlan.

Unless the port is in trunk mode.... which it shouldn't.

You should be able to distinguish between bpdu's from different vlans by checking their priority.

If you are receiving them at irregular intervals, they may even be from different root bridges.

The root bridge is also transmitted in each bpdu. This would definitely be an indication for a crappy stp topology.

Overview of 802.1d bpdu format:

http://www.cisco.com/en/US/tech/tk870/tk136/tk885/technologies_tech_note09186a0080093cc6.shtml

regards,

Leo

[Mike Elliott]

Yes the BPDU's are visible on the access port

The BPDU's come down at regular intervals that are 2 seconds apart. 

I do see multiple BPDU's from the same switch grouped together from time to time.

I do not know if the port is in trunk mode.  I have asked the client to check the configuration of the port and get back to me.

It is 802.1d BPDUs

I've attached a graph that shows the co-relation between BPDU's (in black) with retransmissions (RED).  They don't "ALWAYS" fall on the BPDU packets, but it does happen very often.  What I am not sure of is if the BPDU's are being delayed, a symptom of another issue, or if the BPDU's are causing the delay.

Just an FYI, this is happening to multiple desktops, just not one.  One of the applications in the trace receives about 40Kb every 5 seconds from the server.  Very interesting problem.'

The BPDU looks like this (i've sanitized the MAC):

22499          2011-10-27 12:42:37.263486          Cisco_80:52:99          Spanning-tree-(for-bridges)_00          STP          60          Conf. Root = 32768/906/00:12:00:XX:XX:XX  Cost = 0  Port = 0x804a                    1483.015236

[lgijssel · Accepted solution]

Nope, it looks like you must choose a different approach, this is not your problem.

The bpdu's are arriving at regular intervals, the graph shows this clearly.

Besides, the switch you are on is the root switch (cost=0)

At least I cannot deduce any stp problems from this.

regards,

Leo

btw: sanitized mac's: after sanitized private ip's this is the best I've seen around here.

[Mike Elliott]

RE: sanitized mac - highly secure network.  Surprised they gave me sniffer traces.

What do you think about the retransmissions happening at the same time the BPDU comes down.  Also there is always at least a 2sec pause in packets when a BPDU comes down.

[lgijssel]

But a bpdu is seen every 2s. If what you state is consequently true, in effect there would be no other traffic at all.

This is obviously not the case.

STP bpdu's are the most frequent type of background traffic.

As such, their presence is easily linked to other events although there is no real relation.

More detailed info will be needed to analyze your problem. 

Leo

[cadet alain]

Hi Mike,

receiving  BPDUs every 2 seconds is normal behaviour with PVST+  implementation as it is the default hello interval on the root bridge by default.if you see multiple BPDUs from same switch it is because there are multiple VLANS and the switch is the root bridge for these VLANs.

It shouldn't impact TCP communication like you show in your graph.

Regards.

Alain

[Mike Elliott]

I am not sure if it's impacting TCP, or a result of some other delay.  Confusing to say the least.

[Mike Elliott]

Ok to demonstrate what I am trying to say here... Please view the attached image.

Packet 20712 is part of the stream of packets from my app server

Packets 20713 and 20714 are BPDUs spaced almost 2 seconds apart (normal)

Packet 20715 is the next packet in my stream

The delta between 20712 and 20715 is over 3 seconds.  No packets arrive in the 2 seconds between BPDU's. Meanwhile my app is expecting packets from the server, they have been sent from the server.  And the expected packet arrives 250ms after the 2nd BPDU.

This pattern is repeated throughout the traces.

Now this could be a poorly performing DS3 link, AND I have not gotten a good trace from end to end yet.  I am waiting on this info from the client.  I just find it odd that every time a BPDU comes down the link, my application has to wait for data, and for a number of these delays (the longer ones) retransmissions/lost segs occur.  I am not speculating that the BPDUs are causing the issue, I just find it interesting.

[Mike Elliott]

Damn, I forgot to sanitize the date!

[Kishore Chennupati]

hahha..oh.. I forgot to sanitize my name

[Mike Elliott]

Ok after looking at a couple of traces it is VERY clear the BPDU's are neither the cause, nor a symptom of the problem.  The delay is occurring elsewhere.  I will get the client to sniff both sides of the DS3 and check the timing.  It looks like, from the traces I've been looking at, that there is a burst of data, then a pause, then a burst of data, then a pause.

Thanks for your help guys...

Mike