Re: Unintended traffic seen on SPANed port

jchrisos · ‎11-27-2006

I set up a SPAN/monitor source and destination port. I have a server attached to the source port, and a sniffer attached to the destination port.

When I run my sniffer, I expect to see traffic to or from the server that is on the source port, in addition to broadcast traffic. However, I am also seeing traffic to and from other IP addresses. How can this be if I only have a single server with one IP address on the source port? This is really throwing off my intrusion sensor because it's seeing parts of conversations from servers that are not on this port. I confirmed with a sniffer. Please help!

Source port:

interface GigabitEthernet8/0/36

description blah blah blah

switchport access vlan 17

spanning-tree portfast

Destination port:

interface GigabitEthernet8/0/24

description blah blah blah

SPAN config:

monitor session 2 source interface Ga8/0/36

monitor session 2 destination interface Ga8/0/24

P.S. I tried adding the dest port to the vlan too, but no dice :-(

Richard Burts · ‎11-27-2006

Jim

I do not believe that there is enough information here for us to be able to find the source of your problem. Can you give us specifics of what platform you have configured this on? what version of code is running? can you post the output of show port 8/0/36, show port 8/0/24 and of show vlan 17? Also I note that you are configuring session 2. Is there a session 1 and if so how is it configured? If we had that we might be in better position to give you good advice.

HTH

Rick

HTH

Rick

jchrisos · ‎11-27-2006

Sorry:

3750 switches configured in a stack. Source and dest ports are on the same switch (10/100/1000 swtich). There is one 10/100/1000 switch, the rest are 10/100 switches. let me know what else you need.

switch#sh ver

Cisco Internetwork Operating System Software

IOS (tm) C3750 Software (C3750-I9-M), Version 12.2(20)SE4, RELEASE SOFTWARE (fc1)

Compiled Sun 09-Jan-05 00:09 by antonino

Image text-base: 0x00003000, data-base: 0x0099748C

ROM: Bootstrap program is C3750 boot loader

BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(25r)SE1, RELEASE SOFTWARE (fc)

switch uptime is 1 year, 5 weeks, 3 days, 22 hours, 2 minutes

System returned to ROM by power-on

System image file is "flash:c3750-i9-mz.122-20.SE4.bin"

cisco WS-C3750G-48TS (PowerPC405) processor (revision C0) with 118784K/12280K bytes of memory.

Processor board ID FOC0935U1XS

Last reset from power-on

8 Virtual Ethernet/IEEE 802.3 interface(s)

336 FastEthernet/IEEE 802.3 interface(s)

80 Gigabit Ethernet/IEEE 802.3 interface(s)

The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.

Base ethernet MAC Address : 00:15:62:xx:xx:xx

Motherboard assembly number : 73-9366-08

Power supply part number : 341-0107-01

Motherboard serial number : xxxx

Power supply serial number : xxxx

Model revision number : C0

Motherboard revision number : A0

Model number : WS-C3750G-48TS-S

System serial number : xxxx

SFP Module assembly part number : 73-7757-03

SFP Module revision Number : A0

SFP Module serial number : xxxx

Top Assembly Part Number : 800-25409-02

Top Assembly Revision Number : A0

Version ID : 02

CLEI Code Number : CNMWU00ARB

Hardware Board Revision Number : 0x05

Switch Ports Model SW Version SW Image

------ ----- ----- ---------- ----------

1 52 WS-C3750-48TS 12.2(20)SE4 C3750-I9-M

2 52 WS-C3750-48TS 12.2(20)SE4 C3750-I9-M

3 52 WS-C3750-48TS 12.2(20)SE4 C3750-I9-M

4 52 WS-C3750-48TS 12.2(20)SE4 C3750-I9-M

5 52 WS-C3750-48TS 12.2(20)SE4 C3750-I9-M

6 52 WS-C3750-48TS 12.2(20)SE4 C3750-I9-M

7 52 WS-C3750-48TS 12.2(20)SE4 C3750-I9-M

* 8 52 WS-C3750G-48TS 12.2(20)SE4 C3750-I9-M

Vlan17 is up, line protocol is up

Hardware is EtherSVI, address is 0015.624d.7e41 (bia 0015.624d.7e41)

Internet address is 172.17.1.5/16

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output 00:00:00, output hang never

Last clearing of "show interface" counters never

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 4000 bits/sec, 8 packets/sec

5 minute output rate 5000 bits/sec, 5 packets/sec

27766719 packets input, 4009704270 bytes, 0 no buffer

Received 0 broadcasts (4055984 IP multicast)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

2334939 packets output, 200024168 bytes, 0 underruns

0 output errors, 0 interface resets

0 output buffer failures, 0 output buffers swapped out

Couldnt fit everything, please see post below:

jchrisos · ‎11-27-2006

GigabitEthernet8/0/24 is up, line protocol is down (monitoring)

Hardware is Gigabit Ethernet, address is 0015.62ss.ssss (bia 0015.62ss.ssss)

Description: ssss

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input never, output 00:01:52, output hang never

Last clearing of "show interface" counters never

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 178000 bits/sec, 44 packets/sec

0 packets input, 0 bytes, 0 no buffer

Received 0 broadcasts (0 multicast)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 0 multicast, 0 pause input

0 input packets with dribble condition detected

31919794 packets output, 2342117797 bytes, 0 underruns

0 output errors, 0 collisions, 2 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 PAUSE output

0 output buffer failures, 0 output buffers swapped out

GigabitEthernet8/0/36 is up, line protocol is up (connected)

Hardware is Gigabit Ethernet, address is 0015.62ss.ssss (bia 0015.62ss.ssss)

Description: ssss

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input never, output 00:00:01, output hang never

Last clearing of "show interface" counters never

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 561000 bits/sec, 67 packets/sec

5 minute output rate 99000 bits/sec, 49 packets/sec

1013091294 packets input, 2833088873 bytes, 0 no buffer

Received 137839 broadcasts (0 multicast)

0 runts, 0 giants, 0 throttles

1 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 0 multicast, 0 pause input

0 input packets with dribble condition detected

589915090 packets output, 2305917226 bytes, 0 underruns

0 output errors, 0 collisions, 2 interface resets

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 PAUSE output

0 output buffer failures, 0 output buffers swapped out

mike.bradley · ‎11-27-2006

I believe that if the destination mac is not in the cam table of the switch that the unicast traffic is broadcast until the switch determines the mac and updates its cam table. For more info search the Cisco site for "unicast flooding".

If you review your packets traces you probably only see the start of unicast conversations.

If this causes you problems or concerns try looking into making the server port a private vlan port. I don't have much experience with pvlan ports so take care that it meets your requirements.

Hope that helps...

jchrisos · ‎11-27-2006

Ohh, that seems to make sense. Looking into it now. In the meantime, thanks!

scottmac · ‎11-27-2006

Just a slight correction:

The frames are not "broadcast" they are flooded.

Broadcast implies a MAC destination of all ones (ff.ff.ff.ff.ff.ff); a flood puts the original frame (original source & destination addresses) out all but the ingress port.

Otherwise, I believe Mike is correct.

FWIW

Scott

jchrisos · ‎11-27-2006

Yeah, was a typo. The link he posted is definitely steering me in the right direction. It is indeed flodding as I am no longer SPANing the port, but instead added the port as a standard port on our VLAN and I see flodded traffic (the original source and destination MACs as well as FF:FF:FF:FF:FF:FF).

We have 2 switch stacks which are trunked together via fiber. Not sure why, but the trunks are on their own VLAN (vlan 18 where our workstations and servers are on vlan 17). The problem seems to be traffic on vlan17 from the far switch coming into our main switch on vlan17. This traffic seems to be flooded because the hosts on the far switche's MAC addresses do not show up in the main switch's CAM table.

scottmac · ‎11-27-2006

(Quote)

This traffic seems to be flooded because the hosts on the far switche's MAC addresses do not show up in the main switch's CAM table.

(End Quote)

This would be normal behavior.

If the frame is destined for a host on the switch to which it is connected, and it has "talked" within the timeout value of the forwarding/CAM table, the frame is forwarded to the destination.

If the frame does contains a MAC that is not in the forwarding/CAM table, then the switch would flood the frame (to all ports in the VLAN except the ingress port).

For hosts connected to the far switch, unless they have recently "talked" to a device on our Main switch, then the far switch would have to flood to find the destination host.

I suspect the interconnect was put on its own VLAN to limit the broadcast trafic from one switch to the other (the intermediate L3 process would block broadcasts by default).

It doesn't sound like you are using actual trunks (i.e., 802.1q)to connect the two switches ... just high-speed links from access-port to access-port ... in which case VLANs don't really matter (and the reason you far switch is flooding so much).

If you used 802.1q trunking, the VLANs on the far switch would be part of the same broadcast domain as the same VLAN on the Main switch, and the forwarding/CAM tables would be updated when any host in that VLAN on either switch "talked."

FWIW

Good Luck

Scott

jchrisos · ‎11-28-2006

I believe they are using 802.1q. Here's the config from the main/near switch:

VTP Version : 2

Configuration Revision : 12

Maximum VLANs supported locally : 1005

Number of existing VLANs : 13

VTP Operating Mode : Server

VTP Domain Name : abcd

VTP Pruning Mode : Disabled

VTP V2 Mode : Enabled

VTP Traps Generation : Disabled

MD5 digest : 0x7A 0x56 0x9A 0x41 0x07 0xD6 0x1E 0x6F

Configuration last modified by 0.0.0.0 at 7-5-94 10:05:11

Local updater ID is 172.17.1.5 on interface Vl17 (lowest numbered VLAN interface found)

interface Port-channel1

switchport trunk encapsulation dot1q

switchport mode trunk

!

interface Port-channel2

switchport trunk encapsulation dot1q

switchport mode trunk

!

interface GigabitEthernet2/0/1

description Trunk-Link

switchport trunk encapsulation dot1q

switchport mode trunk

channel-group 2 mode on

!

interface GigabitEthernet2/0/2

description Trunk-Link

switchport trunk encapsulation dot1q

switchport mode trunk

channel-group 2 mode on

!

interface GigabitEthernet7/0/1

description Trunk-Link

switchport trunk encapsulation dot1q

switchport mode trunk

channel-group 1 mode on

!

interface GigabitEthernet7/0/2

description Trunk-Link

switchport trunk encapsulation dot1q

switchport mode trunk

channel-group 1 mode on

!

interface Vlan17

ip address 172.17.1.5 255.255.0.0

no ip unreachables

!

Config from the far switch:

VTP Version : 2

Configuration Revision : 12

Maximum VLANs supported locally : 1005

Number of existing VLANs : 13

VTP Operating Mode : Client

VTP Domain Name : abcd

VTP Pruning Mode : Disabled

VTP V2 Mode : Enabled

VTP Traps Generation : Disabled

MD5 digest : 0x7A 0x56 0x9A 0x41 0x07 0xD6 0x1E 0x6F

Configuration last modified by 0.0.0.0 at 7-5-94 10:05:11

!

interface Port-channel1

switchport trunk encapsulation dot1q

switchport mode trunk

!

interface Port-channel2

switchport trunk encapsulation dot1q

switchport mode trunk

!

interface GigabitEthernet1/0/1

description Trunk-Link

switchport trunk encapsulation dot1q

switchport mode trunk

channel-group 1 mode on

!

interface GigabitEthernet1/0/2

description Trunk-Link

switchport trunk encapsulation dot1q

switchport mode trunk

channel-group 1 mode on

!

interface GigabitEthernet4/0/1

description Trunk-Link

switchport trunk encapsulation dot1q

switchport mode trunk

channel-group 2 mode on

!

interface GigabitEthernet4/0/2

description Trunk-Link

switchport trunk encapsulation dot1q

switchport mode trunk

channel-group 2 mode on

!

jchrisos · ‎11-30-2006

Found the problem!

We have very infrequest commincations going on between several machines. By default, the switch's CAM table removes MAC addresses every 300 seconds (every 5 minutes). The flooding recurred every 10 or so minutes so by that time, the MAC addresses were no longer in the CAM table. I increased the timeout to 900 seconds and it solved my problem!

Thanks everyone for your assistance!!!

glen.grant · ‎11-27-2006

More than likely unicast flooding.

http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a00801d0808.shtml