Solved: trace output shows ! A ! A

Jacob Samuel · ‎01-11-2012

HI

I may need your kind input in the below issue-

i have a customer with 1 Head Office and 100+ branch offices connecting through Telco MPLS network. I am extending one more department to the network under their network by adding another L3 switch and a LAN under that switch. I am using vlan 1 to route between the existing network and the new subnet. Every location, am adding a default route on the L3 switch pointing to the gateway router. In the gateway router i am adding the new subnet in the BGP and add a static reverse route for the new subnet pointing to the vlan 1 ip of the L3 Switch. this is the proceedure i do for adding the new subnet / new dept in the exisitng network. everythign was going fine in this scenario.

But now some locations there is no MPLS network available, instead the wan gateway router is connected to Telco on old Serial link, and susing a static default route to the Telco Router. so the new subnet is not getting routed over the WAN and it is getting droped at the Telco router. i requested Telco to do the necessary to publish this network also in to the MPLS / bgp network so that my new subnet will get router over the MPLS network. as per them they did it ( i dont know what they did exactly, i think they add a static route on their router for my new network like- ip route 172.16.120.0 /24 62.x.x.x my router wan interface).

Now the issue is, when i am trying to ping from the head office to the new subnet (ie: vlan 10 - ip add 172.16.120.1) there is no response, but i can ping 172.16.1.254 (Vlan 1 IP on the L3 Switch, whcih means there is no issue in the connectivity between the L3 Switch and the Router). if i try to trace, it is reaching up to the vlan 1 ip address of the L3 swith and coming back to the router lan interface again (172.16.1.1 - 172.16.1.254 again 172.16.1.1 - 172.16.1.254) like looping. If i try to trace from the Branch Gateway Router to the L3 Switch in the LAN it is giving me output 172.16.1.254 ! A ! A

what could be the exact ssues in this case? what is this ! A in trace output ?

Thanks & Regards

Sunny

rsimoni · ‎01-11-2012

Hi Sunny,

indeed it seems there is some problem on the L3 switch on the new site.

Can you paste its configuration and its routing table?

Riccardo

View solution in original post

rsimoni · ‎01-11-2012

Hi Sunny,

that output means that your trace is blocked by an ACL.

see also

http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a00800a6057.shtml

Riccardo

Jacob Samuel · ‎01-11-2012

Thanks. But

if i try to trace, it is reaching up to the vlan 1 ip address of the L3 swith and coming back to the router lan interface again (172.16.1.1 - 172.16.1.254 again 172.16.1.1 - 172.16.1.254) like looping.

why it is getting looped between Vlan 1 on L3 switch and the Router?

Thanks & Regards

Sunny

rsimoni · ‎01-11-2012

Hi Sunny,

I just answered to the second question without getting in depth with your email. ! A stand for Administevely blocked trace.

Regarding your loop, can you add a topology showing the routers in questions with the addresses showing in the trace as well. That would facilate the understanding of your problem.

Riccardo

Jacob Samuel · ‎01-11-2012

Hi,

I am trying to ping / trace from Head Office LAN (172.16.100.10) to reach the LAN at Branch Office (172.16.120.1).

Sunny

rsimoni · ‎01-11-2012

Hi Sunny,

indeed it seems there is some problem on the L3 switch on the new site.

Can you paste its configuration and its routing table?

Riccardo

Jacob Samuel · ‎01-11-2012

Hi Ricardo,

here is the config-

#sh run
Building configuration...

Current configuration : 10023 bytes
!
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname ac-abc-Sw01
!
boot-start-marker
boot-end-marker
!
enable secret 5
!
username netadmin privilege 15 password
no aaa new-model
system mtu routing 1500
vtp domain PSSP
vtp mode transparent
ip subnet-zero
ip routing
no ip domain-lookup
!
!
!
!
crypto pki trustpoint TP-self-signed-110980608
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-110980608
revocation-check none
rsakeypair TP-self-signed-110980608
!
!
crypto pki certificate chain TP-self-signed-110980608
certificate self-signed 01
30820245 308201AE A0030201 02020101 300D0609 2A864886 F70D0101 04050030
30312E30 2C060355 04031325 494F532D 53656C66 2D536967 6E65642D 43657274
69666963 6174652D 31313039 38303630 38301E17 0D393330 33303130 30303931
305A170D 32303031 30313030 30303030 5A303031 2E302C06 03550403 1325494F
532D5365 6C662D53 69676E65 642D4365 72746966 69636174 652D3131 30393830
36303830 819F300D 06092A86 4886F70D 01010105 0003818D 00308189 02818100
A53BA028 477443E5 2DA31F1A 078573BF 38B4DB39 B3394707 674FE75E 056C10F3
F5731CB7 FC8E3587 A84C1E39 26D3EE43 E9C63F02 22EA0DBA BBD66974 A2AB72F2
163A172E 922DD627 C87D54DA 9B95B6D4 310068DF 7F60EEC1 3734E6B4 E0BC859A
5550EFFF 6EBFE6CD 722D42B1 0A8C367F 0937BE51 7D76A046 944D6F06 30270CE9
02030100 01A36F30 6D300F06 03551D13 0101FF04 05300301 01FF301A 0603551D
11041330 11820F53 502D4441 4D4D414D 2D537730 312E301F 0603551D 23041830
168014A9 675DA580 750227D4 AD488E00 16E744D1 B291A430 1D060355 1D0E0416
0414A967 5DA58075 0227D4AD 488E0016 E744D1B2 91A4300D 06092A86 4886F70D
01010405 00038181 003F891C 04D01988 B18D4C8A 56E35EA2 08E14D88 4C6126E5
06ADB61E 9F446805 33188AE9 BF4A48D0 B2ECE2EF F77D1BB0 688D99E0 7E9B60F2
751A9A64 E02BFE2C 709C4456 A266AA1B 3134DC15 A7F5533A 41A2289C 7EF92FA1
B7A11488 16BE2C02 3F733E32 FAFA247D 8AB981F7 243FF05A FC38860C A1A47635
01686EE1 075A0B8D 7C
quit
!
!
!
!
!
!
spanning-tree mode pvst
spanning-tree loopguard default
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
vlan 10
name Server-Vlan
state suspend
!
vlan 11
name User-VLAN
!
!
!
interface FastEthernet0/1
description *** User Vlan ***
switchport access vlan 11
switchport mode access
!
interface FastEthernet0/2
description *** User Vlan ***
switchport access vlan 11
switchport mode access
!
interface FastEthernet0/3
description *** User Vlan ***
switchport access vlan 11
switchport mode access
!
interface FastEthernet0/4
description *** User Vlan ***
switchport access vlan 11
switchport mode access
!
interface FastEthernet0/5
description *** User Vlan ***
switchport access vlan 11
switchport mode access
!
interface FastEthernet0/6
description *** User Vlan ***
switchport access vlan 11
switchport mode access
!
interface FastEthernet0/7
description *** User Vlan ***
switchport access vlan 11
switchport mode access
!
interface FastEthernet0/8
description *** User Vlan ***
switchport access vlan 11
switchport mode access
!
interface FastEthernet0/9
description *** User Vlan ***
switchport access vlan 11
switchport mode access
!
interface FastEthernet0/10
description *** User Vlan ***
switchport access vlan 11
switchport mode access
!
interface FastEthernet0/11
description *** User Vlan ***
switchport access vlan 11
switchport mode access
!
interface FastEthernet0/12
description *** User Vlan ***
switchport access vlan 11
switchport mode access
!
interface FastEthernet0/13
description *** User Vlan ***
switchport access vlan 11
switchport mode access
!
interface FastEthernet0/14
description *** User Vlan ***
switchport access vlan 11
switchport mode access
!
interface FastEthernet0/15
description *** User Vlan ***
switchport access vlan 11
switchport mode access
!
interface FastEthernet0/16
description *** User Vlan ***
switchport access vlan 11
switchport mode access
!
interface FastEthernet0/17
description *** User Vlan ***
switchport access vlan 11
switchport mode access
!
interface FastEthernet0/18
description *** User Vlan ***
switchport access vlan 11
switchport mode access
!
interface FastEthernet0/19
description *** User Vlan ***
switchport access vlan 11
switchport mode access
!
interface FastEthernet0/20
description *** User Vlan ***
switchport access vlan 11
switchport mode access
!
interface FastEthernet0/21
description *** User Vlan ***
switchport access vlan 11
switchport mode access
!
interface FastEthernet0/22
description *** User Vlan ***
switchport access vlan 11
switchport mode access
!
interface FastEthernet0/23
description *** User Vlan ***
switchport access vlan 11
switchport mode access
!
interface FastEthernet0/24
description *** Uplink to PS Network Switch ***
switchport mode access
!
interface GigabitEthernet0/1
shutdown
!
interface GigabitEthernet0/2
shutdown
!
interface Vlan1
description *** Routing Vlan to PS Network***
ip address 172.16.1.254 255.255.255.0
ip access-group 110 in
!
interface Vlan11
description *** UserS VLAN **
ip address 172.16.120.1 255.255.255.0
ip access-group 100 in
!
ip classless
ip route 0.0.0.0 0.0.0.0 172.16.1.1
ip http server
ip http authentication local
ip http secure-server
ip http timeout-policy idle 180 life 600 requests 10
!
access-list 1 permit 172.16.100.0 0.0.0.255
access-list 1 permit 172.16.155.0 0.0.0.255
access-list 100 remark *** Filtering LAN users to access other network ***
access-list 100 permit ip 172.16.120.0 0.0.0.255 172.16.159.0 0.0.0.255
access-list 100 permit ip 172.16.120.0 0.0.0.255 172.16.158.0 0.0.0.255
access-list 100 permit ip 172.16.120.0 0.0.0.255 172.16.157.0 0.0.0.255
access-list 100 permit ip 172.16.120.0 0.0.0.255 172.16.156.0 0.0.0.255
access-list 100 permit ip 172.16.120.0 0.0.0.255 172.16.155.0 0.0.0.255
access-list 100 permit ip 172.16.120.0 0.0.0.255 172.16.100.0 0.0.0.255
access-list 100 permit ip 172.16.120.0 0.0.0.255 172.16.153.0 0.0.0.255
access-list 100 permit ip 172.16.120.0 0.0.0.255 172.16.121.0 0.0.0.255
access-list 100 permit ip 172.16.120.0 0.0.0.255 172.16.120.0 0.0.0.255
access-list 100 permit ip 172.16.120.0 0.0.0.255 172.16.119.0 0.0.0.255
access-list 100 permit ip 172.16.120.0 0.0.0.255 172.16.118.0 0.0.0.255
access-list 100 permit ip 172.16.120.0 0.0.0.255 172.16.117.0 0.0.0.255
access-list 100 permit ip 172.16.120.0 0.0.0.255 172.16.116.0 0.0.0.255
access-list 100 permit ip 172.16.120.0 0.0.0.255 172.16.115.0 0.0.0.255
access-list 100 permit ip 172.16.120.0 0.0.0.255 172.16.114.0 0.0.0.255
access-list 100 permit ip 172.16.120.0 0.0.0.255 172.16.113.0 0.0.0.255
access-list 100 permit ip 172.16.120.0 0.0.0.255 172.16.112.0 0.0.0.255
access-list 100 permit ip 172.16.120.0 0.0.0.255 172.16.111.0 0.0.0.255
access-list 100 permit icmp any any
access-list 110 remark *** Filter users coming-in-to Routing Vlan 1 ***
access-list 110 permit tcp 172.16.100.0 0.0.0.255 any eq telnet
access-list 110 permit tcp 172.16.155.0 0.0.0.255 any eq telnet
access-list 110 permit tcp 172.16.100.0 0.0.0.255 any eq 22
access-list 110 permit tcp 172.16.155.0 0.0.0.255 any eq 22
access-list 110 permit tcp 172.16.100.0 0.0.0.255 any eq 69
access-list 110 permit tcp 172.16.155.0 0.0.0.255 any eq 69
access-list 110 permit tcp 172.16.100.0 0.0.0.255 any eq www
access-list 110 permit tcp 172.16.155.0 0.0.0.255 any eq www
access-list 110 permit tcp 172.16.100.0 0.0.0.255 any eq 443
access-list 110 permit tcp 172.16.155.0 0.0.0.255 any eq 443
access-list 110 permit icmp 172.16.100.0 0.0.0.255 any
access-list 110 permit icmp 172.16.155.0 0.0.0.255 any
access-list 110 deny   tcp any any eq telnet
access-list 110 deny   tcp any any eq 22
access-list 110 deny   tcp any any eq www
access-list 110 deny   tcp any any eq 443
access-list 110 permit ip 172.16.111.0 0.0.0.255 172.16.120.0 0.0.0.255
access-list 110 permit ip 172.16.112.0 0.0.0.255 172.16.120.0 0.0.0.255
access-list 110 permit ip 172.16.113.0 0.0.0.255 172.16.120.0 0.0.0.255
access-list 110 permit ip 172.16.114.0 0.0.0.255 172.16.120.0 0.0.0.255
access-list 110 permit ip 172.16.115.0 0.0.0.255 172.16.120.0 0.0.0.255
access-list 110 permit ip 172.16.116.0 0.0.0.255 172.16.120.0 0.0.0.255
access-list 110 permit ip 172.16.117.0 0.0.0.255 172.16.120.0 0.0.0.255
access-list 110 permit ip 172.16.118.0 0.0.0.255 172.16.120.0 0.0.0.255
access-list 110 permit ip 172.16.119.0 0.0.0.255 172.16.120.0 0.0.0.255
access-list 110 permit ip 172.16.120.0 0.0.0.255 172.16.120.0 0.0.0.255
access-list 110 permit ip 172.16.121.0 0.0.0.255 172.16.120.0 0.0.0.255
access-list 110 permit ip 172.16.153.0 0.0.0.255 172.16.120.0 0.0.0.255
access-list 110 permit ip 172.16.100.0 0.0.0.255 172.16.120.0 0.0.0.255
access-list 110 permit ip 172.16.155.0 0.0.0.255 172.16.120.0 0.0.0.255
access-list 110 permit ip 172.16.156.0 0.0.0.255 172.16.120.0 0.0.0.255
access-list 110 permit ip 172.16.157.0 0.0.0.255 172.16.120.0 0.0.0.255
access-list 110 permit ip 172.16.158.0 0.0.0.255 172.16.120.0 0.0.0.255
no cdp run
snmp-server community SecPro RW 1
snmp-server enable traps snmp authentication linkdown linkup coldstart warmstart
snmp-server enable traps vtp
snmp-server enable traps vlancreate
snmp-server enable traps vlandelete
snmp-server enable traps port-security
snmp-server enable traps envmon fan shutdown supply temperature status
snmp-server enable traps config-copy
snmp-server enable traps config
snmp-server enable traps vlan-membership
snmp-server enable traps errdisable
!
control-plane
!
!
line con 0
password
login local
line vty 0 4
access-class 1 in
password
login local
line vty 5 15
access-class 1 in
password
login local
!
end

Thanks & regards

Sunny

rsimoni · ‎01-11-2012

Hi Sunny,

you forgot to add the routing table (show ip route).

Please also add 'show interface vlan11'.

First impression is that the ACL's are correct; however can you quickly remove them and retry to ping/trace from head office?

int vlan 1

no ip access-group 110 in

int vlan11

no ip access-group 100 in

(btw users are in vlan11 not in vlan10 - but it does not change anything of course).

Riccardo

Jacob Samuel · ‎01-11-2012

Hi Ricardo,

About the Vlan number, yes sorry it was a mistake.

Regarding the #show ip route and #show interface vlan 11 ouput, i will try from the customer site, i was not able to telnet to the Vlan 1 IP address of the L3 Switch. Let me check it again from the customer site. i will update you with the outputs probably on Saturday.

thanks alot for your support

regards

Sunny

rsimoni · ‎01-11-2012

ok, talk to you later then

Jacob Samuel · ‎01-15-2012

Dear Ricardo,

The issue is resolved, it was becuase the interface vlan 11 was down, think they didnt start using the network, and no ports were up. I was suspecting it but got confused about the erors what i was getting.

thanks a lot for your support.... and the Link.

Regards

Sunny

rsimoni · ‎01-16-2012

Hi Sunny,

glad that you fixed.

I was pretty sure that the issue was something like that; that's why I asked for

Please also add 'show interface vlan11'.

as if no port in that vlan is up STP will keep the SVI down.

Riccardo