I have a simple yet weird problem I could not explain. Corp VPN gateway is ASA5520 running 7.2 code, branch VPN gateway is ASA 5505 running 8.0 code, I created a simple L2L IPsec between the two to protect communications between HQ subnet (10.0.0.0/8) and branch networks (172.30.0.0/20). IPsec tunnel comes up fine. On the branch side, inside VLAN (subnet 172.30.0.0/24) has two devices attached to it, one is a C2801 CCME router on a stick, the other one is a C3560 which terminates all subnets carved out of 172.20.0.0/20. Since the license we have on ASA5505 does not support trunking, the uplink port on C3560 to ASA is configured as switch port, a L3 VLAN is configured on C3560, its IP address is on the same subnet as ASA5505's inside VLAN and C2801's uplink. C3560 has one default route pointing to ASA's Inside VLAN IP address.
Simple enough right? not so, from corp network I can always ping C2801, but I can not always ping C3560's L3 VLAN IP on the same subnet. After I manually clear IPsec SA on either ASA5505 or ASA5520, I will be able to ping C3560, but after a while, it stops ping again.
If I send continuous ping, after a while it would recover.
Debug icmp trace on HQ ASA5520 shows that echo requests were received from inside interface, but ASA5505 in remote end does not seem to receive them. IPsec SA between corp networks and branch networks look fine on both ASAs. I don't understand what is the difference between C3560's IP and C2801's IP from either ASA's point of view?