High cpu load when using l2tpv3

[viktordavityan]

We have couple l2tpv3 tunnels between C2911 and C1921.
which, in turn, pass through a DMVPN tunnel through the provider cloud.
And plain copy operation of a data, say ~100Mb and more in vlan 5 that  xconnected

lead to a high cpu load up to 100% on a C2911.
C2911 by itself DMVPN Hub Phase 1 with a 27 Spokes.
Both installed 15.4(3)M8

Is there something wrong with software hardware ore maybe misuse of technologies?

C2911

l2tp-class class71
authentication
password
!
pseudowire-class pw71
encapsulation l2tpv3
protocol l2tpv3 class71
ip local interface Loopback10
!
interface Loopback0
ip address 10.120.78.242 255.255.255.255
ip ospf 1 area 0
!
interface Loopback10
ip address 10.120.78.101 255.255.255.255
ip ospf 1 area 0
!
interface Tunnel0
ip address 10.120.37.15 255.255.255.0
ip mtu 1400
ip nhrp authentication cisco37
ip nhrp map multicast dynamic
ip nhrp network-id 37
ip nhrp holdtime 300
ip tcp adjust-mss 1360
tunnel source 10.120.0.198
tunnel mode gre multipoint
tunnel key 37
!
interface GigabitEthernet0/1.810
encapsulation dot1Q 810
ip address 10.120.0.198 255.255.255.252
no ip redirects
no ip unreachables
no ip proxy-arp
!
interface GigabitEthernet0/2.5
encapsulation dot1Q 5
no ip redirects
no ip unreachables
no ip proxy-arp
xconnect 10.120.78.102 5 encapsulation l2tpv3 pw-class pw71
!
interface GigabitEthernet0/2.71
encapsulation dot1Q 71
no ip redirects
no ip unreachables
no ip proxy-arp
xconnect 10.120.78.102 71 encapsulation l2tpv3 pw-class pw71

C1921

l2tp-class class71
authentication
password <>
!
pseudowire-class pw71
encapsulation l2tpv3
protocol l2tpv3 class71
ip local interface Loopback10
!
interface Loopback0
ip address 10.120.78.24 255.255.255.255
!
interface Loopback10
ip address 10.120.78.102 255.255.255.255
!
interface Tunnel0
ip address 10.120.37.24 255.255.255.0
no ip redirects
no ip proxy-arp
ip mtu 1400
ip nhrp authentication cisco37
ip nhrp map multicast 10.120.0.198
ip nhrp map 10.120.37.15 10.120.0.198
ip nhrp network-id 37
ip nhrp holdtime 300
ip nhrp nhs 10.120.37.15
ip tcp adjust-mss 1360
tunnel source 10.120.0.21
tunnel destination 10.120.0.198
tunnel key 37
!
interface GigabitEthernet0/0.627
encapsulation dot1Q 627
ip address 10.120.0.21 255.255.255.252
no ip redirects
no ip proxy-arp
!
interface GigabitEthernet0/1.5
encapsulation dot1Q 5
no ip redirects
no ip proxy-arp
xconnect 10.120.78.101 5 encapsulation l2tpv3 pw-class pw71
!
interface GigabitEthernet0/1.71
encapsulation dot1Q 71
no ip redirects
no ip proxy-arp
xconnect 10.120.78.101 71 encapsulation l2tpv3 pw-class pw71
!

PS. On C1921 sometimes there is an error occurs:

Jul 14 00:10:08.350: %SYS-2-MALLOCFAIL: Memory allocation of 18252 bytes failed from 0x2164AD18, alignment 128
Pool: I/O Free: 257984 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "IP Input", ipl= 0, pid= 165
-Traceback= 210115B8z 21648BA0z 21648818z 2164BE3Cz 211DC054z 211DC2C8z 211E2680z 211DA9CCz 211DAC58z 211DAD44z 211DAF90z 21032A48z 21032A2Cz
!

[Julio E. Moisa]

Hi

This messages: -Traceback= 210115B8z 21648BA0z 21648818z 2164BE3Cz 211DC054z 211DC2C8z 211E2680z 211DA9CCz 211DAC58z 211DAD44z 211DAF90z 21032A48z 21032A2Cz

The message above can indicate, lack of memory RAM or a Bug (you could require an upgrade)

Additional, I have tried to understand your topology but I think the L2TPv3 VPN is not passing through the DMVPN i see different used interface, is that right? also your config looks fine but you could simplify this line:

xconnect 10.120.78.102 5 encapsulation l2tpv3 pw-class pw71

with

xconnect 10.120.78.102 5 pw-class pw71

Hope it is useful

[viktordavityan]

I think there is enough memory:

Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 2B098420 333252732 66264692 266988040 48814064 232392796
I/O E000000 33554432 15208908 18345524 1824 3289676

L2TPv3 VPN is indeed passing through the DMVPN because there is no direct connectivity between hub and spokes . That's why we use tunnel in phase 1. 

I feel that something like СoPP is needed here to limit the effect of GRE packets on the processor.

[a.alekseev]

"Sh ip traffic" on both sides

[viktordavityan]

C2911

IP statistics:
Rcvd: 282633437 total, 153748295 local destination
0 format errors, 0 checksum errors, 432540 bad hop count
0 unknown protocol, 0 not a gateway
0 security failures, 0 bad options, 0 with options
Opts: 0 end, 0 nop, 0 basic security, 0 loose source route
0 timestamp, 0 extended security, 0 record route
0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump
0 other
Frags: 128400822 reassembled, 1889 timeouts, 0 couldn't reassemble
169532725 fragmented, 339065647 fragments, 5369 couldn't fragment
0 invalid hole
Bcast: 318 received, 0 sent
Mcast: 12276195 received, 685470 sent
Sent: 13887060 generated, 1517220746 forwarded
Drop: 496780 encapsulation failed, 0 unresolved, 0 no adjacency
0 no route, 0 unicast RPF, 0 forced drop
0 options denied
Drop: 0 packets with source IP address zero
Drop: 0 packets with internal loop back IP address
0 physical broadcast
Reinj: 0 in input feature path, 0 in output feature path

ICMP statistics:
Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 12 unreachable
110928 echo, 330 echo reply, 0 mask requests, 0 mask replies, 0 quench
0 parameter, 0 timestamp, 0 timestamp replies, 0 info request, 0 other
0 irdp solicitations, 0 irdp advertisements
4270 time exceeded, 0 info replies
Sent: 0 redirects, 5262 unreachable, 15 echo, 110928 echo reply
0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 0 timestamp replies
0 info reply, 433052 time exceeded, 0 parameter problem
0 irdp solicitations, 0 irdp advertisements

UDP statistics:
Rcvd: 283703 total, 0 checksum errors, 23 no port 0 finput
Sent: 286992 total, 0 forwarded broadcasts

BGP statistics:
Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh, 0 unrecognized
Sent: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh

TCP statistics:
Rcvd: 24163 total, 0 checksum errors, 7 no port
Sent: 22955 total

EIGRP-IPv4 statistics:
Rcvd: 12062837 total
Sent: 469003 total

PIMv2 statistics: Sent/Received
Total: 0/0, 0 checksum errors, 0 format errors
Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0, Hellos: 0/0
Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
Queue drops: 0
State-Refresh: 0/0

IGMP statistics: Sent/Received
Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
DVMRP: 0/0, PIM: 0/0
Queue drops: 0

OSPF statistics:
Last clearing of OSPF traffic counters never
Rcvd: 232540 total, 0 checksum errors
217488 hello, 9 database desc, 0 link state req
9083 link state updates, 5960 link state acks
Sent: 236661 total
217489 hello, 5 database desc, 3 link state req
12822 link state updates, 6342 link state acks

ARP statistics:
Rcvd: 221 requests, 65 replies, 0 reverse, 0 other
Sent: 207 requests, 233 replies (0 proxy), 0 reverse
Drop due to input queue full: 0

C1921

IP statistics:
Rcvd: 415055020 total, 208991994 local destination
0 format errors, 0 checksum errors, 42113 bad hop count
0 unknown protocol, 0 not a gateway
0 security failures, 0 bad options, 42 with options
Opts: 0 end, 0 nop, 0 basic security, 0 loose source route
0 timestamp, 0 extended security, 0 record route
0 stream ID, 0 strict source route, 42 alert, 0 cipso, 0 ump
0 other
Frags: 205068566 reassembled, 710266 timeouts, 0 couldn't reassemble
145753380 fragmented, 291506760 fragments, 170 couldn't fragment
0 invalid hole
Bcast: 454451 received, 892 sent
Mcast: 1130299 received, 1128979 sent
Sent: 3818279 generated, 3211695759 forwarded
Drop: 0 encapsulation failed, 0 unresolved, 0 no adjacency
2796 no route, 0 unicast RPF, 0 forced drop
0 options denied
Drop: 0 packets with source IP address zero
Drop: 0 packets with internal loop back IP address
0 physical broadcast
Reinj: 0 in input feature path, 0 in output feature path

ICMP statistics:
Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 897 unreachable
278531 echo, 18 echo reply, 0 mask requests, 0 mask replies, 0 quench
0 parameter, 0 timestamp, 0 timestamp replies, 0 info request, 0 other
0 irdp solicitations, 0 irdp advertisements
656 time exceeded, 0 info replies
Sent: 0 redirects, 12136 unreachable, 30 echo, 278531 echo reply
0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 0 timestamp replies
0 info reply, 5740 time exceeded, 0 parameter problem
0 irdp solicitations, 0 irdp advertisements

UDP statistics:
Rcvd: 671543 total, 0 checksum errors, 43608 no port 0 finput
Sent: 210930 total, 417059 forwarded broadcasts

BGP statistics:
Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh, 0 unrecognized
Sent: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh

TCP statistics:
Rcvd: 115123 total, 0 checksum errors, 43 no port
Sent: 202651 total

EIGRP-IPv4 statistics:
Rcvd: 1131671 total
Sent: 1131370 total

PIMv2 statistics: Sent/Received
Total: 0/0, 0 checksum errors, 0 format errors
Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0, Hellos: 0/0
Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
Queue drops: 0
State-Refresh: 0/0

IGMP statistics: Sent/Received
Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
DVMRP: 0/0, PIM: 0/0
Queue drops: 0

OSPF statistics:
Last clearing of OSPF traffic counters never
Rcvd: 0 total, 0 checksum errors
0 hello, 0 database desc, 0 link state req
0 link state updates, 0 link state acks
Sent: 0 total
0 hello, 0 database desc, 0 link state req
0 link state updates, 0 link state acks

ARP statistics:
Rcvd: 3350086 requests, 4250 replies, 0 reverse, 0 other
Sent: 107088 requests, 723536 replies (0 proxy), 0 reverse
Drop due to input queue full: 0

[a.alekseev]

It's because of this
Frags: 128400822 reassembled, 1889 timeouts, 0 couldn't reassemble
169532725 fragmented, 339065647 fragments, 5369 couldn't fragment

https://www.cisco.com/c/en/us/support/docs/dial-access/virtual-private-dialup-network-vpdn/24320-l2tp-mtu-tuning.html

you can try this....

C2911

!
pseudowire-class pw71
encapsulation l2tpv3
protocol l2tpv3 class71
!
!
interface GigabitEthernet0/2.5
encapsulation dot1Q 5
xconnect 10.120.0.21 5 encapsulation l2tpv3 pw-class pw71
!
interface GigabitEthernet0/2.71
encapsulation dot1Q 71
xconnect 10.120.0.21 71 encapsulation l2tpv3 pw-class pw71

c1921
!
interface GigabitEthernet0/1.5
encapsulation dot1Q 5
xconnect 10.120.0.198 5 encapsulation l2tpv3 pw-class pw71
!
interface GigabitEthernet0/1.71
encapsulation dot1Q 71
xconnect 10.120.0.198 71 encapsulation l2tpv3 pw-class pw71
!

[viktordavityan]

And in https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/wan_lserv/configuration/15-mt/wan-lserv-15-mt-book/wan-l2-tun-pro-v3.html

Cisco recommends 

• You must configure a loopback interface on the router for originating and terminating the L2TPv3 traffic. The loopback interface must have an IP address that is reachable from the remote PE device at the other end of an L2TPv3 control channel.

Nevertheless, your idea is clear and I tried to do as you suggested. But without any changes.
But about MTU and PMTUD...  thank you for advice. I somehow did not even value it. But still I don't go to significantly reduce the fragmentation.

[Joseph W. Doherty]

Fragmentation processing often places quite an additional load on the processor.

[a.alekseev]

What do you mean "without any changes"?
Is L2TPv3 still working?

You can ask your the provider to increase MTU on your interfaces...

More over your mGRE tunnels are not optimal...
You are using GRE without tunnel ptotection or IPSec
so the overhead is 24bytes + 4bytes tunnel key

so fot the tunnel you can have
ip mtu 1472
mss 1472-40

[viktordavityan]

Yes it still working. And working fine with VoIP call, Internet surfing, mail ... but when  start to copy the file, a BIG file, in the same vlan, but from different sides of the tunnel problem is rising. I do not even able to get to the console, so the processor loads.

Tunnel already was set to 1400 mtu and 1360 mss.

[a.alekseev]

What is behind the l2tpv3 tunnels? Any routers?

[viktordavityan]

 Just switches with workstation connected from one side(C1921) . And cople of routers on the C2911 side (OSPF).

[a.alekseev]

if you have routers on the C2911 side,
so you can try to use "ip tcp adjust-mss"

[viktordavityan]

We move to the usual scheme without using the xconnect. And everything works as it should. Transmission of Big files and so on. Using xconnect on Low-end platforms is a masochism, pure evil. IMHO. Maybe I'm wrong. However, thank you all for your help.

[viktordavityan]

   Some time has passed. I have additional equipment. And I decided to reproduce the problem at the lab.
The decision was quite prosaic, although it may be not so elegant - service policy hierarchy on both sides of l2tpv3 tunnel. CPU load did not exceed ~45% in the laboratory against almost 100% before.

Just a quick configuration examle from one side.

!
!Service policy definition
!
ip access-list extended FTP
permit tcp any any eq ftp
permit tcp any eq ftp-data any
permit tcp any eq ftp any
permit tcp any any eq ftp-data
ip access-list extended HTTP
permit tcp any any eq 443
permit tcp any any eq www
permit tcp any any eq 8080
permit tcp any any eq 3128
ip access-list extended SMB
permit tcp any any eq 445
permit tcp any eq 445 any
!
class-map match-all HTTP
match access-group name HTTP
class-map match-all FTP
match access-group name FTP
class-map match-all SMB
match access-group name SMB
!
policy-map INNER
class FTP
police 5000000
class HTTP
police 35000000
class SMB
police 25000000
class class-default
police 35000000
!
policy-map RESTRICT_POLICY
class class-default
police 100000000
service-policy INNER
!
! Apply to interface inbound
!
interface GigabitEthernet0/1.5
encapsulation dot1Q 5
xconnect 1.1.1.1 5 encapsulation l2tpv3 pw-class PWC1
service-policy input RESTRICT_POLICY
!