Yes, ASA 9.5(1) on a 5506-X. I have tried ip next-hop, default next-hop, recursive next-hop, and set interface.  I think the debug output confirms that the problem is with this ppoe configured interface, since my default route is out the slow interface, there is no route to the outside world on the fast interface but I thought that PBR would fix that.  Here's the output:

pbr: policy based route lookup called for 192.168.20.22/45321 to 206.168.118.24/0 proto 1 sub_proto 8 received on interface inside
pbr: First matching rule from ACL(15)
pbr: route map SingleHostTest, sequence 50, permit; proceed with policy routing
pbr: evaluating recursive next-hop 207.225.112.15
pbr: no route to next-hop 207.225.112.15 found
pbr: evaluating interface fast
pbr: no route to 206.168.118.24 found on interface fast
pbr: policy based routing could not be applied; proceeding with normal route lookup

<uses default route out slow interface>

Here is the policy map with recursive next hop and set interface configured as was the case when I tested:

route-map SingleHostTest, permit, sequence 50
Match clauses:
ip address (access-lists): test_route_maps
Set clauses:
ip next-hop recursive 207.225.112.15
interface fast

I'm missing something on your explanation. 

Could you send a quick sketch of your infrastructure? And sending out interfaces, routes and route-map configuration?

PBR is a simple thing that's working quite in the same way as router, I've never done any testing with pppoe interface but I'm certainly missing something on your explanation but with a design and some config output it would be clear.

Would you mind to add the output of sh int ip brief ?

Thanks

Here it is, addresses have been changed for anonymity.

I am convinced that PBR is working fine but that the problem is basic routing.  I don't know how to set a route that would direct any traffic reaching the fast interface out to any IP address. And that is complicated by it being a pppoe-defined interface without a directly connected next-hop IP address.

So no matter what I do in PBR, the route lookup will always send out the default (slow) interface.

Thanks so much for helping… this is not my "day job" so I'm over my head.

Thanks,

I don't have any pppoe connection right here. I'm trying to mount my Cisco Virl lab up and simulate a pppoe connection in order to test.

Maybe some issues with ASA and pppoe. Have you raised a TAC case?

I've never tried with pppoe connection. and even if you are doing a set route for the pppoe, as you already have a default route it will not be installed.

Try to see with TAC and in the mean time, I'll try to build up my lab soon.

I've an issue with my virl server. Waiting for Cisco feedback and then I'll build up the lab to test. In the mean time, are you able to configure a fix IP for test pppoe connection? You can set a old router as pppoe server.

Thanks

Thanks so much for taking an interest in this.  I'm starting think that this is a problem with pppoe and routing and not PBR.

I trie switching the default route to the fast ISP / pppoe connection and found problems with the traffic PBR routed out the slow interface, so I tried switching back. ASDM would not let me set the default route to the slow interface because of a "conflict with existing routes."  The state was no static route at all so I essentially had a brick but I can configure it inbound over the fast interface.  I think that there is an implicit default route out the fast/pppoe interface now but it doesn't show up in the static IP list (doing this through ASDM)

I'll raise a TAC case if nothing in this situation sounds familiar to anyone.

Hi

I've tried it in my virtual lab (Cisco VIRL) and I'm facing the exact same issue when my interface out is configured as PPPOE. Be careful, virtual is not the same as reality, however as we have exact same issue and we've not copied our configs, I think you need to raise a TAC case.

Please come back to us with TAC solution if possible.

Thanks so much

Wow, thanks for re-creating the problem.

I see what happened when I removed the default route to the slow ISP, the route out the pppoe interface became the default route because of "ip address pppoe set route."  The other thing that happened is that I believe (and I didn't watch for long before I tried restoring the default route) that inbound traffic to the DMZ over the slow interface was being routed out (and dropped) on the fast interface.

To your knowledge, should PBR be able to override the default route? It seems like if you are setting the next-hop address you are declaring that you know what route to use and the ASA shouldn't reevaluate what route the packet takes.  Clearly if I can get packets out to the next-hop addresses defined by PBR, the upstream routers will know what to do with them.

The goal of pbr is to forward specific traffic out to a different next hop without taking into account the default route. However, the next hop has to be reachable.

PBR will not erase the default route because others will use it if not matching pbr acl.

in your case asa seems to not be able to get out to pppoe interface and lokking towards the routing table. If you test with 2 fixed ip addresses (no pppoe) it will work.

generally we use pbr to forward specific traffic based on source ip and destination port (https, https,....) out to a specific next hop. For example, you want specific traffic to be redirected out to a proxy; in this case pbr is usefull. This is the test I've done but asa don't work as expected, maybe a limitation (but not seen on release notes). TAC should debug it in details if a bug or they will say that's a limitation.

It's along time I'm using pbr but never with pppoe ISP. I never gate such specific case. 

Keep us informed about TAC solution.

Thanks

First word back from TAC is that PBR+pppoe as the secondary route is a known limitation but not documented as such.

We are supposed to be able to allow the pppoe interface to have the primary route and use the static (slow in this example) be a backup route with a higher metric.

When I get this going I will report back.

Ok thanks. This is what I've configured yesterday but not yet tested (run out of time, sorry).

But quite sure that it will works in that way

I have it configured and running with great help from TAC.

As I noted earlier, the pppoe interface cannot be the secondary interface, most likely because it sets a default route with metric 1 when it finds out what the next hop is. This also makes some engineering sense to me because if I configure PBR to set the next hop out pppoe, and the upstream provider changes the next hop address, my configuration would be broken.  

The solution is to allow pppoe (fast in the diagram) to set the default route (ip address pppoe setroute), and use PBR to send any exceptions to the static (slow) interface. This is straightforward because the rules you're creating are just the inverse of using PBR to route out the fast interface. With the fast interface as the default, I just don't bother assigning a next hop for anything I want to go out that route.

The non-intuitive part for me is this: I would think that by setting next hop in PBR, the ASA would leave the route alone because clearly if I set the next hop I know where the packet should be routed. But after setting the next hop to be the slow interface, and the next-hop address is correctly set, the routing tables are still consulted and the packet is sent out the fast interface because it has the default route.

So you need to have a secondary default route with a higher metric to allow packets with next-hop set through PBR to go out the slow interface.  So based on the updated diagram attached, I have:

route slow 0.0.0.0 0.0.0.0 xxx.xxx.xxx.73  2

Hi

Thanks for your comment. This last configs is for sure the one that was working and tested yesterday on my lab.

However, does Cisco TAC talked about roadmap and/or best practices?

Because in some specific cases, we wouldn't have the pppoe as primary and just as backup. In that design, PBR will not work. Workaround is to do what you've done today to make it works,

Thanks

No talk about roadmap but best practices are basically what we implemented.

I think that what you are attempting to do can work just fine, if you want to treat the pppoe as a backup then use PBR to route everything out the non-pppoe interface. I haven't experimented with this, but it seems that you could use "configure next hop verify ability" under Edit Route Map in ASDM to verify that your next hop is available and if it isn't the next hop would not be set and your traffic would fall back to the pppoe interface.

I can see that there are pitfalls in trying to make pppoe primary: you would have to use a fixed address for the next hop based on the routing provided by the ISP. If that routing changes, the pppoe configuration would change but your router would not, so your next-hop IP address would take you nowhere.

A future study question is also what to do if you have two pppoe interfaces. It seems that the best you could do with PBR is to set the next hop interface. But I'm not sure if that would work because you might have two interfaces competing to set a default route with metric 1.  Beyond where I'm going with my configuration.