Thanks for the help, i think the current solution I have in place is along the lines of what you suggest.  However, I'm not using GRE tunnels currently - though I should point out, the EIGRP is only running locally at the remote site, we do not have EIGRP employed at all at our head office.

However, I think I've sussed out why it went wrong (preventing similar from happening again may be another matter!)

Our head office internet pipe is provided by ISP1 and the ADSL circuit at the remote site is also provided by ISP1.

The 10Mbps circuit at the remote site is provided by ISP2.

So, what I think has happened is that (for only around a few minutes) ISP1 lost it's external internet connectivity (long and complicated story but this kind of thing  has happened a few times in the past, thus we are using ISP2 for this new connection!) so the ASA dead peer detection has kicked in, seeing ISP2 peer as down and formed tunnel to ISP1 peer (ADSL router) Since ADSL router is using same ISP as head office internet pipe connection it wasn't affected by ISP1's "internet" outage.

BUT, at remote site, nothing is actually wrong - HSRP and EIGRP status is all fine as far as it's concerned.

So, the question remains really - is there any way of specifiying a preferred peer on the ASA?  When the ASA does fail over to the 2nd peer, can I have it automatically go back to the "primary" peer when comms are restored to it?

I think my current solution "works" other than in the case where there is a problem in the "cloud" so to speak.  Unfortunately, the reliability of ISP1 is such that this is probably the more likely fault scenario!   Is there anything that would help me overcome this sort of issue?

Thanks.

Sorry I am confused with what you have said regarding your

setup and what you think has gone wrong.

Are you monitoring any HSRP interfaces? Are you running any dynamic routing protocols over the VPN?

Why router the primary router choose to use the secondary router to reach the head office?

Apologies, I realise I was rambling a bit – it’s quite a difficult set-up to try to explain concisely though!

Head office – main internet pipe, large number of IPSEC VPN connections to a large number of remote sites, typically with ADSL connections.  The IPSEC VPN tunnels terminate on our ASA at head office.  For the vast majority of our sites, this set-up is fine and we can “live” with any problems that may arise from a site having a router or circuit failure or whatever.

However, one particular site is more critical than the others.  Hence we want to ensure connectivity with this office and introduce some added resilience for it.

This remote office has 2 routers – Cisco 1841s – one is “primary” with a 10Mbps circuit connected (via ISP2) and one is “secondary” with an ADSL circuit connected (via ISP1 – the same ISP used for our main head office circuit)   They are running HSRP and EIGRP between them (running locally only, not across the VPN tunnels) so that if the primary router or 10Mbps circuit should fail the secondary router will take over as active and traffic should flow across the ADSL connection.   From, admittedly limited, testing, the resilient aspect of this part of the solution seems to work. 

Back at head office, on the ASA, I have configured 2 peers in the VPN set-up for this critical remote office.  One peer is the IP address of the primary remote router (ISP2) and the other peer is the IP address of the secondary router (ISP1).  The thought behind that being, if the 10Mbps circuit or primary router itself was to fail then obviously that peer would be unavailable and the ASA’s dead peer detection would make it move onto the other peer.

So, normal operation - all traffic from the remote site should flow via its primary router over the 10Mbps circuit to head office.  Only if there is a problem with the primary router or 10Mbps circuit should traffic flow via the secondary router and ADSL line instead.

However, in this situation, what has happened is – there was NO fault with the 10Mbps circuit or router so locally everything at the remote site was fine.  BUT, ISP1 has had some sort of “internet outage” meaning that their connectivity with other networks has been lost for a few minutes.  The impact of this was that the Head Office ASA could no longer “see” the primary router at the remote site so, because of dead peer detection, it moved onto the secondary peer (the ADSL router) which it could still “see” because it uses the same ISP (connectivity internal to the ISP was ok, only connectivity external to the ISP was affected at that time)

So, even though there was no issue at the remote office, it’s the head office ASA that’s decided to use the “secondary” peer to establish the tunnel to (and it continued to use that secondary peer even when comms were restored to the primary i.e. when the ISP's "internet outage" was over)  What I want to know is whether there is a way for the ASA to “prefer” one peer over the other and, in the event of a failure, whether I can have the ASA automatically fall back to the "primary" peer once comms are re-established to it?  (Or any other way round this?)

Does that make more sense?  Sorry, it’s quite difficult to explain!

Ps just in case anyone is thinking “why are you bothering with resilience for the remote office, what about your head office?” we do already have resilience for our main head office circuits too!

What I am havin trouble with is this:-

The routers are running EIGRP and HSRP - fine

They have VPN tunnels - with NO dynamic routing running.

so IF the VPN tunnel from router 1 is OK - but the ASA in the head office decides its dead and creates a VPN connection to router 2, where is the logical failover between router 1 and 2 that they decide between them that router 1 is still the primary local router, but it passes off traffic to the head office via router 2 - this makes no sense to me?

Unless you either have some monitoring or dynamic failover routing?

Yes, that’s what I don’t understand either.  But that’s what seems to have happened.  (As is typical with these kind of things, when the incident occurred, I was concentrating more on getting service back up and running again properly than finding out what had actually happened but what seemed to be happening was that the tunnel between the secondary remote router and the ASA was established and stayed up whereas it looked like the primary router was trying to build a tunnel to the ASA again but it was constantly being torn down each time?)  Like you, I’m confused as to why/how the traffic from the remote site could still be directed via the secondary router when there were no HSRP or EIGRP changes at that end? (Having said that, the site was complaining of “slowness/connectivity” issues so it may have been that, despite the tunnel being “up” on the secondary router, traffic was not actually flowing properly through it from the remote site?)

Maybe I’ll rephrase my original question slightly though:

·         In a set-up like ours, what should happen if the ASA detects the 1st peer in its list as down and moves onto the 2^nd peer?

·         When the 1st peer becomes available again, what should happen from the ASA perspective?

I still get the feeling (or maybe it’s just clutching at straws!) that there is something glaringly simple and obvious that I’m missing which is at  the root of all this.  However, I’m trying to organise a maintenance window at some stage so that I can look into it in more depth though as it seems there are too many questionmarks over what actually occurred at present.

Thanks for your advice (and patience!) so far.

Message was edited by: mitchen - corrected typo - should have been "when the 1st peer becomes available again" NOT "when the 2nd peer becomes available again"

Thanks (and yes, sorry, that was a typo above, I meant the "1st peer"). So, forgetting about the initial problem just for a moment and trying to clarify exactly what should happen in this set-up...

If 1st peer is marked as failed, ASA builds tunnel to 2nd peer - this means, in a situation where there is a problem with the remote site's primary router or circuit, the solution will work in that the ASA will no longer be able to detect the 1st peer so will failover to the 2nd peer and HSRP/EIGRP at the remote site will also failover to the secondary router so comms will be established successfully via the slower ADSL link.

However, the problem with my set-up will be when the primary router or circuit returns to service again (and HSRP/EIGRP at the remote site updates accordingly)  The remote site will be trying to send traffic via the primary router again while the ASA will still be trying to use the 2nd peer.  Is that correct?  So what would happen in this scenario?  Presumably there will be a conflict of some sort and that would affect the comms between the remote site and head office?

And, what I'm trying to establish is - is there any way round this from the ASA perspective? i.e. is there any way I could get it to recognise that when the 1st peer becomes available again that it should prefer it over the 2nd peer?  If not, is there a simple manual workaround e.g. could I simply clear the secondary peer SA using "clear crypto ipsec sa peer w.x.y.z" from the ASA CLI to have it rebuild the tunnel using the 1st peer again (rather than having to remove the 2nd peer from the config altogether to force it back over to the 1st peer!)