cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
23765
Views
45
Helpful
11
Replies

APIC Cluster - Why minimum of 3 controllers?

ericbkchng
Level 1
Level 1

Hi!

I'm just getting started on learning about the Cisco ACI, and one of the things that struck me was that Cisco has recommended (or mandated?) a minimum of 3 APICs in a cluster. Is this a requirement? If so, why can't we just have 2 controllers in a cluster?

In this webpage (http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/unified-fabric/white-paper-c11-730021.html), there was a discussion on the number of controllers vs data loss, but there is no explanation about why 2 controllers can't be used.

Thanks!

1 Accepted Solution

Accepted Solutions

mtimm
Cisco Employee
Cisco Employee

"with the cluster in this state it is unadvised to make changes in the GUI, i don't remember if its even allowed."  

 

Changes to policy are not allowed by the backend until the cluster is fully fit.  So it doesn't matter if you try to make changes via the GUI or the REST API or the CLI, they will not be allowed.

View solution in original post

11 Replies 11

Sebastian Pesic
Level 1
Level 1

Hi,

I think the recommended minimum is 3 APIC's in a cluster. I believe you can run less than that but it is best explained why 3 is the minimum recommendation in the whitepaper under "Effect of Replication on Reliability".

A good thing to understand is also that the Spine/Leaf infrastructure will still be able to process packets with policys previously configured even if all the APIC's malfunction but you will not be able to enforce new policys.

Hope that helps!

 

dpita
Cisco Employee
Cisco Employee

Hello 

To understand why three APICs is the recommended minimum you must understand how the APICs distribute information between the three. All parts of ACI are datasets generated and processed by the Distributed Policy Repository and that data for those APICs functions are partitioned into logically banded subsets called shards (like  DB shard). a Shard is then broken into three replicas or copies. each APIC has a replica for every shard but only 1 APIC is the master for a particular replica/shard. This is a way to distribute the workload evenly and load balance processing across the cluster of 3 as well as a fail safe in case an APIC goes down.

Now that the theory is out of the way, imagine one of your three APICs goes down. the remaining two will negotiate who will now be the master for the shards that the down APIC was in charge of. Workload is then load balance to the two and the cluster becomes fully fit again. Working with 2 APICs is really unadvised due to the split brain condition. This occurs when APIC 1 and APIC 2 thing they are both leaders for a shard and cannot agree so the shard is in contention and the cluster is unfit/"data layer partially diverged". with the cluster in this state it is unadvised to make changes in the GUI, i don't remember if its even allowed. 

With the case of only 1 APIC, that APIC does all the work, it is the leader for all shards but if it goes down then you can not make any changes at all. data plane will continue forwarding but since no APIC, theres no way to create new policies or changes. 

 

Thanks for using the support forums! hope this helps!

mtimm
Cisco Employee
Cisco Employee

"with the cluster in this state it is unadvised to make changes in the GUI, i don't remember if its even allowed."  

 

Changes to policy are not allowed by the backend until the cluster is fully fit.  So it doesn't matter if you try to make changes via the GUI or the REST API or the CLI, they will not be allowed.

dpita
Cisco Employee
Cisco Employee

Thanks Mike for clearing that up!  I didn't remember =)

My lab setup seems to run "fully fit" with 2 controllers.  See attached screen shot.  The cluster is setup for 3 but only two have ever been connected.  The fabric is showing fully fit and policy can be applied.  Just wanted to give "ericbkchng" confirmation that it works; the caveat would be that this should only be done in a lab or testing environment.

 

Here is a good article from Cisco's KB showing how a 2 APIC setup might be used during a site failure on an ACI Stretched Fabric.  http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_kb-aci-stretched-fabric.html

 

 

In future code you may see a warning/alert in the GUI to the affect of:

 

"Your Cluster contains less than 3 in-service Controllers. Please Backup the cluster and not utilize the fabric in its current state for production"

Hi dpita,

Thanks a lot for your detailed reply. I did some reading up myself about sharding and understand better about the whole concept. However, I'm still unclear about the split-brain condition... is it that if the 2 APICs fail to communicate with each other (but each are able to communicate with the fabric), then a policy update on one of the APICs could result in a scenario that the leaf switches are confused because they are receiving conflicting policy updates from 2 different APICs?

In relation to the above, I'd also like to ask if Cisco actually does allow us to set up a cluster with just 2 APICs, if the customer does not mind taking on the risk of a split-brain condition? Or is it that because Cisco has fixed the replication factor to be 3, we have to have at least 3 APICs in the cluster?

I read in the white paper "The APIC supports from 3 to 31 nodes in a cluster. The current recommended minimum is a 3-node cluster.", so I'm a bit confused about this as I'm tnot sure if "recommended" actually means "mandatory". 

Thanks!

 

Hello,

Technically there is nothing stopping you from running a two node cluster. It is just ill advised as you are putting yourself in a dangerous situation. I have a  feeling the setup script when building the APICs might not let you choose two, but again, not 100% sure. If you really wanted two, you might have to build the fabric as a three node cluster then shrink the size to two when you can log into to the GUI and register all the nodes, wait for everything to be "fully fit" then shrink the size.

The fact is, there is no hard limitation imposed by Cisco from running a two node cluster. When the need to replace an APIC comes around, the nodes must be removed sequentially therefore a cluster of two will occur at some point. 

A three node cluster is most definitely recommended and not mandatory, your fabric will still operate with one, or two (caveat with two is that your fabric is in a bubble so nothing could possibly happen to upset the APICs)

Finally, i would like to once again let you know it is really not advised that you run a two node cluster. You would be much better off with a single node cluster. 

Hope this helps!

 

Hi Dpita,

 

Thanks again for your detailed explanation. It is indeed possible to have a 2-node cluster situation occurring in the case that 1 APIC is down, now I realize that :)

But do you think you could help me understand why "1" is better than "2"? Is it just because of the split-brain condition? I don't quite get what exactly is the risk and possible problems that having 2 APICs in the cluster could cause, and what makes it worse off than having just 1 APIC.

Thank you!

Anyone has an explanation for my question as above? Thank you!

The valid size for your cluster is typically at 3 or more.  We only allow a single APIC in lab/test environments where redundancy is not required.  The next size the system will allow you to configure up from 1 is 3.  This means if you configure 3 but only have 2, the cluster will never be fully fit. 

Three or more cluster members for any production system.

Robert

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Save 25% on Day-2 Operations Add-On License