07-30-2014 01:43 PM - edited 03-01-2019 06:01 AM
Communication between the various nodes and processes in the ACI Fabric uses Inter-Fabric Messaging (IFM). IFM uses SSL-encrypted TCP communication. Each APIC and fabric node has 1024-bit SSL keys embedded in secure storage. The SSL certificates are signed by Cisco Manufacturing Certificate Authority (CMCA).
Problems with IFM SSL communication can prevent fabric nodes from successfully joining the fabric. Some of the symptoms include:
admin@APIC-B-1:~> acidiag fnvread ID Name Serial Number IP Address Role State LastUpdMsgId ------------------------------------------------------------------------------------------------- 101 leaf-1-B SAL00000001 10.0.117.160/32 leaf active 0 102 leaf-2-B SAL00000002 10.0.117.191/32 leaf active 0 161 spine-1-B FGE00000003 10.0.117.190/32 spine inactive 0x100 Total 3 nodes admin@APIC-B-1:~>
User Access Verification
spine-1-B login: admin
************************************************** ******************************
Fabric discovery in progress, show commands are not fully functional
Logout and Login after discovery to continue to use show commands.
************************************************** ******************************
spine-1-B#
spine-1-B# netstat -ant | grep :12
tcp 0 0 10.0.117.190:12439 0.0.0.0:* LISTEN
tcp 0 0 10.0.117.190:12119 0.0.0.0:* LISTEN
tcp 0 0 10.0.117.190:12887 0.0.0.0:* LISTEN
tcp 0 0 10.0.117.190:12151 0.0.0.0:* LISTEN
tcp 0 0 10.0.117.190:12407 0.0.0.0:* LISTEN
tcp 0 0 10.0.117.190:12183 0.0.0.0:* LISTEN
tcp 0 0 10.0.117.190:12440 0.0.0.0:* LISTEN
tcp 0 0 10.0.117.190:12120 0.0.0.0:* LISTEN
tcp 0 0 10.0.117.190:12888 0.0.0.0:* LISTEN
tcp 0 0 10.0.117.190:12152 0.0.0.0:* LISTEN
tcp 0 0 10.0.117.190:12408 0.0.0.0:* LISTEN
tcp 0 0 10.0.117.190:12184 0.0.0.0:* LISTEN
spine-1-B#
Problems with IFM SSL communication are logged in the following logs:
For example, looking at the logs on the fabric node shows:
spine-1-B# tail -f svc_ifc_policyelem.log | grep SSL 3910||14-06-19 21:40:11.748+00:00||ifm||DBG4||co=ifm||openssl error during SSL_accept()||../dme/common/src/ifm/./IFMSSL.cc||173 3910||14-06-19 21:40:11.748+00:00||ifm||DBG4||co=ifm||openssl: error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca||../dme/common/src/ifm/./IFMSSL.cc||178
Likewise if we check the logs on the APIC shows:
admin@APIC-B-1:log> tail -f svc_ifc_appliancedirector.bin.log | grep SSL 32509||14-06-19 06:42:21.707+00:00||ifm||DBG4||co=ifm||openssl error during SSL_connect()||../common/src/ifm/./IFMSSL.cc||173 32509||14-06-19 06:42:21.707+00:00||ifm||DBG4||co=ifm||openssl: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed||../common/src/ifm/./IFMSSL.cc||178
In this example there is a problem with the SSL certificate verification. You can view the details of the SSL communication by using the openssl command from the APIC CLI. First we'll check the APIC itself by using openssl to connect internally:
admin@APIC-B-1:~> openssl s_client -state -connect 10.0.0.1:12151 CONNECTED(00000003) < output removed for brevity> --- Certificate chain 0 s:/C=US/ST=CA/L=SanJose/O=Insieme Networks/CN=Insieme i:/C=XX/L=Default City/O=Default Company Ltd 1 s:/C=XX/L=Default City/O=Default Company Ltd i:/C=XX/L=Default City/O=Default Company Ltd --- Server certificate -----BEGIN CERTIFICATE----- MIICkzCCAXsCCQCGN3jYiLGTgzANBgkqhkiG9w0BAQUFADBCMQswCQYDVQQGEwJY WDEVMBMGA1UEBwwMRGVmYXVsdCBDaXR5MRwwGgYDVQQKDBNEZW ZhdWx0IENvbXBh bnkgTHRkMB4XDTEzMDcxOTIwNDAzMloXDTE0MDcxOTIwNDAzMl owWTELMAkGA1UE BhMCVVMxCzAJBgNVBAgMAkNBMRAwDgYDVQQHDAdTYW5Kb3NlMR kwFwYDVQQKDBBJ bnNpZW1lIE5ldHdvcmtzMRAwDgYDVQQDDAdJbnNpZW1lMIGfMA 0GCSqGSIb3DQEB AQUAA4GNADCBiQKBgQC8y/ZR+pPVKQs69rc9t3/BRq+rLsYjAq hy6lz4rIxBO1sW 2eRXXX8BgvzDeeHqEjQiyjHvTUpybUDZ7ES1tF6sv3lsMgJAJi AuG3ecfWarh7qf 6vKx8U6H/shZDjQP4kCv70oMm1gdC3yonuLwJhwBGK0McHEifm YcSucSUUSavQID AQABMA0GCSqGSIb3DQEBBQUAA4IBAQA82c2iAKwbpdgxk7PxqV RnREh5C2E5xEO2 ermIPsNjSXtQwEnjmvts003KPzkx7ND1SLatw39fGL5ZksSOjm Kc7a303rDqh2qq nAvjyIpEshBzJxXJdhZyPI6L4j6a+0yMuXYpGMlezfFrRFImrf Fs26+W4bcdL+x2 Yo5Ez7KEauvim2yj7TSERJB3s+r3nRUCCGfquPIBfPWrXzbYzj 2rXv/QPoR8W/xk A667jKPHtC7SAzIKWZjyiRwi1yVs4vDkyMyFxCurMiiF3A/gTj x/097L0JYX/MJK 7lkuIod1WG3sKtQbbc+Uh6GdZkHQghAIHukDhzyckPhvqLxmWj 48 -----END CERTIFICATE----- subject=/C=US/ST=CA/L=SanJose/O=Insieme Networks/CN=Insieme issuer=/C=XX/L=Default City/O=Default Company Ltd < output removed for brevity>
In the output above, openssl is used to connect to a listening TCP port using IFM (12151) locally on the same device where the command is being run. The specific TCP port used isn't important, so long as it is associated with a DME server using IFM SSL (use 'netstat -an | grep :12' to find one).
The output shows that the certificates used by the APIC are not valid CMCA certificates. Checking the fabric node that is in an inactive state shows:
admin@APIC-B-1:~> openssl s_client -state -connect spine-1-B:12440 CONNECTED(00000003) <output removed for brevity> --- Certificate chain 0 s:/serialNumber=PID:N9K-C9508 SN:FGE00000003/CN=FGE00000003 i:/O=Cisco Systems/CN=Cisco Manufacturing CA 1 s:/O=Cisco Systems/CN=Cisco Manufacturing CA i:/O=Cisco Systems/CN=Cisco Root CA 2048 2 s:/O=Cisco Systems/CN=Cisco Root CA 2048 i:/O=Cisco Systems/CN=Cisco Root CA 2048 --- Server certificate -----BEGIN CERTIFICATE----- MIIDuzCCAqOgAwIBAgIKGRJjSAAAAAbyczANBgkqhkiG9w0BAQUFADA5MRYwFAYD VQQKEw1DaXNjbyBTeXN0ZW1zMR8wHQYDVQQDExZDaXNjbyBNYW 51ZmFjdHVyaW5n IENBMB4XDTE0MDUyNjE0MTg0OVoXDTI0MDUyNjE0Mjg0OVowPT ElMCMGA1UEBRMc UElEOk45Sy1DOTUwOCBTTjpGR0UxODE4MEFDWTEUMBIGA1UEAx MLRkdFMTgxODBB Q1kwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAM8HvlkZVs NPMZ9mwBuiM5yD YNIlf4Qwm/J01Z2zzPaSFFy4nZweqq0OVWPOgEeadKPShLLsJO LnRPxVYFZLyOU2 gmCmrch3HcGJks0YrLW+s7XbeAzx6JDggumoO03C64eKEy9HdS dzc2VM3j5wVBiF 8+de253WptdRzd6Nj+KPAgMBAAGjggFDMIIBPzAOBgNVHQ8BAf 8EBAMCBPAwHQYD VR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMB0GA1UdDgQWBB TWbUbPa7vySop5 y9capeR1+OgvFjAfBgNVHSMEGDAWgBTQxSImq09GYOyuBZHH3F rRsEf3bDA/BgNV HR8EODA2MDSgMqAwhi5odHRwOi8vd3d3LmNpc2NvLmNvbS9zZW N1cml0eS9wa2kv Y3JsL2NtY2EuY3JsMEwGCCsGAQUFBwEBBEAwPjA8BggrBgEFBQ cwAoYwaHR0cDov L3d3dy5jaXNjby5jb20vc2VjdXJpdHkvcGtpL2NlcnRzL2NtY2 EuY2VyMD8GCSsG AQQBgjcUAgQyHjAASQBQAFMARQBDAEkAbgB0AGUAcgBtAGUAZA BpAGEAdABlAE8A ZgBmAGwAaQBuAGUwDQYJKoZIhvcNAQEFBQADggEBAGrKs3J8j8 /cNXk9INw2nD/J 74Am55POxP4+ZSDS7v+6U8pRvykQIw8YuPoTWOXzd4JGfli838 poVBjhNoMg10Vx FKZ+UUSH6mYQr6+COAKo7KRVgsRUncla5m+ibf+vJNGKyO3tIc pYPLUPdCUo5u95 QJ0Vs8wYHcXbLkVrDKDm0J6S6oKJg4QE6DLqlXHNArHIcMUwlE tB0TdTaJKgSD7Y 4gOTsSuuFExbtfhbFRooJCOPCR+kwyHR0zQjOvAcDtWNtiKomW 5Nij2sWQO+nAJJ 8OarjXW6459k25CYumaF8LXCEdrFGWnmN5iICF4ngDl37wT0mx S8v/gjksAdrjk= -----END CERTIFICATE----- subject=/serialNumber=PID:N9K-C9508 SN:FGE00000003/CN=FGE00000003 issuer=/O=Cisco Systems/CN=Cisco Manufacturing CA <output removed for brevity>
In this output, the inactive fabric node is using certificates signed by the Cisco Manufacturing Certificate Authority, which is expected. Certificates must meet the following criteria:
In this case there are mismatched certificates between the APIC and fabric node that is inactive. Incidentally, since there are other fabric nodes properly registered with the fabric in this example, those nodes are also using invalid certificates. The invalid certificates need to be updated with valid CMCA certificates.
Hi facing same issue in some leafs,
working ones: Certificate chain
0 s:/serialNumber=PID:N9K-C9396PX SN:SAL18474R7Q/CN=SAL18474R7Q
i:/O=Cisco Systems/CN=Cisco Manufacturing CA
1 s:/O=Cisco Systems/CN=Cisco Manufacturing CA
i:/O=Cisco Systems/CN=Cisco Root CA 2048
2 s:/O=Cisco Systems/CN=Cisco Root CA 2048
i:/O=Cisco Systems/CN=Cisco Root CA 2048
and failed ones:
Certificate chain
0 s:/C=US/ST=CA/L=SanJose/O=Insieme Networks/CN=Insieme
i:/C=XX/L=Default City/O=Default Company Ltd
How can we solve this? Do we need to contact TAC? or any way to solve, runninf 1.2.1m
You need to open a Cisco TAC Case so the ACI TAC engineer can use this information to generate the new CERT files for your switches in this condition.
Thank you for using the Cisco Support Forum for ACI.
T.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: