MSE stopped responding - Oracle DB doesn't shutdown
I assume we hit bug CSCud77760 on customer's MSE VA.
I tried to shutdown services but Oracle DB hasn't shutdown. I found there are lots of oracle processes running one of them is utilizing one core to 100%. If I try to connect to Oracle DB to shutdown it manually sqlplus / AS SYSDBA hangs connecting to DB (it's probably the step where scripts got frozen). Only sqlplus -prelim AS SYSDBA is possible to perform oradebug commands.
I'm not Oracle expert to find in dump/traces what process to kill to unlock DB and perform gracefull shutdown. Could someone give me advice how to shutdown the DB without breaking it down? I can eventually upload hanganalyze and systemstate dump but it's a big.
No resource issue because this is not pushed to production yet....out of 1000 AP's only 10 are synced to MSE. TAC case has been sitting for more than a month. Weekly I get an email with some log request.
I went ahead and did a fresh install on another VM machine. Also upgraded code. No issues for last month. BUT I do not have HA enabled so not sure how much is HA is causing Oracle issues.
As there were no useful answer for 2 days I started to find Oracle DB specialist in sw. devel. division of our company. He advised over the phone try to kill process what utilizes CPU but another process started utilizing CPU. Therefore I shared him connection to the system he looked at some process information. He told that Oracle is probably in state when it's going down but for some reason it cannot there is no other way then kill it and I didn't worry because Oracle is really resilient. He killed process called Oracle "something" manager - all Oracle processes went down immediately. Then he started Oracle it recovered DB from transaction log. We verified DB was absolutely OK. Then I just applied CSCud77760 workaround setting, shutdown DB and started MSE.
It was resolved just in 20 minutes what is unattainable with TAC as probably everybody who is in regular connection with it can confirm ;-)
Transferring Crash file from standby: Login to the Active WLC in HA.
From CLI: (Cisco Controller) >transfer upload datatype crash (Cisco
Controller) >transfer upload filename (Cisco
Controller) >transfer upload mode tftp (Cisco Controller) >transfer
This is the start of a display filter cross reference between Wireshark
and OmniPeek. The 1st installment is a table of advanced filters. More
filters will be added as time allows. It is a living doc, so check back
for changes every so often Please feel f...