Cisco Support Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Announcements

Welcome to Cisco Support Community. We would love to have your feedback.

For an introduction to the new site, click here. And see here for current known issues.

New Member

VSM 7.5 Upgrade Application Failed Still initializing

After upgrading from 7.1 to 7.5 and all reports appeared fine when trying to login to the VSOM page I get the attached error. It never actually initializes after 24 hours. I can login to VSM and see 2 of 3 service sucessfully started. The Operations Manager overall status report critical. Image 2 attached. Has anyone done and upgrade from 7.1 and seen this and come up with a fix?

Thanks for any input.

Everyone's tags (1)
1 ACCEPTED SOLUTION

Accepted Solutions
Cisco Employee

cwhill,   Looks like you

cwhill,

   I did some additional digging for you and it looks like yesterday 3/15 a bug was filed:

https://tools.cisco.com/bugsearch/bug/CSCun40024

The work around is locted in the link if you feel comfortable executing it. If not feel free to open up a TAC case and we will be happy to get you up and running.

15 REPLIES
New Member

Some more info in case this

Some more info in case this helps for advice. When I looked at the catalina.log 

/usr/BWhttpd/tomcat/logs/catalina.log

I see this at the tail of the log.

caused by: com.cisco.psbu.vs.ism.model.services.VsomException: Operation failed:

Cisco Employee

in the CLI run:sudo su

in the CLI run:

sudo su -

service cisco status

------------------------

You will most likely see multiple services not running. If you do see some services not running

run:

service cisco restart

---------------------------

this can take up to ten minutes. Please let us know the outcome.

New Member

[localadmin@VSOM01 ~]$ sudo

[localadmin@VSOM01 ~]$ sudo su -
Thank you so much for some help on this. Its driving me crazy. I'm suprised others haven't seen this issue as well. I have a fairly new, clean install and this was the first upgrade so there isn't alot of baggage on this server yet. I thought it would have went clean. It looks like all services are running. I pasted the console output below.

[root@VSOM01 ~]# service cisco status
Checking for VSOM database server ... Running.
Checking for tomcat...Running.
Checking for VSOM... running.
Checking for database server ... Running.
Checking for event engine daemon... Running.
Checking for snmp daemon... Running.
Checking for Snmp Server... Running.
Checking for http server... Running.
Checking for LocalAdmin Server... Running.
Checking for lighttpd... Running.
Checking for msi rest... Running.
Checking for msi daemon... Running.
Checking for pending upgrade request
    No Pending Upgrade Request
Checking for vsrecorder
    Checking for xvcrmanMonitor... Running.
    Checking for xvcrman... Running.
    Checking for gap server... Running.
Checking for vsms daemons... running.
Checking for scheduler daemon... Running.
Checking for FailoverServer... Running.
Checking for Storage Monitor... Running.
Checking for MediaOut Handler... Running.
Checking for DiscoveryServer... Running.
Checking for Backup Server... Running.
Checking for CmapiServer... Running.
Checking for Analytics Server... Running.
Checking for Analytics Metadata Handler... Running.
Checking for External Trigger Server... Running.
Checking for Device Info Server... Running.
Checking for mrtg... Running.
Checking for System Monitor... Running.

 

New Member

Anyone else with thoughts on

Anyone else with thoughts on the issue? I'm going to try tech support at Cisco today to see if I can get anywhere with that. I was hoping to avoid that.  =(

New Member

Dear All,

Dear All,

I tried the same given in this link  https://bst.cloudapps.cisco.com/bugsearch/bug/CSCun40024

Also restarted the cisco service but to no avail. Please do help me.

I version of softwares installed is

 rpm -aq | grep Cisco
Cisco_VSBase-7.5.1-043d
Cisco_VSMS-7.5.1-043d
Cisco_VSDrivers-7.5.1-043d
Cisco_Tomcat-7.0.37-2
Cisco_VSRecorder-7.5.1-043d
Cisco_MPClient-7.5.0-24
Cisco_SASD-7.5.0-26
Cisco_VSOM-7.5.1-4
Cisco_AMQBroker-7.5.1-1
Cisco_VSMC-7.5.1-004
Cisco_VSTools-7.5.1-043

When checking the logs catalina.log, I get to see the following

Caused by: com.cisco.psbu.vs.ism.model.services.VsomException: Operation failed: Application is still initializing. Try again after some time
        ... 25 more

Please help me.

New Member

In detail, I get the error in

In detail, I get the error in catalina.log file as follows

May 17, 2017 4:33:00 PM org.apache.cxf.binding.http.interceptor.DispatchInterceptor handleMessage
INFO: Invoking POST on /getServiceStatus
May 17, 2017 4:33:00 PM org.apache.cxf.phase.PhaseInterceptorChain doIntercept
INFO: Interceptor has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault: Operation failed: Application is still initializing. Try again after some time
        at com.cisco.psbu.vs.ism.interceptors.LoginInterceptor.createFault(LoginInterceptor.java:218)
        at com.cisco.psbu.vs.ism.interceptors.LoginInterceptor.handleMessage(LoginInterceptor.java:108)
        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:236)
        at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:89)
        at org.apache.cxf.transport.servlet.ServletDestination.invoke(ServletDestination.java:99)
        at com.cisco.psbu.vs.ism.main.VsomServletController.invokeDestination(VsomServletController.java:551)
        at com.cisco.psbu.vs.ism.main.VsomServletController.invoke(VsomServletController.java:136)
        at org.apache.cxf.transport.servlet.AbstractCXFServlet.invoke(AbstractCXFServlet.java:163)
        at org.apache.cxf.transport.servlet.AbstractCXFServlet.doPost(AbstractCXFServlet.java:141)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Caused by: com.cisco.psbu.vs.ism.model.services.VsomException: Operation failed: Application is still initializing. Try again after some time

Please help.

New Member

error in vsom_be.log

error in vsom_be.log

2017-05-17 16:02:06,600 [Thread-38] ERROR c.c.p.v.i.s.u.AuditLogMgmtUtil: Internal Error
org.springframework.jdbc.UncategorizedSQLException: CallableStatementCallback; uncategorized SQLException for SQL [{call partitionAuditLogs(?, ?)}]; SQL state [HY000]; error code [1526]; Table has no partition for value 1495015326000; nested exception is java.sql.SQLException: Table has no partition for value 1495015326000
        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:83) ~[spring-jdbc-2.5.6.jar:2.5.6]
        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80) ~[spring-jdbc-2.5.6.jar:2.5.6]
        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80) ~[spring-jdbc-2.5.6.jar:2.5.6]
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:952) ~[spring-jdbc-2.5.6.jar:2.5.6]
        at org.springframework.jdbc.core.JdbcTemplate.call(JdbcTemplate.java:985) ~[spring-jdbc-2.5.6.jar:2.5.6]
        at org.springframework.jdbc.object.StoredProcedure.execute(StoredProcedure.java:117) ~[spring-jdbc-2.5.6.jar:2.5.6]
        at com.cisco.psbu.vs.ism.services.auditlog.impl.AuditLogMgmt.partitionAuditLogs(AuditLogMgmt.java:101) ~[AuditLogMgmt.class:na]
        at com.cisco.psbu.vs.ism.services.util.AuditLogMgmtUtil.partitionAuditLogs(AuditLogMgmtUtil.java:335) ~[AuditLogMgmtUtil.class:na]
        at com.cisco.psbu.vs.ism.services.internal.VsomServicesInitializer.vsomRestarted(VsomServicesInitializer.java:153) [VsomServicesInitializer.class:na]
        at com.cisco.psbu.vs.ism.main.BootServlet$VsomInitializer.run(BootServlet.java:78) [BootServlet$VsomInitializer.class:na]
        at java.lang.Thread.run(Unknown Source) [na:1.7.0_17]
Caused by: java.sql.SQLException: Table has no partition for value 1495015326000
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1055) ~[mysql-connector-java-5.1.6.jar:na]
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956) ~[mysql-connector-java-5.1.6.jar:na]
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3491) ~[mysql-connector-java-5.1.6.jar:na]
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3423) ~[mysql-connector-java-5.1.6.jar:na]
        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1936) ~[mysql-connector-java-5.1.6.jar:na]
        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2060) ~[mysql-connector-java-5.1.6.jar:na]
        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2542) ~[mysql-connector-java-5.1.6.jar:na]
        at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1734) ~[mysql-connector-java-5.1.6.jar:na]
        at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:995) ~[mysql-connector-java-5.1.6.jar:na]
        at com.mysql.jdbc.CallableStatement.execute(CallableStatement.java:859) ~[mysql-connector-java-5.1.6.jar:na]
        at com.mchange.v2.c3p0.impl.NewProxyCallableStatement.execute(NewProxyCallableStatement.java:3044) ~[c3p0-0.9.1.jar:0.9.1]
        at org.springframework.jdbc.core.JdbcTemplate$5.doInCallableStatement(JdbcTemplate.java:987) ~[spring-jdbc-2.5.6.jar:2.5.6]
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:936) ~[spring-jdbc-2.5.6.jar:2.5.6]
        ... 7 common frames omitted
2017-05-17 16:02:06,602 [Thread-38] ERROR c.c.p.v.i.m.BootServlet: Error in handling vsomRestarted
com.cisco.psbu.vs.ism.model.services.VsomException: System error: Server error. Cause org.springframework.jdbc.UncategorizedSQLException CallableStatementCallback; uncategorized SQLException for SQL [{call partitionAuditLogs(?, ?)}]; SQL state [HY000]; error code [1526]; Table has no partition for value 1495015326000; nested exception is java.sql.SQLException: Table has no partition for value 1495015326000
        at com.cisco.psbu.vs.ism.services.util.AuditLogMgmtUtil.partitionAuditLogs(AuditLogMgmtUtil.java:350) ~[AuditLogMgmtUtil.class:na]
        at com.cisco.psbu.vs.ism.services.internal.VsomServicesInitializer.vsomRestarted(VsomServicesInitializer.java:153) ~[VsomServicesInitializer.class:na]
        at com.cisco.psbu.vs.ism.main.BootServlet$VsomInitializer.run(BootServlet.java:78) ~[BootServlet$VsomInitializer.class:na]
        at java.lang.Thread.run(Unknown Source) [na:1.7.0_17]

Please help.

Silver

My initial concern is that

My initial concern is that your VSOM DB is corrupt.  Do you have a recent backup you could restore with via the VSMC/CDAF console?

Scott Olsen Solutions Specialist Bulletproof Solutions Inc. Web: www.bulletproofsi.com
New Member

Hi Scott,

Hi Scott,

Thank you for the advice. Well ! I don't have any backup of VSOM DB. What I am thinking is to reset the whole lot and begin the fresh recording/DB. Or can I have the backup of the VSOM DB, whether it's corrupted or not ?

Please advice.

Regards

New Member

Dear Team,

Dear Team,

I am still with the same issue. Please do help.

Or else, please do help me to reset the DB from scratch so that I can do away with the corrupted DB and have a fresh one please.

Regards

Cisco Employee

cwhill,   Looks like you

cwhill,

   I did some additional digging for you and it looks like yesterday 3/15 a bug was filed:

https://tools.cisco.com/bugsearch/bug/CSCun40024

The work around is locted in the link if you feel comfortable executing it. If not feel free to open up a TAC case and we will be happy to get you up and running.

New Member

Thanks so much. I'll call an

Thanks so much. I'll call an open up a case on it! I aprpeciate the help and the response.

Silver

This is terrific feedback!I'm

This is terrific feedback!

I'm also amazed someone managed to find an actual public facing bug where the contents aren't restricted.  

Could there actually be positive change happening?  Here's to hoping!

Scott Olsen Solutions Specialist Bulletproof Solutions Inc. Web: www.bulletproofsi.com
New Member

Thank you again Branden for

Thank you again Branden for the help in resolving this. Things are running great again.

New Member

Dear Branden,

Dear Branden,

The solution didn't work me, it is still giving same problem .  Please find the output of 

tail -f /usr/BWhttpd/tomcat/logs/catalina.log
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
    at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
    at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
    at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: com.cisco.psbu.vs.ism.model.services.VsomException: Operation
failed: Application is still initializing. Try again after some time
    ... 25 more

Please advise me to resolve this issue.

Regards,

1196
Views
10
Helpful
15
Replies
CreatePlease login to create content