VSOM 6.3.2 - RAM + Swap practically full

[xsijwl_spiac]

Hello,

All is in the title.

Do you have any idea the problem ?

[Scott Olsen]

Interesting.  Could we get a bit more information?

What MSP model?  1RU, 2RU, etc?

How many I/O streams are flowing through the box?  (Both inbound proxys, and outbound "MediaOut" viewing)

Are all the proxy streams configured identically? 

Based on the snapshot above, I'd venture to guess that the proxy process under PID 10506 has "gone rogue" based on its metrics.

If you have a support agreement, I'd recommend opening a TAC case.  Perhaps they would want to investigate.

If not, I can propose two different courses of action:

1 - Quick and Dirty - '/etc/init.d/cisco stop && /etc/init.d/cisco start' to punt services.

2 - Surgical (based on available information) - 'kill -11 10506' this will kill the bloated proxy process, create a crashdump for it, and VSMS will spawn a new one.

Let me know either way... I'm curious!

Cheers,

Scott.

[harsandh]

Scott idea is a good one.

Are you get any Core Dumps?

# ls -l /usr/BWhttpd/cores

What is the bit rate on that proxy that it is take such high memory?

Something is not right ..

[xsijwl_spiac]

Hi !

Thanks for your response.

Interesting.  Could we get a bit more information?

Yes

What MSP model?  1RU, 2RU, etc?

2RU

How many I/O streams are flowing through the box?  (Both inbound proxys, and outbound "MediaOut" viewing)

Are all the proxy streams configured identically? 

Yes

Based on the snapshot above, I'd venture to guess that the proxy process under PID 10506 has "gone rogue" based on its metrics.

How trace this proxy with configured camera ?

If you have a support agreement, I'd recommend opening a TAC case.  Perhaps they would want to investigate.

I opening a TAC case, but I have not yet received a response

If not, I can propose two different courses of action:

1 - Quick and Dirty - '/etc/init.d/cisco stop && /etc/init.d/cisco start' to punt services.

Use

2 - Surgical (based on available information) - 'kill -11 10506' this will kill the bloated proxy process, create a crashdump for it, and VSMS will spawn a new one.

Not Use

Let me know either way... I'm curious!

######################################################################################

Scott idea is a good one.

Are you get any Core Dumps?

# ls -l /usr/BWhttpd/cores

total 94M

-rw------- 1 root nobody 134M Nov 22 10:59 core-httpd.17296

What is the bit rate on that proxy that it is take such high memory?

Something is not right ..

Thanks a lot !

[Balakumar Tj]

I have cores dump issue, Should I delete all these files, will it create any issues in vsom? Please help

:/usr/BWhttpd/cores # ls -ltrh

total 6.2G

-rw------- 1 root root 168M Nov 28 12:52 core-MediaOut.4089

-rw------- 1 root root  56M Nov 28 13:04 core-MediaOut.23210

-rw------- 1 root root  60M Dec  2 18:13 core-MediaOut.24953

-rw------- 1 root root 209M Feb  4 18:20 core-MediaOut.28248

-rw------- 1 root root 169M Feb  5 11:15 core-MediaOut.11399

-rw------- 1 root root 238M Feb  5 15:40 core-MediaOut.26571

-rw------- 1 root root 201M Feb  6 09:54 core-MediaOut.16854

-rw------- 1 root root 217M Feb  6 13:23 core-MediaOut.3638

-rw------- 1 root root 246M Feb  6 15:50 core-MediaOut.3662

-rw------- 1 root root  54M Feb  6 19:41 core-MediaOut.10718

-rw------- 1 root root  82M Feb  7 08:14 core-MediaOut.31558

-rw------- 1 root root 158M Feb  7 09:46 core-MediaOut.2340

-rw------- 1 root root 143M Feb  7 10:04 core-MediaOut.682

-rw------- 1 root root 232M Feb  7 11:00 core-MediaOut.10039

-rw------- 1 root root 225M Feb  7 15:41 core-MediaOut.16259

-rw------- 1 root root  59M Feb  7 19:04 core-MediaOut.307

-rw------- 1 root root  40M Feb  8 11:23 core-MediaOut.28113

-rw------- 1 root root  39M Feb  8 11:26 core-MediaOut.28396

-rw------- 1 root root  27M Feb  8 12:22 core-MediaOut.28898

-rw------- 1 root root  24M Feb  8 12:22 core-MediaOut.4841

-rw------- 1 root root  26M Feb  8 12:23 core-MediaOut.4890

-rw------- 1 root root  26M Feb  8 12:25 core-MediaOut.5036

-rw------- 1 root root  25M Feb  8 12:26 core-MediaOut.5360

-rw------- 1 root root  25M Feb  8 12:27 core-MediaOut.5512

-rw------- 1 root root  25M Feb  8 12:28 core-MediaOut.5638

-rw------- 1 root root  25M Feb  8 12:34 core-MediaOut.5724

-rw------- 1 root root  26M Feb  8 14:26 core-MediaOut.6683

-rw------- 1 root root  25M Feb  8 14:27 core-MediaOut.21107

-rw------- 1 root root  25M Feb  8 14:27 core-MediaOut.21181

-rw------- 1 root root  26M Feb  8 14:45 core-MediaOut.21231

-rw------- 1 root root  26M Feb  8 14:47 core-MediaOut.24108

-rw------- 1 root root  25M Feb  8 14:49 core-MediaOut.24382

-rw------- 1 root root  26M Feb  9 20:31 core-MediaOut.24616

-rw------- 1 root root  25M Feb  9 20:33 core-MediaOut.21111

-rw------- 1 root root  26M Feb  9 20:51 core-MediaOut.21328

-rw------- 1 root root  25M Feb  9 21:03 core-MediaOut.24057

-rw------- 1 root root 225M Feb 10 11:33 core-MediaOut.25711

-rw------- 1 root root 268M Feb 10 14:00 core-MediaOut.22339

-rw------- 1 root root 252M Feb 10 14:07 core-MediaOut.26844

-rw------- 1 root root 230M Feb 10 14:10 core-MediaOut.32549

-rw------- 1 root root 213M Feb 10 14:15 core-MediaOut.3187

-rw------- 1 root root 206M Feb 10 14:16 core-MediaOut.7385

-rw------- 1 root root 262M Feb 10 15:37 core-MediaOut.8419

-rw------- 1 root root 123M Feb 10 16:34 core-MediaOut.20799

-rw------- 1 root root  38M Feb 10 19:17 core-MediaOut.21550

-rw------- 1 root root  27M Feb 10 22:33 core-MediaOut.21130

-rw------- 1 root root 149M Feb 11 09:48 core-MediaOut.17415

-rw------- 1 root root 274M Feb 11 15:38 core-MediaOut.4229

-rw------- 1 root root 226M Feb 11 15:39 core-MediaOut.11924

-rw------- 1 root root 223M Feb 11 15:42 core-MediaOut.12732

-rw------- 1 root root  63M Feb 11 18:34 core-MediaOut.15013

-rw------- 1 root root  55M Feb 12 08:17 core-MediaOut.7053

-rw------- 1 root root  74M Feb 12 08:47 core-MediaOut.26753

-rw------- 1 root root 128M Feb 13 10:57 core-MediaOut.4184

-rw------- 1 root root 146M Feb 13 11:27 core-MediaOut.24358

-rw------- 1 root root 133M Feb 13 11:52 core-MediaOut.8537

-rw------- 1 root root 102M Feb 13 12:04 core-MediaOut.21834

-rw------- 1 root root 188M Feb 14 06:04 core-MediaOut.26702

-rw------- 1 root root 197M Feb 14 06:23 core-MediaOut.19684

-rw------- 1 root root 205M Feb 14 08:06 core-MediaOut.850

-rw------- 1 root root 178M Feb 14 08:41 core-MediaOut.12013

-rw------- 1 root root 171M Feb 14 09:25 core-MediaOut.2545

-rw------- 1 root root 235M Feb 14 09:58 core-MediaOut.30856

-rw------- 1 root root 193M Feb 14 09:59 core-MediaOut.22961

-rw------- 1 root root 216M Feb 14 10:45 core-MediaOut.24054

-rw------- 1 root root 190M Feb 14 11:23 core-MediaOut.29586

-rw------- 1 root root 150M Feb 14 11:26 core-MediaOut.25222

-rw------- 1 root root    0 Feb 14 11:35 core-MediaOut.27255

[Scott Olsen]

Hi there Balakumar,

This should really be a new post, but I'll offer some feedback regardless;

Unless you are submitting these cores to TAC for debugging, they really aren't of much use to you.  I'd be more concerned with the frequency with which they are accumulating, and the disk space that they are using up on your root file system.  If you allow that disk to fill... bad things will happen.  Given the size of some of them, I'd say you are already encountering this.

If you want to play it safe, pull them off with WinSCP or FileZilla, or any other transport method you have (so as to have a backup of them), then delete them from the VSM server.

You may also want to grab a support report from the server in question and open a TAC case regardless.  In my opinion, something strange is certainly happening to be causing all the MediaOut process crashing.

Best of luck,

Scott Olsen
Solutions Specialist
Bulletproof Solutions Inc.
Web: www.bulletproofsi.com

[Balakumar Tj]

Thanks Scott, Yes you are correct, My users were already facing issues

Users not able to see the live videos and get DB Error.

I have removed the cores dump, now it seems good. Thanks for your help.