Things that go *bump* in the DC

[Leo Laohoo]

NOTICE:  The intention for this thread is for anyone to post anything "bad" that happened in your DC. 

Why you should never let a non-IT person into a DC

Recently (as in August 2013), the people who manages our three DC escorted a few non-IT people into the DC.  (Politics was the ONLY reason why they are there.  Their "task" or intention event was to go through each and everything with a power plug and note down the asset number and see if it's the list.  If it's not, it's "ammunition" against the BU that manages or owns the appliance(s).)

Anyway, the DC has one outter door and an inner door.  The group was about to walk into the inner door when one of the asset guy went, "Hold on, I forgot someone in the vehicle."  He turns around and pushes a RED button, which he hoped was the door release.  Anyone want to fanthom what a RED button in a DC (picture below)?

Literally, everything went quiet.  And dark.

The second non-IT guy went, "Did we just do something BAD?"  The DC guy, a large guy you don't want to meet in a dark alley, goes "Yes.  The bloke just cut ALL POWER to the DC."

To which the guy-who-pushed-the-button then responded with, "So when can we go in and do OUR job?"

In a calm voice, the DC guy told them to "get the f*ck out of the DC" or some major accident will befall upon them.

PS:  Door release button in Australia is normally colored GREEN.  The Emergency Powered Off (EPO) button is red (like the one pictured above) and, by law, has to be labelled properly.   In this particular case, the DC doesn't have/need a door-release button. Just push the door-kn0b and the door opens.

[Jouni Forss]

Reminded me of this

On a similiar note, we had some people come do maintanance related to one of our datacenters (electrical). Long story short, they broke both of our main VPN modules and we could not get any replacement devices during the same day (seemed there were none available in the whole country).

After a couple of attempts to rebuild VPN setup with 2 separate devices (which ofcourse werent licensed to support all the needed features) I ended up sitting 17 hours straight configuring all the VPNs on an ASR router. Oh joy

- Jouni

[Leo Laohoo]

Reminded me of this

Funny you posted this pix.  The day after the incident, this pix was circulated all over the office.  LOL!

[George Stefanick]

So what did the 50 users it impacted do ? And why would they put it close to the door and not have a BIGGER sign .. gezz

__________________________________________________________________________________________
"Satisfaction does not come from knowing the solution, it comes from knowing why." - Rosalind Franklin
__________________________________________________________________________________________
‎"I'm in a serious relationship with my Wi-Fi. You could say we have a connection."

[Leo Laohoo]

So what did the 50 users it impacted do ?

The DC house all the servers to a medical centre.  No, the servers were not redundant.   Politics.

And why would they put it close to the door and not have a BIGGER sign

The EPO was placed FAR from the door.  The non-IT guy didn't read.  He just pushed the button.  I believe that we are not allowed to put a "door release" button if we have an EPO.

[George Stefanick]

Let me know how this ended up .. Was he escorted out never to come back ?

__________________________________________________________________________________________
"Satisfaction does not come from knowing the solution, it comes from knowing why." - Rosalind Franklin
__________________________________________________________________________________________
‎"I'm in a serious relationship with my Wi-Fi. You could say we have a connection."

[Leo Laohoo]

We he escorted out never to come back

He's still got a job.

Whether or not he'll be allowed back into a DC is another question.  Whether or not he'll be allowed to touch anything electrical is another question. 

[Daniel Bruhn]

Clearly red buttons don't work in your office.  Have you considered trying painting it green?

Cheers,

Dan

[shillings]

Thanks for recounting the story Leo. What a nightmare!

One of the owners of the company I work for recounted a similar story. Many years ago he pressed a similar button and the entire place went dead. It was his second day in the job and I don't think things had been very well explained to him. Anyway, he took the stairs down and got out before anyone realised who'd done it. Next day he went back to work and owned up.

[Leo Laohoo]

Clearly red buttons don't work in your office.

Note to myself:  RED BUTTONS do NOT open the door.

Have you considered trying painting it green?

I'm recommending they put a sign with flashing neon lights.

One of the owners of the company I work for recounted a similar story. Many years ago he pressed a similar button and the entire place went dead. It was his second day in the job and I don't think things had been very well explained to him. Anyway, he took the stairs down and got out before anyone realised who'd done it. Next day he went back to work and owned up.

One of my previous supervisor did exactly this same thing.  He was on a high-pressure call.  It was a one-person operation NOC and the link alarms were going off and far from where the phone was.  He was hoping that if he'd hit the red button, the alarms would go away.  So he hit the red button.  The alarms stopped.  So did the lights.  And the DC next door. 

Anyway, he still got his job.

[stubinski]

The second non-IT guy went, "Did we just do something BAD?"  The DC guy, a large guy you don't want to meet in a dark alley, goes "Yes.  The bloke just cut ALL POWER to the DC."
To which the guy-who-pushed-the-button then responded with, "So when can we go in and do OUR job?"

Classic

This is why we have a clear hard plastic lid covering the Almighty Red Button that you need to lift before you can press it.  Its sad when you have to "user proof" the simple things

[Leo Laohoo]

This is why we have a clear hard plastic lid covering the Almighty Red Button that you need to lift before you can press it.

We have it too.  Didn't work.  He lifted the lid and hit the button.