Alrighty, here is a story (this is not really hardware but software related and not PC related and not and not and not
But I am going to share anyways.
This story is of a major screwup but luckily NOT by me.....
I came onto a project at the 50% IPR (software about 50% complete). The vendor stated during one of the meetings that they INSTALL and RUN their software as ROOT. (they said the performance was better that way :roll: :roll: ) We had ~40+ unix based servers running a RTOS (RealTime Operating System).
I told them immediately that I would NOT accept the software if the install kit and runtime environment were all under root. I was taken aside by my director and the vendors Lead Engineer and told that basically I do not know what I am talking about... Shhhhhh.... The vendor knows what they are doing. /me being new to the organization said OK???? :roll: :roll: :roll:
My director (not me) accepts the software and the vendor starts their install/upgrade scripts. One crucial part of the scripts, was the base directory that they needed to execute the scripts from. The script writer instead of at least hardcoding the $BASEDIR environment variable just expected people to know to start the script three levels down.
The first thing the script does is an (rm -rf *) as user ROOT to ReMove the previous installation. It also does this from WHICHEVER directory they start the script.
Since they were user ROOT and since they were at / (the ROOT directory) they totally deleted the entire system.
If this were not bad enough - they were supposed to have an ~45 minute validation period between the first server install and the rest of the 40+ servers. This validation would make sure the install went well and that the servers restarted with everything fine.
The best part about it was, they were SO confident in the scripts that they were were pushing and running the scripts on the rest of the 40+ servers at the same time they were doing the first install. 38 of the 40+ servers were totally dropped before they realized what they had done.
The second best part was they did this during busy hour because their engineers did not want to work late. The director that allowed all of this (mine) was reprimanded pretty badly. The vendor had to pay for "ALL" lost transactions that the customers would normally be paying. Roughly $150,000 and 16 hours later (engineers had to work late
) they had all servers back online (OUCH).
Moral of the story:
Always be careful when doing anything as root.
Always assume that anything that you create will not run as root (drivers aside).
Never perform, ANY upgrade during busy hour or even close to busy hour.
Always check your work (kinda like the measure 5 times cut once mentality).
If you are an executive level manager (regardless of your previous experience) leave the tough stuff to your lower level managers/tech leads.
Last but not least:
ALWAYS remember that the new guy may actually have a good idea and may actually know what he is doing. (you can also extrapolate this to "always listen to Ches111