Please create an account to participate in the Slashdot moderation system


Forgot your password?
It's funny.  Laugh. Idle

Bone-Headed IT Mistakes 259

snydeq writes "PCs preconfigured with stone-age malware, backups without recovery, Social Security numbers stored in plain view of high school students — Andy Brandt gives InfoWorld's Stupid Users series a new IT admin twist. Call it fratricide if you will, but getting paid to know better is no guarantee against IT idiocy, as these stories attest."
This discussion has been archived. No new comments can be posted.

Bone-Headed IT Mistakes

Comments Filter:
    • Re: (Score:3, Informative)

      by Applekid ( 993327 )
      Even the printer friendly version has text ads sliming it up, and they were practically more distracting than regular ads since they look identical to heading nodes within the article.

      Eh, is it time to just hosts out so I don't frustrate myself trying to read anything they product?
      • Re: (Score:2, Insightful)

        by mollymoo ( 202721 )

        Even the printer friendly version has text ads sliming it up [...]

        Those evil, evil bastards. Imagine wanting to get paid for your work. They should be like you and work for free. You do your day job for free, yes? I mean, you don't mind people taking your work without paying, even if the price is as mind-bogglingly low as a fraction of a second of mindshare, do you?

      • by Emperor Zombie ( 1082033 ) on Monday June 16, 2008 @06:20PM (#23816195)
        I [Do you like things that start with "I"? Take our IT IQ test!] don't know [For more stories about people not knowing things, check out "Stupid user tricks" and "More stupider user tricks"] what you're talking about [Are people talking about you behind your back? Read our "Top 10 reasons to be paranoid" and find out]. Those text [If you enjoy reading text, you might enjoy "Stupid hacker tricks" and "Stupid hacker tricks 2: The folly of youth"] ads [Is malware putting your system at risk? Take our Network Security IQ Test] weren't irritating [Is your job getting on your nerves? Check out "The 7 dirtiest jobs in IT" to see how much worse it could be] at all!
    • then 7 pages
      There are seven more pages after the printer friendly version? Sheesh. Talk about a long article.
    • Is it me.. or was this article HORRIBLY difficult to read? (excluding the ads)
    • by halcyon1234 ( 834388 ) <> on Tuesday June 17, 2008 @09:03AM (#23821981) Journal
      Plain Old Text, no ads:

      For those of us who make our living behind a keyboard in IT, it's hard to imagine a more time-tested vulnerability than the end-user. Armed with network access, these IT viruses wreak havoc nearly everywhere you look -- havoc borne of tech idiocy.

      Of course, not all computer users live to cause mayhem, sowing the seeds of destruction in our metaverse, merely by clicking every last Storm worm variant that appears in their inboxes. In fact, sometimes the worst offenses spring from our own ranks, hatched by individuals whose stated mission is to help technology work better: the IT admin.

      For the most part, we IT folks toil away unsung in often miserable conditions just to make workplaces more efficient, secure, and supportive of end-user needs. But then, a few of us -- well, we can be caught doing some really dumb things.

      So having kicked the user to the brain-dead curb in "Stupid user tricks: Eleven IT horror stories" and "More stupider user tricks: IT horror stories redux," it's only fair that we turn the spotlight inward to expose a few legendary IT brain farts committed by those who are paid to know better.

      Preconfiguring PCs with stone-age malware

      Incident: Toward the end of 2006, several high-profile consumer electronics companies -- both makers and retailers -- ended up with egg on their faces when reports surfaced that they were shipping to consumers devices infected with malware. Apple's Video iPod and several models of digital photo frames were found to be infecting the computers of unsuspecting users the first time they were plugged in. The risk associated with those infections was significant. In the end, however, the damage was limited.

      A year later, though, that wasn't the case. In September 2007, German computer maker Medion announced that as many as 100,000 laptop computers sold through Aldi superstores in Germany and Denmark came preinstalled with Windows Vista, the Bullguard anti-virus program -- and a virus.

      The case could have been devastating for the privacy or information security of anyone who bought one of the laptops. Modern malware, highly adept at stealing information such as bank account log-ins or credit card numbers, poses a real risk to consumers and companies alike.

      Only, it wasn't, because the virus, Stoned.Angelina, dates back to 1994, a full year prior to the launch of Windows 95, let alone the advent of widespread Internet access or online commerce.

      Thankfully, Stoned.Angelina isn't a particularly dangerous virus, at least not to anything more recent than DOS. It's a boot-sector virus that replicates itself by copying itself to floppy disks. Remember those? The Medion laptops didn't even have floppy drives.

      Medion never said exactly how this historic malware relic ended up in the default image on so many laptops. In the case of the iPod and photo-frame infections, the malware came from an infected machine in the factory in China that assembled the final products and installed the software onto the devices' internal storage.

      When you consider just how difficult it must be to load Stoned.Angelina onto a modern computer, you get a sense at how boneheaded the IT guy would need to be in order to infect a drive image used in tens of thousands of hard drives.

      Fallout: With no way to spread and no effect whatsoever on Windows Vista, Stoned.Angelina took its toll mainly on Medion, making the company a laughingstock. The punch line: Even though the machine came preloaded with an anti-virus app, the anti-virus engine couldn't clean the system. Bullguard later released a repair program that cleaned out the boot sector, just in case you, someday, somehow, found a floppy drive that worked with the laptop and inserted a disk.

      Moral: One, don't let the guy running an old copy of DOS on his computer build your drive images. And two, if you're going to deliberately infect thousands of computers, pick malware that's actually going to do something.

      Oh, you wanted to recover those b
  • by ferrellcat ( 691126 ) on Monday June 16, 2008 @04:13PM (#23814733)
    Deleting hundreds of thousands of White House emails, and not having a backup?
  • by mmkkbb ( 816035 ) on Monday June 16, 2008 @04:17PM (#23814799) Homepage Journal
    The RISKS Digest [] never gets old.
  • ... they do features....
    • by cashman73 ( 855518 ) on Monday June 16, 2008 @04:35PM (#23815009) Journal
      I agree. Most true, seasoned, and well-educated IT guys generally know what they're doing, and don't make mistakes. What should be discussed here are the most common mistakes by guys like Bob in the fifth cubicle on the right that was promoted to "head IT guy" because either (a) he was screwing the office manager who put in a good word to the head boss for him or (b) somebody heard him talking about "computers" around the water cooler and the company needed somebody to babysit their systems (most likely, it's (b), because he's probably more of a nerd than a true geek, and therefore won't be screwing anybody, except the users under him).

      Either that, or we should be discussing the boneheaded shiat done by lusers that IT guys have to clean up after. But that's probably already been done before around here, ad nauseum,...

  • the Daily WTF (Score:5, Interesting)

    by El_Muerte_TDS ( 592157 ) on Monday June 16, 2008 @04:17PM (#23814811) Homepage []

    pretty much a new bone head story every day
    • Re: (Score:2, Insightful)

      by Tweenk ( 1274968 )
      The Daily WTF is not the best place for open sourcerers, RMS worshippers and other idealists, and sometimes smells of Visual Basic and other vile secretions of a certain company, but is very fun nonetheless.

      Be sure to first look up the fundamental memes: picture of a printout on a wooden table, The Real WTF is..., brillant (sic), and Oracle NULL=''.
  • by Lonewolf666 ( 259450 ) on Monday June 16, 2008 @04:19PM (#23814835) []. Even if some of the stories are probably made up.
    • by eln ( 21727 ) on Monday June 16, 2008 @04:45PM (#23815135)
      Ah yes, the Daily WTF: the Penthouse Forum of the IT world.
    • "Made up"? It's so refreshing seeing an optimist in this day and age ;)
      • Well, some of them. A few of the stories sound a little constructed. But I guess that maybe 80% are real. Which is bad enough ;-)
        • by Moraelin ( 679338 ) on Monday June 16, 2008 @07:52PM (#23817009) Journal
          The sad thing is that each time I think about a story, "nah, nobody can be _that_ clueless", someone just has to selflessly offer himself as an example of even greater lack of clue. Seriously, I've seen so much WTF code in practice -- what with being the guy brought over when everything else failed miserably -- that now nothing seems unbelievable any more.

          There are people who simply don't know even the basic syntax out there, much less the basic CS notions, and still got hired because they were the cheapest. Sadder still, only a minority of them get fired for gross incompetence.

          Seriously, I've seen people who didn't even know what quotes do in Java, pretend they're Java gurus. Literally. One needed an explanation of why Java complains when he writes something like getUserData(John Smith), Java gives him a syntax error.

          Another one needed some explaining as to why if he declares a variable in the constructor, it's not visible in another method. Seemed to essentially assume that since the constructor has the same name as the class, that's where you declare class members. Right? Mind you, the whole concept of scope seemed a bit fuzzy to him.

          One particularly promising young padawan tried to "fix" a bug by changing every single if in the program from

          if (someCondition()) {

          if (someCondition() == true) {
          Actually insisted that the bug was now fixed. 'Cause Java generates different code when you write "== true." Ookaayy.

          An inventive guy tried to get around some data objects being invariant (you know, all getters and no setters) by writing basically a method like this:

          public void nuller(String x) {
                  x = null;
          Was genuinely surprised that calling "nuller(someDataObject.getName())" didn't actually set the name to null. Took some explaining to understand that it's not some Java bug, but, really, how it's supposed to work.

          An _architect_ made a whole team use the boxed objects (Integer, Character, Boolean) instead of the primitive types (int, char, boolean) in all method calls, as a speed optimization. See, if you have an Integer parameter, Java only copies a pointer, not the whole int. (That was before Java 5 and its automatic boxing and unboxing, too, btw.) Sadder even, nobody in that team had any objections.

          And that's just the simple ones, the ones that can be told in one paragraph. There are more, but let's not write a whole tome.

          So, really, there are some truly monumentally clueless people out there. And they do random clueless things, until by sheer brute force they arrive at something which survives their testing with a couple of clicks in the GUI. Yay, they solved the problem. (Not.) Give them enough time and lack of interest to actually get a book and learn, and it'll grow into an "experience" of such witch-doctor tricks that worked once, and cargo cult code that tries to look like something they saw once, but they never understood why.

          So, well, if you see some code sample that looks like it _must_ be a fabricated story... well, it is at least _possible_ that it's true. And know that someone somewhere probably wrote an even bigger abomination.
  • by zehnra ( 1076641 ) on Monday June 16, 2008 @04:20PM (#23814847)
    Information Security isn't going to get better without a major shift in how people work. As a society, we need to examine who really needs what data and then truly limit everyone to what they need. Until we can define these roles/access levels in black and white terms and permanently adhere to the controls put in place, there will always be IT blunders.

    The problem is that these changes are rarely permanent, but more of a pendulum that swings back and forth as events like this occur. If Bob is taking home Social Security numbers on his laptop and someone steals it, controls may be put in place to prevent people from saving files to their laptops (and Bob is let go). Six months later, Suzie complains that she needs to be able to copy a proposal she's working on so that she can work on her flight to Japan. An exception is made. This typically snowballs until we're back to where Joe can copy the accounting records with SSNs.

    Ease of access and efficiency nearly always trump security when these breaches aren't fresh in everyone's minds.
    • Re: (Score:3, Insightful)

      by jd ( 1658 )
      This is why you have mandatory access controls, so that copying within the confines of what is needed to perform the job is permitted, whilst copying outside the confines of what is needed for work OR copying onto devices less secure than required for that type of data is not.

      The problem with MAC is that it is time-consuming to set up and very difficult to get absolutely right. If it isn't absolutely right, it ends up needing to be hacked to bypass unnecessary limitations, which will have al kinds of unpred

  • by Torinaga-Sama ( 189890 ) on Monday June 16, 2008 @04:20PM (#23814853) Homepage
    When a company simply accepts what the sales drone says about a given product as a fact.
  • by DRAGONWEEZEL ( 125809 ) on Monday June 16, 2008 @04:21PM (#23814871) Homepage
    more privileges than you need mistake! This one plagues IT guys day in and out.
    Whoops, I mis-clicked and deleted a domain. Sorry Doc, I accidentally selected all your patients then declared them to have a clean bill of health. Oops I deleted a block of user accounts.

    And a few I really did do....
    Double "oh sh!t":
                                            I just accidentally removed all my own rights... (I'll never forget the time I made that mistake... )
                                            Setting a block of users to the wrong group, giving them Admin rights.
                                            Clicking on a link that my trusted IT friend sent me...
  • by COMON$ ( 806135 ) * on Monday June 16, 2008 @04:24PM (#23814895) Journal
    1. Hire competent IT people, don't promote mailroom boy to Admin because he can fix spyware.

    2. Continuing education for your IT people.

    3. Just because someone looks old, doesn't make them a competent 'seasoned' IT guy.

    4. Respect your IT pro's opinions.

    We all have a plethora of stories of users, but even more of fellow co-workers in over their heads causing massive damage. Sometimes it goes unseen, other times it can desecrate a business. Make sure your IT people are educated, have a passion for what they do. Not just a paycheck monkey draining your resources.

    A good test here, if your IT head is an ex-HR manager, mailroom clerk, secretary, or other far removed profession and have yet to get any certifications or degrees to prove their competence after 10 years then you probably are in trouble. Not in every case, but enough to make you worry.

    Im not saying that a cert or degree proves that you are competent, but it at least shows that you try to be.

    • by Mordok-DestroyerOfWo ( 1000167 ) on Monday June 16, 2008 @04:39PM (#23815053)
      I cannot stress your point #4 enough. Sometimes it seems like every decision that I and our IT staff make gets voted down by management because they'd have to remember another password, or encryption is just to darn difficult to use on the road. Just because you're paranoid does not mean that everybody is not out to get you.
    • Re: (Score:3, Insightful)

      by CompMD ( 522020 )
      "4. Respect your IT pro's opinions."

      That has always been my most sincere wish. However, I'm young, not as highly educated as the chief engineer/company president, and so that doesn't happen.

      Never mind the fact that all the workstations and servers work, all the strange high-end scientific and engineering software works, and the network never goes down.
      • Catch-22. Your boss will not respect your position until there is a major problem with the systems. Once there is a major problem with the systems, you will be fired, and the new guy who fixes the problems will be seen a savior.
        Solution? Try and outline all the things that can possibly be going wrong; all the script kiddies hitting the firewall for naught, all the times the servers might have been brought down by bugs you patched, etc. Problem? Now you've spent a lot of time and resources twiddling y
    • This is slightly off topic, but...

      Respect your IT pro's opinions.

      Dude, if you can figure out how to make that happen, you will become an IT hero.

      I had a client who called me to help build a network for her new business. I interviewed the client to determine her needs, asked a lot of follow-up questions to make sure I really understood what she wanted and expected from her network, then started drafting up a design to meet these goals. She then became the Client from Hell.

      It wasn't bad enought that she ignored most of what I s

      • She found a pretty, $5000 software product that is the core of her business, but didn't listen when I pointed out the (many) design flaws in the program. For example, who uses DHCP to assign an address to a standalone host when the client software that talks to that host has to have the IP address (not FQDN of the host, the IP address!) statically set in the configuration file?!?!

        To be fair, you can use DHCP to set a static IP address, you just plug in the machine's MAC address. However, that's still a bad
    • by Belial6 ( 794905 ) on Monday June 16, 2008 @05:18PM (#23815507)

      A good test here, if your IT head is an ex-HR manager, mailroom clerk, secretary, or other far removed profession and have yet to get any certifications or degrees to prove their competence after 10 years then you probably are in trouble. Not in every case, but enough to make you worry. Im not saying that a cert or degree proves that you are competent, but it at least shows that you try to be.
      I would say the opposite. If after 10 years in the industry, your IT guys are still chasing the meaningless certifications, then you are probably in trouble.
  • by Anonymous Coward on Monday June 16, 2008 @04:25PM (#23814913)
    I was new to the whole *nix thing but had been let loose as root on all the boxes at work. Someone suggested I could/should create a script to customise my environment so that I could run it when I logged on. Problem was I named the script "df" (my initials) and then promptly decided that it needed to go in to the /usr/bin/ directory. Yeah - now you know why I posted anonymously. :-D
    • by Cro Magnon ( 467622 ) on Monday June 16, 2008 @05:26PM (#23815579) Homepage Journal
      Could have been worse. At least your name wasn't "Richard Morton". Imagine the havok a script with those initials would do!
      • Or rather... wouldn't do? I can sure think of several times where I've typed "rm" and ended up wishing /bin/rm had gone missing.
    • Should have called it AC
  • Did anyone else give up on the tedious page clicking and entirely unfunny "IT Geek Quiz" that was clearly thrown together by the same sort of folks these people are mocking?
  • by Anonymous Coward on Monday June 16, 2008 @04:34PM (#23814991)
    At my middle school, there was a policy to give every student an ID card. That's fine. They decided that the best number to use for their ID is their Social Security Card. That's bad. They printed out a sheet every day listing the absent students for the day, with their names and their school id's. That is worse. Teachers threw these into their trashcans when they were done. Yes, the train wreck isn't over yet. The spreadsheet containing all of these numbers was on a public share. It was also accessible from the school website.

    Or how about 3 years later, in my high school. All of the teachers user names and default passwords were on a spreadsheet on a network share. A publicly accessible network share. If a teacher didn't change their default password (a 4 digit number), A student would have full reign over their data.

    Worse off, the grade book program was accessible from any networked machine (thanks Novell)
    Thank god this was nearly a decade ago... So, one could pick a random terminal in the school and make subtle changes to their own (or perhaps someone elses) grades.

    I used to think "I wish that I was alive during the 80's so that I could have been part of the cracking scene there". In hindsight, I could have done such bad things during the 90's, when I grew up.
    • Ah yes... my first social engineering... getting into the grading program at school. All the teachers knew I was the guy to ask about computers (even though I wasn't really that big of a geek in school). So it was really pretty easy to confuse a teacher about which password to give me (system/app - but either is still bad to give to a student). I prevented a few of my buddies from failing English that year.
    • Worse off, the grade book program was accessible from any networked machine (thanks Novell)
      Don't blame novell, the system admin should have:
      1) put the sensitive files in a directory
      2) put the teachers in a group
      3) only granted rights for that directory to the teacher group

      It's not Novell's fault. It's the Netadmin's fault.
  • by Phroggy ( 441 ) <.slashdot3. .at.> on Monday June 16, 2008 @04:37PM (#23815023) Homepage
    Hold on a minute here.

    The IT guy blames his boss for installing the Alexa toolbar, which lead to the deletion of all dynamic content on the company's web site.

    No it didn't.

    Yes, the Alexa toolbar isn't something anybody needs to run, and yes, Alexa should respect robots.txt, but whoever set up their web site is clearly incompetent:

    1) Never rely on robots.txt for security.
    2) The article says the Alexa spider captured usernames and passwords? What the hell were usernames and passwords doing unprotected on the web site?
    3) The Alexa spider clicked all the Delete links. Never ever use links to delete things! Always use a submit button with POST, not GET. Generally, most spiders won't submit POST forms.

    Security through obscurity is even less effective when the obscurity is poor.
  • Lot of stores sound like stupid PHB driven ones and the tech are just along for the ride.
  • My favorite (Score:5, Funny)

    by hal9000(jr) ( 316943 ) on Monday June 16, 2008 @04:39PM (#23815063)
    Not as major is the Infoworld examples, but I still to this day sometimes forget to set-up a virtual interface when configuring a cisco router. This little command me more often than I care to admit:

    cisco-router$ en
    cisco-router$ config t
    cisco-router(config)# int g0/1
    cisco-router(config-if)# ip address mask
    Connection Closed

    • Easy fix:

      cisco-router$ en
      cisco-router$ reload in 15
      isco-router$ config t
      cisco-router(config)# int g0/1
      cisco-router(config-if)# ip address mask
      Connection Closed

      IT Admin: Gah...Now I have to wait 15 minutes for the router to reload. Oh, well...time to get a soda.
      • Re: (Score:3, Insightful)

        Easier solution.

        Turn it off, turn it on. Nothing was written to running-config.

        Now wait the same 15 minutes, only 15 seconds earlier.
        • Turn it off, turn it on. Nothing was written to running-config.

          In this case that's probably going to work.

          When the router is at a power station in Guangzhou and you'd have to wait until 3AM and call in one of the people in the company who speaks the language so you can call the local office during their business hours and get them to power-cycle it for you... you're a lot happier that you remembered "reload in 15".
          • Exactly.

            In my environment [], the hub for my service area is an hour away by 737 (no roads there, either). The outlying villages are all accessible either by charter air service, boat (in the summer, and only some of the outlying villages) or snowmobile (in the winter). Some of my central offices are on mountain tops that in really bad weather may not even be reachable for a week or more.

            In other words, just reaching over to turn the router off isn't always an option. In fact, for me, it usually isn't an op
        • How do you do this when your router is sitting in an unmanned wire center 500 miles away? If the router is sitting right next to you, why are you using telnet rather than a console cable to get into the thing?

          In either case, it is because nothing was written to the running config that the "reload in 15" command works.
  • My experiences (Score:5, Interesting)

    by HappySmileMan ( 1088123 ) on Monday June 16, 2008 @04:40PM (#23815071)
    My school once had a folder called "Vice-Principal" in the network folders, what did it contain? Why, the C: drive of the vice-principal's computer of course, they didn't let you access "Program Files\" or "Windows\" of course, but what WAS accessible, was a Microsoft Access database containing every student in the school, their PPN number (equivalent of Social Security in Ireland I think), their home phone number, medical conditions, exam results etc. Of course this year they got new computers and completely re-setup the network, this time it seems substantially more secure.
  • by thatskinnyguy ( 1129515 ) on Monday June 16, 2008 @04:41PM (#23815079)
    Database take a dump? No backup of the transaction log? Fear not! With just two easy steps, your life will be back on track:

    1. Update Resume`
    2. Leave Town!
  • by gEvil (beta) ( 945888 ) on Monday June 16, 2008 @04:41PM (#23815081)
    I used to work with a guy who did the "useless backup" thing. He set up an automated backup system that encrypted the files to tape. It ran fine for a long while. But when we had a server failure and needed to recover from the backup tapes, he couldn't remember what the decryption password was. All he could do was sit there saying "I remember that it was a good one." I just wanted to smack him...
    • I did something just as bad at the beginning of my IT career:

      We had a horrendous Clipper-based database that contained all of our company's purchase orders, sales orders, customer invoices, etc. It was about 900MB back when the original Pentium was still new and the biggest consumer grade hard drive you could buy was just over a gig. The database used to have a lot of corruption problems, and it was my job to fix it when that happened. Once in a while, the tools I had to fix the corruption wouldn't work,
  • Anonymously :) (Score:5, Interesting)

    by Anonymous Coward on Monday June 16, 2008 @04:58PM (#23815265)

    A company decides to run an internal check to see how many people will respond to a phishing scam. They send out an email to a group looking like the intranet page, "reminding" everyone to submit their username and password for the upcoming upgrade this weeken.

    The email is actually an HTML form, but users being users, some of course hit reply instead of filling out the form and hitting submit. Worse yet, some hit "Reply All". Worse yet, some had HTML turned off, so the password wasn't even hidden in HTML source, it was in plain text for all on the list to see.

    Yes, testing internally to see how many people are susceptible to phishing attacks is a good thing. However, send it via bcc, so group replies won't have passwords spreading around the company like a bad joke.

    Next up, inform some people you are running your test. We have two different security groups, corporate, and the one I'm in. We didn't know about it, and all but shut down corporate security's access to the network. We traced the originating IP to their network, as well as the form submission IP. Since they weren't answering their phones, we didn't have much choice.

    I found out because a supposedly "technical" engineer called me saying he had responded to it, and realized some people were replying and he could see other people's passwords. He didn't think there was anything wrong with submitting it, because it looked so real it couldn't be fake.

  • by steveha ( 103154 ) on Monday June 16, 2008 @05:07PM (#23815379) Homepage
    This one really wasn't the IT staff's fault, so this is slightly off topic, but this is my all time favorite Daily WTF story. []

    • by Illbay ( 700081 )
      On the bright side, think of the reduction in the carbon footprint for that weekend! They oughta have Algore give him a medal.
  • ...the linked story is more than a year old?
  • by hedley ( 8715 ) <> on Monday June 16, 2008 @05:20PM (#23815527) Homepage Journal
    I could not access my mbox, the file was gone. Soon a co-worker stopped by... same... mbox gone. 2+2 together a quick visit to IT. "Hello, did you do anything to the company mbox's?", IT: "Oh yes, I observed they took up a lot of space on the disk so I *removed* them all"!


    • I had something similar happen, once, when I was doing tech support for an ISP. We were told to keep important data on out Network Share rather than our computers so that if anything happened, they could image our hard drives without data loss. It was only after that saved data vanished (With, I might add, about two years of saved tech tips.) that I found out that "We don't back up the Network Shares. You should have kept it on your own machine."
  • From memories past (Score:5, Interesting)

    by Macka ( 9388 ) on Monday June 16, 2008 @05:28PM (#23815605)
    I used to work in Unix Support for a large multi-national. Had loads of customers ring in with cock ups over the years. Some of them were silly, like a developer with root access typing rogue spaces where they shouldn't be. e.g: "chmod -R me / foobar". Conversations always started like "OMG I own the whole system, HELP!". Others were more obtuse, like a world renowned news reporting organisation who allowed one of their developers to install a very important database in his own account. System management got outsourced to Singapore, he then left the company, so Singapore deleted his account. We were left trying to reconstruct was was left from a dd image copy of the disk.

    Another one I remember (about 20 years ago) was where one customer had systems that would crash at about 10am every monday morning. After a very long trouble shooting experience (i.e. months) the cause was found to be a delivery lorry that arrived every monday morning. He would back up to the loading bay, where some rubber bumpers (fenders) had been installed. He had the habit of stopping the lorry when he banged into the bumpers. Unfortunately this sent a shock wave through the building sufficient to cause some of the disks in the computer room throw a hissy fit and park their heads in the middle of whatever I/O they were doing.

    In the early 90's I found myself having to pick up SCO Unix support for my sin's. Thankfully it only lasted 4 years. Two specific customer incidents I remember from that time. One was a call from a hospital who's system seemed in a right state. The guy was panicing, so I cut short my usual trouble shooting routine, got in the car and drove down there. Took one look at the system, typed ^D and then left after it'd finished booting to multi-user. Taught me a lesson; embarrassed the hell out of the customer and I never heard from him again.

    The second was more interesting. I had a customer in the MoD at HMS Dolphin in Gosport. A number of their systems would crash simultaneously at certain times during the week. There was no real pattern to when, but when one of them went, they all did. I couldn't find the problem. No common denominators. Power monitors didn't show anything. Nothing. That was until one day the customer was staring out the window when the systems crashed. He remembered seeing one of the warships leaving the harbor and sailing right past his window. He also remembered seeing the ship starting its RADAR as it went past; and as the beam swept the computer room, all the systems crashed. The fix: a snotty email dictating that captains don't start their radar until they've cleared the harbor and made it out to sea.

    I could go on typing for another hour straight with stories like this that either I've seen, or have happened to friends/colleagues :-)

    • Re: (Score:3, Insightful)

      by pclminion ( 145572 )

      System management got outsourced to Singapore, he then left the company, so Singapore deleted his account. We were left trying to reconstruct was was left from a dd image copy of the disk.

      This one drives me CRAZY. Yes, it's downright stupid to have critical things running under employee accounts. But the worse failing, I think, is this silly idea that once somebody has left all traces of them must be eradicated from the universe, as if the ghost of their keypresses will arise from the ashes of their wor

  • by digitalhermit ( 113459 ) on Monday June 16, 2008 @05:57PM (#23815973) Homepage
    I once got called to help another technician with a system restore. Over the weekend a server had crashed and we had to rebuild it. First thing we do is to re-install the server. This took a few hours. Then we had to restore the data. No problem. We pull in the tapes but for some reason, it cannot find any files. The tech says that he's sure the backups were successful. Even the previous days and weeks had the same problem.

    Figuring we had a busted tape drive, we drive 60 miles to pick up a tape drive from another location. Plug it up and bleah, same results. I ask for the backup log. Sure enough, everything is successful. Only problem is that nothing is configured to be backed up. So every hour, every day, every week, every month the job would complete successfully. Successfully backed up nothing.

    The worst I've ever done personally was to install a CIFS module on AIX. This inadvertently updated a TCPIP package. This package had an obscure bug that was only triggered with long running sessions. It tooks hours to determine that the failure wasn't related to another patch that had gone in, and wasn't related to a very similar issue related to the connector...

  • Now there's a bone-headed [] idea.
  • I have been bashing people who tell me that all IT jobs will be in India and China and Russia. This is not going to happen to every freaking job because each field depends on people who are competent. You may have a Ph.D. in Comp. Sci. or Mathematics, but you're completely useless if you cannot perform job related functions in a competent manner. That is why the number of jobs is always greater than the number of candidates who can do those jobs well. This applies to every country, not just the United Sta

  • It's ironic that just this morning I received an email from a user with just this line:

    "are we able to get email right now?"

    I resisted the urge to reply back with "no."
  • I thought that "boot.ini" didn't arrive on Windows until NT4.0, 2000, and XP? (The article says "windows 3.1" - I call shenanigans)
  • by FranTaylor ( 164577 ) on Monday June 16, 2008 @06:43PM (#23816385)
    One of my co-workers once decided to install a beta version of Windows NT on the company's Novell file server, which EVERYBODY used for EVERYTHING. He did this in the evening when noone would notice and then he left for two weeks' vacation!!! I have never in my entire life met a more arrogant SOB. The entire company was down for over a day as we restored the server from a backup.

    The boss refused to fire him (out of a cannon), so we filled the entire volume of his office with computer boxes. We went up and over the drop ceiling to deposit the last few boxes so he could not even open the door. When he returned from vacation, it took him a whole day to figure out how to get the boxes out.
    • Re: (Score:3, Funny)

      by shakah ( 78118 )
      Speaking of Windows NT beta versions, best I saw was a Q/A lab with over 100 Windows boxes. All the boxes were mistakenly installed/configured over the course of a few days with a beta (or trial) version of Windows 4.0 which timed-out after 180 days (I think) with a "blue screen of death" (no licensing issue, the tech just grabbed the wrong CD and kept using it) . All was fine for quite some time until boxes stared BSOD-ing one-by-one -- once we realized what happened it was kind of humorous to watch them f
  • by DigiShaman ( 671371 ) on Monday June 16, 2008 @07:56PM (#23817053) Homepage
    Never assume that just because Backup Exec (or other backup utility) has backed up your data, that you don't indeed have problematic tapes and/or other hardware issues.

    Test your god damn tapes people! When a company loses two years worth of data because backups were *never* verified to be working correctly, they're fucked. Needless to say, you'll be out of a job too.

    Again. Restore from tape and verify!

    Note: this just happened to a company I know. They called me asking for help because their last few IT contractors never verified backups are taking place properly. I really feel sorry for this company, and I've only met the owner once. Sad...
  • How About This? (Score:4, Insightful)

    by Anonymous Coward on Monday June 16, 2008 @11:18PM (#23818419)
    I work for a Very Large Power Company, mostly hydro-based generation. We've been running our Generation Control System on *nix for about as long as anybody can remember. It's robust, secure and dependable.

    However, we're beginning to see issues, especially with subsystems on old(er) proprietary hardware (cough*Alphas*cough) and replacement components are either scarce and expensive, or just plain unobtainable.

    So we've recently completed the first phase(s) of a major GCS-upgrade project and the decisions have been rubber-stamped by the Government. (We are what's known as a "State-Owned Enterprise.) The new GCS system will be running on a Microsoft Windows Server platform.


    Because the two contractor chicks who presented the choices to a Government-run committee, whose members have no desire to be held responsible or accountable in any way, shape or form, heavily promoted Microsoft Windows Server, via a bunch of garish PowerPoint presentations and Word documents.


    Because, as one of the contractor chicks candidly admitted not long after, "[I] only know Windows."

    So, a national infrastructure control system, one which epitomises the very notion of "Mission Critical", is to be based upon what is quite probably the absolute worst choice of NOS imaginable.

    The (unaffiliated) national power distribution company migrated from *nix to MWS a few years ago, for what were essentially the same reasons. Their admins are not envied. Much of their time is spent coaxing the backup-backup-backup-backup servers back up.

    One immediate result of the recent decision is that three of this company's best-and-brightest IT people resigned and "moved on". The departure of several more is imminent. I can't call them rats, but they are certainly escaping a ship that's heading straight for the iceberg, full steam ahead.

    It's highly likely that this country's governing party will change at the forthcoming national election, although it will change nothing else. If anything, the soon-to-be-incoming party is likely to be even more MS-friendly than the current one, so I don't foresee any likelihood of sanity prevailing anywhere near the top in the near future.

    Instead, what's likely to happen is that once the system begins falling apart - as it surely will - MWS will be quietly shelved by lower echelon IT management (avoiding any embarrassment to anybody in an expensive suit) and a *nix-based one will be restored. Estimates of when that will occur range from "Within a year" to "It has to happen eventually."

    I use Win XP Pro at home. It's fine for general purpose family use. But MS Windows does not belong on a server: Or, at least, not on any which are expected to remain functional most of the time.

    True story and, yes MS fanboys, I know you'll be modding this down to "-1: Troll" and "Flamebait". I can cope with it, thanks. I have bigger worries right now.
  • PHB Edicts (Score:3, Funny)

    by PPH ( 736903 ) on Tuesday June 17, 2008 @12:15AM (#23818815)

    Back when I used to work for a major aerospace manufacturer, we had an interesting incident:

    We had a production control system hosted on a series of HP-UX servers. The IT department had just reorganized, placing a new (inexperienced) manager in charge of our systems. One day, all the servers went off line. As the factory ground to a halt, I managed to log in to one via telnet. It seemed to be up, but many functions were failing. I traced the problem to: no /tmp directories remained on any of the systems. I contacted the on-duty admin. with his tidbit of information. She informed be that, "as ordered by management, all /tmp directories were to be removed." Apparently, the new boss had read somewhere, that /tmp was for storage of "junk". He deemed the storage of "junk" to be an inappropriate use of company resources and, to prevent it, all /tmp directories were to be removed.

Machines that have broken down will work perfectly when the repairman arrives.