Edit: This is still an interesting dilemma. Maybe more organizations would just be more cloud centric and not worry about any of it these days.
Originally posted August 4, 2009 on AIXchange
I was recently talking with someone who works for a non-profit organization. He was tasked with being the AIX administrator. He didn’t necessarily want those extra duties. He didn’t have a background in systems administration, and he had no training or real inclination to do the job. Still, he did his best to administer the non-profit’s machines on top of his normal duties.
Most of us who’ve worked on computers for a while understand that backups are critical. If we haven’t experienced the horror stories first-hand, we’ve certainly heard them. We know that the capability to restore your system–on different hardware that, preferably, is in a different location–is essential.
The non-profit stored its mksysb and savevg images on another machine. However, the organization ran out of disk space on this secondary machine and decided it would recover some disk space by moving the backup images to the production machine. In short, they gambled that nothing would go wrong with the production machine, and they lost everything. Disaster struck on that primary machine. The filesystems were erased, and the data and backups vanished. No offsite tapes. No way to recover the data.
All the while, those in charge at the non-profit maintained that there was no budget to hire a system administrator. They were doing their best.
I know of another company that figured, because it had primary and backup sites, it didn’t need to worry about backups. The problem was, due to the way the machines were configured, when data was deleted at the primary site, it was deleted at the backup site as well. Redundant machines are great, but you must have a way to access older data if your current data becomes lost or corrupted.
I don’t expect people juggling sysadmin duties with a “regular” job to get all of these issues. Hopefully though, the importance of backups is universally understood. Data must be protected.
Along those lines, backups are useless if they’ve never been tested. Be sure to verify that you can restore your data, however you choose to back it up.
I keep wondering, what’s the solution for organizations that cannot afford administrators? Do we ask consultants with experience to step in, pro bono, or at least at a reduced rate, to help non-profits manage their machines?
If nothing else, it might be nice for these part-time administrators to have someone that they can turn to when they have questions. These folks can benefit from Internet forums, IRC channels etc., although it can be hard for seasoned professionals to give strangers more than general advice about machines in which they have no vested interest.
In this case, had the non-profit asked whether moving these images back to the primary machine was a good idea, they would have had received a resounding “no,” and disaster could have been averted. To organizations that claim that they cannot afford proper help, I have to wonder about the value of that lost data. I’m willing to bet that trying to recreate data–or being forced to live without it–is far more expensive than finding the appropriate assistance in the first place.