The big headlines have been all about Sony’s security breach, and the massive data leak that occurred. What you didn’t hear about was how large parts of their technology infrastructure were rendered unusable. Most of their workstations were severely infected and inoperable for at least several days (some for weeks) and a large portion of their network and server infrastructure was compromised. Even If the hardware was functional, everything still had to be taken offline, scrutinized and analyzed for evidence, reprogrammed then finally redeployed. Qualified or not, Sony’s IT department had a gigantic mess to clean up, and they had to do this quickly (and improve security along the way) as the company was hemorrhaging money every minute their operations were offline.
If there is one thing that is certain (besides Death & Taxes) is that hardware will fail, and probably at the worst possible time. Why it fails is not important – but how you recover from failure is critical and can mean the difference between an inconvenience and a catastrophe. Sony’s disastrous breach is more of an exception in terms of hardware failure – it’s unlikely every single machine in your company will fail at once, but there’s always the chance that a catastrophe – natural or man-made – can wipe out multiple machines at a time. Preventing this type of event from happening is largely beyond your control. What you can do is control how you recover from it, which is a mixture of preparation, training and flexibility.
- Have a current, offsite backup of all your critical data.
The words “offsite” and “current” cannot be emphasized enough. Onsite backups are better than no backups, but if they get destroyed alongside the equipment they were backing up, it’s the same as having no backups. Depending on your business, current can mean different things – old data might be better than no data, but it could still mean many hours of lost work to get back to where you were before the data loss, and then you have to make up for that lost time. Make sure you are backing up the right data as well. Backing up email that is already stored on a server (which is itself being backed up) is a waste of time and money that could be focused on backing up your work documents. - Understand where your data resides.
Where is your data stored? Where is your email stored? What about your applications? You don’t have to understand the technical details, but you should know whether your data is stored onsite, offsite, in the cloud, or some mixture of all of the above. More importantly, you should know how to get to it – either from an alternate location and hardware, or – in the case of backups – who to contact to have data restored. If your critical business data resides at a single point of failure (e.g. your laptop hard drive), consider what would happen if you were to lose that laptop or if the drive was to fail. - Document your infrastructure.
If your business or organization relies heavily on technology-supported processes, rebuilding your infrastructure from scratch could result in serious disruption, especially if it is built differently, and given the pace of technology advancement, this is almost a guarantee. Older equipment and software may not be replaceable, so plan for replacing them on a non-emergent timeline, and prepare your employees for the change. At minimum, you should know that even if you are able to get equipment and software quickly, there will still be a ramp-up period while everyone gets acclimated to the new environment. Making changes in a stable calm environment is a lot less disruptive than doing so in a disaster recovery situation. - Train yourself and your employees to be flexible.
While it may not be possible for all jobs and functions (and some businesses), the crux of disaster preparedness (and recovery) is knowing how to get things done with the tools you have at hand. Most folks don’t realize that their email can be accessed via other methods than the one or two ways they use currently. The same could be said for accessing organizational data. This is not to say that everyone needs to know exactly how to get it done (technology can be complicated, especially tech that isn’t used on a regular basis), but to be open to doing their jobs differently by using alternate tools and methods.
Whether your company relies on racks of equipment or a single laptop, all of the above applies. Catastrophes come in all shapes and sizes, but hardware failure is always a disaster when you are ill-prepared.
I can’t tie a knot that would safely secure a boat, nor can I carve a race-winning pinewood racer, but I’m pretty sure my time as a Boy Scout primed me for a career in technology. Their motto, “be prepared” made a deep and lasting impression on me, and I try to exemplify that attitude in how I conduct my business, and encourage my clients to do the same. This can take all forms – planning for the safety and security of your loved ones is something everyone should take very seriously – but many businesses are less than ideally prepared for adverse events. Though most folks think in terms of actual disasters – fires, floods, earthquakes and so on (welcome to Southern California!) – you should also consider smaller-scale catastrophes such as data loss, security breaches, employee malfeasance, theft, vandalism, and virus infections. Every business should have a Disaster Recovery and Business Continuity Plan, and if that business or organization relies on technology, those plans should include technology recovery and continuity as well. Don’t have a plan? Here are five important items to get you started on writing one:
- Back up your data – most folks have learned the hard lesson of data loss and at a minimum try to back up their most important data to a separate drive. But if that backup is stored on premise, it is just as susceptible to whatever might damage your source data. At minimum, a copy of your backups should be stored offsite in a secure location, and the best solution is a combination of cloud-based backups and regular rotation of local backups to an offsite location.
- Keep track of critical logins and passwords – most organizations that can’t afford to maintain a full-time IT person on staff often suffer from a blind spot in their operation manuals and documentation: logins and passwords for important technology services, as well as contact numbers or email addresses for critical vendor services. Keeping these small bits of information current and stored offsite can mean the difference between hours and days in recovering from a disastrous event.
- Identify your technology weak spots – if your business relies on physical technology to conduct business, consider how hard it would be to operate without that technology for days, or even weeks. Email or web server on premise? Payroll checks printed on special printers? Even if you don’t use any specialized hardware, can your business operate without internet or electricity? Identifying these potential vulnerabilities will go a long way to helping you minimize or eliminate them before they can cripple your business during adverse circumstances.
- Evaluate vendor preparedness – if you rely on service providers for crucial technology services, you should have at least a basic understanding of how prepared they are for disasters. Though you have less to worry about with large, experienced providers (even Gmail goes down from time to time), if one of your “weak spots” is a service provided by someone else, you should know if they are prepared to handle a disaster, and how the loss of this service would affect your own operations.
- Train your people – if you or someone in a leadership position is incapacitated or isolated from the organization, others need to be prepared to fill those shoes. This means training them or at least preparing documentation for them on all of the above. Nothing is worse than watching an organization flounder while everyone stands around staring at each other not knowing what to do.
These are only a few aspects of a well-formed DR/BC Plan. The larger the business, the more detailed and complex it will become, but every organization large or small, should have one. It may seem expensive or a waste of time, but putting the effort into a DR/BCP will be the difference between your organization overcoming a challenge or succumbing to a disaster. Be prepared!
Image courtesy of winnond at FreeDigitalPhotos.net