This week, one of our managed services customers sites went down due to a power failure. The site has a backup generator, but guess what? Their generator failed to start! While there is not much we can do for a customer that is entirely without power, there are things we have in place to get clients back up and running, the moment stable power returns.
As soon as I received the notification that the power was down at the site, I knew something had gone wrong because I knew the site had a backup generator that should have automatically engaged within 30 seconds of a grid power loss. Except for this time, it didn’t kick on!
I spoke with personnel at the site, and National Grid will not restore power for at least six hours. Downtime is extremely costly for this organization, so I immediately broke my schedule and headed to their location. Our plan was simple. We would get the servers back up and running as soon as power was restored to the site. At the time, we weren’t sure if it would be National Grid or the generator that would come on first, but one thing was for sure; downtime would be minimized as much as possible with this strategy. All we needed was for power to come back and charge the battery backup systems to a safe level. I was confident that the information systems could quickly be brought online once the power came back because we had configured the battery backups to report to the servers that they were low on energy and that the servers should perform an orderly shutdown. Having this failsafe in place minimized risk by limiting the possibility of corruption of the server’s operating system.
To summarize, there are a few things you should keep in mind when it comes to battery backups: First, they are not intended to be a substitute for grid power or in place of a generator. Battery backups are not sized to be a long-term solution, but merely a stopgap intended to run the IT systems while you get your generator started or perform an orderly shutdown of your servers. Second, UPS systems help you avoid sudden failures due to momentary blinks and brownouts. Third, when properly sized and configured, the UPS will send signals to your equipment to turn back on once the battery has recharged enough that it is safe to go back on the grid.