Introduction

During the great power blackout of the summer of 2003, many data centers experienced significant system failures that theoretically should not have happened. In too many cases, the battery backup systems that should have provided adequate power protection for the Uninterruptible Power Supply (UPS) failed in the moment of need. The cause: inadequate monitoring and maintenance of the batteries.

The UPS is your data insurance policy and business lifeline. The justification to spend between $50,000 to $200,000 or more on the UPS is to assure critical power continuity during the 15 to 45 second lapse between main power outage and the start-up of the backup power generator. When quantifying the risk of loss from a UPS failure, the decision to maintain the UPS at its highest state of readiness is critical - as is making sure the batteries will perform. When the battery is compromised within the UPS, your data insurance policy disappears.

The US power grid is said to be 99.9 percent availability, which means the average consumer across the US experiences 8 ¾ hours of outages per year. That 0.1 percent risk factor represents $125 billion per year in direct losses. Indirect losses for recovery can easily double that figure. Unplanned, catastrophic UPS failures play a contributing role in that dollar figure, with power quality outages estimated to cost $25 billion to the US economy per year. When the UPS fails, facility engineers are fired, unrecoverable business losses are suffered, and in worst-case scenarios, lives are jeopardized. This risk need not exist, since meaningful, real-time battery performance data can now be obtained 24/7. With access to this data, users are able to accurately pinpoint and replace weak batteries before they pose a risk to the power backup system.

UPS Failure Modes are 98% due to batteries A typical UPS module consists of a two components: the electronic power management and the backup battery power. The common knowledge failure rates stated within the backup power industry show that the battery is the root of failure 98 percent of the time (see Figure 1). However, the industry is caught in an apparent catch-22: unless current battery management practices change, the 98% won't change either. The data proves new battery management techniques are required if these risks are to be eliminated.

Figure 1: Batteries fail more often than electronics. UPS suppliers and data from a battery monitoring center indicate that the cause of UPS failure is battery failure 98 percent of the time, and electronics failure responsible for just 2 percent.

The Unpredictable Nature of Battery Failure

Early battery failure can occur for a variety of reasons, and seems to be more common among VRLA (Valve Regulated Lead Acid), commonly known as sealed or "maintenance-free" batteries. These batteries are being specified more often than wet cell types due to their much lower cost, smaller footprint and apparent ease of maintenance. Trend data collected from continuous remote monitoring of such battery systems has shown that individual battery jars can fail within a two week period (see Figure 2) - and it takes just one failed jar to bring down the entire string when under load! Some of the most common reasons for failure include:

1) Cracks in battery jar cases, cell dry-out, post seal leaks, stuck valves, plate sulfation, dendritic
shorts, cracked plates or separators and poor plate welds
2) Environmental factors, especially temperature - in one case monitoring information exposed administrative staff that shut off the air conditioning on weekends, resulting in temperatures exceeding 100 F in the battery room
3) Indefinite length of time sitting without a charge between the battery manufacture date and UPS system installation and commissioning
4) Battery manufacturing defects - monitoring data has shown up to 3% of batteries to be defective "out of the box" and others failing shortly after the warranty period
5) Frequent cycling
6) Inappropriate UPS float charge and equalization settings
7) Battery interconnections not properly torqued by maintenance personnel

Battery health has traditionally been difficult to detect. Standard methods - often included in many advanced UPS systems - use total system voltage or discharge cycle measurement to find bad batteries. However, these methods are unable to detect individual battery jar weakness in advance. By the time a fault is detected, the battery system has already been compromised for an indefinite length of time.

Current battery management practices supported by major UPS manufacturers and the IEEE recommend quarterly, semi-annual or annual battery maintenance, including visual inspections, voltage checks, specific gravity readings, rewatering, resistance tests, retorquing connections and a general cleaning. Battery strings are replaced well in advance of the end battery service life to minimize risk. These services are done by the UPS manufacturer's service department or an independent battery service provider. As Figure 1 illustrates, it is obvious that these practices do not assure adequate protection - there is simply too much time between service intervals when the batteries could have posed a risk.

An advanced battery management system using battery impedance measurement data as the principal method of battery failure prediction is the only way battery users can be sure their systems will work. When this data is collected and used for trend analysis, individual problem jars can be identified and swapped out before the system is at risk. The results of this data can be used to develop a new battery management program that can reduce or eliminate risk due to battery failure.

The graph illustrates a 2-week, on-line battery monitor reading that documents an increasingly common failure trend from monitoring data in VRLA batteries. Yellow diamonds document weekly impedance readings of a 12-volt VRLA jar in a cabinet of 40 jars supporting a UPS. Within a two week period, the jar went from a stable 2.95 milli ohm reading, to a 189 percent rise in the first week and a continued rise to 322 percent in the second week before finally being replaced. The graph shows a return to acceptable levels upon replacement.

A New Battery Management Recommendation for Data Centers

A revised battery management program recommendation has been developed using continuous remote battery monitoring as its cornerstone. By implementing this program, risk due to battery failure can be eliminated at the same time as battery maintenance and replacement costs are reduced. Batteries are managed by data rather than a set schedule. The features of such a program follow:

1) Continuous, remote monitoring of battery parameters including battery temperature, voltage, and impedance - in real time
2) Trained personnel analyze the data from a central control room, at least on a weekly basis, and contact the facilities engineer or local battery service provider when a problem occurs
3) Problem batteries are replaced or serviced as needed
4) Physical inspection of batteries reduced to once per year
5) Battery string replacement extended to the point where total string impedance has increased 10%

Summary

The quest in search of 99.99999 percent power reliability realizes great equipment costs associated with designing systems that reduce the risk of battery failure. The number of redundant battery strings and UPS systems designed into today's mission critical backup power systems clearly illustrate this. Trend data obtained from online battery monitoring supports industry claims that the battery, by an overwhelming margin, should be the ultimate concern in today's UPS systems - especially since just one rogue battery is able to cause a complete system failure. As a result, UPS systems stand at risk of failure a greater portion of time than is realized.

This insight justifies a shift in the way batteries are monitored and maintained. By moving to a battery management program based on real-time, data-based, continuous impedance measurement, risk due to battery failure can be virtually eliminated while lowering maintenance costs. Redundant UPS and battery systems may no longer be required as the risk of premature battery failure disappears. It is time to change the way batteries are managed.

BTECH has just completed a white paper entitled "Remote Monitoring of VRLA Batteries - Uncovering a Hidden Risk". Contact BTECH for a copy of the paper.

Thomas Leonard, Vice President, COO is the managing partner and Gene Bohensky the Sales Manager of BTECH, Inc.

Contact BTECH, Inc. at 973-683-9950 or visit www.btechinc.com