APC BX1000 Click-of-Death Repair

I was generally impressed by the build and component quality that went into APC's 10-year-old APC BX1000. Little did I know I'd be writing a repair story about it two years later.

If there's something you should have learned about me by now, it's that I despise hardware the manufacturer cut corners to make. Also, I hate giving up on what should still be usable hardware. When my BX1000 failed in December 2015, despite what appeared to be great design and execution, I was surprised. I cracked it open and couldn’t find anything visually wrong with it. Quick spot-checks failed to reveal anything obvious. So, I bought the Cyber Power LX1500 and tossed the BX1000 in my “fix it later” pile. After nine months of hunting down lower-end to mid-range alternatives, tearing them down and being repeatedly disappointed by their quality, I decided it'd be a good time to revisit the APC unit for a more serious attempt at UPS necromancy.

What symptoms did my BX1000 present back then? A picture is worth a thousand words, so I’ll let this short video tell the story before I go into more depth:

APC BX1000

The clicking did not start quite so bad. It began around October 2015, coincidentally when the landlord turned the electric boiler back on. The UPS responded with a few clicks each time the boiler and its pump kicked on. I measured line voltage and found it to dip as low as 107V on cold nights, so I initially shrugged it off as the boiler start-up dragging line voltage low enough to make the UPS’ voltage regulation kick in.

It got progressively worse from there until December, when every minor line voltage fluctuation caused the UPS to go into a clicking frenzy, occasionally followed by the UPS switching to battery backup for an increasingly long time, and then going back to line power. Checking AC voltage with my oscilloscope revealed nothing to explain the UPS’ behavior, and probing its output showed that the clicking happened to be the AVR transformer getting switched in and out of boost mode, which was consistent with the line voltage sags I measured previously. By the time I retired the BX1000, it would no longer reliably power up while connected to AC power. When it did, it took forever for the UPS to switch back to line power, even when my instrumentation said the input looked perfectly fine.

What could cause a UPS to fail in this manner? Mainly, I suspected bad power delivery to the circuitry managing input voltage detection, loss of power or bad power at the UPS micro-controller, or a loss of filtering at the detection circuitry’s inputs causing it to trip on input noise. I don't know how line voltage monitoring issues could be related to the UPS failing to start, but perhaps we'll find out.

Since everything appears functional, aside from the erratic behavior, I am optimistic that today's exploration will end with a simple, definitive fix.

MORE: APC BR1000G Tear-Down
MORE: APC BN650M1-CA Tear-Down
MORE: APC BGE90M-CA Tear-Down

This thread is closed for comments
37 comments
    Your comment
  • Andragoras99
    That was awesome troubleshooting, thank you for sharing.
  • Jacob_130
    When you do fix it, can you YouTube it?
  • Jacob_130
    You should post a video showing the repairs
  • Zaxx420
    'Now on to the next one: why did the capacitor fail in the first place?'

    That's one aspect that sooo many folks never consider or can't be bothered to investigate. Just like a fuse that keeps blowing...there is something that's causing it to blow...even if it lasts a while before blowing. Even worse is when they decide to bridge it with a piece of wire or wrap the blown (glass) fuse with aluminum foil. Now I'm guilty as well but ONLY long enuff to get myself out of a jam...like being stranded on the side of the road or something. Just never do it with a high current load or if it's blowing the instant you turn it on...and never with a sensitive/expensive critical component like the ignition ECU of your self semi-restored muscle car project...don't ask.
  • turkey3_scratch
    You are so smart.
  • Daniel Sauvageau
    2358213 said:
    When you do fix it, can you YouTube it?

    The repair is already done and I'm not set up for recording - don't even own a video camera or webcam. Even if I did own one, I'm no good with talking and working at the same time. Also, if I posted repairs on YouTube, I wouldn't be able to post them on THG since THG is paying me for exclusive publication.

    1712875 said:
    You are so smart.

    Too much credit :) This is basic troubleshooting, anyone going through electronics tech training should be able to do as before graduation.
  • jcamilo70
    I have a 2005 RS1200 that died on me, making clicking sound constantly, and only turning on directly into battery mode.

    Where did you manage to get the replacement pieces? Where could I find the micro-controller for this unit or how should I identify its part number in order to look for it?
  • Daniel Sauvageau
    2358467 said:
    I have a 2005 RS1200 that died on me, making clicking sound constantly, and only turning on directly into battery mode. Where did you manage to get the replacement pieces? Where could I find the micro-controller for this unit or how should I identify its part number in order to look for it?

    The only part I needed to replace here was that single capacitor. Where did I get the capacitor from? It was just one of many leftover parts I've accumulated either from ordering more parts than necessary for past repairs/projects, or parts I had ordered for potential future projects to pad orders for free S&H. I may have over $2000 worth of parts in my spare parts and samples boxes.

    For the micro-controller, I didn't touch it. Even if I had wanted to replace it, they can only be ordered blank from parts distributors (or pre-programmed with your own firmware for large enough orders or distributors that offer such a service for an additional fee) and those would be no good without the correct firmware to put on them. If the controller died, it is game over unless you can find a donor UPS based on the same PCB.
  • powernod
    This example showed us the importance of high quality caps to be used in such critical components such as a UPS/PSU. A single cap-failure (*from the many a UPS uses) and caused total malfunction of the UPS !!

    P.S. This review has also shown us the importance/need for UPSs reviews. Since PSUs are being thoroughly reviewed for many years, i don't understand why UPSs shouldn't be reviewed as well!! A PSU's proper operation is in consequence with the proper operation of the hardware that "feeds" a PSU (*UPS)
    P.S.2 : Great article by the way!!
  • AllyWatts
    Nice!
  • unclebun
    I just sent back in a trade-UPS to APC the exact same unit for the exact same reason. Nice to know the reason why. Mine even started out the same way--clicking when the air conditioner/heat pump cycled, but got much worse over several months' time.
  • Daniel Sauvageau
    1621541 said:
    I just sent back in a trade-UPS to APC the exact same unit for the exact same reason. Nice to know the reason why. Mine even started out the same way--clicking when the air conditioner/heat pump cycled, but got much worse over several months' time.

    While we don't know for certain if it was truly the exact same problem, I agree that there is a strong probability.

    Makes you wonder how well all of those UPSes I got over the past year with (mostly) lower quality capacitors are going to fare over the next 5-10 years. I would be surprised to see any of them match the BX1000's reliability.
  • nukemaster
    Another great one.

    I did not know that a charge pump was THAT inefficient since Maxim rates the ones they sell(add some caps and call it a day) pretty high.

    My USB sound card also uses one to get more voltage swing(thanks to a negative voltage) for the line driver(opamp + charge pump in one.).
  • zodiacfml
    I like how the article is written without worry of the general audience of Tom's.

    I feel bad for my two motherboards and routers which died prematurely, more likely due to capacitors. The bad capacitor in the photo above has a slight bulge.
  • Daniel Sauvageau
    35532 said:
    I did not know that a charge pump was THAT inefficient since Maxim rates the ones they sell(add some caps and call it a day) pretty high.

    Charging capacitors from a voltage source is only 50% efficient. In a boost pump though, that energy gets added to the source's voltage and for a voltage doubling charge pump, half of the total energy output is provided by the capacitor. If that half is 50% efficient, then overall efficiency becomes 75% under ideal conditions.

    In the BX1000's case though, it is a voltage inverting charge pump, not a boost pump, so direct contribution from the source to output power does not apply. I should have been more specific. I have never really worked with charge pumps and the details skipped my mind. Also, APC's implementation here is cobbled up from a bunch of discrete diodes and transistors which loses about 4V, making efficiency that much worse - clearly a no-concern here.

    Bottom line: if you are genuinely concerned with efficiency, use a magnetics-based switching converter. With modern micro-power switchers operating at frequencies beyond 1MHz, you can use inductors under 4.7uH which are nearly as small and inexpensive as capacitors without charge pumps' caveats and higher efficiency across a much broader input-output voltage range.

    Historically, charge pumps came about to avoid using large and expensive inductors for low power supplies. Ironically, today's low-power switchers can use smaller inductors than those recommended in many charge pumps' reference design input EMI chokes.
  • Daniel Sauvageau
    191196 said:
    The bad capacitor in the photo above has a slight bulge.

    If by 'bulge' you mean that the bad capacitor's top isn't a sharp square bend around the edge as they often look like they are on larger capacitors, they are all this way. Small capacitors have rounded edges to reduce tearing when forming the casings with a die press and rounded edges also help the forming dies last longer as there is no sharp edge biting into the marterial.
  • JackNaylorPE
    Tho, yours appears to be just a bit older ... I basically wrote off APC after they were purchased by Schneider Electric .... around 2006-2007 IIRC. All of our APC equipment has since been phased out as it's usually been more expensive to replace the batteries than to replace the unit.
  • Daniel Sauvageau
    35894 said:
    All of our APC equipment has since been phased out as it's usually been more expensive to replace the batteries than to replace the unit.

    It is a shame that getting proper replacement batteries in small quantities through official channels often borders on prohibitively expensive. You won't have the dilemma of replacing batteries on many of the UPSes I have torn down over the past year though as I would not be surprised if the batteries outlived their respective UPS' capacitors.

    For modestly loaded UPS, you can rebuild packs with more readily available lower performance batteries for less than half the cost of official replacements.
  • texastim65
    I have a 2005/2006 era 1500 model.

    A couple months ago it started endlessly switching from line mode to battery mode (making that annoying click sound while switching). I brought home a meter from work and verified my outlet power was fine and I event plugged the UPS into another UPS at work just to be absolutely sure.

    At that point I gave up and bought a replacement 1500 model. Now I wonder if I have the same problem you did (bad capacitor, maybe even the same one) and if I could get it working again if I replaced the capacitors (I don't have a scope at work to figure out which capacitor).

    The fact both yours and mine worked for 10 years (mine was on 24x7 over that period) makes me think it's the life of the capacitor reason it failed.
  • Daniel Sauvageau
    1111004 said:
    The fact both yours and mine worked for 10 years (mine was on 24x7 over that period) makes me think it's the life of the capacitor reason it failed.

    The symptoms are common enough. Are they all C41 or equivalent failures? We'd have to hear about other people's repair successes on this model range to tell. I would not be surprised.
  • JackNaylorPE
    1736083 said:
    For modestly loaded UPS, you can rebuild packs with more readily available lower performance batteries for less than half the cost of official replacements.


    Unfortunately, determining which offerings will make a suitable replacement is not an easy task ... one thing about APC was ... back in the day before Schneider, everyone used them. An electrician friend of mine had contracts to replace UPSs at several large NYC firms which they would do in a 2 year cycle. For fear of resale I guess, his contract required that he install the new units, leave the old but properly "dispose of" the old batteries. And he'd dispose of a few of them by giving them to me :)

    But since APCs fall from grace,with no clear favorite., the firms began opening the specs and spreading out the replacement cycle... buying a certain quantity of replacements every 6 months. With a hodgepodge of makes and models knowing what might work in another unit he started using a disposal company to get rid of them. But I don't even think he has the contract anymore.

    I have checked a few sites "compatibility lists" but saw many instances where the claimed replacement had different specs and even dimensions. Given my lack of knowledge of what differences might matter and which wouldn't, I just feel safer not buying something for half price that I am not so sure of. I am down to just 2 UPS's now... one on my CAD workstation which also serves as SOHO file / media server and an old one that I use to extra time outta my lappie when power is out.
  • felix666
    A lot of Sony equipment in the 90s went bad because of bad smt electrolytics (including my costly semipro Hi-8 camcorder). Foiled components can be easily affected by manufacturing variances of their internals. The best examples nowadays are specific batches of lithium batteries that catch fire.
    These SMT electrolytics caps are easy to remove with a regular soldering iron. The leads are just folded back on their plastic base; you reflow one pin and bend the part backwards. It unfolds the soldered lead, and allows to raise the melted one. I've done it often. No damage to either the part on the pc board. The hot air stresses the other parts around and I only use it when I can't do otherwise, and still I will put tape over the other parts to reduce the thermal shock.
    I like your articles about those UPSes as I have two of them, and they just keep up working fine.
  • allrock
    Given the symptoms and voltage drop from the boiler starting I would check and make sure all the neutral bus connections in the electrical panel are not loose , a loose neutral return to the power pole transformer will cause all sorts of strange voltages (And voltage drops) to appear in household electrical circuits when the neutral reference goes high resistance it will try and return via the bonded ground rod instead (which is not directly referenced to the transformers center tap) and power sensing electronics take a beating balance wise when this happens...
  • Daniel Sauvageau
    35894 said:
    Unfortunately, determining which offerings will make a suitable replacement is not an easy task

    How easy or difficult it is will depend on your requirements, The first concern is matching the size. The next concern would be the batteries maximum discharge current rating. If you only use the UPS up to about half of its original rating though, this will rarely be an issue and that's how I'm not particularly worried about using some generic batteries in my BX1000.

    566328 said:
    The leads are just folded back on their plastic base; you reflow one pin and bend the part backwards. It unfolds the soldered lead, and allows to raise the melted one.

    The pins on these capacitors are thin tape that goes through the plastic coaster and fold outwards from the middle. Their pads extend under the coaster and "folding the capacitor backwards" after flowing one pin would not be possible - you would likely rip the other pin's pad off. If you want to discuss thermal shock, ask Louis Rossman who does all of his SMD Apple MacBook rework at maximum soldering iron and hot air temperature :)

    376938 said:
    Given the symptoms and voltage drop from the boiler starting I would check and make sure all the neutral bus connections in the electrical panel are not loose

    I did check that back then and found no difference at the building's utility bus bar. The boiler operates on 240V with no neutral, which means no possible effect on neutral balance. It also has a separate pole drop from the rest of the building.