benjojo.co.uk

benjojo posted 14 Jan 2025 17:39 +0000

The PDU, it screams, and honestly... Me too buddy

Two PDUs each with a seven segment display, one displays 04, the other displays AA

benjojo replied 14 Jan 2025 17:40 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/y29pt4T3Rr4VqCY182

The mischief maker that took down chunks of my infrastructure at 3AM has been successfully caught

1U APC PDU being held by the cable like you would a dead animal

alarig@hostux.social replied 14 Jan 2025 18:32 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/rG21Hj5V31P9L2D1g2

@benjojo How did the PDU blow?

benjojo replied 14 Jan 2025 20:15 +0000
in reply to: https://hostux.social/users/alarig/statuses/113828138902177836

@alarig Seems that the thing just died of age/temperature.

I noticed it went into "ER" state in mid December, but I didnt have the time to replace it then, well, I paid the price last night as it finally blew itself apart and shut all of the relays for a hour!

benjojo replied 14 Jan 2025 17:44 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/rG21Hj5V31P9L2D1g2

Power Disturbed Unit

benjojo replied 14 Jan 2025 20:08 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/g85R4vT85Xp3Mgn2Z5

PDU hell is a real place where PDUs will be sent at the first sign of defiance.

Anyway, This AP7921 is quite interesting inside:

1) They have a "custom" chip, that based on vibes is actually some Atmel SKU

2) They love these yellow vampire taps thing, The dual PSUs inside for the control are powered via a vampire tap to the live

3) Ground and Neutral is just one long wire with vampire taps

Visually this unit seems fine, I assume the caps have gone bad (as I've fixed other APC PDUs via recapping). No idea how old this unit it, copy right on the boards say 2003 and 2005 though. So they are either very lazy with updating their board revisions, or this thing had a long life before screaming into the abyss as it failed

Various internal photos of a AP7921, best described on the main post

jamesog@mastodon.soc.. replied 14 Jan 2025 20:16 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/DX1s62dls13mZ44cXx

@benjojo Did I ever tell you about the time @vaelen discovered a bad PDU in (I think) SJC? He noticed a bad smell and some smouldering.

A ticket got created with Equinix, I took the pop offline overnight so they could swap it out. The tech got back to me and said "we couldn't find the bad PDU” and I replied "it's the one that is PRACTICALLY ON FIRE”.

Good times.

benjojo replied 14 Jan 2025 20:25 +0000
in reply to: https://mastodon.social/users/jamesog/statuses/113828546317134705

@jamesog I've heard similar stories someone being sent out to investigate why a PDU dropped out to find it quite literally "oxidising" in real time in front of their eyes

Scary shit!

vaelen@tech.lgbt replied 15 Jan 2025 21:36 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/1YKf98wKB2h41s1hZt

@benjojo @jamesog
In this case I was not there for the PDU at all! it was "fine", smelled awful, the C19 connector had become gooey and soft.. sparked the second I touched it (with gloves :) )

cuzzo@mastodon.onlin.. replied 14 Jan 2025 20:30 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/DX1s62dls13mZ44cXx

@benjojo I would think that’s a date code (0610) on the chip in the third pic? 6th week of 2010?

benjojo replied 14 Jan 2025 20:42 +0000
in reply to: https://mastodon.online/users/cuzzo/statuses/113828602077722064

@cuzzo yeah nice catch!

penguin42@mastodon.o.. replied 14 Jan 2025 21:21 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/DX1s62dls13mZ44cXx

@benjojo That thing with one long wire is weird; especially the way it just ends with a cut, uncapped wire.

benjojo replied 14 Jan 2025 21:23 +0000
in reply to: https://mastodon.org.uk/users/penguin42/statuses/113828801655748897

@penguin42 in theory it's fine because it's not live (assuming the plugs have been correctly wired up etc etc etc), but yeah it's certainly a choice on APCs part

wolf480pl@mstdn.io replied 14 Jan 2025 23:53 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/DX1s62dls13mZ44cXx

@benjojo are the vampire taps on live visible in the photos? All I see is vampire taps on neutral and grpund, while the live seems to just go straight from the inlet to the second long PCB behind the one with relays

benjojo replied 15 Jan 2025 00:20 +0000
in reply to: https://mstdn.io/users/wolf480pl/statuses/113829402186004994

@wolf480pl There is a single tap on the live, see the photo where my two fingers straddle the live, in the middle of that link there is a tap for the control plane power supplies!

bencc@morehammer.uk replied 14 Jan 2025 22:03 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/DX1s62dls13mZ44cXx

@benjojo First 2 digits of serial number seem to be year of manufacture.

WiteWulf@cyberplace... replied 14 Jan 2025 22:14 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/DX1s62dls13mZ44cXx

@benjojo we stopped buying APC switched PDUs after one too many failures. They’re just too damn flaky, particularly after power outages

TheDragon@hachyderm... replied 14 Jan 2025 22:36 +0000
in reply to: https://cyberplace.social/users/WiteWulf/statuses/113829011422954012

@WiteWulf @benjojo any recommendations for more reliable ones?

WiteWulf@cyberplace... replied 14 Jan 2025 22:39 +0000
in reply to: https://hachyderm.io/users/TheDragon/statuses/113829099139951177

@TheDragon @benjojo we just decided we didn’t need per outlet managed PDUs, and just buy metered APC ones instead. They’re a lot more reliable.

benjojo replied 14 Jan 2025 22:44 +0000
in reply to: https://hachyderm.io/users/TheDragon/statuses/113829099139951177

@TheDragon @WiteWulf my desk is now a Raritan Per Port Metered PDU and I've been extremely impressed at it.

Raritan seeming actually gives a shit about the software they are shipping, the whole thing seems light years ahead of anything APC have put out, I didnt know I could be actually happy about PDUs!

WiteWulf@cyberplace... replied 14 Jan 2025 22:46 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/1kXx8mXGHXzf9t871c

@benjojo @TheDragon I have an ex-DC 7921 in my home entertainment centre. I’ll look at the raritan kit when it eventually gives up

astraleureka@social... replied 15 Jan 2025 01:07 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/DX1s62dls13mZ44cXx

@benjojo ah APC, out of all the power delivery vendors they've been responsible for the most fires. almost everybody I know who has done DC tech work has had some sort of smoking APC experience. my favorite was installing a half rack sized symmetra UPS and not 30 minutes later, wheeling a smoldering regulator module out of the building

asl@mastodon.launay... replied 14 Jan 2025 17:59 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/g85R4vT85Xp3Mgn2Z5

@benjojo I had the same problem with an older unit of the same type and manufacturer... I only trust redundant psu in servers now, eventually combined with two of those things.

In the same line, I had a ip power 9280 from... someone... which sent packet from 0.0.0.0, and mac 0:0:0 etc. Took me weeks to find the culprit.

ledeuns@bsd.network replied 14 Jan 2025 18:14 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/y29pt4T3Rr4VqCY182

@benjojo So you have an A feed and an AA feed 😅