Computer gore! Some network device between San Francisco and London is sometimes corrupting packets (in a way that the packet checksum survives), causing my collectd metrics packets to show up with broken hostnames (and often internal metric values...) and also causing fun named files to be made on my metrics server... I also wonder sometimes if the destination is sometimes corrupted and my metrics packet zoops off somewhere else on the internet (The right hostname is
airmail .benjojo .co.uk)
one honk maybe more
yadt@tech.lgbt
replied 23 Jun 2025 18:58 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/xb5wSS8wBNpx1m1NGx
benjojo
replied 23 Jun 2025 18:59 +0000
in reply to: https://tech.lgbt/users/yadt/statuses/114734211192258071
zev@honk.bewilderbee..
replied 23 Jun 2025 19:05 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/xb5wSS8wBNpx1m1NGx
@benjojo Fun that it's almost always length-preserving (single bit flips in most of the cases shown), but occasionally (e.g.
beljjo) drops a whole byte...wonder what's going on there?
benjojo
replied 23 Jun 2025 19:12 +0000
in reply to: https://honk.bewilderbeest.net/u/zev/h/gYlz8gZVGyFHGNt3Q4
@zev I suspect (but have not looked) that collectd's packet format is checking string lengths, so only the ones that sorta vibe check out will survive
famfo@frogs.lgbt
replied 23 Jun 2025 19:07 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/xb5wSS8wBNpx1m1NGx
@benjojo are you loosing metric data or does most of it arrive as expected... just sometimes a bit mangled?
benjojo
replied 23 Jun 2025 19:13 +0000
in reply to: https://frogs.lgbt/users/famfo/statuses/114734246034372058
@famfo Na, it happens in bursts and when it does all kinds of stuff gets corrupt, so I'm pretty sure large areas of the packet is subject to some corruption
domi@donotsta.re
replied 23 Jun 2025 19:08 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/xb5wSS8wBNpx1m1NGx
benjojo
replied 23 Jun 2025 19:14 +0000
in reply to: https://donotsta.re/objects/2b3d17d6-2ff7-4959-adb5-92d91d9ad4f6
@domi it's highly rude for some bitfips to trigger some char to turn into a
*, because that makes deleting these files a bit scary :P
jeroen@secluded.ch
replied 23 Jun 2025 19:18 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/xb5wSS8wBNpx1m1NGx
benjojo
replied 23 Jun 2025 19:20 +0000
in reply to: https://secluded.ch/users/jeroen/statuses/114734288750702079
@jeroen collectd's basic protocol is just UDP "fire and forget", this is useful (collectd will often be the last surviving process on a machine as it's blowing up), but yeah, the basic protocol has no signing and encryption. I have ingress ACL's of course, but that wont help me if something is chewing up the packets (I have prometheus too, but collectd+RRD has it's own useful charms)
erincandescent@akko...
replied 23 Jun 2025 19:33 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/xb5wSS8wBNpx1m1NGx
benjojo
replied 23 Jun 2025 22:15 +0000
in reply to: https://akko.erincandescent.net/objects/26ec5fee-9e43-48c5-9fb9-cc5feefb1b13
fazalmajid@social.vi..
replied 23 Jun 2025 23:24 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/xb5wSS8wBNpx1m1NGx
@benjojo I've seen this behavior before. Packet with security hash failing but TCP checksum intact. My best guess is a Singapore deep packet inspection device was experiencing random bit corruptions, but since it recalculates the TCP checksum, that one survived unscathed.
jbaert@mastodon.soci..
replied 23 Jun 2025 23:25 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/xb5wSS8wBNpx1m1NGx
PiiiepsBrummm@chaos...
replied 24 Jun 2025 05:39 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/xb5wSS8wBNpx1m1NGx
@benjojo Ok, my breakfast clown seems to be a little bit off ... ;-)
Perhaps it is the conversion of you metrics packages to imperial ones and back?