benjojo.co.uk

one honk maybe more

benjojo posted 21 Nov 2024 15:33 +0000

Mildly interesting, got a alert of a box going mental on the load avg

 Tasks: 155, 243 thr; 2 running
 Load average: 5922.82 4616.68 2131.84
 Uptime: 151 days(!), 22:30:04

dmesg said

[12507760.522357] INFO: task kcompactd0:32 blocked for more than 1208 seconds.
[12507760.522411]       Not tainted 5.10.[redacted]
[12507760.522446] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[12507760.522491] task:kcompactd0      state:D stack:    0 pid:   32 ppid:     2 flags:0x00004000

It turned out that one sshfs PID had decided to become a slow moving fork bomb..? Sure I guess, that's a new one

benjojo replied 21 Nov 2024 15:38 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/hp6jkxCtlt7189Z44P

also TIL that somewhere in my RRD/CollectD setup it maxes out at 5000~ load average before it gives up

A RRD graph showing 3 lines for load average over a 1 min, 5 min, and 15 min, the 1 min line goes rapidly all the way to 5.0k, before stopping being reported, while all of the other lines slowly climb there before all crashing down

karppinen@mastodon.o.. replied 22 Nov 2024 05:52 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/PZpB2Tt4XhDBpj926c

@benjojo speaking of monitoring limits, just got a Zabbix alert for a bunch of my MX204s having rebooted ~simultaneously. Very disconcerting.

They didn’t though, they just had 497 days of uptime which seems to roll around a 32-bit unsigned counter with the unit being a 1/100th of second(?!)

benjojo replied 22 Nov 2024 10:09 +0000
in reply to: https://mastodon.online/users/karppinen/statuses/113525045876262570

@karppinen yeah @IPngNetworks had the same thing https://ublog.tech/@IPngNetworks/113086850154843521