@dee So using https://github.com/benjojo/numa I made my bgpd's:

(A) Detect the amount of CPU Sockets
(B) Randomly pick one
(C) Set the scheduling mask to use that CPU sockets threads/cores only
(D) Set the memory policy (MPOL_LOCAL) to only allocate RAM from the socket it's scheduled on
(E) re-exec the whole program, you need to do this in go because set_mempolicy only works on the thread, or if you can run it on the root thread (hint: in go this is hard) you can have it apply to the whole application. Just re-exec'ing yourself it easier.

It is worth pointing out this only makes sense for memory heavy workloads (all that bgp.tools really does is memory access), and in places where NUMA is actually exposed to you, so cloud environments this is not useful. Given that most people are in the cloud/IAAS environments, this is not that applicable to most people anymore (other than the cheap bastards like me!)

I'm 98% sure this saves CPU "%" because while a CPU Thread is stuck waiting on the other sockets memory access, it can't do anything else, so it's not like the threads are sitting doing stuff in this case, they are just blocked. So what I've really done here is reduce memory access latency.

The only downside is that I have to be a little more careful with memory usage now, since I can now "OOM" on a single socket:

~# numastat -m
                          Node 0          Node 1           Total
                 --------------- --------------- ---------------
MemTotal               256568.49       258038.88       514607.36
MemFree                 23292.68          464.24        23756.92
MemUsed                233275.81       257574.63       490850.45

But I like my current odds.

@dee Play IaaS games, get IaaS prizes :)

The last time I priced bgp.tools to run on cloud IaaS it was like 12K GBP a month, I've never spent 12K GBP in hardware capex, and I spend less than 10% of that in opex every month

@jamesog Can't beat a rrd graph! They are written to a tmpfs and every 12 hours sync'd to disk. That way you get the upsides of RRD (long haul downsampling) with little downside (disk destruction). I do have prometheus too, but they are different use cases.

@jamesog Most things are less insane that datadog.

RRD just has different "arrays" for different sample ranges at different intervals. it's retro, but at least you know what you are getting

I also loved watching the NUMA stats on the machine slowly drop as the deployment rolled out. Intel make some nice (and easy to compile) Prometheus exporters that exports out some CPU/RAM metrics, very useful for performance tuning your application (and seeing what is hurting)

https://github.com/intel/pcm/

I have EYPC machines in the fleet too, but their tools have been harder to get working annoyingly, Luckily none of those machines are multi socket.