@dee So using https://github.com/benjojo/numa I made my bgpd's:

(A) Detect the amount of CPU Sockets
(B) Randomly pick one
(C) Set the scheduling mask to use that CPU sockets threads/cores only
(D) Set the memory policy (MPOL_LOCAL) to only allocate RAM from the socket it's scheduled on
(E) re-exec the whole program, you need to do this in go because set_mempolicy only works on the thread, or if you can run it on the root thread (hint: in go this is hard) you can have it apply to the whole application. Just re-exec'ing yourself it easier.

It is worth pointing out this only makes sense for memory heavy workloads (all that bgp.tools really does is memory access), and in places where NUMA is actually exposed to you, so cloud environments this is not useful. Given that most people are in the cloud/IAAS environments, this is not that applicable to most people anymore (other than the cheap bastards like me!)

I'm 98% sure this saves CPU "%" because while a CPU Thread is stuck waiting on the other sockets memory access, it can't do anything else, so it's not like the threads are sitting doing stuff in this case, they are just blocked. So what I've really done here is reduce memory access latency.

The only downside is that I have to be a little more careful with memory usage now, since I can now "OOM" on a single socket:

~# numastat -m
                          Node 0          Node 1           Total
                 --------------- --------------- ---------------
MemTotal               256568.49       258038.88       514607.36
MemFree                 23292.68          464.24        23756.92
MemUsed                233275.81       257574.63       490850.45

But I like my current odds.