From 446db7ac10cba5aa8576ff32531eee2a7826706b Mon Sep 17 00:00:00 2001 From: Andrew Clayton Date: Sun, 11 Aug 2024 16:54:13 +0100 Subject: [PATCH] lib: Better available cpu count determination on Linux At startup, the unit router process creates a number of threads, it tries to create the same number of threads (not incl the main thread) as there are 'cpus' in the system. On Linux the number of available cpus is determined via a call to sysconf(_SC_NPROCESSORS_ONLN); in a lot of cases this produces the right result, i.e. on a four cpu system this will return 4. However this can break down if unit has been restricted in the cpus it's allowed to run on via something like cpuset()'s and/or sched_setaffinity(2). For example, on a four 'cpu' system, starting unit will create an extra 4 router threads $ /opt/unit/sbin/unitd $ ps -efL | grep router andrew 234102 234099 234102 0 5 17:00 pts/10 00:00:00 unit: router andrew 234102 234099 234103 0 5 17:00 pts/10 00:00:00 unit: router andrew 234102 234099 234104 0 5 17:00 pts/10 00:00:00 unit: router andrew 234102 234099 234105 0 5 17:00 pts/10 00:00:00 unit: router andrew 234102 234099 234106 0 5 17:00 pts/10 00:00:00 unit: router Say we want to limit unit to two cpus, i.e. $ taskset -a -c 2-3 /opt/unit/sbin/unitd $ ps -efL | grep router andrew 235772 235769 235772 0 5 17:08 pts/10 00:00:00 unit: router andrew 235772 235769 235773 0 5 17:08 pts/10 00:00:00 unit: router andrew 235772 235769 235774 0 5 17:08 pts/10 00:00:00 unit: router andrew 235772 235769 235775 0 5 17:08 pts/10 00:00:00 unit: router andrew 235772 235769 235776 0 5 17:08 pts/10 00:00:00 unit: router So despite limiting unit to two cpus $ grep Cpus_allowed_list /proc/235772/status Cpus_allowed_list: 2-3 It still created 4 threads, probably not such an issue in this case, but if we had a 64 'cpu' system and wanted to limit unit two cpus, then we'd have 64 threads vying to run on two cpus and with our spinlock implementation this can cause a lot of thread scheduling and congestion overhead. Besides, our intention is currently to create nr router threads == nr cpus. To resolve this, on Linux at least, this patch makes use of sched_getaffinity(2) to determine what cpus unit is actually allowed to run on. We still use the result of sysconf(_SC_NPROCESSORS_ONLN); as a fallback, we also use its result to allocate the required cpuset size (where sched_getaffinity() will store its result) as the standard cpu_set_t only has space to store 1023 cpus. So with this patch if we try to limit unit to two cpus we now get $ taskset -a -c 2-3 /opt/unit/sbin/unitd $ ps -efL | grep router andrew 236887 236884 236887 0 3 17:20 pts/10 00:00:00 unit: router andrew 236887 236884 236888 0 3 17:20 pts/10 00:00:00 unit: router andrew 236887 236884 236889 0 3 17:20 pts/10 00:00:00 unit: router This also applies to the likes of docker, if you run docker with the --cpuset-cpus="" option, unit will now create a number of router threads that matches the cpu count specified. Perhaps useful if you are running a number of unit docker instances on a high cpu count machine. Link: Signed-off-by: Andrew Clayton --- src/nxt_lib.c | 27 +++++++++++++++++++++++---- 1 file changed, 23 insertions(+), 4 deletions(-) diff --git a/src/nxt_lib.c b/src/nxt_lib.c index aba07dda4..de23ce0ab 100644 --- a/src/nxt_lib.c +++ b/src/nxt_lib.c @@ -32,7 +32,7 @@ const char *malloc_conf = "junk:true"; nxt_int_t nxt_lib_start(const char *app, char **argv, char ***envp) { - int n; + int n = 0; nxt_int_t flags; nxt_bool_t update; nxt_thread_t *thread; @@ -87,13 +87,32 @@ nxt_lib_start(const char *app, char **argv, char ***envp) #ifdef _SC_NPROCESSORS_ONLN /* Linux, FreeBSD, Solaris, MacOSX. */ n = sysconf(_SC_NPROCESSORS_ONLN); +#endif + +#if (NXT_HAVE_LINUX_SCHED_GETAFFINITY) + if (n > 0) { + int err; + size_t size; + cpu_set_t *set; + + set = CPU_ALLOC(n); + if (set == NULL) { + return NXT_ERROR; + } + + size = CPU_ALLOC_SIZE(n); + + err = sched_getaffinity(0, size, set); + if (err == 0) { + n = CPU_COUNT_S(size, set); + } + + CPU_FREE(set); + } #elif (NXT_HPUX) n = mpctl(MPC_GETNUMSPUS, NULL, NULL); -#else - n = 0; - #endif nxt_debug(&nxt_main_task, "ncpu: %d", n);