To get the best HAPEE performance for your particular environment, you can configure some processes to run with certain CPU cores for optimal results.
Enable SYSCTL features
In Unix-like operating systems, the feature sysctl reads and modifies the attributes of the system kernel such as its version number, maximum limits, and security settings.
The package hapee-1.6r2-base adds some recommended sysctl settings to the file
/etc/sysctl.d/30-hapee-1.6.conf. These sysctl features are not enabled by default. As system administrator, you can open this file and enable the sysctls that you need.
Each sysctl has comments to explain its usage and benefits.
The Linux daemon irqbalance is a Linux daemon helps to balance the CPU load generated by interrupts across all CPUs. It identifies the highest volume interrupt sources and isolates them to a single CPU, so that the load is spread as much as possible over an entire processor set while minimizing cache hit rates for IRQ handlers.
However, irqbalance tends to use all CPU cores for network interrupts, which can be counterproductive with HAProxy.
So, before you perform CPU pinning, you must stop and uninstall irqbalance as follows:
service irqbalance stop yum remove irqbalance
service irqbalance stop apt-getremove irqbalance
Achieve Processor Affinity
Processor affinity, or CPU pinning, enables the binding and unbinding of a process or a thread to a CPU or a range of CPUs, so that the process or thread executes only on the designated CPU or CPUs rather than on any CPU.
In the case of HAPEE, it is required to place network IOs and certain processes on different CPU cores. You can contact HAProxy for assistance if necessary.
Understand processes and CPU cores
Before you pin processes to CPU cores, it is important to know how the kernel sees physical CPUs, cores, and threads in dual-core machines.
To know which cores are hosted on the same physical CPU, run:
cat /sys/bus/cpu/devices/cpu0/topology/core_siblings_list 0-3,8-11
In this example, core0 to core3 and core8 to core11 are on the same physical CPU, hence they share the same CPU L3 memory.
Pin network interrupts to cores
To get the best performance from HAProxy, it is important to pin HAProxy and network interrupts on the same physical CPU, but on different cores. This way, they can benefit from the maximum CPU cycles and share the CPU L3 memory.
Pinning network interrupts depends on the network card driver. Here we introduce a generic method.
To know which IRQ your network interface uses, run:
fgrep eth /proc/interrupts CPU0 CPU1 CPU2 CPU3 20: 47088 293673 27185 36396 IO-APIC-fasteoi eth0
- The first column gives the IRQ associated to the network interface.
- The other columns report the number of interrupts reported by CPU core.
With this information, you can pin this IRQ number to a CPU core using the
/proc file system:
/proc/irq/<IRQ number>, locate a file called smp_affinity. It contains a bit mask that represents the processors to which this
<IRQ number> can be routed. Each field in the bit mask corresponds to a processor.
<IRQ number> in the smp_affinity file is in hexadecimal format. Hence, you must convert the bit patterns from binary to hex before you set them in the
/proc file system.
Pin eth0 interrupts on CPU0 only, bit pattern:
echo 1 > /proc/irq/20/smp_affinity
Pin eth0 interrupts on CPU1 only, bit pattern:
echo 2 > /proc/irq/20/smp_affinity
Pin eth0 interrupts on CPU0 and CPU1, bit pattern:
echo 3 > /proc/irq/20/smp_affinity
Pin HAProxy to a core
HAProxy can pin itself to a CPU core:
In the global section of the file
/etc/hapee-1.6/hapee-lb.cfg, add the
cpu-map instruction which takes two arguments: the
<process id> and the
<CPU core id>.
The following example pins HAProxy process 1 onto CPU core 2:
cpu-map 1 2
Pin other processes
Other processes that could require intensive CPU cycles are best moved to other cores.
The following is a non-exhaustive list of processes or software that you may want to avoid using on same CPU core as HAProxy or network IOs:
- backup clients
- munin client
- nagios plugins or clients
- snmpd / net-snmp
- syslog / rsyslogd / syslog-ng
- zabbix clients
You can use
taskset to pin daemons or tools to different CPU cores.