Performing System Tuning for HAPEE

To get the best HAPEE performance for your particular environment, you can configure some processes to run with certain CPU cores for optimal results.

Enable sysctl features

Sysctl is a feature of certain Unix-like operating systems that reads and modifies the attributes of the system kernel such as its version number, maximum limits, and security settings.

The package hapee-1.6r2-base adds some system tuning to the file /etc/sysctl.conf. These sysctl features are not enabled by default. As system administrator, you can open this file and enable the sysctls that you need.

Each sysctl has with comments to explain its usage and benefits.

Disable irqbalance

Irqbalance is a Linux daemon that helps to balance the CPU load generated by interrupts across all CPUs. Irqbalance identifies the highest volume interrupt sources and isolates them to a single CPU, so that the load is spread as much as possible over an entire processor set while minimizing cache hit rates for IRQ handlers.

However, irqbalance tends to use all CPU cores for network interrupts, which can be counterproductive with HAProxy.

So, before you perform CPU pinning, you must stop and uninstall irqbalance as follows:

On CentOS/RedHat:
service irqbalance stop
yum remove irqbalance
On Debian/Ubuntu
service irqbalance stop
apt-getremove irqbalance

Achieving Processor Affinity

Processor affinity, or CPU pinning, enables the binding and unbinding of a process or a thread to a CPU or a range of CPUs, so that the process or thread executes only on the designated CPU or CPUs rather than on any CPU.

In the case of HAPEE, it is required to place network IOs and certain processes on different CPU cores. You can contact HAProxy for assistance if necessary.

Understanding processes and CPU cores

Before you pin processes to CPU cores, it is important to know how the kernel sees physical CPUs, cores, and threads in dual-core machines.

To know which cores are hosted on the same physical CPU, run:

cat /sys/bus/cpu/devices/cpu0/topology/core_siblings_list
0-3,8-11

In this example, core0 to core3 and core8 to core11 are on the same physical CPU, hence they share the same CPU L3 memory.

Pinning network interrupts to cores

To get the best performance from HAProxy, it is important to pin HAProxy and network interrupts on the same physical CPU, but on different cores. This way, they can benefit from the maximum CPU cycles and share the CPU L3 memory.

Pinning network interrupts depends on the network card driver. Here we introduce a generic method.

To know which IRQ your network interface uses, run:

fgrep eth /proc/interrupts
CPU0       CPU1       CPU2       CPU3
20:       47088     293673      27185      36396   IO-APIC-fasteoi   eth0
  • The first column gives the IRQ associated to the network interface.
  • The other columns report the number of interrupts reported by CPU core.

With this information, you can pin this IRQ number to a CPU core using the /proc file system:

1. In /proc/irq/[IRQ number], locate a file called smp_affinity. It contains a bit mask that represents the processors to which this [IRQ number] can be routed. Each field in the bit mask corresponds to a processor.

The [IRQ number]in the smp_affinityfile is in hexadecimal format. Hence, you must convert the bit patterns from binary to hex before you set them in the /proc file system.

Example: Pin eth0 interrupts on CPU0 only, bit pattern: 0001:

echo 1 > /proc/irq/20/smp_affinity

Example: Pineth0 interrupts on CPU1 only, bit pattern: 0010:

echo 2 > /proc/irq/20/smp_affinity

Example: Pin eth0 interrupts on CPU0 and CPU1, bit pattern: 0011:

echo 3 > /proc/irq/20/smp_affinity

Pinning HAProxy to a core

HAProxy can pin itself to a CPU core:

1. In the global section of the file /etc/hapee-1.6/hapee-lb.cfg, add the cpu-map instruction which takes two arguments: the process id and the CPU core id.

The following example pins HAProxy process 1 onto CPU core 2:

cpu-map 1 2

Pinning other processes

Other processes that could require intensive CPU cycles are best moved to other cores.

The following is a non-exhaustive list of processes or software that you may want to avoid using on same CPU core as HAProxy or network IOs:

  • backup clients
  • munin client
  • nagios plugins or clients
  • snmpd / net-snmp
  • syslog / rsyslogd / syslog-ng
  • zabbix clients

You can use taskset to pin daemons or tools to different CPU cores.