HAProxy Runtime API
Master CLI
Use the master CLI to supervise HAProxy processes and access their UNIX sockets.
The master CLI allows you to explore and manage the HAProxy processes as system processes without regard to the load balancer application itself. For example, you can display the number of reloads that have occurred and display details about the master and worker processes, including PIDs, runtimes, and whether a worker is attached as the current worker or if it’s an old connection that’s either hung or still processing connections before terminating.
The master CLI is more of a management and debugging tool providing insight into the HAProxy processes and the reload process, unlike the HAProxy Runtime API Unix Socket commands, which provide load balancer application-aware commands. The master CLI does provide “wrapper” commands that you can use to send HAProxy Runtime API commands to the running application, but if you need to perform application-aware actions rather than process-management actions, the HAProxy Runtime API is the preferred tool.
List the master CLI commands using the help command:
nix
nix
outputtext
outputtext
The master CLI is different from the HAProxy Runtime API, which allows you to access the dynamic (in-memory) features of the running HAProxy application, such as stick tables and map files. The master CLI is for managing the workers and UNIX sockets. You can still, however, use the master CLI to send HAProxy Runtime API commands using the master CLI @ and @@ commands.
Understanding the HAProxy master and worker architecture Jump to heading
HAProxy supports graceful, no-impact reloads by overlapping two load balancer workers in a way that allows existing connections to continue normally on the original process while the new process becomes the destination for new connections. These two workers and the reload command are supervised by a master process.
When HAProxy starts, it starts the master process and one worker process. The master manages the worker process, and it manages reloads. It does not handle load balancer application traffic and does not even validate the load balancer configuration. The worker process handles validation of the configuration and execution of the load balancer application.
When you issue a reload command for HAProxy, the master designates the current worker as the leaving worker, and it starts a new worker as the destination for future connections. These two events happen at the same time:
- The leaving worker no longer receives new connections. Its existing connections are allowed to continue uninterrupted. When the last connection closes, the master process terminates the leaving worker. Dynamic (in-memory) configuration changes applied using the HAProxy Runtime API are lost. If you wish to preserve dynamic changes made in the leaving process, you must either save those changes to configuration files or be prepared to use the HAProxy Runtime API to repeat the changes as soon as the reload creates the new process.
- The new worker initializes itself using files from persistent storage: load balancer configuration files, map files, certs, WAF rules files, and so on. It inherits no dynamic (in-memory) configuration from the leaving process. It becomes the destination for all new incoming connections.
There’s a period of time when the leaving worker and the current worker overlap. You can direct master CLI commands at either one of these workers using the @ or @@ prefix. See Address a specific process.
Master CLI invocation Jump to heading
The master CLI is a socket bound to the master process in master-worker mode. This CLI gives access to the UNIX sockets in every running process and allows basic supervision of those workers.
The master CLI is configurable only from the hapee or haproxy program arguments using the -S option to define the socket.
For HAProxy Enterprise: to see the invocation command, view the Systemd unit file: /lib/systemd/system/hapee-<version>-lb.service.
You can also use systemctl to display the unit file:
nix
nix
outputtext
outputtext
Tip
The -S option also accepts bind options separated by commas, for example:
nix
nix
Master CLI commands Jump to heading
Like HAProxy Runtime API commands, you issue master CLI commands by using echo to write them via socat to the socket specified in the unit file:
nix
nix
Address a specific process Jump to heading
The master CLI provides two special command prefixes allowing you to control how the command is executed:
- The
@prefix (single “at” sign) allows you to send an isolated command to a specific process: the master, the current worker, or a leaving worker. - The
@@prefix (double “at” sign) allows you to open a persistent session with a process, sending it one or more commands and waiting for them to finish.
Send an isolated command Jump to heading
The master CLI uses a special prefix notation to access the multiple processes. This notation begins with @.
A @ prefix can be followed by a relative process number or by an exclamation point and a PID, for example, @1 or @!1271. A @ alone specifies the master. The leaving process is accessible only using its PID because it no longer has a relative process number. Only the current worker has a relative process number (@1).
This prefix may be used as a wrapper before a command, indicating that this command and only this one will be sent to the designated process. In this case the full command ends at the end of line or semi-colon like any regular command.
Example: address the current worker
nix
nix
outputtext
outputtext
Example: address the leaving worker by PID during a reload
The leaving worker has no relative process number, so you must address it by PID. Here, connections are still draining while a reload is in progress:
nix
nix
outputtext
outputtext
nix
nix
outputtext
outputtext
The prefix may also be used as a standalone command to switch the default execution context to the designated process, indicating that all subsequent HAProxy Runtime API commands will be executed in that process, until a new @-prefixed command changes the execution context again.
Example: switch context
nix
nix
outputtext
outputtext
Limitations
Some HAProxy Runtime API commands alter a CLI session’s state (for example, set anon, set timeout) and may not behave the same way when run from the master CLI, as commands sent through the master CLI are sent one at a time on their own CLI session. Similarly, commands that actively monitor the CLI for input or closure and are immediately interrupted when the CLI is closed (for example, show events, wait) won’t work as expected through the master CLI because the command’s input is closed after each command. For cases such as these, consider opening a persistent session with @@ instead.
Open a persistent session Jump to heading
Available since:
- HAProxy Runtime API 3.2r1
This prefix, optionally followed by an HAProxy Runtime API command, is very similar to the @ prefix documented above except that it enters the worker process, delivers the whole command line into it as-is and stays there until the command finishes. Semi-colons are delivered as well, allowing you to execute a full pipelined command in a worker process. The connection with the work remains open until the list of commands completes. Any data sent after the commands will be forwarded to the worker process’ CLI and may be consumed by the commands being executed and won’t be parsed by the master process’ CLI, offering a truly bidirectional connection with the worker process. As such, users of such commands must be very careful to wait for the command’s completion before sending new commands to the master CLI.
Instead of executing a single HAProxy Runtime API command, it’s also possible to open a fully interactive session on the worker process by not specifying any command (for example, @@1 on its own line). The prompt mode of the master socket (interactive, prompt, timed) is propagated into the worker process. Terminate the session either by closing the connection or by quitting the worker process (using the quit command).
When to use @@?
You can use @ for most cases. Use @@ when a command needs to see the effect of an earlier command in the same chain, needs to block waiting for something (as in wait), or needs to keep watching (as in show events -w).
Example: gracefully drain and remove a server
nix
nix
outputtext
outputtext
Example: forcefully close connections and remove a server immediately
nix
nix
Example: stream events in real time (a tail -f equivalent)
nix
nix
Session modes Jump to heading
You can enable any of several modes in your master CLI session. This section describes the commands used to manage session modes.
prompt Jump to heading
The master CLI can operate in several modes at once, which you toggle on or off individually. The prompt command shows you which modes are currently active.
The master is identified by the master string, and other processes are identified by their PID. If the last reload attempt failed, the master prompt is set to master[ReloadFailed]>. Under these circumstances, the process is still running on the previous configuration, and the new configuration didn’t take effect.
The prompt of the master CLI shows which modes are currently enabled: d for mcli-debug-mode, e for expert-mode, x for experimental-mode.
Example: show active modes
Tip
The active modes show in the workers’ prompts as well.
expert-mode Jump to heading
This command activates expert-mode for every worker accessed from the master CLI. Combined with mcli-debug-mode, it also activates the mode on the master. The e flag displays in the master CLI prompt.
experimental-mode Jump to heading
This command activates experimental-mode for every worker accessed from the master CLI. Combined with mcli-debug-mode, it also activates the mode on the master. The x flag displays in the master CLI prompt.
mcli-debug-mode Jump to heading
This keyword enables a special mode in the master CLI that enables every keyword meant for a worker CLI on the master CLI as well, letting you debug the master process. Once activated, list the newly available keywords with help. Combined with experimental-mode or expert-mode, it enables even more keywords. The d flag displays in the master CLI prompt.
Process lifecycle Jump to heading
The master CLI provides several commands for reloading the load balancer and displaying the state of the HAProxy processes.
reload Jump to heading
You can also reload the HAProxy master process with the reload command, which does the same as a kill -USR2 on the master process, provided that the user has at least operator or admin privileges.
This command allows you to perform a synchronous reload where the command returns a reload status once the reload completes. Be careful with the timeout if a tool is used to parse the output because it returns the output only after it has parsed the configuration and forked the new worker. The socat command uses a timeout of 0.5s by default, for example, so it will quit before showing the message if the reload takes too long. ncat does not have a timeout by default. When compiled with USE_SHM_OPEN=1, the reload command is also able to dump the startup-logs of the master.
Example
nix
nix
outputtext
outputtext
Caution
The reload command is the last one executed on the master CLI, and every other command after it is ignored. Once the reload command returns its status, it closes the connection to the CLI.
Note that a reload will close all connections to the master CLI. See also the hard-reload command.
hard-reload Jump to heading
This command does the same as the reload command over the master CLI with the exception that it does a hard-stop (-st) instead of a soft-stop (-sf) of the previous process. This means the previous process does not wait to achieve anything before exiting, so all connections will be closed.
show proc Jump to heading
Use the show proc command to list the processes.
Example
nix
nix
outputtext
outputtext
In this example, the master has been reloaded 5 times but the old worker is still running and survived 1 reload. You could access the CLI of the worker to investigate and determine why the old worker is running.
The debug parameter is useful to show debug details. It currently shows the FDs for IPC communication. Note that the debug output is not guaranteed to be stable between HAProxy versions.
show startup-logs Jump to heading
Like its counterpart on the stats socket, this command is able to show the startup messages of HAProxy. However, it doesn’t dump the startup messages of the current worker, but the startup messages of the latest startup or reload, which means it’s able to dump the parsing messages of a failed reload.
Those messages are also dumped with the reload command.
Info
To ensure that all messages are logged, HAProxy must be compiled with USE_SHM_OPEN=1.
Troubleshooting Jump to heading
Can't find the target PID matching the prefix '@!<pid>'
The process you addressed no longer exists. The leaving worker that already exited before you reached it, since it’s terminated as soon as its last connection closes. Re-run show proc immediately before addressing a leaving worker by PID.
Unknown command: '<command>'
The command exists but is gated behind a mode that isn’t enabled, commonly mcli-debug-mode, expert-mode, or experimental-mode. Enable the relevant session mode and retry.
Connection refused, or no such file or directory, when connecting to the master CLI socket
The process wasn’t started with -S, or you’re pointing at the wrong path. For HAProxy Enterprise, see Master CLI invocation to confirm the actual invocation.
See also Jump to heading
- For the complete master CLI reference, see Master CLI in the HAProxy Management Guide.
- For the complete HAProxy Runtime API UNIX socket command reference, see Unix Socket commands in the HAProxy Management Guide.
- For information on
show cli sockets, a worker-level command referenced in the master CLIhelpoutput, see show cli sockets.