Troubleshooting

Debugging

On this page

Available since:

  • HAProxy Data Plane API 3.1r1

The debug socket provides commands for use in diagnosing API issues. The HAProxy Support team may ask you to configure the debug socket in your HAProxy Data Plane API configuration and execute commands that generate debug data they can use to diagnose issues. The pprof command also provides an HTTP interface allowing the Support team to examine profiling data through a dedicated port.

Info

The debug socket is distinct from the socket used for normal HAProxy Data Plane API calls.

Configure the socket Jump to heading

The debug socket is disabled by default. To enable it, define it in one of these ways:

  • On the dataplaneapi command line, use the –debug-socket-path argument.
  • In the dataplaneapi.yml configuration file, use the debug_socket_path option. This method overrides the command line option when both are set.

Restart the service:

nix
sudo systemctl restart hapee-extras-dataplaneapi
nix
sudo systemctl restart hapee-extras-dataplaneapi

The socket path is reported in the dataplaneapi log at startup:

startup log
text
"level":"info","msg":"-- command socket Starting on /var/run/dataplane-debug.sock","time":"2023-09-13T11:44:08+02:00"}
startup log
text
"level":"info","msg":"-- command socket Starting on /var/run/dataplane-debug.sock","time":"2023-09-13T11:44:08+02:00"}

Once the socket is configured, you can issue a variety of commands to generate debug information or to observe process behavior in real-time. To access pprof using the web interface, see pprof web interface.

Commands Jump to heading

Use the help command to display the debug socket runtime commands:

nix
echo "help" | sudo socat  - /var/run/dataplane-debug.sock
nix
echo "help" | sudo socat  - /var/run/dataplane-debug.sock
output
text
Dataplaneapi runtime commands:
           conf   show HAProxy configuration
     goroutines   display number of goroutines
          stack   output stack trace
        version   show dataplaneapi version
          pprof   pprof dumps
       dapiconf   show dataplaneapi configuration
type help <command> for more info
output
text
Dataplaneapi runtime commands:
           conf   show HAProxy configuration
     goroutines   display number of goroutines
          stack   output stack trace
        version   show dataplaneapi version
          pprof   pprof dumps
       dapiconf   show dataplaneapi configuration
type help <command> for more info

conf command Jump to heading

Use the conf command to fetch the HAProxy load balancer configuration in either raw or structured format. The default format is raw.

nix
echo "conf" | sudo socat  - /var/run/dataplane-debug.sock
nix
echo "conf" | sudo socat  - /var/run/dataplane-debug.sock
output
text
global
    maxconn            10000
    # Syslog daemon
    log                127.0.0.1 local0
    log                127.0.0.1 local1 notice
    ## Systemd journal
    #log               /dev/log local0
    #log               /dev/log local1 notice
    user               hapee-lb
    group              hapee
...
output
text
global
    maxconn            10000
    # Syslog daemon
    log                127.0.0.1 local0
    log                127.0.0.1 local1 notice
    ## Systemd journal
    #log               /dev/log local0
    #log               /dev/log local1 notice
    user               hapee-lb
    group              hapee
...

The conf raw and conf structured commands return the current (latest) version unless you specify another version. Get the current configuration version number the same way you do for normal HAProxy Data Plane API calls:

nix
curl --silent --user admin:adminpwd http://localhost:5555/v3/services/haproxy/configuration/version
nix
curl --silent --user admin:adminpwd http://localhost:5555/v3/services/haproxy/configuration/version
output
text
1
output
text
1

Specify the version number as an argument to conf raw or conf structured:

nix
echo "conf structured 1" | sudo socat  - /var/run/dataplane-debug.sock
nix
echo "conf structured 1" | sudo socat  - /var/run/dataplane-debug.sock
output
text
acmeproviders: {}
backends:
  be_app:
    backendbase:
      errorfiles: []
      errorfilesfromhttperrors: []
      forcepersistlist: []
      ignorepersistlist: []
      abortonclose: ""
      acceptinvalidhttpresponse: ""
...
output
text
acmeproviders: {}
backends:
  be_app:
    backendbase:
      errorfiles: []
      errorfilesfromhttperrors: []
      forcepersistlist: []
      ignorepersistlist: []
      abortonclose: ""
      acceptinvalidhttpresponse: ""
...

To fetch the configuration from a transaction, specify 0 for version and then the transaction ID.

To get the array of transactions, use the GET method with the transactions endpoint:

nix
curl -s -u admin:adminpwd http://localhost:5555/v3/services/haproxy/transactions | jq
nix
curl -s -u admin:adminpwd http://localhost:5555/v3/services/haproxy/transactions | jq
output
text
[
  {
    "_version": 1,
    "id": "5655273c-d099-4537-9992-8f1243627186",
    "status": "in_progress"
  }
]
output
text
[
  {
    "_version": 1,
    "id": "5655273c-d099-4537-9992-8f1243627186",
    "status": "in_progress"
  }
]

Specify 0 for version, then the transaction ID:

nix
echo "conf structured 0 5655273c-d099-4537-9992-8f1243627186" | sudo socat  - /var/run/dataplane-debug.sock
nix
echo "conf structured 0 5655273c-d099-4537-9992-8f1243627186" | sudo socat  - /var/run/dataplane-debug.sock
output
text
acmeproviders: {}
backends:
  be_app:
    backendbase:
      errorfiles: []
      errorfilesfromhttperrors: []
      forcepersistlist: []
      ignorepersistlist: []
      abortonclose: ""
      acceptinvalidhttpresponse: ""
...
output
text
acmeproviders: {}
backends:
  be_app:
    backendbase:
      errorfiles: []
      errorfilesfromhttperrors: []
      forcepersistlist: []
      ignorepersistlist: []
      abortonclose: ""
      acceptinvalidhttpresponse: ""
...

goroutines command Jump to heading

List the number of Go routines in the HAProxy Data Plane API:

nix
echo "goroutines" | sudo socat  - /var/run/dataplane-debug.sock
nix
echo "goroutines" | sudo socat  - /var/run/dataplane-debug.sock
output
text
26
output
text
26

stack command Jump to heading

Use the stack command to produce stack traces of current Go routines:

nix
echo "stack" | sudo socat  - /var/run/dataplane-debug.sock
nix
echo "stack" | sudo socat  - /var/run/dataplane-debug.sock
output
text
1: running [Created by runtime.(*DebugServer).Start.func2 @ runtime.go:104]
    commands      stack.go:63                   MakeStackDump()
    commands      stack.go:45                   Stack.Command({}, {0x1e925e7381b0, 0x1e925cdfd900, 1})
    runtime       serve.go:46                   serve(0x1e925a2be028, {0x34a28e0, 0x1e92617a0008})
1: sync.WaitGroup.Wait [249 minutes]
    sync          sema.go:114                   runtime_SemacquireWaitGroup(#6, 0x40)
    sync          waitgroup.go:206              (*WaitGroup).Wait(#43)
    dataplaneapi  server.go:341                 (*Server).Serve(#24)
    main          main.go:287                   startServer(#19, #14)
    main          main.go:71                    main()
...
output
text
1: running [Created by runtime.(*DebugServer).Start.func2 @ runtime.go:104]
    commands      stack.go:63                   MakeStackDump()
    commands      stack.go:45                   Stack.Command({}, {0x1e925e7381b0, 0x1e925cdfd900, 1})
    runtime       serve.go:46                   serve(0x1e925a2be028, {0x34a28e0, 0x1e92617a0008})
1: sync.WaitGroup.Wait [249 minutes]
    sync          sema.go:114                   runtime_SemacquireWaitGroup(#6, 0x40)
    sync          waitgroup.go:206              (*WaitGroup).Wait(#43)
    dataplaneapi  server.go:341                 (*Server).Serve(#24)
    main          main.go:287                   startServer(#19, #14)
    main          main.go:71                    main()
...

Display stack traces in raw format:

nix
echo "stack raw" | sudo socat  - /var/run/dataplane-debug.sock
nix
echo "stack raw" | sudo socat  - /var/run/dataplane-debug.sock
output
text
goroutine 106 [running]:
github.com/haproxytech/dataplaneapi/runtime/commands.Stack.Command({}, {0x1e925f6e81e0?, 0x1e925cdfd900?, 0x2?})
        github.com/haproxytech/dataplaneapi/runtime/commands/stack.go:53 +0x9c
github.com/haproxytech/dataplaneapi/runtime.serve(0x1e925a2be028, {0x34a28e0, 0x1e92617a0008})
        github.com/haproxytech/dataplaneapi/runtime/serve.go:46 +0x20f
created by github.com/haproxytech/dataplaneapi/runtime.(*DebugServer).Start.func2 in goroutine 51
        github.com/haproxytech/dataplaneapi/runtime/runtime.go:104 +0x4e
goroutine 1 [sync.WaitGroup.Wait, 329 minutes]:
sync.runtime_SemacquireWaitGroup(0x262ccf9?, 0x40?)
...
output
text
goroutine 106 [running]:
github.com/haproxytech/dataplaneapi/runtime/commands.Stack.Command({}, {0x1e925f6e81e0?, 0x1e925cdfd900?, 0x2?})
        github.com/haproxytech/dataplaneapi/runtime/commands/stack.go:53 +0x9c
github.com/haproxytech/dataplaneapi/runtime.serve(0x1e925a2be028, {0x34a28e0, 0x1e92617a0008})
        github.com/haproxytech/dataplaneapi/runtime/serve.go:46 +0x20f
created by github.com/haproxytech/dataplaneapi/runtime.(*DebugServer).Start.func2 in goroutine 51
        github.com/haproxytech/dataplaneapi/runtime/runtime.go:104 +0x4e
goroutine 1 [sync.WaitGroup.Wait, 329 minutes]:
sync.runtime_SemacquireWaitGroup(0x262ccf9?, 0x40?)
...

You can also get a stack trace from the /debug/pprof/trace endpoint in the pprof web interface.

version command Jump to heading

Display version details for HAProxy Data Plane API:

nix
echo "version" | sudo socat  - /var/run/dataplane-debug.sock
nix
echo "version" | sudo socat  - /var/run/dataplane-debug.sock
output
text
v3.3.1-ee1 7d90e2d1
output
text
v3.3.1-ee1 7d90e2d1

pprof command Jump to heading

Use the pprof command to capture profiling data in pprof format.

Typically, you’ll be sending the pprof output to the HAProxy Support team. If you want to examine the data yourself, you’ll need to use your own pprof tool, for example, the Go pprof web tool.

CPU profiling Jump to heading

Capture CPU profiling data for a specific period of time and write it to a file:

nix
pprof cpu start <FILENAME> [<TIME>]
nix
pprof cpu start <FILENAME> [<TIME>]

The default <TIME> value is 10m.

For example, capture CPU profiling data for 10 minutes, writing it to the file mypprof-cpu:

nix
echo "pprof cpu start mypprof-cpu" | sudo socat  - /var/run/dataplane-debug.sock
nix
echo "pprof cpu start mypprof-cpu" | sudo socat  - /var/run/dataplane-debug.sock
output
text
CPU Profile with duration of 10m0s
output
text
CPU Profile with duration of 10m0s

For example, capture CPU profiling data for 1 hour 20 minutes, writing it to the file mypprof-cpu:

nix
echo "pprof cpu start mypprof-cpu 1h20m" | sudo socat  - /var/run/dataplane-debug.sock
nix
echo "pprof cpu start mypprof-cpu 1h20m" | sudo socat  - /var/run/dataplane-debug.sock
output
text
CPU Profile with duration of 1h20m0s
output
text
CPU Profile with duration of 1h20m0s

While the profiler is running, issue any API calls causing behavior that requires debugging. The profiler stops gathering data when the specified time period expires. Optionally, you can terminate pprof at any time using the pprof cpu stop command:

nix
echo "pprof cpu stop" | sudo socat  - /var/run/dataplane-debug.sock
nix
echo "pprof cpu stop" | sudo socat  - /var/run/dataplane-debug.sock
output
text
CPU Profile stopped
output
text
CPU Profile stopped

Examine the profile data file using the pprof tool of your choice, for example, the Go pprof web tool:

nix
go tool pprof -web /tmp/mypprof-cpu
nix
go tool pprof -web /tmp/mypprof-cpu

You can also get CPU profile data from the /debug/pprof/profile endpoint in the pprof web interface.

Memory/heap dump Jump to heading

Dump memory to a file:

nix
pprof mem <FILENAME>
nix
pprof mem <FILENAME>

For example, dump memory to file mypprof-mem:

nix
echo "pprof mem mypprof-mem" | sudo socat  - /var/run/dataplane-debug.sock
nix
echo "pprof mem mypprof-mem" | sudo socat  - /var/run/dataplane-debug.sock

To examine the memory dump, you can use the pprof tool of your choice, for example, the Go pprof tool. View the data in a browser:

nix
go tool pprof -web /tmp/mypprof-mem
nix
go tool pprof -web /tmp/mypprof-mem

You can also get a memory dump from the /debug/pprof/heap endpoint in the pprof web interface.

pprof web interface Jump to heading

The pprof web interface allows you to get data from an HTTP server running in the HAProxy Data Plane API.

To enable the server, use the pprof web start command. Specify the port and the time duration that the server will run.

nix
pprof web start <PORT> <TIME>
nix
pprof web start <PORT> <TIME>

For example, run the HTTP server at port 8888 for 1 hour:

nix
echo "pprof web start 8888 1h" | sudo socat  - /var/run/dataplane-debug.sock
nix
echo "pprof web start 8888 1h" | sudo socat  - /var/run/dataplane-debug.sock
output
text
pprof web started on port 8888 with duration of 1h0m0s
output
text
pprof web started on port 8888 with duration of 1h0m0s

The server creates the following endpoints where you can get data:

  • /debug/pprof/profile: CPU profiling data. Optionally, specify the ?seconds variable. The default value is 10 seconds.
  • /debug/pprof/heap: Memory allocation
  • /debug/pprof/trace: Stack trace. Optionally, specify the ?seconds variable. The default value is 10 seconds.

You can specify an endpoint as an argument to a Go tool or to an HTTP utility like curl.

  • For example, collect 30 seconds of CPU data and open it in interactive pprof:

    nix
    go tool pprof http://localhost:8888/debug/pprof/profile?seconds=30
    nix
    go tool pprof http://localhost:8888/debug/pprof/profile?seconds=30

  • For example, perform a memory dump and open it in the pprof browser interface:

    nix
    go tool pprof -web http://localhost:8888/debug/pprof/heap
    nix
    go tool pprof -web http://localhost:8888/debug/pprof/heap

  • For example, collect 60 seconds of stack trace data, write it to a file, and open the file in the Go trace tool:

    nix
    curl -s http://127.0.0.1:8888/debug/pprof/trace?seconds=60 > ./trace.out
    go tool trace ./trace.out
    nix
    curl -s http://127.0.0.1:8888/debug/pprof/trace?seconds=60 > ./trace.out
    go tool trace ./trace.out

You can terminate the HTTP server at any time using pprof web stop:

nix
echo "pprof web stop" | sudo socat  - /var/run/dataplane-debug.sock
nix
echo "pprof web stop" | sudo socat  - /var/run/dataplane-debug.sock

dapiconf command Jump to heading

The dapiconf command dumps the HAProxy Data Plane API configuration in Golang dump format.

Dump the HAProxy Data Plane API configuration:

nix
echo "dapiconf" | sudo socat  - /var/run/dataplane-debug.sock
nix
echo "dapiconf" | sudo socat  - /var/run/dataplane-debug.sock
output
text
&configuration.Configuration{
    Cluster: configuration.ClusterConfiguration{},
    Notify:  configuration.NotifyConfiguration{
        BootstrapKeyChanged: &configuration.ChanNotify{
            subscribers: {"monitorBootstrapKey":0x144c6bb85570},
            mu:          sync.RWMutex{},
        },
        ServerStarted: &configuration.ChanNotify{
            subscribers: {"clusterMonitor":0x144c6bb85500, "commandSocket":0x144c691fa000},
            mu:          sync.RWMutex{},
...
output
text
&configuration.Configuration{
    Cluster: configuration.ClusterConfiguration{},
    Notify:  configuration.NotifyConfiguration{
        BootstrapKeyChanged: &configuration.ChanNotify{
            subscribers: {"monitorBootstrapKey":0x144c6bb85570},
            mu:          sync.RWMutex{},
        },
        ServerStarted: &configuration.ChanNotify{
            subscribers: {"clusterMonitor":0x144c6bb85500, "commandSocket":0x144c691fa000},
            mu:          sync.RWMutex{},
...

Do you have any suggestions on how we can improve the content of this page?

Previous page Troubleshooting Next page Reference
© 2026 HAProxy Technologies, LLC. All Rights Reserved
Manage Cookie Preferences