OpenTelemetry

Manage the HAProxy OpenTelemetry filter with the HAProxy Runtime API

This page applies to:

  • HAProxy 3.4 and newer

After you’ve enabled the HAProxy OpenTelemetry filter, you can call the following HAProxy Runtime API commands.

Command Description
flt-otel debug [level] Set or show the debug level (if compiled with OTEL_DEBUG).
flt-otel disable Disable the filter.
flt-otel enable Enable the filter.
flt-otel flush Force-export buffered telemetry.
flt-otel hard-errors Enable hard-errors mode.
flt-otel instruments Show configured metric instruments.
flt-otel logging [state] Set the logging state.
flt-otel rate [value] Set or show the rate limit.
flt-otel scopes Show configured scopes and groups.
flt-otel soft-errors Disable hard-errors mode.
flt-otel status Show the filter status.

flt-otel debug Jump to heading

Set or show the debug level (if compiled with OTEL_DEBUG).

Show the current debug level

nix
echo "flt-otel debug" | sudo socat stdio tcp4-connect:127.0.0.1:9999
nix
echo "flt-otel debug" | sudo socat stdio tcp4-connect:127.0.0.1:9999
output
text
flt-otel : current debug level is 1919 (0x077f)
output
text
flt-otel : current debug level is 1919 (0x077f)

flt-otel disable Jump to heading

Disable the filter.

nix
echo "flt-otel disable" | sudo socat stdio tcp4-connect:127.0.0.1:9999
nix
echo "flt-otel disable" | sudo socat stdio tcp4-connect:127.0.0.1:9999
output
text
flt-otel : filter disabled
output
text
flt-otel : filter disabled

flt-otel enable Jump to heading

Enable the filter.

nix
echo "flt-otel enable" | sudo socat stdio tcp4-connect:127.0.0.1:9999
nix
echo "flt-otel enable" | sudo socat stdio tcp4-connect:127.0.0.1:9999
output
text
flt-otel : filter enabled
output
text
flt-otel : filter enabled

flt-otel flush Jump to heading

Force-export buffered telemetry.

nix
echo "flt-otel flush" | sudo socat stdio tcp4-connect:127.0.0.1:9999
nix
echo "flt-otel flush" | sudo socat stdio tcp4-connect:127.0.0.1:9999
output
text
flt-otel : flushed proxy mysite, filter my-otel-filter
output
text
flt-otel : flushed proxy mysite, filter my-otel-filter

flt-otel hard-errors Jump to heading

Enable hard-errors mode. In hard-errors mode, processing of the stream stops as soon as the first error occurs. In soft-errors mode, processing of the stream continues without interruption if there is an error. Errors are caused by incorrect configuration of the instrumentation or are related to the operation of HAProxy.

nix
echo "flt-otel hard-errors" | sudo socat stdio tcp4-connect:127.0.0.1:9999
nix
echo "flt-otel hard-errors" | sudo socat stdio tcp4-connect:127.0.0.1:9999
output
text
flt-otel : filter set hard-errors
output
text
flt-otel : filter set hard-errors

flt-otel instruments Jump to heading

Show configured metric instruments.

nix
echo "flt-otel instruments" | sudo socat stdio tcp4-connect:127.0.0.1:9999
nix
echo "flt-otel instruments" | sudo socat stdio tcp4-connect:127.0.0.1:9999
output
text
opentelemetry filter instruments
------------------------------------------------------------------------------
proxy mysite, filter my-otel-filter
scope instrument type unit state
frontend_http_request haproxy.sessions.active udcnt_int {session} created (idx 0)
frontend_http_request haproxy.sessions.active update
frontend_http_request haproxy.http.requests cnt_int {request} created (idx 1)
frontend_http_request haproxy.http.requests update
frontend_http_request haproxy.fe.connections gauge_int {connection} created (idx 2)
frontend_http_request haproxy.fe.connections update
server_session_start haproxy.server.session.start cnt_int {session} created (idx 3)
server_session_start haproxy.server.session.start update
server_session_end haproxy.sessions.active update
output
text
opentelemetry filter instruments
------------------------------------------------------------------------------
proxy mysite, filter my-otel-filter
scope instrument type unit state
frontend_http_request haproxy.sessions.active udcnt_int {session} created (idx 0)
frontend_http_request haproxy.sessions.active update
frontend_http_request haproxy.http.requests cnt_int {request} created (idx 1)
frontend_http_request haproxy.http.requests update
frontend_http_request haproxy.fe.connections gauge_int {connection} created (idx 2)
frontend_http_request haproxy.fe.connections update
server_session_start haproxy.server.session.start cnt_int {session} created (idx 3)
server_session_start haproxy.server.session.start update
server_session_end haproxy.sessions.active update

flt-otel logging Jump to heading

Show the current logging state or set the logging state to on, off, or dontlog-normal. This is enabled when you have a log directive in the otel-instrumentation section, such as log global.

Show the current logging state

nix
echo "flt-otel logging" | sudo socat stdio tcp4-connect:127.0.0.1:9999
nix
echo "flt-otel logging" | sudo socat stdio tcp4-connect:127.0.0.1:9999
output
text
flt-otel : logging is currently enabled
output
text
flt-otel : logging is currently enabled

Disable logging

nix
echo "flt-otel logging off" | sudo socat stdio tcp4-connect:127.0.0.1:9999
nix
echo "flt-otel logging off" | sudo socat stdio tcp4-connect:127.0.0.1:9999
output
text
flt-otel : logging is disabled
output
text
flt-otel : logging is disabled

Enable logging

nix
echo "flt-otel logging on" | sudo socat stdio tcp4-connect:127.0.0.1:9999
nix
echo "flt-otel logging on" | sudo socat stdio tcp4-connect:127.0.0.1:9999
output
text
flt-otel : logging is enabled
output
text
flt-otel : logging is enabled

flt-otel rate Jump to heading

Set or show the rate limit.

Show the current rate limit

nix
echo "flt-otel rate" | sudo socat stdio tcp4-connect:127.0.0.1:9999
nix
echo "flt-otel rate" | sudo socat stdio tcp4-connect:127.0.0.1:9999
output
text
flt-otel : current rate limit is 100.00
output
text
flt-otel : current rate limit is 100.00

Set the rate limit to 50%

nix
echo "flt-otel rate 50.0" | sudo socat stdio tcp4-connect:127.0.0.1:9999
nix
echo "flt-otel rate 50.0" | sudo socat stdio tcp4-connect:127.0.0.1:9999
output
text
flt-otel : rate limit set to 50.00
output
text
flt-otel : rate limit set to 50.00

flt-otel scopes Jump to heading

Show configured scopes and groups.

nix
echo "flt-otel scopes" | sudo socat stdio tcp4-connect:127.0.0.1:9999
nix
echo "flt-otel scopes" | sudo socat stdio tcp4-connect:127.0.0.1:9999
output
text
opentelemetry filter scopes
------------------------------------------------------------------------------
proxy mysite, filter my-otel-filter
scopes:
scope event used
frontend_http_request on-frontend-http-request yes
backend_http_request on-backend-http-request yes
process_request on-http-process-request yes
finish_http_request on-http-end-request yes
server_session_start on-server-session-start yes
wait_for_response on-http-wait-response yes
http_response on-http-response yes
http_response_end on-http-end-response yes
server_session_end on-server-session-end yes
(no groups)
events:
event dispatched
on-stream-start 1
on-stream-stop 1
on-client-session-start 1
on-backend-set 1
on-frontend-http-request 1
on-backend-http-request 1
on-http-process-request 1
on-http-headers-request 1
on-http-end-request 1
on-client-session-end 1
on-server-session-start 1
on-http-wait-response 1
on-http-response 1
on-http-headers-response 1
on-http-end-response 1
on-server-session-end 1
output
text
opentelemetry filter scopes
------------------------------------------------------------------------------
proxy mysite, filter my-otel-filter
scopes:
scope event used
frontend_http_request on-frontend-http-request yes
backend_http_request on-backend-http-request yes
process_request on-http-process-request yes
finish_http_request on-http-end-request yes
server_session_start on-server-session-start yes
wait_for_response on-http-wait-response yes
http_response on-http-response yes
http_response_end on-http-end-response yes
server_session_end on-server-session-end yes
(no groups)
events:
event dispatched
on-stream-start 1
on-stream-stop 1
on-client-session-start 1
on-backend-set 1
on-frontend-http-request 1
on-backend-http-request 1
on-http-process-request 1
on-http-headers-request 1
on-http-end-request 1
on-client-session-end 1
on-server-session-start 1
on-http-wait-response 1
on-http-response 1
on-http-headers-response 1
on-http-end-response 1
on-server-session-end 1

flt-otel soft-errors Jump to heading

Disable hard-errors mode. In soft-errors mode, processing of the stream continues without interruption if there is an error. In hard-errors mode, processing of the stream stops as soon as the first error occurs. Errors are caused by incorrect configuration of the instrumentation or are related to the operation of HAProxy.

nix
echo "flt-otel soft-errors" | sudo socat stdio tcp4-connect:127.0.0.1:9999
nix
echo "flt-otel soft-errors" | sudo socat stdio tcp4-connect:127.0.0.1:9999
output
text
flt-otel : filter set soft-errors
output
text
flt-otel : filter set soft-errors

flt-otel status Jump to heading

Show the filter status.

nix
echo "flt-otel status" | sudo socat stdio tcp4-connect:127.0.0.1:9999
nix
echo "flt-otel status" | sudo socat stdio tcp4-connect:127.0.0.1:9999
output
text
opentelemetry filter status
------------------------------------------------------------------------------
library: C++ 1.26.0, C wrapper 2.0.0-862
debug level: 0x7f
dropped count: spans 0, logs 0, sdk 0
proxy mysite, filter my-otel-filter
configuration: /etc/haproxy/instrumentation.cfg
groups/scopes: 0/9
instrumentation my-instrumentation
configuration: /etc/haproxy/otel-client.yaml
tracer: active
meter: active
logger: active
rate limit: 100.00 %
hard errors: no
disabled: no
logging: disabled
idle timeout: 0 ms
analyzers: 01400910
counters
attached: run 1, rate-limit 0, disabled 0, error 0
disabled: scope 0, hard-error 0
output
text
opentelemetry filter status
------------------------------------------------------------------------------
library: C++ 1.26.0, C wrapper 2.0.0-862
debug level: 0x7f
dropped count: spans 0, logs 0, sdk 0
proxy mysite, filter my-otel-filter
configuration: /etc/haproxy/instrumentation.cfg
groups/scopes: 0/9
instrumentation my-instrumentation
configuration: /etc/haproxy/otel-client.yaml
tracer: active
meter: active
logger: active
rate limit: 100.00 %
hard errors: no
disabled: no
logging: disabled
idle timeout: 0 ms
analyzers: 01400910
counters
attached: run 1, rate-limit 0, disabled 0, error 0
disabled: scope 0, hard-error 0

See also Jump to heading

Do you have any suggestions on how we can improve the content of this page?