OpenTelemetry

Export traces using the HAProxy OpenTelemetry filter

On this page

This page applies to:

  • HAProxy 3.4 and newer

The HAProxy OpenTelemetry filter enables you to see what’s happening inside the load balancer as it processes a live request. This view is called a trace.

OpenTelemetry HAProxy trace

A trace shown in Jaeger

A trace is composed of time spans. For example, between a client starting a session and later sending a full HTTP request, we can measure a span of time. With numerous events that have start and end times, we record the time spans between each and link them to form a complete picture of the actions within the load balancer, the order in which they occurred, and how long each action took.

Some spans are nested within, or children of, other time spans. HAProxy processes an HTTP request-response session in two phases: receive, process, and relay the request; and receive, process, and relay the response. Both are parts of the client’s overall HAProxy session. So, we could configure spans for the request and response phases as children of the HAProxy session span.

In other cases, one action causes another, but the two don’t have a parent-child relationship. For instance, if you develop a Lua script that sends an asynchronous HTTP request to an IP reputation service, you could define a span link that indicates that the HAProxy session causes that Lua action, but that it happens concurrently.

In the upcoming sections, you’ll learn how to define spans and their relationships.

Data delivery not guaranteed

One important assumption when working with OpenTelemetry data is that there is no guarantee that all telemetry will be successfully exported to the destination. The OpenTelemetry specification doesn’t mandate guaranteed delivery; therefore, data loss can occur at various stages of collection, processing, or export and should be considered a normal operational possibility. If you have very high traffic, a large number of active connections, and you send a lot of data to the OpenTelemetry system, some of it may be lost.

Enable OpenTelemetry traces Jump to heading

In this section, we cover how to configure HAProxy for OpenTelemetry traces.

Configure the OpenTelemetry filter Jump to heading

The filter opentelemetry directive enables instrumentation of HAProxy, which means the emission of signals: traces, logs, and metrics.

haproxy
frontend myfrontend
  filter opentelemetry  id my-otel-filter  config /etc/haproxy/instrumentation.cfg 
  ...
haproxy
frontend myfrontend
  filter opentelemetry  id my-otel-filter  config /etc/haproxy/instrumentation.cfg 
  ...
  • id sets a unique identifier for this filter.
  • config indicates the path to the instrumentation file.

Next, let’s examine the instrumentation file.

Create an instrumentation file Jump to heading

HAProxy exposes events as they happen within the load balancer so that you can add them to a trace. You must define which events to watch by creating an OpenTelemetry instrumentation file. Add a file with a .cfg file extension at the location specified in the HAProxy configuration from the previous section. In our example, we name it /etc/haproxy/instrumentation.cfg.

instrumentation.cfg
ini
[my-otel-filter]
otel-instrumentation my-instrumentation
  config /etc/haproxy/otel-client.yaml
instrumentation.cfg
ini
[my-otel-filter]
otel-instrumentation my-instrumentation
  config /etc/haproxy/otel-client.yaml

  • At the start of the file, add your filter’s ID in square brackets. It should match the filter opentelemetry directive’s id argument.

  • Below that, add an otel-instrumentation directive to mark the beginning of your settings. Its only argument is a name.

  • Nested under otel-instrumentation, set the required directive config to the location of your OpenTelemetry C++ client library configuration. This is a YAML file that passes your telemetry data through a pipeline of providers, processors, samplers, and exporters. As an example, you can export your data to a running instance of OpenTelemetry Collector or to Jaeger. To learn about that file, see Configure the client library.

    See the HAProxy OpenTelemetry filter configuration guide for other directives that you can place in an otel-instrumentation section.

Instrumentation file formatting

A few items to note about formatting:

  • Indentation is optional, but it helps make your file easier to read.
  • End the file with a blank newline, or else you’ll get an error.
  • You can add comments to the file (documentation text) by prefixing the text with a hash (#).
  • Set the debug-level directive to 0x77f to set the debug bitmask. This is effective only when compiled with OTEL_DEBUG=1.

Define scopes Jump to heading

A scope is a configuration section in your instrumentation file that defines a span and associates it with an HAProxy event. To define a span, you must add it inside a scope. A scope can also include a span’s metadata, such as attributes, span events, span links, baggage, ACLs, span status, and injected context.

Let’s start with a set of predefined scopes that you can customize.

instrumentation.cfg
ini
[my-otel-filter]
otel-instrumentation my-instrumentation
  config /etc/haproxy/otel-client.yaml
  # Add scopes that reference otel-scope sections...
  scopes frontend_http_request
  scopes backend_http_request
  scopes process_request
  scopes finish_http_request
  scopes server_session_start
  scopes wait_for_response
  scopes http_response
  scopes http_response_end
  scopes server_session_end
# Define otel-scope sections...
otel-scope frontend_http_request
  span "HAProxy session" root
  span "Client session" parent "HAProxy session"
  span "HTTP request" parent "Client session"
  otel-event on-frontend-http-request
otel-scope backend_http_request
  span "Process backend request rules" parent "HTTP request"
  otel-event on-backend-http-request
otel-scope process_request
  span "Process request" parent "HTTP request"
  finish "Process backend request rules"
  otel-event on-http-process-request
otel-scope finish_http_request
  span "Forward request" parent "HAProxy session"
  finish "Process request"
  finish "HTTP request"
  otel-event on-http-end-request
otel-scope server_session_start
  span "Server session" parent "HAProxy session"
  finish "Forward request"
  otel-event on-server-session-start
otel-scope wait_for_response
  span "Wait for response" parent "Server session"
  otel-event on-http-wait-response
otel-scope http_response
  span "HTTP response" parent "Server session"
  finish "Wait for response"
  otel-event on-http-response
otel-scope http_response_end
  span "Forward response" parent "HAProxy session"
  finish "HTTP response"
  otel-event on-http-end-response
otel-scope server_session_end
  finish *
  otel-event on-server-session-end
instrumentation.cfg
ini
[my-otel-filter]
otel-instrumentation my-instrumentation
  config /etc/haproxy/otel-client.yaml
  # Add scopes that reference otel-scope sections...
  scopes frontend_http_request
  scopes backend_http_request
  scopes process_request
  scopes finish_http_request
  scopes server_session_start
  scopes wait_for_response
  scopes http_response
  scopes http_response_end
  scopes server_session_end
# Define otel-scope sections...
otel-scope frontend_http_request
  span "HAProxy session" root
  span "Client session" parent "HAProxy session"
  span "HTTP request" parent "Client session"
  otel-event on-frontend-http-request
otel-scope backend_http_request
  span "Process backend request rules" parent "HTTP request"
  otel-event on-backend-http-request
otel-scope process_request
  span "Process request" parent "HTTP request"
  finish "Process backend request rules"
  otel-event on-http-process-request
otel-scope finish_http_request
  span "Forward request" parent "HAProxy session"
  finish "Process request"
  finish "HTTP request"
  otel-event on-http-end-request
otel-scope server_session_start
  span "Server session" parent "HAProxy session"
  finish "Forward request"
  otel-event on-server-session-start
otel-scope wait_for_response
  span "Wait for response" parent "Server session"
  otel-event on-http-wait-response
otel-scope http_response
  span "HTTP response" parent "Server session"
  finish "Wait for response"
  otel-event on-http-response
otel-scope http_response_end
  span "Forward response" parent "HAProxy session"
  finish "HTTP response"
  otel-event on-http-end-response
otel-scope server_session_end
  finish *
  otel-event on-server-session-end
  • Below the otel-instrumentation section, add otel-scope sections. These define one or more spans and associate them with HAProxy events. The firing of the event triggers the start of the span.
  • One span must be labeled root, designating it the top-level span to which other spans are children.
  • Some otel-scope sections define finish directives, which indicate spans to stop. You can also set finish to an asterisk (*) to stop all spans. To stop all request-channel spans, use *req*. To stop all response-channel spans, use *res*. Spans continue until a finish directive stops them.
  • In the otel-instrumentation section, set scopes directives to add otel-scope sections to the trace.

There are many events that you can include in your trace. The table below covers them.

otel-event Description
on-stream-start This event fires before any channel processing begins. No channel is available at that point, so context injection/extraction via HTTP headers cannot be used in scopes bound to these events. When tracing HTTP services, if you want to extract context from an HTTP header, then it’s better to use on-frontend-http-request as the event attached to the root span instead of this event. Sample fetches in these scopes aren’t direction-constrained.
on-client-session-start This event fires when the analysis of the request channel (data pipe between the client and HAProxy) begins. It corresponds to the start of a new client session.
on-frontend-tcp-request This event fires during TCP request content inspection on the frontend. During this stage, any tcp-request content actions that are defined on the frontend run, which can determine whether to accept or reject the TCP request.
on-http-wait-request This event fires after the load balancer has waited for the complete HTTP request and has now received it. This wait period could stop analysis if it exceeds timeout http-request or timeout client.
on-http-body-request This event fires after the load balancer has waited for the HTTP request body, which is now available for inspection. This stage can continue if the buffer is full or the advertised contents have reached the buffer.
on-frontend-http-request This event fires during the processing of the HTTP request rules defined on the frontend. At this stage, the load balancer processes ACLs, filters, statistics, redirects, and HTTP header rules including the Connection header. There can be http-request deny rules that stop processing.
on-switching-rules-request This event fires when the load balancer implements default_backend and use_backend rules to choose the backend.
on-backend-set This event fires when a backend is assigned to the stream. It isn’t called if the proxy is a listen section.
on-backend-tcp-request This event fires during TCP request content inspection on the backend. During this stage, any tcp-request content actions that are defined on the backend run, which can determine whether to accept or reject the TCP request.
on-backend-http-request This event fires during the processing of the HTTP request rules defined on the backend. At this stage, the load balancer processes ACLs, filters, statistics, redirects, and HTTP header rules including the Connection header. There can be http-request deny rules that stop processing.
on-process-server-rules-request This event fires when the load balancer evaluates use-server directives in a backend.
on-http-process-request This event fires when the load balancer performs all the processing enabled for the request. At this point, unwanted requests have been filtered out.
on-tcp-rdp-cookie-request This event fires when the load balancer applies Windows Remote Desktop Protocol (RDP) server persistence rules to the current stream, which happens when persist rdp-cookie is set.
on-process-sticking-rules-request This event fires when the load balancer applies stick table server persistence rules, such as for stick on and stick store-request directives.
on-http-headers-request This event fires after all HTTP request headers have been parsed and analyzed.
on-http-end-request This event fires when all HTTP request data has been processed and forwarded to the backend server.
on-client-session-end This event fires when the request channel analysis ends.
on-server-unavailable This event fires during request channel end-analysis when response analyzers were configured but never executed because the server wasn’t reached.
on-server-session-start This event fires when the analysis of the response channel (data pipe between HAProxy and the backend server) begins, after a server connection has been established.
on-tcp-response This event fires during TCP response content analysis on the current response. During this stage, any tcp-response content actions that are defined on the backend run, which can determine whether to accept or reject the TCP response.
on-http-wait-response This event fires after the load balancer has waited for the complete HTTP response and has now received it. This wait period could stop analysis if it exceeds timeout server.
on-process-store-rules-response This event fires when the load balancer evaluates stick store-response directives.
on-http-response This event fires when the load balancer performs all the processing enabled for the response.
on-http-headers-response This event fires after all HTTP response headers have been parsed and analyzed.
on-http-end-response This event fires when all HTTP response data has been processed and forwarded.
on-server-session-end This event fires when the response channel analysis ends.
on-stream-stop This event fires after all channel processing ends. It typically finishes all active spans. No channel is available at that point, so context injection/extraction via HTTP headers cannot be used in scopes bound to these events. Sample fetches in these scopes aren’t direction-constrained.
on-idle-timeout This event fires periodically when the stream has no data transfer activity. It requires the idle-timeout directive to set the interval. This event is useful for heartbeat spans, idle-time metrics, and idle-time log records.

Configure the client library Jump to heading

To send your telemetry data to software that can process it, such as OpenTelemetry Collector or Jaeger, you must add a client library configuration YAML file. This file controls the underlying OpenTelemetry C++ client and configures export settings for traces, metrics, and logs. You can set more than just where to export data though, by configuring these sections:

  • exporters define where to send telemetry data. Supported YAML exporters are otlp_grpc, otlp_http, otlp_file, zipkin, elasticsearch, ostream, and memory.
  • samplers control trace the sampling strategy (traces only).
  • processors define how telemetry is batched before export (traces and logs only).
  • readers configure periodic metric collection intervals (metrics only).
  • providers set resource attributes attached to all telemetry.
  • signals bind the above components together per signal type.

See the HAProxy OpenTelemetry C Wrapper documentation for full details.

In the example below, we export the data by using the OpenTelemetry Protocol (OTLP) exporter type. It targets localhost:4318. You can run OpenTelemetry Collector at that address, or other software, such as Jaeger.

otel-client.yaml
yaml
# Exporters - defines protocols and target endpoints for traces, metrics, and logs
exporters:
  exporter_otlp_http:
    type:         otlp_http
    endpoint:     "http://localhost:4318/v1/traces"
    http_headers: []
# Metric readers - defines interval and timeout for collecting metrics
readers:
  reader_metrics:
    export_interval: 10000
    export_timeout:  5000
# Trace samplers - defines whether to sample trace data or send 100%
samplers:
  sampler_traces:
    type:  parent_based
    ratio: 1.0
    
# Processors - batch data and set parameters for when to emit
processors:
  processor_traces_batch:
    type:                  batch
    max_queue_size:        2048
    schedule_delay:        5000
    max_export_batch_size: 512
  processor_logs_batch:
    type:                  batch
    max_queue_size:        2048
    schedule_delay:        5000
    max_export_batch_size: 512
# Providers - how to represent the service sending the data
providers:
  provider_traces:
    resources:
      - service.version:     "3.4" # example that uses HAProxy version, but value is up to you
      - service.instance.id: "id-haproxy"
      - service.name:        "haproxy"
      - service.namespace:   "HAProxy traces example"
  provider_metrics:
    resources:
      - service.version:     "3.4"
      - service.instance.id: "id-haproxy"
      - service.name:        "haproxy"
      - service.namespace:   "HAProxy metrics example"
  provider_logs:
    resources:
      - service.version:     "3.4"
      - service.instance.id: "id-haproxy"
      - service.name:        "haproxy"
      - service.namespace:   "HAProxy logs example"
# Signals - Pull all the setttings together in the signals section
signals:
  traces:
    scope_name: "HAProxy OTEL - traces"
    exporters:  exporter_otlp_http
    samplers:   sampler_traces
    processors: processor_traces_batch
    providers:  provider_traces
  metrics:
    scope_name: "HAProxy OTEL - metrics"
    exporters:  exporter_otlp_http
    readers:    reader_metrics
    providers:  provider_metrics
  logs:
    scope_name: "HAProxy OTEL - logs"
    exporters:  exporter_otlp_http
    processors: processor_logs_batch
    providers:  provider_logs
otel-client.yaml
yaml
# Exporters - defines protocols and target endpoints for traces, metrics, and logs
exporters:
  exporter_otlp_http:
    type:         otlp_http
    endpoint:     "http://localhost:4318/v1/traces"
    http_headers: []
# Metric readers - defines interval and timeout for collecting metrics
readers:
  reader_metrics:
    export_interval: 10000
    export_timeout:  5000
# Trace samplers - defines whether to sample trace data or send 100%
samplers:
  sampler_traces:
    type:  parent_based
    ratio: 1.0
    
# Processors - batch data and set parameters for when to emit
processors:
  processor_traces_batch:
    type:                  batch
    max_queue_size:        2048
    schedule_delay:        5000
    max_export_batch_size: 512
  processor_logs_batch:
    type:                  batch
    max_queue_size:        2048
    schedule_delay:        5000
    max_export_batch_size: 512
# Providers - how to represent the service sending the data
providers:
  provider_traces:
    resources:
      - service.version:     "3.4" # example that uses HAProxy version, but value is up to you
      - service.instance.id: "id-haproxy"
      - service.name:        "haproxy"
      - service.namespace:   "HAProxy traces example"
  provider_metrics:
    resources:
      - service.version:     "3.4"
      - service.instance.id: "id-haproxy"
      - service.name:        "haproxy"
      - service.namespace:   "HAProxy metrics example"
  provider_logs:
    resources:
      - service.version:     "3.4"
      - service.instance.id: "id-haproxy"
      - service.name:        "haproxy"
      - service.namespace:   "HAProxy logs example"
# Signals - Pull all the setttings together in the signals section
signals:
  traces:
    scope_name: "HAProxy OTEL - traces"
    exporters:  exporter_otlp_http
    samplers:   sampler_traces
    processors: processor_traces_batch
    providers:  provider_traces
  metrics:
    scope_name: "HAProxy OTEL - metrics"
    exporters:  exporter_otlp_http
    readers:    reader_metrics
    providers:  provider_metrics
  logs:
    scope_name: "HAProxy OTEL - logs"
    exporters:  exporter_otlp_http
    processors: processor_logs_batch
    providers:  provider_logs

Restart the OpenTelemetry filter Jump to heading

To apply changes to the OpenTelemetry filter or its associated configuration files, restart HAProxy. For example:

nix
sudo systemctl reload haproxy
nix
sudo systemctl reload haproxy

Customize your trace Jump to heading

You can customize tracing, as explained in the following sections.

Add other spans Jump to heading

You can add other spans to your trace.

Example:

instrumentation.cfg
ini
[my-otel-filter]
otel-instrumentation my-instrumentation
  ...
  scopes switching_rules_request
  scopes backend_set
otel-scope switching_rules_request
  span "Backend switching rules" parent "HTTP request"
  otel-event on-switching-rules-request
otel-scope backend_set
  span "Backend set" parent "HTTP request"
  finish "Backend switching rules"
  otel-event on-backend-set
instrumentation.cfg
ini
[my-otel-filter]
otel-instrumentation my-instrumentation
  ...
  scopes switching_rules_request
  scopes backend_set
otel-scope switching_rules_request
  span "Backend switching rules" parent "HTTP request"
  otel-event on-switching-rules-request
otel-scope backend_set
  span "Backend set" parent "HTTP request"
  finish "Backend switching rules"
  otel-event on-backend-set

To add a span:

  1. Add an otel-scope section to your instrumentation file. On the line where you declare the otel-scope, set a name.

  2. Inside that otel-scope section, set a span directive. The name you give the span will appear in the trace. For example, use Backend switching rules; set its parent argument to indicate which span this span is a child of.

    Choosing the parent span

    How do you know which span is the parent? The choice is yours. You can make a deeply nested trace or create a shallow hierarchy. It helps to remember that a span starts when an event fires and it continues until you stop it with a finish directive. That means you can keep a span going to serve the purpose of being a parent.

    From our original example, we started three spans from the same event, and they can all be parents to other spans: The HAProxy session can be the top-most parent; the Client session span can be the parent to any spans that relate to work on the frontend; the HTTP request span can be the parent to other spans that relate to processing the HTTP request.

    instrumentation.cfg
    ini
    otel-scope frontend_http_request
      span "HAProxy session" root
      span "Client session" parent "HAProxy session"
      span "HTTP request" parent "Client session"
      otel-event on-frontend-http-request
    instrumentation.cfg
    ini
    otel-scope frontend_http_request
      span "HAProxy session" root
      span "Client session" parent "HAProxy session"
      span "HTTP request" parent "Client session"
      otel-event on-frontend-http-request

    Without a parent argument, your span will become a root span and start an entirely new trace.

  3. Inside the otel-scope section, set an otel-event directive to indicate the HAProxy event that will trigger the start of the span. See the list of available otel-events.

  4. Optional: Add a finish directive if the start of this span should be the end of another span. You’ll likely need to finish the span you’re creating in another span.

  5. In the otel-instrumentation section, add a scopes directive with its argument being the name you gave to the otel-scope. This adds the scope to the trace.

Set attributes on spans Jump to heading

You can set attributes on a span to capture extra, relevant information. Use the attribute directive in an otel-scope section to set a key and its value. There’s a convention to use dots in the name as a way to organize related attributes into categories, but the name is up to you. When declaring attributes, indentation is optional, but improves readability. An attribute applies to the active span, meaning the span declared prior to the attribute directive.

The syntax for an attribute is:

text
attribute <key> <sample> ... [{ if | unless } <condition>]
text
attribute <key> <sample> ... [{ if | unless } <condition>]

The value is a fetch. See the configuration tutorial on fetches.

Here’s an example:

instrumentation.cfg
ini
otel-scope frontend_http_request
  span "HAProxy session" root
  span "Client session" parent "HAProxy session"
    attribute "frontend.name" fe_name
  span "HTTP request" parent "Client session"
    attribute "http.method" method
    attribute "http.url" url
    attribute "http.version" str("HTTP/") req.ver
  otel-event on-frontend-http-request
otel-scope backend_http_request
  span "Process backend request rules" parent "HTTP request"
    attribute "backend.name" be_name
  otel-event on-backend-http-request
otel-scope server_session_start
  span "Server session" parent "HAProxy session"
    attribute "server.name" srv_name
  finish "Forward request"
  otel-event on-server-session-start
instrumentation.cfg
ini
otel-scope frontend_http_request
  span "HAProxy session" root
  span "Client session" parent "HAProxy session"
    attribute "frontend.name" fe_name
  span "HTTP request" parent "Client session"
    attribute "http.method" method
    attribute "http.url" url
    attribute "http.version" str("HTTP/") req.ver
  otel-event on-frontend-http-request
otel-scope backend_http_request
  span "Process backend request rules" parent "HTTP request"
    attribute "backend.name" be_name
  otel-event on-backend-http-request
otel-scope server_session_start
  span "Server session" parent "HAProxy session"
    attribute "server.name" srv_name
  finish "Forward request"
  otel-event on-server-session-start

Trigger spans with an otel-group action Jump to heading

You can group one or more scopes and then trigger them from your HAProxy configuration file via an otel-group action. These directives support this action:

  • http-request otel-group <filter-id> <group> [condition]
  • http-response otel-group <filter-id> <group> [condition]
  • http-after-response otel-group <filter-id> <group> [condition]
  • tcp-request content otel-group <filter-id> <group> [condition]
  • tcp-response content otel-group <filter-id> <group> [condition]

Suppose you want to pause processing of a request for three seconds and have that represented as a span in the trace. Add the following lines to your HAProxy configuration.

haproxy
frontend myfrontend
  filter opentelemetry  id my-otel-filter  config /etc/haproxy/instrumentation.cfg 
  
  http-request otel-group my-otel-filter start_pause_group
  http-request pause 3s
  http-request otel-group my-otel-filter end_pause_group
haproxy
frontend myfrontend
  filter opentelemetry  id my-otel-filter  config /etc/haproxy/instrumentation.cfg 
  
  http-request otel-group my-otel-filter start_pause_group
  http-request pause 3s
  http-request otel-group my-otel-filter end_pause_group
  • First, the http-request otel-group action triggers a group of scopes named start_pause_group.
  • Second, we start a pause via http-request pause 3s.
  • Third, we use an http-request otel-group action to trigger the group of scopes named end_pause_group.

In your instrumentation file, add these lines:

instrumentation.cfg
ini
[my-otel-filter]
otel-instrumentation my-instrumentation
  ...
  groups start_pause_group end_pause_group
  
otel-group start_pause_group
  scopes start_pause
otel-group end_pause_group
  scopes end_pause
otel-scope start_pause
  span "Paused pipeline" parent "HTTP request"
otel-scope end_pause
  finish "Paused pipeline"
instrumentation.cfg
ini
[my-otel-filter]
otel-instrumentation my-instrumentation
  ...
  groups start_pause_group end_pause_group
  
otel-group start_pause_group
  scopes start_pause
otel-group end_pause_group
  scopes end_pause
otel-scope start_pause
  span "Paused pipeline" parent "HTTP request"
otel-scope end_pause
  finish "Paused pipeline"
  • The groups directive adds groups to the trace.
  • The otel-group sections defines the groups. Inside each group, we set the scopes.
  • The otel-scope sections define the scopes. Note that when triggered exclusively by an otel-group action in your HAProxy configuration, you don’t need to set an otel-event in the otel-scope.

Your trace will show a new span named Paused pipeline that lasts for three seconds.

Set timestamped events on spans Jump to heading

A span represents a unit of work that has a start and an end time, but you can also define span events, which are events that happen during a single point in time. The event directive, which goes into an otel-scope, has this syntax:

text
event <name> [time [<unit>] <sample>] <key> <sample> ... [{ if | unless } <condition>]
text
event <name> [time [<unit>] <sample>] <key> <sample> ... [{ if | unless } <condition>]

It expects a name for the event and a key-value, which sets an event attribute.

For example, suppose you wanted to capture the point in time when an HTTP redirect happens. Add these lines to your HAProxy configuration to perform the redirect:

haproxy
frontend myfrontend
  filter opentelemetry  id my-otel-filter  config /etc/haproxy/instrumentation.cfg
  http-request set-var(req.targetpath) str(/newurl)
  acl redirected path,strcmp(req.targetpath) eq 0
  http-request redirect code 302  location http://%[hdr(host)]%[var(req.targetpath)] unless redirected
haproxy
frontend myfrontend
  filter opentelemetry  id my-otel-filter  config /etc/haproxy/instrumentation.cfg
  http-request set-var(req.targetpath) str(/newurl)
  acl redirected path,strcmp(req.targetpath) eq 0
  http-request redirect code 302  location http://%[hdr(host)]%[var(req.targetpath)] unless redirected
  • The http-request set-var directive sets a variable named req.targetpath to the URL path, /newurl, to which we’ll redirect the user. Using a variable simplifies the configuration, allowing us to reuse the value in multiple spots, including a line that checks if the user has already been redirected to this URL, preventing a loop.
  • The ACL named redirected checks if the current path is the redirect URL. This lets us know if we’ve already performed the redirect.
  • The http-request redirect directive performs the HTTP redirect, unless the redirect has already been performed. For more information, see the HTTP redirects configuration tutorial.

In the instrumentation file, the otel-scope named finish_http_request comes from a previous example. This contains the span that we’ll add an event to.

instrumentation.cfg
ini
otel-scope finish_http_request
  span "Forward request" parent "HAProxy session"
  finish "Process request"
  finish "HTTP request"
  otel-event on-http-end-request
  acl redirected path,strcmp(req.targetpath) eq 0
  event "redirected" "redirect.path" var(req.targetpath) if redirected
instrumentation.cfg
ini
otel-scope finish_http_request
  span "Forward request" parent "HAProxy session"
  finish "Process request"
  finish "HTTP request"
  otel-event on-http-end-request
  acl redirected path,strcmp(req.targetpath) eq 0
  event "redirected" "redirect.path" var(req.targetpath) if redirected
  • The acl line defines the redirected ACL in the instrumentation file. It returns true if the currently requested URL path matches the target redirect path, indicating if we’ve already performed the redirect.
  • The event directive adds a timestamped event named redirected with an attribute named redirect.targetpath, which contains the target redirect path. We add the event only if the ACL is true.

Your span will now show an event named redirected attached to the span named Forward request. The event will capture the redirect URL path.

Set the span status Jump to heading

A span has a status that indicates whether it succeeded or if an error occurred. By default, this status remains unset unless you set it. The status directive, which goes into an otel-scope section, has this syntax:

text
status <code> [<sample> ...] [{ if | unless } <condition>]
text
status <code> [<sample> ...] [{ if | unless } <condition>]

It can have one of these codes:

Status Description
error The operation contains an error. When using error, you can set one or more sample fetches to add an error message. For example: status "error" str("http.status_code: ") status.
ignore Don’t set a status. (Default)
ok The operation completed successfully.
unset Explicitly mark the status as unset.

Suppose you wanted to check whether the backend server returned a successful HTTP response or an error. If the response contains an error, you can set a description on the status line, such as to capture the HTTP status code. If the response was successful, you’ll set the span’s status to ok and also set attributes that capture the message’s content-type and content-length.

In your instrumentation file, add these lines:

instrumentation.cfg
ini
otel-scope http_response
  acl acl-http-status-ok status 100:399
  span "HTTP response" parent "Server session"
    status "error" str("http.status_code: ") status if !acl-http-status-ok
    status "ok" if acl-http-status-ok
    attribute "content.type" res.hdr("content-type") if acl-http-status-ok
    attribute "content.length" res.hdr("content-length") if acl-http-status-ok
  finish "Wait for response"
  otel-event on-http-response
instrumentation.cfg
ini
otel-scope http_response
  acl acl-http-status-ok status 100:399
  span "HTTP response" parent "Server session"
    status "error" str("http.status_code: ") status if !acl-http-status-ok
    status "ok" if acl-http-status-ok
    attribute "content.type" res.hdr("content-type") if acl-http-status-ok
    attribute "content.length" res.hdr("content-length") if acl-http-status-ok
  finish "Wait for response"
  otel-event on-http-response
  • The acl directive checks whether the HTTP status is between 100 and 399, which indicates a success.
  • The otel-scope named http_response comes from our previous example. It contains the span named HTTP response that we want to set the status of.
  • The status directive sets the status of the span. We use an if statement that checks the ACL named acl-http-status-ok to know whether to set an ok or error status.
  • When the acl-http-status-ok ACL is true, we also set attributes for the content-type and content-length.

Your span will now show a status of either OK or ERROR, depending on the HTTP response received from the server. If it’s ok, we set attributes that capture the content-type and content-length. If it’s an error, we set an error status description like http.status_code: 500.

In some cases, a span has a cause-effect relationship with another span instead of a parent-child relationship, such as when the processing of an HTTP request triggers the start of an asynchronous task. In those cases, you can define a span link. Span links appear in your trace as hyperlinks to the related span, making it easy to jump from one span to the other in the trace.

For example, suppose you wanted to use the HAProxy Enterprise Traffic Mirroring module to send a copy of the HTTP request to a QA server to process it through a new version of HAProxy, prior to deploying that version to your production servers. This traffic mirroring happens asynchronously. Linking this unit of work to your HAProxy session span offers a way to represent this relationship.

Add these lines to your HAProxy configuration:

haproxy
frontend myfrontend
  filter opentelemetry  id my-otel-filter  config /etc/haproxy/instrumentation.cfg
  # Directives to mirror traffic go here (not shown)...
  # Trigger the OpenTelemetry group
  http-request otel-group my-otel-filter mirror_traffic
haproxy
frontend myfrontend
  filter opentelemetry  id my-otel-filter  config /etc/haproxy/instrumentation.cfg
  # Directives to mirror traffic go here (not shown)...
  # Trigger the OpenTelemetry group
  http-request otel-group my-otel-filter mirror_traffic
  • The http-request otel-group action triggers the OpenTelemetry group named mirror_traffic.

In your instrumentation file, add these lines:

instrumentation.cfg
ini
[my-otel-filter]
otel-instrumentation my-instrumentation
  config /etc/haproxy/otel-client.yaml
  ...
  groups mirror_traffic
otel-group mirror_traffic
  scopes mirror_traffic
otel-scope mirror_traffic
  span "Mirror traffic" parent "HAProxy session"
  link "HTTP request" attr "link.kind" str("follows_from")  "mirror.url" str("mirror.mysite.com")
instrumentation.cfg
ini
[my-otel-filter]
otel-instrumentation my-instrumentation
  config /etc/haproxy/otel-client.yaml
  ...
  groups mirror_traffic
otel-group mirror_traffic
  scopes mirror_traffic
otel-scope mirror_traffic
  span "Mirror traffic" parent "HAProxy session"
  link "HTTP request" attr "link.kind" str("follows_from")  "mirror.url" str("mirror.mysite.com")
  • In the otel-instrumentation section, the groups directive adds the group named mirror_traffic to the trace.
  • The otel-group section named mirror_traffic contains the scope named mirror_traffic.
  • The otel-scope section named mirror_traffic defines a new span, Mirror traffic, that’s a child of the HAProxy session span. The link directive links this span to the span named HTTP request. We also add attributes to the link, which allow you to set metadata.

Your trace will show a span named Mirror traffic that has a link to the HTTP request span.

Propagate context across services Jump to heading

OpenTelemetry HAProxy context propagation

Context propagation shown in Jaeger

The power of OpenTelemetry lies in its support for distributed tracing, which is the ability to track a request as it moves between services, crossing process and system boundaries, perhaps traversing a series of microservices or application servers in a network. By collecting telemetry data from each point along the way and correlating it via a shared context (the trace ID and parent span ID), you can see the full picture. Your trace can reveal the system’s overall health and pinpoint bottlenecks.

To add the spans that HAProxy produces to a larger trace, you can extract the shared context from an HTTP header that was set by an upstream service. To pass the context to a service downstream, you can inject the context into an HTTP header.

To extract context from an HTTP header, change the otel-scope section that defines your root span:

instrumentation.cfg
ini
otel-scope frontend_http_request
  extract -mycontext use-headers
  span "HAProxy session" parent -mycontext
  span "Client session" parent "HAProxy session"
  span "HTTP request" parent "Client session"
  otel-event on-frontend-http-request
instrumentation.cfg
ini
otel-scope frontend_http_request
  extract -mycontext use-headers
  span "HAProxy session" parent -mycontext
  span "Client session" parent "HAProxy session"
  span "HTTP request" parent "Client session"
  otel-event on-frontend-http-request
  • Add an extract directive with a use-headers argument to get the context from the traceparent HTTP header and store it in a variable. We prefix the variable name with a dash, -mycontext. When the first character is a dash, the context name will be excluded from the HTTP header, making it compatible with non-HAProxy services. If the upstream service is HAProxy, you can omit the dash, allowing you to prefix the header value with a name.
  • If you’re propagating context between two OpenTelemetry filters in the same HAProxy configuration (not inter-process), you can pass the context via variables, which is faster than HTTP headers. Use the use-vars argument instead of use-headers.
  • Instead of setting the root argument on the top-most span, set parent to the name of the variable.
Test it with Python

This Python script, based on the OpenTelemetry Python propagation example, starts a web server on port 5002. It passes the context in an HTTP header to http://127.0.0.1:80/, where it expects HAProxy to be listening. It sends its spans via the OTLP exporter to http://localhost:4318/v1/traces. With this running, if you access the application at port 5002, it will combine its spans with the HAProxy spans, which are sent to the collector at port 4318.

upstream.py
py
from flask import Flask
import requests
from opentelemetry import trace, baggage
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
from opentelemetry.baggage.propagation import W3CBaggagePropagator
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
app = Flask(__name__)
# Service name is required for most backends
resource = Resource.create(attributes={
  SERVICE_NAME: "upstream-service"
})
trace.set_tracer_provider(TracerProvider(resource=resource))
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")))
tracer = trace.get_tracer(__name__)
@app.route('/')
def hello():
  with tracer.start_as_current_span("api1_span") as span:
    ctx = baggage.set_baggage("hello", "world")
    
    headers = {}
    W3CBaggagePropagator().inject(headers, ctx)
    TraceContextTextMapPropagator().inject(headers, ctx)
    print(headers)
    
    response = requests.get('http://127.0.0.1:80/', headers=headers)
    return f"Hello from upstream! Response from downstream: {response.text}"
if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5002)
upstream.py
py
from flask import Flask
import requests
from opentelemetry import trace, baggage
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
from opentelemetry.baggage.propagation import W3CBaggagePropagator
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
app = Flask(__name__)
# Service name is required for most backends
resource = Resource.create(attributes={
  SERVICE_NAME: "upstream-service"
})
trace.set_tracer_provider(TracerProvider(resource=resource))
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")))
tracer = trace.get_tracer(__name__)
@app.route('/')
def hello():
  with tracer.start_as_current_span("api1_span") as span:
    ctx = baggage.set_baggage("hello", "world")
    
    headers = {}
    W3CBaggagePropagator().inject(headers, ctx)
    TraceContextTextMapPropagator().inject(headers, ctx)
    print(headers)
    
    response = requests.get('http://127.0.0.1:80/', headers=headers)
    return f"Hello from upstream! Response from downstream: {response.text}"
if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5002)

To run this on Debian, try these commands:

sh
mkdir myproject
cd myproject
sudo apt install python3.13-venv
python3 -m venv .venv
. .venv/bin/activate
pip install flask opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http requests
python upstream.py
sh
mkdir myproject
cd myproject
sudo apt install python3.13-venv
python3 -m venv .venv
. .venv/bin/activate
pip install flask opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http requests
python upstream.py

To inject the context into an HTTP header, add the inject directive to an otel-scope that uses one of the following HAProxy events:

  • on-client-session-start
  • on-frontend-tcp-request
  • on-http-wait-request
  • on-http-body-request
  • on-frontend-http-request
  • on-switching-rules-request
  • on-backend-tcp-request
  • on-backend-http-request
  • on-http-tarpit-request
  • on-process-server-rules-request
  • on-http-process-request
  • on-tcp-rdp-cookie-request
  • on-process-sticking-rules-request
  • on-http-headers-request
  • on-http-wait-response
  • on-process-store-rules-response
  • on-http-response
  • on-http-headers-response

Example

instrumentation.cfg
ini
otel-scope process_request
  span "Process request" parent "HTTP request"
    inject "-mycontext" use-headers
    baggage "hello" str(world)
  finish "Process backend request rules"
  otel-event on-http-process-request
instrumentation.cfg
ini
otel-scope process_request
  span "Process request" parent "HTTP request"
    inject "-mycontext" use-headers
    baggage "hello" str(world)
  finish "Process backend request rules"
  otel-event on-http-process-request

The active span becomes the parent to the downstream service’s spans. In this case, we use the Process requests span.

Test it with Python

This Python script, based on the OpenTelemetry Python propagation example, starts a web server at localhost on port 5001. It receives the context in an HTTP header. It sends its spans via the OTLP exporter to http://localhost:4318/v1/traces. With this running, if you set 127.0.0.1:5001 as a backend server in your HAProxy configuration, it will combine its spans with the HAProxy spans, which are sent to the collector at port 4318.

downstream.py
py
from flask import Flask, request
from opentelemetry import trace, baggage
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.baggage.propagation import W3CBaggagePropagator
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
app = Flask(__name__)
# Service name is required for most backends
resource = Resource.create(attributes={
  SERVICE_NAME: "downstream-service"
})
trace.set_tracer_provider(TracerProvider(resource=resource))
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")))
tracer = trace.get_tracer(__name__)
@app.route('/')
def hello():
  # Example: Log headers received in the request in API 2
  headers = dict(request.headers)
  print(f"Received headers: {headers}")
  carrier ={'traceparent': headers['Traceparent']}
  ctx = TraceContextTextMapPropagator().extract(carrier=carrier)
  print(f"Received context: {ctx}")
  
  b2 ={'baggage': headers['Baggage']}
  ctx2 = W3CBaggagePropagator().extract(b2, context=ctx)
  print(f"Received context2: {ctx2}")
  
  # Start a new span
  with tracer.start_span("api2_span", context=ctx2):
    # Use propagated context
    print(baggage.get_baggage('hello', ctx2))
    return "Hello from downstream!"
    
if __name__ == '__main__':
  app.run(port=5001)
downstream.py
py
from flask import Flask, request
from opentelemetry import trace, baggage
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.baggage.propagation import W3CBaggagePropagator
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
app = Flask(__name__)
# Service name is required for most backends
resource = Resource.create(attributes={
  SERVICE_NAME: "downstream-service"
})
trace.set_tracer_provider(TracerProvider(resource=resource))
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")))
tracer = trace.get_tracer(__name__)
@app.route('/')
def hello():
  # Example: Log headers received in the request in API 2
  headers = dict(request.headers)
  print(f"Received headers: {headers}")
  carrier ={'traceparent': headers['Traceparent']}
  ctx = TraceContextTextMapPropagator().extract(carrier=carrier)
  print(f"Received context: {ctx}")
  
  b2 ={'baggage': headers['Baggage']}
  ctx2 = W3CBaggagePropagator().extract(b2, context=ctx)
  print(f"Received context2: {ctx2}")
  
  # Start a new span
  with tracer.start_span("api2_span", context=ctx2):
    # Use propagated context
    print(baggage.get_baggage('hello', ctx2))
    return "Hello from downstream!"
    
if __name__ == '__main__':
  app.run(port=5001)

To run this on Debian, try these commands:

sh
mkdir myproject
cd myproject
sudo apt install python3.13-venv
python3 -m venv .venv
. .venv/bin/activate
pip install flask opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http requests
python downstream.py
sh
mkdir myproject
cd myproject
sudo apt install python3.13-venv
python3 -m venv .venv
. .venv/bin/activate
pip install flask opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http requests
python downstream.py

Use baggage to pass data across services Jump to heading

Baggage is data that you can pass between services when propagating context. For example, you can use it to pass the name of the upstream service to HAProxy. Another example: You can use it to pass the name of the HAProxy backend to a downstream service. Baggage makes it easy to share information that can help during debugging.

When HAProxy gets baggage from an extracted context, it becomes automatically available to child spans and downstream services. You can also add new key-value pairs to baggage. Beware that because baggage that you add will be transferred to downstream services, and is in a sense global data, you should check that you’re not sharing sensitive information like passwords and keys.

  • When your extract directive uses use-headers, HAProxy gets baggage from the baggage HTTP header.
  • When your extract directive uses use-vars, HAProxy gets baggage from variables stored in the context.

To add a key-value pair to baggage, use the baggage directive.

instrumentation.cfg
ini
otel-scope process_request
  span "Process request" parent "HTTP request"
    inject "-mycontext" use-headers
    baggage "myheader" req.fhdr("x-myheader")
    baggage "backendname" be_name
    baggage "mystring" str(somevalue)
  finish "Process backend request rules"
  otel-event on-http-process-request
instrumentation.cfg
ini
otel-scope process_request
  span "Process request" parent "HTTP request"
    inject "-mycontext" use-headers
    baggage "myheader" req.fhdr("x-myheader")
    baggage "backendname" be_name
    baggage "mystring" str(somevalue)
  finish "Process backend request rules"
  otel-event on-http-process-request

Set variables Jump to heading

To keep your instrumentation file tidy, it can help to store fields in variables. Variables propagate to child spans, so you could declare variables in a parent span and use them in child spans.

instrumentation.cfg
ini
otel-scope frontend_http_request
  extract -mycontext use-headers
  span "HAProxy session" parent -mycontext
  span "Client session" parent "HAProxy session"
  span "HTTP request" parent "Client session"
    set-var     txn.otel.method method
    set-var-ctx txn.otel.trace_id    "HTTP request" trace-id
    set-var-ctx txn.otel.traceparent "HTTP request" traceparent
    set-var-ctx txn.otel.sampled     "HTTP request" sampled
    set-var-ctx txn.otel.baggage     "HTTP request" baggage
    set-var-ctx txn.otel.haproxy_id  "HTTP request" baggage(haproxy_id)
  otel-event on-frontend-http-request
otel-scope backend_http_request
  span "Process backend request rules" parent "HTTP request"
    attribute "otel.method"      var(txn.otel.method)
    attribute "otel.trace_id"    var(txn.otel.trace_id)
    attribute "otel.traceparent" var(txn.otel.traceparent)
    attribute "otel.sampled"     var(txn.otel.sampled)
    attribute "otel.baggage"     var(txn.otel.baggage)
    attribute "otel.haproxy_id"  var(txn.otel.haproxy_id)
  otel-event on-backend-http-request
instrumentation.cfg
ini
otel-scope frontend_http_request
  extract -mycontext use-headers
  span "HAProxy session" parent -mycontext
  span "Client session" parent "HAProxy session"
  span "HTTP request" parent "Client session"
    set-var     txn.otel.method method
    set-var-ctx txn.otel.trace_id    "HTTP request" trace-id
    set-var-ctx txn.otel.traceparent "HTTP request" traceparent
    set-var-ctx txn.otel.sampled     "HTTP request" sampled
    set-var-ctx txn.otel.baggage     "HTTP request" baggage
    set-var-ctx txn.otel.haproxy_id  "HTTP request" baggage(haproxy_id)
  otel-event on-frontend-http-request
otel-scope backend_http_request
  span "Process backend request rules" parent "HTTP request"
    attribute "otel.method"      var(txn.otel.method)
    attribute "otel.trace_id"    var(txn.otel.trace_id)
    attribute "otel.traceparent" var(txn.otel.traceparent)
    attribute "otel.sampled"     var(txn.otel.sampled)
    attribute "otel.baggage"     var(txn.otel.baggage)
    attribute "otel.haproxy_id"  var(txn.otel.haproxy_id)
  otel-event on-backend-http-request
  • Use set-var for storing sample fetch values. Its arguments are the variable’s name, then the fetch. This directive supports if and unless condition statements.
  • Use set-var-ctx for storing a field from a referenced span or span context. Its arguments are the variable name, a reference to a span or context, and the field to store. This directive supports if and unless condition statements.
  • Use the var fetch to read a variable by its name.
  • Use unset-var to remove a variable.

The set-var-ctx directive supports storings these fields:

Field Description
trace-id trace id, lowercase hex
span-id span id, lowercase hex
trace-flags W3C trace-flags byte, lowercase hex
traceparent full W3C 00-<trace>-<span>-<flags> string
tracestate full tracestate header (context references only)
tracestate() a single tracestate entry (context references only)
baggage the whole baggage carrier (all entries)
baggage() a single baggage entry value
sampled 1 if the sampled flag is set, else 0
valid 1 if the context is valid, else 0
remote 1 if the context is remote, else 0

Rate limit filter activations Jump to heading

Traces help you understand how requests flow through your application. They help identify bottlenecks or other problem areas. But you don’t always need to capture every request to get these insights. To record telemetry data for only a percentage of your traffic, set a rate limit to activate the filter for that percentage.

In your instrumentation file, under the otel-instrumentation section, set the rate-limit directive to a floating-point number between 0.0 and 100.0. It defaults to 100.0. This directive sets the percentage of streams for which the filter will be active. In the example below, the filter is active for 50% of streams.

instrumentation.cfg
ini
[my-otel-filter]
otel-instrumentation my-instrumentation
  config /etc/haproxy/otel-client.yaml
  rate-limit 50.0
instrumentation.cfg
ini
[my-otel-filter]
otel-instrumentation my-instrumentation
  config /etc/haproxy/otel-client.yaml
  rate-limit 50.0

Define ACLs Jump to heading

You can define named, reusable ACL expressions in the instrumentation file to use with if and unless conditions. For a full description of ACLs, see the ACLs configuration tutorial.

In the otel-instrumentation section or any of the otel-scope sections, use the acl directive. It compares a sample fetch value against a known value. If there’s a match, the expressions return true, otherwise false.

In this example, we define an ACL named acl-http-status-ok and then use it when deciding how to set the status of the HTTP response span. We could have instead set this ACL in the otel-scope section.

instrumentation.cfg
ini
[my-otel-filter]
otel-instrumentation my-instrumentation
  config /etc/haproxy/otel-client.yaml
  acl acl-http-status-ok status 100:399
  ...
otel-scope http_response
  span "HTTP response" parent "Server session"
    status "error" str("http.status_code: ") status unless acl-http-status-ok
    status "ok" if acl-http-status-ok
    attribute "content.type" res.hdr("content-type") if acl-http-status-ok
    attribute "content.length" res.hdr("content-length") if acl-http-status-ok
  finish "Wait for response"
  otel-event on-http-response
instrumentation.cfg
ini
[my-otel-filter]
otel-instrumentation my-instrumentation
  config /etc/haproxy/otel-client.yaml
  acl acl-http-status-ok status 100:399
  ...
otel-scope http_response
  span "HTTP response" parent "Server session"
    status "error" str("http.status_code: ") status unless acl-http-status-ok
    status "ok" if acl-http-status-ok
    attribute "content.type" res.hdr("content-type") if acl-http-status-ok
    attribute "content.length" res.hdr("content-length") if acl-http-status-ok
  finish "Wait for response"
  otel-event on-http-response

See also Jump to heading

Do you have any suggestions on how we can improve the content of this page?

Previous page Installation Next page Logs
© 2026 HAProxy Technologies, LLC. All Rights Reserved
Manage Cookie Preferences