Export metrics using the HAProxy OpenTelemetry filter

The HAProxy OpenTelemetry filter can emit measurements that depict the activity happening in the load balancer. For example, you can count the number of active HAProxy sessions at the same time as you emit a span in a trace that marks the start of the client’s session. The measurements are tied to the events you subscribed to in otel-scope sections in your instrumentation file, which you set up for tracing.

These metrics doesn’t replace those you get with the Statistic dashboard or Prometheus metrics, since those provide a finer level of detail. Instead, use HAProxy OpenTelemetry metrics as a supplement, to monitor the health of client requests at a coarse level, in conjunction with tracing. Some logging frameworks support OpenTelemetry metrics natively. For other software, you can relay the data through OpenTelemetry Collector, which can convert it into the Prometheus format for wider compatibility.

Enable OpenTelemetry metrics Jump to heading #

To enable OpenTelemetry metrics:

1. The metrics correspond to the same scopes you would define for traces, which are triggered by HAProxy events. To see how to configure scopes, see the configuration tutorial on traces.

2. In your instrumentation file, add an instrument directive to an otel-scope section. This prepares a metric by setting its type, aggregation, description, unit of measurement, and fetch sample. This declaration doesn’t emit the measurement; to do that, you’ll add a separate instrument update directive.

   The syntax is:

   text
   instrument <type> <name> [aggr <aggregation>] [desc <description>] [unit <unit>] value <sample> [bounds <bounds>] [{ if | unless } <condition>]

   text
   instrument <type> <name> [aggr <aggregation>] [desc <description>] [unit <unit>] value <sample> [bounds <bounds>] [{ if | unless } <condition>]

   • The supported instrument types are:

     Type	Description	Default Aggregation
     cnt_int	A counter (uint64) measures a non-negative, increasing value.	sum
     gauge_int	A gauge (int64) measures non-additive values when changes occur.	last_value
     hist_int	A histogram (uint64) measures a distribution of values over time.	histogram
     udcnt_int	An up-down counter (int64) can increment and decrement, allowing you to observe a cumulative value that goes up or down.	sum

   • Set a name in double quotes, such as "haproxy.sessions.active". By convention, use dots to create categories of metrics.

   • The aggr argument sets how to aggregate the measurements for this metric. Typically, you can leave aggr unset and allow it to use the instrument’s default type of aggregation. The supported aggregation values are:

     Aggregations	Description
     default	Use the SDK default for the instrument type.
     drop	Discard the measurements.
     exp_histogram	Use a base-2 exponential histogram.
     histogram	Use an explicit bucket histogram where you must set the bounds argument.
     last_value	Use the last recorded value.
     sum	Use a sum of the recorded values.

   • Set a description in double quotes, such as "Active sessions".

   • Optional: Set the unit of measurement, such as ms for milliseconds or ns for nanoseconds. Or, enclose the unit in curly braces, such as {session} and {request}, to set an annotation that describes the data, but will be ignored by parsers.

   • Set the value to a literal integer, int(1), or a fetch method like lat_ns_tot, which returns the latency in total nanoseconds. For up-down counters, you can set a negative integer to decrement the metric, such as int(-1).

   • For histograms, set the upper bounds of the buckets. For example, 1000 1000000 1000000000 will create four buckets: -∞ to 1000, 1001 to 1000000, 1000001 to 1000000000, and 1000000000 to +∞.

   • Optional: Set an if or unless statement to enable the metric only when the condition is true. The condition is an ACL expression.

3. Add an instrument update directive to emit a measurement when the event in the same otel-scope section fires. Because you must name the metric to update, your instrument update directive can be in a different otel-scope section than where you declared the original instrument directive. You might separate them if, for example, you wanted to increment the metric at the start of a session and decrement it at the end, which would key off two distinct events.

   The syntax is:

   text
   instrument update <name> [attr <key> <sample> ...] [{ if | unless } <condition>]

   text
   instrument update <name> [attr <key> <sample> ...] [{ if | unless } <condition>]

   • Specify the name of the metric to emit.

   • Optional: Add one or more attr arguments to set attributes on the measurement.

   • Optional: Set an if or unless statement to emit the metric only when the condition is true. The condition is an ACL expression.

4. Configure the client library to send your measurements to software that can process it.

Examples Jump to heading #

This instrumentation file shows examples of metrics.

• Active client sessions (up-down counter)
• HTTP request count (counter)
• Frontend connections (gauge)
• HTTP request latency (explicit bucket histogram)

Your metrics can utilize any of the HAProxy fetch methods, depending on when the event fires.

instrumentation.cfg
ini
[my-otel-filter]
otel-instrumentation my-instrumentation
  config /etc/haproxy/otel-client.yaml
  ...
otel-scope frontend_http_request
  span "HAProxy session" root
  span "Client session" parent "HAProxy session"
  span "HTTP request" parent "Client session"
  # Increment the number of active HAProxy sessions (use a variable to hold the value)
  set-var sess.otel_udcnt_delta int(1)
  instrument udcnt_int "haproxy.sessions.active" desc "Active client sessions" value var(sess.otel_udcnt_delta) unit "{session}"
  instrument update "haproxy.sessions.active"
  # Count the number of HTTP requests
  instrument cnt_int  "haproxy.http.requests" desc "HTTP request count" value int(1) unit "{request}"
  instrument update "haproxy.http.requests" attr "phase" str("request")
  # Measure the current number of frontend connections
  instrument gauge_int "haproxy.fe.connections" desc "Frontend connections" value fe_conn unit "{connection}"
  instrument update "haproxy.fe.connections"
  otel-event on-frontend-http-request
  ...
otel-scope http_response
  span "HTTP response" parent "Server session"
  finish "Wait for response"
  # Measure latency in three buckets
  instrument hist_int "haproxy.http.latency" desc "HTTP request latency" value lat_ns_tot unit "ns" aggr "histogram" bounds "1000 1000000 1000000000"
  instrument update "haproxy.http.latency" attr "phase" str("response")
  otel-event on-http-response
  ...
otel-scope server_session_end
  finish *
  # Decrement the number of active HAProxy sessions (update the variable before calling update)
  set-var sess.otel_udcnt_delta int(-1)
  instrument update "haproxy.sessions.active"
  otel-event on-server-session-end

instrumentation.cfg
ini
[my-otel-filter]
otel-instrumentation my-instrumentation
  config /etc/haproxy/otel-client.yaml
  ...
otel-scope frontend_http_request
  span "HAProxy session" root
  span "Client session" parent "HAProxy session"
  span "HTTP request" parent "Client session"
  # Increment the number of active HAProxy sessions (use a variable to hold the value)
  set-var sess.otel_udcnt_delta int(1)
  instrument udcnt_int "haproxy.sessions.active" desc "Active client sessions" value var(sess.otel_udcnt_delta) unit "{session}"
  instrument update "haproxy.sessions.active"
  # Count the number of HTTP requests
  instrument cnt_int  "haproxy.http.requests" desc "HTTP request count" value int(1) unit "{request}"
  instrument update "haproxy.http.requests" attr "phase" str("request")
  # Measure the current number of frontend connections
  instrument gauge_int "haproxy.fe.connections" desc "Frontend connections" value fe_conn unit "{connection}"
  instrument update "haproxy.fe.connections"
  otel-event on-frontend-http-request
  ...
otel-scope http_response
  span "HTTP response" parent "Server session"
  finish "Wait for response"
  # Measure latency in three buckets
  instrument hist_int "haproxy.http.latency" desc "HTTP request latency" value lat_ns_tot unit "ns" aggr "histogram" bounds "1000 1000000 1000000000"
  instrument update "haproxy.http.latency" attr "phase" str("response")
  otel-event on-http-response
  ...
otel-scope server_session_end
  finish *
  # Decrement the number of active HAProxy sessions (update the variable before calling update)
  set-var sess.otel_udcnt_delta int(-1)
  instrument update "haproxy.sessions.active"
  otel-event on-server-session-end

See also Jump to heading #

• The HAProxy OpenTelemetry GitHub documentation describes the metrics directives and arguments.