OpenTelemetry
Export logs using the HAProxy OpenTelemetry filter
This page applies to:
- HAProxy 3.4 and newer
The HAProxy OpenTelemetry filter supports forwarding HAProxy logs to an OpenTelemetry compatible logging system, such as OpenTelemetry Collector. The benefit of this is that your logs will gain metadata that links them to your traces, allowing you to correlate a client’s request across both types of signals.
OpenTelemetry approaches logging with the intent of not reinventing the wheel. Logging protocols and frameworks are often well established within an organization, helping operations, security, and engineering teams to understand events happening within their infrastructure and applications. Instead of trying to change how organizations capture and format logs, OpenTelemetry wants only to correlate logs with traces so that you can easily go from one to the other for a given request. To achieve that, you need to add metadata to your logs. Then, while still relaying logs to the servers you’ve used before, you can begin to send your logs to software that understands how to interpret the OpenTelemetry protocol and link your logs to your traces.
Enable OpenTelemetry logs Jump to heading
To enable OpenTelemetry logging:
-
The advantage of adding OpenTelemetry metadata to your logs is to correlate logs with traces. Begin by configuring traces, which will set up the instrumentation file in which you’ll set logging.
-
In your instrumentation file, add a
log-recorddirective to anotel-scopesection. The syntax is:textlog-record <severity> [id <integer> event <name>] [time [<unit>] <sample>] [span <ref>] [attr <key> <sample>] ... <sample> ... [{ if | unless } <condition>]textlog-record <severity> [id <integer> event <name>] [time [<unit>] <sample>] [span <ref>] [attr <key> <sample>] ... <sample> ... [{ if | unless } <condition>] -
Configure the client library to send your log data to software that can process it.
Example: HTTP log format Jump to heading
In the example below, we add a log-record to the otel-scope section that subscribes to the on-server-session-end event, which runs at the end of the session. By emitting the log record at the end of the session, we ensure that all log variables are available. We’re using the HTTP log format. You can, however, place log-record directives into any of your otel-scope sections. Having more than one log-record in your instrumentation file lets you emit messages for various events.
instrumentation.cfgini
instrumentation.cfgini
- We set the severity to the level we’d like to assign to this log message, in this case
info. Supported severity levels aretrace,trace2,trace3,trace4,debug,debug2,debug3,debug4,info,info2,info3,info4,warn,warn2,warn3,warn4,error,error2,error3,error4,fatal,fatal2,fatal3, andfatal4. For more information, see the entry onSeverityNumberin the OpenTelemetry Logs Data Model documentation. - The
idandeventfields are up to you. Choose any number and string, respectively, that helps identify the origin of this log record. - With the
spanargument, we set the name of the span with which we want to correlate this log record. Notice how we’re defininglog-recordin anotel-scopesection that doesn’t define theHTTP requestspan. That’s fine; it’s defined in a differentotel-scopein our instrumentation file. - The rest of the line sets the body of the log record. In this case, we store the details of the current HTTP request, using the HTTP log format.
The generated LogRecord stores the log message in its Body field. Other fields are metadata that estabish the correlation with trace data and help to categorize the message, including the timestamp, severity, event name, trace ID, and span ID. For example:
text
text
Example: JSON log format Jump to heading
The log-record directive can set a JSON log format. In the example below, we add a log-record to the otel-scope section that subscribes to the on-server-session-end event, which runs at the end of the session when all log variables are available.
instrumentation.cfgini
instrumentation.cfgini
- We set the severity to the level we’d like to assign to this log message, in this case
info. Supported severity levels aretrace,trace2,trace3,trace4,debug,debug2,debug3,debug4,info,info2,info3,info4,warn,warn2,warn3,warn4,error,error2,error3,error4,fatal,fatal2,fatal3, andfatal4. For more information, see the entry onSeverityNumberin the OpenTelemetry Logs Data Model documentation. - The
idandeventfields are up to you. Choose any number and string, respectively, that helps identify the origin of this log record. - With the
spanargument, we set the name of the span with which we want to correlate this log record. Notice how we’re defininglog-recordin anotel-scopesection that doesn’t define theHTTP requestspan. That’s fine; it’s defined in a differentotel-scopein our instrumentation file. - The rest of the line sets the body of the log record. In this case, we store the details of the current HTTP request, using a JSON format.
The log record will look like this:
text
text
Add attributes to log records Jump to heading
You can add attributes to your log records to serve as extra metadata. Use the log-record directive’s attr argument to add an attribute. In this example, we add one attribute named app.comment with a string literal value and another named app.version set to a captured response header.
instrumentation.cfgini
instrumentation.cfgini
The LogRecord will have attributes:
text
text
See also Jump to heading
- The HAProxy OpenTelemetry GitHub documentation describes the logging directives and arguments.