Prometheus metrics

The load balancer exposes a Prometheus endpoint that publishes metrics that you can scrape with a Prometheus-compatible agent such as the Prometheus server, Fluentd, Telegraf, and Metricbeat. It is enabled by default, so you only need to configure the IP address and port where it listens.

Configure the Prometheus exporter Jump to heading #

To configure the Prometheus exporter:

1. Add a new frontend section to your configuration file. In it, include the http-request use-service prometheus-exporter directive. In the following example, the Prometheus exporter page becomes available on all IP addresses at port 8405:

   haproxy
   frontend prometheus
     bind :8405
     mode http
     http-request use-service prometheus-exporter
     no log

   haproxy
   frontend prometheus
     bind :8405
     mode http
     http-request use-service prometheus-exporter
     no log

   The no log directive disables logging for these requests.

2. Optional: Add a conditional statement to the end of the line to show the metrics page only on a certain URL path. In the next example, the metrics become available at the URL path /metrics on port 8405:

   haproxy
   frontend prometheus
     bind *:8405
     mode http
     http-request use-service prometheus-exporter if { path /metrics }
     no log

   haproxy
   frontend prometheus
     bind *:8405
     mode http
     http-request use-service prometheus-exporter if { path /metrics }
     no log

Configure the Prometheus server Jump to heading #

The Prometheus server collects your load balancer metrics and stores them in its database. From there, you can use software like Grafana to graph them. To configure your Prometheus server to scrape the metrics page:

1. Follow the First Steps with Prometheus guide to install the Prometheus server.

2. Edit its prometheus.yml file so that it includes a job for scraping load balancer metrics. Add the following to the scrape_configs section, replacing localhost with your load balancer’s address. If you run multiple load balancer servers, include them in the targets array.

   prometheus.yml
   yaml
   - job_name: 'load-balancer-metrics'
     static_configs:
       - targets: ['localhost:8405']

   prometheus.yml
   yaml
   - job_name: 'load-balancer-metrics'
     static_configs:
       - targets: ['localhost:8405']

   By default, the Prometheus server scrapes the URL /metrics. To change this path, set the metrics_path parameter in the scrape_configs section of the Prometheus configuration file.

Host over HTTPS Jump to heading #

To serve the Prometheus endpoint over HTTPS:

1. Edit the load balancer configuration and add the ssl parameter to the bind line to enable HTTPS. Also, set the crt parameter to the path where you’ve stored your TLS key and certificate file:

   haproxy
   frontend prometheus
     bind :8405 ssl crt /certs/site.pem
     mode http
     http-request use-service prometheus-exporter
     no log

   haproxy
   frontend prometheus
     bind :8405 ssl crt /certs/site.pem
     mode http
     http-request use-service prometheus-exporter
     no log

2. On the Prometheus server, edit the prometheus.yml configuration file. Set scheme to https:

   prometheus.yml
   yaml
   - job_name: 'load-balancer-metrics'
     scheme: https
     static_configs:
       - targets: ['localhost:8405']

   prometheus.yml
   yaml
   - job_name: 'load-balancer-metrics'
     scheme: https
     static_configs:
       - targets: ['localhost:8405']

3. Optional: You can also configure the tls_config block to set a CA certificate for verifying the load balancer’s certificate, enable client certificates, set a ServerName extension, or disable validation of the load balancer’s certificate. See the Prometheus configuration documentation for details.

   Below, we skip validation of the certificate.

   prometheus.yml
   yaml
   - job_name: 'load-balancer-metrics'
     scheme: https
     static_configs:
       - targets: ['localhost:8405']
     tls_config:
       # Disable validation of server certificate
       insecure_skip_verify: true

   prometheus.yml
   yaml
   - job_name: 'load-balancer-metrics'
     scheme: https
     static_configs:
       - targets: ['localhost:8405']
     tls_config:
       # Disable validation of server certificate
       insecure_skip_verify: true

Filter metrics Jump to heading #

To reduce the volume of metrics Prometheus scrapes from the load balancer, you can specify filter parameters in the Prometheus configuration file. The filters are applied on the load balancer, thereby reducing the load on the network and the Prometheus server.

In the following section, we describe several use cases.

Specify metrics scope Jump to heading #

Use the scope parameter to specify which categories of metrics to scrape. Categories include:

• global
• frontend
• listener
• backend
• server
• sticktable
• asterisk (*) = all scopes
• empty = no metrics

In the load balancer job shown below, we are scraping only frontend and sticktable metrics:

prometheus.yml
yaml
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
scrape_configs:
  - job_name: 'load-balancer-metrics'
    static_configs:
      - targets: ['192.168.56.31:8405']
    params: 
      scope: 
        - frontend
        - sticktable

prometheus.yml
yaml
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
scrape_configs:
  - job_name: 'load-balancer-metrics'
    static_configs:
      - targets: ['192.168.56.31:8405']
    params: 
      scope: 
        - frontend
        - sticktable

Omit servers in maintenance mode Jump to heading #

To reduce the volume of metrics returned, omit results from servers in maintenance mode. Add the no-maint key. The no-maint key is particularly useful with load balancer configurations using server-template to define servers dynamically, wherein unused server slots are left in maintenance mode.

There is no consistency check on server state. So if the state of a server changes while the exporter is running, only a portion of the metrics for this server will be reported.

Below, we define no-maint to omit metrics from servers in maintenance mode:

prometheus.yml
yaml
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
scrape_configs:
  - job_name: 'load-balancer-metrics'
    static_configs:
      - targets: ['192.168.56.31:8405']
    params: 
      scope: 
        - frontend
        - sticktable
     no-maint:
       - empty

prometheus.yml
yaml
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
scrape_configs:
  - job_name: 'load-balancer-metrics'
    static_configs:
      - targets: ['192.168.56.31:8405']
    params: 
      scope: 
        - frontend
        - sticktable
     no-maint:
       - empty

Scrape health status only Jump to heading #

To scrape only server health status without other server metrics, configure the job with a metric_relabel_configs section, as follows:

prometheus.yml
yaml
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: 'load-balancer-metrics'
    static_configs:
      - targets: ['192.168.56.31:8405']
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'haproxy_(process_|frontend_|listener_|backend_|server_check_status).*'
        action: keep

prometheus.yml
yaml
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: 'load-balancer-metrics'
    static_configs:
      - targets: ['192.168.56.31:8405']
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'haproxy_(process_|frontend_|listener_|backend_|server_check_status).*'
        action: keep

Exported metrics Jump to heading #

The Prometheus exporter generates the following metrics.

Global metrics Jump to heading #

• haproxy_process_active_peers
• haproxy_process_build_info
• haproxy_process_busy_polling_enabled
• haproxy_process_bytes_out_rate
• haproxy_process_bytes_out_total
• haproxy_process_connected_peers
• haproxy_process_connections_total
• haproxy_process_current_backend_ssl_key_rate
• haproxy_process_current_connection_rate
• haproxy_process_current_connections
• haproxy_process_current_frontend_ssl_key_rate
• haproxy_process_current_run_queue
• haproxy_process_current_session_rate
• haproxy_process_current_ssl_connections
• haproxy_process_current_ssl_rate
• haproxy_process_current_tasks
• haproxy_process_current_zlib_memory
• haproxy_process_dropped_logs_total
• haproxy_process_failed_resolutions
• haproxy_process_frontend_ssl_reuse
• haproxy_process_hard_max_connections
• haproxy_process_http_comp_bytes_in_total
• haproxy_process_http_comp_bytes_out_total
• haproxy_process_idle_time_percent
• haproxy_process_jobs
• haproxy_process_limit_connection_rate
• haproxy_process_limit_http_comp
• haproxy_process_limit_session_rate
• haproxy_process_limit_ssl_rate
• haproxy_process_listeners
• haproxy_process_max_backend_ssl_key_rate
• haproxy_process_max_connection_rate
• haproxy_process_max_connections
• haproxy_process_max_fds
• haproxy_process_max_frontend_ssl_key_rate
• haproxy_process_max_memory_bytes
• haproxy_process_max_pipes
• haproxy_process_max_session_rate
• haproxy_process_max_sockets
• haproxy_process_max_ssl_connections
• haproxy_process_max_ssl_rate
• haproxy_process_max_zlib_memory
• haproxy_process_nbproc
• haproxy_process_nbthread
• haproxy_process_pipes_free_total
• haproxy_process_pipes_used_total
• haproxy_process_pool_allocated_bytes
• haproxy_process_pool_failures_total
• haproxy_process_pool_used_bytes
• haproxy_process_recv_logs_total
• haproxy_process_relative_process_id
• haproxy_process_requests_total
• haproxy_process_spliced_bytes_out_total
• haproxy_process_ssl_cache_lookups_total
• haproxy_process_ssl_cache_misses_total
• haproxy_process_ssl_connections_total
• haproxy_process_start_time_seconds
• haproxy_process_stopping
• haproxy_process_unstoppable_jobs
• haproxy_process_uptime_seconds

Frontend metrics Jump to heading #

• haproxy_frontend_bytes_in_total
• haproxy_frontend_bytes_out_total
• haproxy_frontend_connections_rate_max
• haproxy_frontend_connections_total
• haproxy_frontend_current_sessions
• haproxy_frontend_denied_connections_total
• haproxy_frontend_denied_sessions_total
• haproxy_frontend_failed_header_rewriting_total
• haproxy_frontend_http_cache_hits_total
• haproxy_frontend_http_cache_lookups_total
• haproxy_frontend_http_comp_bytes_bypassed_total
• haproxy_frontend_http_comp_bytes_in_total
• haproxy_frontend_http_comp_bytes_out_total
• haproxy_frontend_http_comp_responses_total
• haproxy_frontend_http_requests_rate_max
• haproxy_frontend_http_requests_total
• haproxy_frontend_http_responses_total
• haproxy_frontend_intercepted_requests_total
• haproxy_frontend_internal_errors_total
• haproxy_frontend_limit_session_rate
• haproxy_frontend_limit_sessions
• haproxy_frontend_max_session_rate
• haproxy_frontend_max_sessions
• haproxy_frontend_request_errors_total
• haproxy_frontend_requests_denied_total
• haproxy_frontend_responses_denied_total
• haproxy_frontend_sessions_total
• haproxy_frontend_status

Listener metrics Jump to heading #

• haproxy_listener_bytes_in_total
• haproxy_listener_bytes_out_total
• haproxy_listener_current_sessions
• haproxy_listener_denied_connections_total
• haproxy_listener_denied_sessions_total
• haproxy_listener_failed_header_rewriting_total
• haproxy_listener_internal_errors_total
• haproxy_listener_limit_sessions
• haproxy_listener_max_sessions
• haproxy_listener_request_errors_total
• haproxy_listener_requests_denied_total
• haproxy_listener_responses_denied_total
• haproxy_listener_sessions_total
• haproxy_listener_status

Backend metrics Jump to heading #

• haproxy_backend_active_servers
• haproxy_backend_agg_check_status
• haproxy_backend_agg_server_status
• haproxy_backend_backup_servers
• haproxy_backend_bytes_in_total
• haproxy_backend_bytes_out_total
• haproxy_backend_check_last_change_seconds
• haproxy_backend_check_up_down_total
• haproxy_backend_client_aborts_total
• haproxy_backend_connect_time_average_seconds
• haproxy_backend_connection_attempts_total
• haproxy_backend_connection_errors_total
• haproxy_backend_connection_reuses_total
• haproxy_backend_current_queue
• haproxy_backend_current_sessions
• haproxy_backend_downtime_seconds_total
• haproxy_backend_failed_header_rewriting_total
• haproxy_backend_http_cache_hits_total
• haproxy_backend_http_cache_lookups_total
• haproxy_backend_http_comp_bytes_bypassed_total
• haproxy_backend_http_comp_bytes_in_total
• haproxy_backend_http_comp_bytes_out_total
• haproxy_backend_http_comp_responses_total
• haproxy_backend_http_requests_total
• haproxy_backend_http_responses_total
• haproxy_backend_internal_errors_total
• haproxy_backend_last_session_seconds
• haproxy_backend_limit_sessions
• haproxy_backend_loadbalanced_total
• haproxy_backend_max_connect_time_seconds
• haproxy_backend_max_queue
• haproxy_backend_max_queue_time_seconds
• haproxy_backend_max_response_time_seconds
• haproxy_backend_max_session_rate
• haproxy_backend_max_sessions
• haproxy_backend_max_total_time_seconds
• haproxy_backend_queue_time_average_seconds
• haproxy_backend_redispatch_warnings_total
• haproxy_backend_requests_denied_total
• haproxy_backend_response_errors_total
• haproxy_backend_response_time_average_seconds
• haproxy_backend_responses_denied_total
• haproxy_backend_retry_warnings_total
• haproxy_backend_server_aborts_total
• haproxy_backend_sessions_total
• haproxy_backend_status
• haproxy_backend_total_time_average_seconds
• haproxy_backend_uweight
• haproxy_backend_weight

Server metrics Jump to heading #

• haproxy_server_bytes_in_total
• haproxy_server_bytes_out_total
• haproxy_server_check_code
• haproxy_server_check_duration_seconds
• haproxy_server_check_failures_total
• haproxy_server_check_last_change_seconds
• haproxy_server_check_status
• haproxy_server_check_up_down_total
• haproxy_server_client_aborts_total
• haproxy_server_connect_time_average_seconds
• haproxy_server_connection_attempts_total
• haproxy_server_connection_errors_total
• haproxy_server_connection_reuses_total
• haproxy_server_current_queue
• haproxy_server_current_sessions
• haproxy_server_current_throttle
• haproxy_server_downtime_seconds_total
• haproxy_server_failed_header_rewriting_total
• haproxy_server_http_responses_total
• haproxy_server_idle_connections_current
• haproxy_server_idle_connections_limit
• haproxy_server_internal_errors_total
• haproxy_server_last_session_seconds
• haproxy_server_limit_sessions
• haproxy_server_loadbalanced_total
• haproxy_server_max_connect_time_seconds
• haproxy_server_max_queue
• haproxy_server_max_queue_time_seconds
• haproxy_server_max_response_time_seconds
• haproxy_server_max_session_rate
• haproxy_server_max_sessions
• haproxy_server_max_total_time_seconds
• haproxy_server_need_connections_current
• haproxy_server_queue_limit
• haproxy_server_queue_time_average_seconds
• haproxy_server_redispatch_warnings_total
• haproxy_server_response_errors_total
• haproxy_server_response_time_average_seconds
• haproxy_server_responses_denied_total
• haproxy_server_retry_warnings_total
• haproxy_server_safe_idle_connections_current
• haproxy_server_server_aborts_total
• haproxy_server_sessions_total
• haproxy_server_status
• haproxy_server_total_time_average_seconds
• haproxy_server_unsafe_idle_connections_current
• haproxy_server_used_connections_current
• haproxy_server_uweight
• haproxy_server_weight

Stick table metrics Jump to heading #

• haproxy_sticktable_size
• haproxy_sticktable_used