Alerts and monitoring

Prometheus metrics

On this page

Available since

  • HAProxy 2.0
  • HAProxy Enterprise 2.0r1
  • HAProxy ALOHA 11.5

The load balancer exposes a Prometheus endpoint that publishes metrics that you can scrape with a Prometheus-compatible agent such as the Prometheus server, Fluentd, Telegraf, and Metricbeat. It is enabled by default, so you only need to configure the IP address and port where it listens.

Configure the Prometheus exporter Jump to heading

To configure the Prometheus exporter:

  1. Add a new frontend section to your configuration file. In it, include the http-request use-service prometheus-exporter directive. In the following example, the Prometheus exporter page becomes available on all IP addresses at port 8405:

    haproxy
    frontend prometheus
      bind :8405
      mode http
      http-request use-service prometheus-exporter
      no log
    haproxy
    frontend prometheus
      bind :8405
      mode http
      http-request use-service prometheus-exporter
      no log

    The no log directive disables logging for these requests.

  2. Optional: Add a conditional statement to the end of the line to show the metrics page only on a certain URL path. In the next example, the metrics become available at the URL path /metrics on port 8405:

    haproxy
    frontend prometheus
      bind *:8405
      mode http
      http-request use-service prometheus-exporter if { path /metrics }
      no log
    haproxy
    frontend prometheus
      bind *:8405
      mode http
      http-request use-service prometheus-exporter if { path /metrics }
      no log

Configure the Prometheus server Jump to heading

The Prometheus server collects your load balancer metrics and stores them in its database. From there, you can use software like Grafana to graph them. To configure your Prometheus server to scrape the metrics page:

  1. Follow the First Steps with Prometheus guide to install the Prometheus server.

  2. Edit its prometheus.yml file so that it includes a job for scraping load balancer metrics. Add the following to the scrape_configs section, replacing localhost with your load balancer’s address. If you run multiple load balancer servers, include them in the targets array.

    prometheus.yml
    yaml
    - job_name: 'load-balancer-metrics'
      static_configs:
        - targets: ['localhost:8405']
    prometheus.yml
    yaml
    - job_name: 'load-balancer-metrics'
      static_configs:
        - targets: ['localhost:8405']

    By default, the Prometheus server scrapes the URL /metrics. To change this path, set the metrics_path parameter in the scrape_configs section of the Prometheus configuration file.

Host over HTTPS Jump to heading

To serve the Prometheus endpoint over HTTPS:

  1. Edit the load balancer configuration and add the ssl parameter to the bind line to enable HTTPS. Also, set the crt parameter to the path where you’ve stored your TLS key and certificate file:

    haproxy
    frontend prometheus
      bind :8405 ssl crt /certs/site.pem
      mode http
      http-request use-service prometheus-exporter
      no log
    haproxy
    frontend prometheus
      bind :8405 ssl crt /certs/site.pem
      mode http
      http-request use-service prometheus-exporter
      no log

  2. On the Prometheus server, edit the prometheus.yml configuration file. Set scheme to https:

    prometheus.yml
    yaml
    - job_name: 'load-balancer-metrics'
      scheme: https
      static_configs:
        - targets: ['localhost:8405']
    prometheus.yml
    yaml
    - job_name: 'load-balancer-metrics'
      scheme: https
      static_configs:
        - targets: ['localhost:8405']

  3. Optional: You can also configure the tls_config block to set a CA certificate for verifying the load balancer’s certificate, enable client certificates, set a ServerName extension, or disable validation of the load balancer’s certificate. See the Prometheus configuration documentation for details.

    Below, we skip validation of the certificate.

    prometheus.yml
    yaml
    - job_name: 'load-balancer-metrics'
      scheme: https
      static_configs:
        - targets: ['localhost:8405']
      tls_config:
        # Disable validation of server certificate
        insecure_skip_verify: true
    prometheus.yml
    yaml
    - job_name: 'load-balancer-metrics'
      scheme: https
      static_configs:
        - targets: ['localhost:8405']
      tls_config:
        # Disable validation of server certificate
        insecure_skip_verify: true

Filter metrics Jump to heading

To reduce the volume of metrics Prometheus scrapes from the load balancer, you can specify filter parameters in the Prometheus configuration file. The filters are applied on the load balancer, thereby reducing the load on the network and the Prometheus server.

In the following section, we describe several use cases.

Specify metrics scope Jump to heading

Use the scope parameter to specify which categories of metrics to scrape. Categories include:

  • global
  • frontend
  • listener
  • backend
  • server
  • sticktable
  • asterisk (*) = all scopes
  • empty = no metrics

In the load balancer job shown below, we are scraping only frontend and sticktable metrics:

prometheus.yml
yaml
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
scrape_configs:
  - job_name: 'load-balancer-metrics'
    static_configs:
      - targets: ['192.168.56.31:8405']
    params: 
      scope: 
        - frontend
        - sticktable
prometheus.yml
yaml
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
scrape_configs:
  - job_name: 'load-balancer-metrics'
    static_configs:
      - targets: ['192.168.56.31:8405']
    params: 
      scope: 
        - frontend
        - sticktable

Omit servers in maintenance mode Jump to heading

To reduce the volume of metrics returned, omit results from servers in maintenance mode. Add the no-maint key. The no-maint key is particularly useful with load balancer configurations using server-template to define servers dynamically, wherein unused server slots are left in maintenance mode.

There is no consistency check on server state. So if the state of a server changes while the exporter is running, only a portion of the metrics for this server will be reported.

Below, we define no-maint to omit metrics from servers in maintenance mode:

prometheus.yml
yaml
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
scrape_configs:
  - job_name: 'load-balancer-metrics'
    static_configs:
      - targets: ['192.168.56.31:8405']
    params: 
      scope: 
        - frontend
        - sticktable
     no-maint:
       - empty
prometheus.yml
yaml
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
scrape_configs:
  - job_name: 'load-balancer-metrics'
    static_configs:
      - targets: ['192.168.56.31:8405']
    params: 
      scope: 
        - frontend
        - sticktable
     no-maint:
       - empty

Scrape health status only Jump to heading

To scrape only server health status without other server metrics, configure the job with a metric_relabel_configs section, as follows:

prometheus.yml
yaml
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: 'load-balancer-metrics'
    static_configs:
      - targets: ['192.168.56.31:8405']
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'haproxy_(process_|frontend_|listener_|backend_|server_check_status).*'
        action: keep
prometheus.yml
yaml
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: 'load-balancer-metrics'
    static_configs:
      - targets: ['192.168.56.31:8405']
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'haproxy_(process_|frontend_|listener_|backend_|server_check_status).*'
        action: keep

Exported metrics Jump to heading

The Prometheus exporter generates the following metrics.

Global metrics Jump to heading

  • haproxy_process_active_peers
  • haproxy_process_build_info
  • haproxy_process_busy_polling_enabled
  • haproxy_process_bytes_out_rate
  • haproxy_process_bytes_out_total
  • haproxy_process_connected_peers
  • haproxy_process_connections_total
  • haproxy_process_current_backend_ssl_key_rate
  • haproxy_process_current_connection_rate
  • haproxy_process_current_connections
  • haproxy_process_current_frontend_ssl_key_rate
  • haproxy_process_current_run_queue
  • haproxy_process_current_session_rate
  • haproxy_process_current_ssl_connections
  • haproxy_process_current_ssl_rate
  • haproxy_process_current_tasks
  • haproxy_process_current_zlib_memory
  • haproxy_process_dropped_logs_total
  • haproxy_process_failed_resolutions
  • haproxy_process_frontend_ssl_reuse
  • haproxy_process_hard_max_connections
  • haproxy_process_http_comp_bytes_in_total
  • haproxy_process_http_comp_bytes_out_total
  • haproxy_process_idle_time_percent
  • haproxy_process_jobs
  • haproxy_process_limit_connection_rate
  • haproxy_process_limit_http_comp
  • haproxy_process_limit_session_rate
  • haproxy_process_limit_ssl_rate
  • haproxy_process_listeners
  • haproxy_process_max_backend_ssl_key_rate
  • haproxy_process_max_connection_rate
  • haproxy_process_max_connections
  • haproxy_process_max_fds
  • haproxy_process_max_frontend_ssl_key_rate
  • haproxy_process_max_memory_bytes
  • haproxy_process_max_pipes
  • haproxy_process_max_session_rate
  • haproxy_process_max_sockets
  • haproxy_process_max_ssl_connections
  • haproxy_process_max_ssl_rate
  • haproxy_process_max_zlib_memory
  • haproxy_process_nbproc
  • haproxy_process_nbthread
  • haproxy_process_pipes_free_total
  • haproxy_process_pipes_used_total
  • haproxy_process_pool_allocated_bytes
  • haproxy_process_pool_failures_total
  • haproxy_process_pool_used_bytes
  • haproxy_process_recv_logs_total
  • haproxy_process_relative_process_id
  • haproxy_process_requests_total
  • haproxy_process_spliced_bytes_out_total
  • haproxy_process_ssl_cache_lookups_total
  • haproxy_process_ssl_cache_misses_total
  • haproxy_process_ssl_connections_total
  • haproxy_process_start_time_seconds
  • haproxy_process_stopping
  • haproxy_process_unstoppable_jobs
  • haproxy_process_uptime_seconds

Frontend metrics Jump to heading

  • haproxy_frontend_bytes_in_total
  • haproxy_frontend_bytes_out_total
  • haproxy_frontend_connections_rate_max
  • haproxy_frontend_connections_total
  • haproxy_frontend_current_sessions
  • haproxy_frontend_denied_connections_total
  • haproxy_frontend_denied_sessions_total
  • haproxy_frontend_failed_header_rewriting_total
  • haproxy_frontend_http_cache_hits_total
  • haproxy_frontend_http_cache_lookups_total
  • haproxy_frontend_http_comp_bytes_bypassed_total
  • haproxy_frontend_http_comp_bytes_in_total
  • haproxy_frontend_http_comp_bytes_out_total
  • haproxy_frontend_http_comp_responses_total
  • haproxy_frontend_http_requests_rate_max
  • haproxy_frontend_http_requests_total
  • haproxy_frontend_http_responses_total
  • haproxy_frontend_intercepted_requests_total
  • haproxy_frontend_internal_errors_total
  • haproxy_frontend_limit_session_rate
  • haproxy_frontend_limit_sessions
  • haproxy_frontend_max_session_rate
  • haproxy_frontend_max_sessions
  • haproxy_frontend_request_errors_total
  • haproxy_frontend_requests_denied_total
  • haproxy_frontend_responses_denied_total
  • haproxy_frontend_sessions_total
  • haproxy_frontend_status

Listener metrics Jump to heading

  • haproxy_listener_bytes_in_total
  • haproxy_listener_bytes_out_total
  • haproxy_listener_current_sessions
  • haproxy_listener_denied_connections_total
  • haproxy_listener_denied_sessions_total
  • haproxy_listener_failed_header_rewriting_total
  • haproxy_listener_internal_errors_total
  • haproxy_listener_limit_sessions
  • haproxy_listener_max_sessions
  • haproxy_listener_request_errors_total
  • haproxy_listener_requests_denied_total
  • haproxy_listener_responses_denied_total
  • haproxy_listener_sessions_total
  • haproxy_listener_status

Backend metrics Jump to heading

  • haproxy_backend_active_servers
  • haproxy_backend_agg_check_status
  • haproxy_backend_agg_server_status
  • haproxy_backend_backup_servers
  • haproxy_backend_bytes_in_total
  • haproxy_backend_bytes_out_total
  • haproxy_backend_check_last_change_seconds
  • haproxy_backend_check_up_down_total
  • haproxy_backend_client_aborts_total
  • haproxy_backend_connect_time_average_seconds
  • haproxy_backend_connection_attempts_total
  • haproxy_backend_connection_errors_total
  • haproxy_backend_connection_reuses_total
  • haproxy_backend_current_queue
  • haproxy_backend_current_sessions
  • haproxy_backend_downtime_seconds_total
  • haproxy_backend_failed_header_rewriting_total
  • haproxy_backend_http_cache_hits_total
  • haproxy_backend_http_cache_lookups_total
  • haproxy_backend_http_comp_bytes_bypassed_total
  • haproxy_backend_http_comp_bytes_in_total
  • haproxy_backend_http_comp_bytes_out_total
  • haproxy_backend_http_comp_responses_total
  • haproxy_backend_http_requests_total
  • haproxy_backend_http_responses_total
  • haproxy_backend_internal_errors_total
  • haproxy_backend_last_session_seconds
  • haproxy_backend_limit_sessions
  • haproxy_backend_loadbalanced_total
  • haproxy_backend_max_connect_time_seconds
  • haproxy_backend_max_queue
  • haproxy_backend_max_queue_time_seconds
  • haproxy_backend_max_response_time_seconds
  • haproxy_backend_max_session_rate
  • haproxy_backend_max_sessions
  • haproxy_backend_max_total_time_seconds
  • haproxy_backend_queue_time_average_seconds
  • haproxy_backend_redispatch_warnings_total
  • haproxy_backend_requests_denied_total
  • haproxy_backend_response_errors_total
  • haproxy_backend_response_time_average_seconds
  • haproxy_backend_responses_denied_total
  • haproxy_backend_retry_warnings_total
  • haproxy_backend_server_aborts_total
  • haproxy_backend_sessions_total
  • haproxy_backend_status
  • haproxy_backend_total_time_average_seconds
  • haproxy_backend_uweight
  • haproxy_backend_weight

Server metrics Jump to heading

  • haproxy_server_bytes_in_total
  • haproxy_server_bytes_out_total
  • haproxy_server_check_code
  • haproxy_server_check_duration_seconds
  • haproxy_server_check_failures_total
  • haproxy_server_check_last_change_seconds
  • haproxy_server_check_status
  • haproxy_server_check_up_down_total
  • haproxy_server_client_aborts_total
  • haproxy_server_connect_time_average_seconds
  • haproxy_server_connection_attempts_total
  • haproxy_server_connection_errors_total
  • haproxy_server_connection_reuses_total
  • haproxy_server_current_queue
  • haproxy_server_current_sessions
  • haproxy_server_current_throttle
  • haproxy_server_downtime_seconds_total
  • haproxy_server_failed_header_rewriting_total
  • haproxy_server_http_responses_total
  • haproxy_server_idle_connections_current
  • haproxy_server_idle_connections_limit
  • haproxy_server_internal_errors_total
  • haproxy_server_last_session_seconds
  • haproxy_server_limit_sessions
  • haproxy_server_loadbalanced_total
  • haproxy_server_max_connect_time_seconds
  • haproxy_server_max_queue
  • haproxy_server_max_queue_time_seconds
  • haproxy_server_max_response_time_seconds
  • haproxy_server_max_session_rate
  • haproxy_server_max_sessions
  • haproxy_server_max_total_time_seconds
  • haproxy_server_need_connections_current
  • haproxy_server_queue_limit
  • haproxy_server_queue_time_average_seconds
  • haproxy_server_redispatch_warnings_total
  • haproxy_server_response_errors_total
  • haproxy_server_response_time_average_seconds
  • haproxy_server_responses_denied_total
  • haproxy_server_retry_warnings_total
  • haproxy_server_safe_idle_connections_current
  • haproxy_server_server_aborts_total
  • haproxy_server_sessions_total
  • haproxy_server_status
  • haproxy_server_total_time_average_seconds
  • haproxy_server_unsafe_idle_connections_current
  • haproxy_server_used_connections_current
  • haproxy_server_uweight
  • haproxy_server_weight

Stick table metrics Jump to heading

  • haproxy_sticktable_size
  • haproxy_sticktable_used

See also Jump to heading

Do you have any suggestions on how we can improve the content of this page?

Previous page Monitor URI Next page AuthN / authZ
© 2024 HAProxy Technologies, LLC. All Rights Reserved
Manage Cookie Preferences