Controller metrics

HAProxy Unified Gateway serves Prometheus metrics to aid in monitoring the health, performance, and operations of your services. This guide will focus on HAProxy Unified Gateway controller metrics.

Looking for load balancer metrics?

Go to the guide on using HAProxy stats metrics.

View controller metrics Jump to heading

Controller metrics are served by HAProxy Unified Gateway’s controller-runtime metrics server on the controller port, by default on port 31060 using the /metrics path. If your use case necessitates configuring to a different port, refer to the HAProxy Unified Gateway installation guide on how to change the port number.

Send a request to the controller metrics endpoint. For example, using curl:

nix
curl http://<host>:31060/metrics

nix
curl http://<host>:31060/metrics

output
nix
# HELP certwatcher_read_certificate_errors_total Total number of certificate read errors
# TYPE certwatcher_read_certificate_errors_total counter
certwatcher_read_certificate_errors_total 0
# HELP certwatcher_read_certificate_total Total number of certificate reads
# TYPE certwatcher_read_certificate_total counter
certwatcher_read_certificate_total 0
# HELP controller_runtime_active_workers Number of currently used workers per controller
# TYPE controller_runtime_active_workers gauge
controller_runtime_active_workers{controller="BackendCR"} 0
controller_runtime_active_workers{controller="ConfigMap"} 0
controller_runtime_active_workers{controller="DefaultsCR"} 0
...

output
nix
# HELP certwatcher_read_certificate_errors_total Total number of certificate read errors
# TYPE certwatcher_read_certificate_errors_total counter
certwatcher_read_certificate_errors_total 0
# HELP certwatcher_read_certificate_total Total number of certificate reads
# TYPE certwatcher_read_certificate_total counter
certwatcher_read_certificate_total 0
# HELP controller_runtime_active_workers Number of currently used workers per controller
# TYPE controller_runtime_active_workers gauge
controller_runtime_active_workers{controller="BackendCR"} 0
controller_runtime_active_workers{controller="ConfigMap"} 0
controller_runtime_active_workers{controller="DefaultsCR"} 0
...

Tip

When accessing the controller metrics endpoint from a node inside your Kubernetes cluster, you can set <host> to localhost. For example: curl http://localhost:31060/metrics.

Authentication modes Jump to heading

There are three authentication modes: none, kube-rbac, and basic.

Enable none authentication mode Jump to heading

The none authentication mode is the default if deployed via controller.yaml manifest file (the default is kube-rbac if deployed via Helm), and metrics are served over plain HTTP with no authentication. Once enabled, the controller metrics are available at the endpoint http://<host>:31060/metrics.

Use cases include:

Development and testing environments.
Environments where the metrics port isn’t exposed outside the cluster.
Trusted network environments with network policies restricting access.

Example network policy

The controller metrics endpoint is accessible to anyone who can reach its port. If needed for your use case, consider applying a Kubernetes NetworkPolicy to restrict access. For example, here is a NetworkPolicy that limits access from only specific pods within the cluster:

nix
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-metrics-from-prometheus
  namespace: haproxy-unified-gateway
spec:
  podSelector:
    matchLabels:
      run: haproxy-unified-gateway
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
      ports:
        - port: 31060
          protocol: TCP

nix
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-metrics-from-prometheus
  namespace: haproxy-unified-gateway
spec:
  podSelector:
    matchLabels:
      run: haproxy-unified-gateway
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
      ports:
        - port: 31060
          protocol: TCP

This NetworkPolicy is applied to the example haproxy-unified-gateway namespace, where pods in that namespace with the label run: haproxy-unified-gateway allow ingress only from pods in another namespace labeled monitoring, only on TCP port 31060.

To enable the none authentication mode, confirm the --metrics-auth=none argument is enabled in the controller configuration. For example:

nix
kubectl edit deployment -n haproxy-unified-gateway haproxy-unified-gateway

nix
kubectl edit deployment -n haproxy-unified-gateway haproxy-unified-gateway

controller.yaml
nix
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: haproxy-unified-gateway
  name: haproxy-unified-gateway
  namespace: haproxy-unified-gateway
spec:
  ...
  template:
    metadata:
      labels:
        run: haproxy-unified-gateway
    spec:
      serviceAccountName: haproxy-unified-gateway
      containers:
        - name: haproxy-unified-gateway
          ...
          args:
            ...
            - --metrics-auth=none

controller.yaml
nix
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: haproxy-unified-gateway
  name: haproxy-unified-gateway
  namespace: haproxy-unified-gateway
spec:
  ...
  template:
    metadata:
      labels:
        run: haproxy-unified-gateway
    spec:
      serviceAccountName: haproxy-unified-gateway
      containers:
        - name: haproxy-unified-gateway
          ...
          args:
            ...
            - --metrics-auth=none

If you installed HAProxy Unified Gateway without using Helm, edit your controller.yaml file with the configuration above and apply it:

nix
kubectl apply -f controller.yaml

nix
kubectl apply -f controller.yaml

Enable kube-rbac authentication mode Jump to heading

The kube-rbac authentication mode is the default if deployed via Helm, and it secures the controller metrics endpoint using Kubernetes’ RBAC (role-based access control) authorization over HTTPS. The controller validates requests with Kubernetes TokenReview and SubjectAccessReview API calls, ensuring only ServiceAccounts with the correct RBAC permissions can access the controller metrics endpoint.

Use cases include:

Production environments.
In-cluster Prometheus with a dedicated ServiceAccount.
Environments requiring an audit trail for the controller metrics endpoint access.

To enable the kube-rbac authentication mode:

Configure the controller to use kube-rbac authentication mode by enabling its argument.

nix
kubectl edit deployment -n haproxy-unified-gateway haproxy-unified-gateway

nix
kubectl edit deployment -n haproxy-unified-gateway haproxy-unified-gateway

controller.yaml
nix
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: haproxy-unified-gateway
  name: haproxy-unified-gateway
  namespace: haproxy-unified-gateway
spec:
  ...
  template:
    metadata:
      labels:
        run: haproxy-unified-gateway
    spec:
      serviceAccountName: haproxy-unified-gateway
      containers:
        - name: haproxy-unified-gateway
          ...
          args:
            ...
            - --metrics-auth=kube-rbac

controller.yaml
nix
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: haproxy-unified-gateway
  name: haproxy-unified-gateway
  namespace: haproxy-unified-gateway
spec:
  ...
  template:
    metadata:
      labels:
        run: haproxy-unified-gateway
    spec:
      serviceAccountName: haproxy-unified-gateway
      containers:
        - name: haproxy-unified-gateway
          ...
          args:
            ...
            - --metrics-auth=kube-rbac

If you installed HAProxy Unified Gateway without using Helm, edit your controller.yaml file with the configuration above and apply it:

nix
kubectl apply -f controller.yaml

nix
kubectl apply -f controller.yaml

Prometheus requires a ClusterRole that grants access to the controller metrics endpoint’s nonResourceURL and a ClusterRoleBinding to its ServiceAccount. This configuration is found in rbac.yaml (or if you deployed with Helm, check this configuration with kubectl edit clusterrolebinding haproxy-unified-gateway-metrics-reader):

nix
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: haproxy-unified-gateway-metrics-reader
rules:
  - nonResourceURLs:
      - "/metrics"
    verbs:
      - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: haproxy-unified-gateway-metrics-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: haproxy-unified-gateway-metrics-reader
subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: monitoring

nix
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: haproxy-unified-gateway-metrics-reader
rules:
  - nonResourceURLs:
      - "/metrics"
    verbs:
      - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: haproxy-unified-gateway-metrics-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: haproxy-unified-gateway-metrics-reader
subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: monitoring

Optional: Edit the subjects section to match your Prometheus ServiceAccount name and namespace. You can use the configuration above as a template or change it as necessary to meet your required security policies. For example, with kube-prometheus-stack, the ServiceAccount is typically named prometheus-kube-prometheus-prometheus in the monitoring namespace. Apply the configuration change with kubectl -f rbac.yaml.

If you only have namespace-level access, you will need a cluster admin to create the ClusterRole and ClusterRoleBinding resources. Otherwise, consider using basic authentication mode because it’s configured entirely within the namespace scope.

Verify kube-rbac authentication

Create an ephemeral ServiceAccount bearer token for Prometheus with the kubectl create token command.
```
nix
TOKEN=$(kubectl create token SERVICE_ACCOUNT_NAME -n NAMESPACE)
```
```
nix
TOKEN=$(kubectl create token SERVICE_ACCOUNT_NAME -n NAMESPACE)
```
Where:
- The TOKEN variable can be changed to a more descriptive name.
- Replace SERVICE_ACCOUNT_NAME with your already existing ServiceAccount name.
- Replace NAMESPACE with the namespace where the ServiceAccount exists.
For example:
```
nix
TOKEN=$(kubectl create token prometheus -n monitoring)
```
```
nix
TOKEN=$(kubectl create token prometheus -n monitoring)
```

Verify by providing the bearer token to access the controller metrics endpoint on any node.

nix
curl -k -H "Authorization: Bearer $TOKEN" https://<host>:31060/metrics

nix
curl -k -H "Authorization: Bearer $TOKEN" https://<host>:31060/metrics

Where:

The -k option makes curl skip the verification step, which is used here because we’re using a self-signed certificate.
The -H option specifies an extra header, the bearer token for authorization.
Note the use of the https protocol.

For example:

nix
curl -k -H "Authorization: Bearer $TOKEN" https://<host>:31060/metrics

nix
curl -k -H "Authorization: Bearer $TOKEN" https://<host>:31060/metrics

output
nix
# HELP certwatcher_read_certificate_errors_total Total number of certificate read errors
# TYPE certwatcher_read_certificate_errors_total counter
certwatcher_read_certificate_errors_total 0
# HELP certwatcher_read_certificate_total Total number of certificate reads
# TYPE certwatcher_read_certificate_total counter
certwatcher_read_certificate_total 0
# HELP controller_runtime_active_workers Number of currently used workers per controller
# TYPE controller_runtime_active_workers gauge
controller_runtime_active_workers{controller="BackendCR"} 0
...

output
nix
# HELP certwatcher_read_certificate_errors_total Total number of certificate read errors
# TYPE certwatcher_read_certificate_errors_total counter
certwatcher_read_certificate_errors_total 0
# HELP certwatcher_read_certificate_total Total number of certificate reads
# TYPE certwatcher_read_certificate_total counter
certwatcher_read_certificate_total 0
# HELP controller_runtime_active_workers Number of currently used workers per controller
# TYPE controller_runtime_active_workers gauge
controller_runtime_active_workers{controller="BackendCR"} 0
...

Enable basic authentication mode Jump to heading

The basic authentication mode secures the controller metrics endpoint using HTTP Basic Authentication over HTTPS. Credentials can be provided via CLI flags or environment variables.

Use cases include:

A Prometheus deployment is external to HAProxy Unified Gateway, and its Prometheus instances are scraping from outside the Kubernetes cluster.
Deploying a quick, secure setup without a Kubernetes-specific configuration.

To enable basic authentication mode:

Create a Kubernetes generic type secret from literal values by executing the following command. Set these values:

SECRET_NAME is an identifiable name.
NAMESPACE is the namespace where HAProxy Unified Gateway is deployed.
The --from-literal=username= option sets the username for basic authentication.
The --from-literal=password= option sets the password for basic authentication. Always use strong passwords avoid default or example passwords in production environments.

nix
kubectl create secret generic SECRET_NAME \
  -n NAMESPACE \
  --from-literal=username=USERNAME \
  --from-literal=password='PASSWORD'

nix
kubectl create secret generic SECRET_NAME \
  -n NAMESPACE \
  --from-literal=username=USERNAME \
  --from-literal=password='PASSWORD'

An example

nix
kubectl create secret generic hug-metrics-auth \
  -n haproxy-unified-gateway \
  --from-literal=username=prometheus \
  --from-literal=password='changeme'

nix
kubectl create secret generic hug-metrics-auth \
  -n haproxy-unified-gateway \
  --from-literal=username=prometheus \
  --from-literal=password='changeme'

output
nix
secret/hug-metrics-auth created

output
nix
secret/hug-metrics-auth created

Configure the controller to use basic authentication mode and enviornment variables from the secret you created.

nix
kubectl edit deployment -n haproxy-unified-gateway haproxy-unified-gateway

nix
kubectl edit deployment -n haproxy-unified-gateway haproxy-unified-gateway

controller.yaml
nix
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: haproxy-unified-gateway
  name: haproxy-unified-gateway
  namespace: haproxy-unified-gateway
spec:
  ...
  template:
    metadata:
      labels:
        run: haproxy-unified-gateway
    spec:
      serviceAccountName: haproxy-unified-gateway
      containers:
        - name: haproxy-unified-gateway
          ...
          args:
            - --hugconf-crd=haproxy-unified-gateway/hugconf
            - --metrics-auth=basic
          env:
            ...
            - name: METRICS_BASIC_AUTH_USER
              valueFrom:
                secretKeyRef:
                  name: hug-metrics-auth
                  key: username
            - name: METRICS_BASIC_AUTH_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: hug-metrics-auth
                  key: password

controller.yaml
nix
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: haproxy-unified-gateway
  name: haproxy-unified-gateway
  namespace: haproxy-unified-gateway
spec:
  ...
  template:
    metadata:
      labels:
        run: haproxy-unified-gateway
    spec:
      serviceAccountName: haproxy-unified-gateway
      containers:
        - name: haproxy-unified-gateway
          ...
          args:
            - --hugconf-crd=haproxy-unified-gateway/hugconf
            - --metrics-auth=basic
          env:
            ...
            - name: METRICS_BASIC_AUTH_USER
              valueFrom:
                secretKeyRef:
                  name: hug-metrics-auth
                  key: username
            - name: METRICS_BASIC_AUTH_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: hug-metrics-auth
                  key: password

If you installed HAProxy Unified Gateway without using Helm, edit your controller.yaml file with the configuration above and apply it:

nix
kubectl apply -f controller.yaml

nix
kubectl apply -f controller.yaml

Verify by providing the credentials to access the controller metrics endpoint from any node.

nix
curl -k -u USERNAME:PASSWORD https://<host>:31060/metrics

nix
curl -k -u USERNAME:PASSWORD https://<host>:31060/metrics

Where:

The -k option makes curl skip the verification step, which is used here because we’re using a self-signed certificate.
The -u option specifies the user name and password to use for server authentication.
Note the use of the https protocol.

An example

nix
curl -k -u prometheus:changeme https://<host>:31060/metrics

nix
curl -k -u prometheus:changeme https://<host>:31060/metrics

output
nix
# HELP certwatcher_read_certificate_errors_total Total number of certificate read errors
# TYPE certwatcher_read_certificate_errors_total counter
certwatcher_read_certificate_errors_total 0
# HELP certwatcher_read_certificate_total Total number of certificate reads
# TYPE certwatcher_read_certificate_total counter
certwatcher_read_certificate_total 0
# HELP controller_runtime_active_workers Number of currently used workers per controller
# TYPE controller_runtime_active_workers gauge
controller_runtime_active_workers{controller="BackendCR"} 0
...

output
nix
# HELP certwatcher_read_certificate_errors_total Total number of certificate read errors
# TYPE certwatcher_read_certificate_errors_total counter
certwatcher_read_certificate_errors_total 0
# HELP certwatcher_read_certificate_total Total number of certificate reads
# TYPE certwatcher_read_certificate_total counter
certwatcher_read_certificate_total 0
# HELP controller_runtime_active_workers Number of currently used workers per controller
# TYPE controller_runtime_active_workers gauge
controller_runtime_active_workers{controller="BackendCR"} 0
...

Troubleshooting Jump to heading

401 Unauthorized

If you are accessing the controller metrics endpoint with kube-rbac authentication mode:
1. Verify if Prometheus is sending the Authorization: Bearer <token> header.
2. Confirm the target ServiceAccount exists.

If you’re accessing the controller metrics endpoint with basic authentication mode:

Verify that you’re including credentials. For example, curl -k https://<host>:31060/metrics is missing -u USERNAME:PASSWORD.
Verify the spelling of the credentials.

Check if the credentials are loading from the pod environment variables.

nix
kubectl exec -n <NAMESPACE> <POD_NAME> -- env | grep METRICS_BASIC_AUTH_USER

nix
kubectl exec -n <NAMESPACE> <POD_NAME> -- env | grep METRICS_BASIC_AUTH_USER

An example

nix
kubectl exec -n haproxy-unified-gateway haproxy-unified-gateway-59fc455c6d-vmxhm -- env | grep METRICS_BASIC_AUTH_USER METRICS_BASIC_AUTH_USER=prometheus

nix
kubectl exec -n haproxy-unified-gateway haproxy-unified-gateway-59fc455c6d-vmxhm -- env | grep METRICS_BASIC_AUTH_USER METRICS_BASIC_AUTH_USER=prometheus

403 Forbidden

If you’re accessing the controller metrics endpoint with kube-rbac authentication mode:

The Prometheus ServiceAccount is authenticated but not authorized. With the following command, check that the haproxy-unified-gateway-metrics-reader ClusterRole exists and the ClusterRoleBinding references the correct ServiceAccount name and namespace:
```
nix
kubectl get clusterrolebinding haproxy-unified-gateway-metrics-reader -o yaml
```
```
nix
kubectl get clusterrolebinding haproxy-unified-gateway-metrics-reader -o yaml
```

Connection refused

Verify the controller pod is running and the controller metrics port is configured properly.

nix
kubectl get pod -n haproxy-unified-gateway <POD_NAME> -o yaml

nix
kubectl get pod -n haproxy-unified-gateway <POD_NAME> -o yaml

example output
nix
...
ports:
    - containerPort: 31080
      name: http
      protocol: TCP
    - containerPort: 31443
      name: https
      protocol: TCP
    - containerPort: 31060
      name: metrics
      protocol: TCP
    - containerPort: 31024
      name: stat
      protocol: TCP

example output
nix
...
ports:
    - containerPort: 31080
      name: http
      protocol: TCP
    - containerPort: 31443
      name: https
      protocol: TCP
    - containerPort: 31060
      name: metrics
      protocol: TCP
    - containerPort: 31024
      name: stat
      protocol: TCP

Verify the Service’s controller metrics port configuration.

nix
kubectl get service -n haproxy-unified-gateway haproxy-unified-gateway -o yaml

nix
kubectl get service -n haproxy-unified-gateway haproxy-unified-gateway -o yaml

example output
nix
  ...
  ports:
      - name: stat
        nodePort: 32375
        port: 31024
        protocol: TCP
        targetPort: 31024
      - name: metrics
        nodePort: 31060
        port: 31060
        protocol: TCP
        targetPort: 31060

example output
nix
  ...
  ports:
      - name: stat
        nodePort: 32375
        port: 31024
        protocol: TCP
        targetPort: 31024
      - name: metrics
        nodePort: 31060
        port: 31060
        protocol: TCP
        targetPort: 31060

Check pod logs for start up errors.

nix
kubectl logs -n NAMESPACE deploy/haproxy-unified-gateway | grep -i metric

nix
kubectl logs -n NAMESPACE deploy/haproxy-unified-gateway | grep -i metric

Controller metrics reference Jump to heading

The controller metrics shown in the versioned table below labels fields with their names, type of metric, and a description. Controller metrics specific to HAProxy Unified Gateway use the hug_ namespace prefix.

HAProxy Unified Gateway Version

certwatcher_read_certificate_errors_total
counter

Total number of certificate read errors.

certwatcher_read_certificate_total
counter

Total number of certificate reads.

controller_runtime_active_workers
gauge

Number of currently used workers per controller. Label: controller=controller name.

controller_runtime_conversion_webhook_panics_total
counter

Total number of conversion webhook panics.

controller_runtime_max_concurrent_reconciles
gauge

Maximum number of concurrent reconciles per controller. Label: controller=controller name.

controller_runtime_reconcile_errors_total
counter

Total number of reconciliation errors per controller. Label: controller=controller name.

controller_runtime_reconcile_panics_total
counter

Total number of reconciliation panics per controller. Label: controller=controller name.

controller_runtime_reconcile_time_seconds
histogram

Length of time per reconciliation per controller. Time series: _bucket, _sum, _count. Labels: controller=controller name, le=observation buckets of 0.005, 0.01, 0.025, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.25, 1.5, 1.75, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5, 6, 7, 8, 9, 10, 15, 20, 25, 30, 40, 50, 60, +Inf.

controller_runtime_reconcile_timeouts_total
counter

Total number of reconciliation timeouts per controller. Label: controller=controller name.

controller_runtime_reconcile_total
counter

Total number of reconciliations per controller. Labels: controller=controller name, result=reconcile result of success, error, requeue, requeue_after.

controller_runtime_terminal_reconcile_errors_total
counter

Total number of terminal reconciliation errors per controller. Label: controller=controller name.

controller_runtime_webhook_panics_total
counter

Total number of webhook panics.

go_cgo_go_to_c_calls_calls_total
counter

Count of calls made from Go to C by the current process. Sourced from /cgo/go-to-c-calls:calls.

go_cpu_classes_gc_mark_assist_cpu_seconds_total
counter

Estimated total CPU time Go routines spent performing GC tasks to assist the GC and prevent it from falling behind the application. This metric is an overestimate and not directly comparable to system CPU time measurements. Compare only with other /cpu/classes metrics. Sourced from /cpu/classes/gc/mark/assist:cpu-seconds.

go_cpu_classes_gc_mark_dedicated_cpu_seconds_total
counter

Estimated total CPU time spent performing GC tasks on processors (as defined by GOMAXPROCS) dedicated to those tasks. This metric is an overestimate and not directly comparable to system CPU time measurements. Compare only with other /cpu/classes metrics. Sourced from /cpu/classes/gc/mark/dedicated:cpu-seconds.

go_cpu_classes_gc_mark_idle_cpu_seconds_total
counter

Estimated total CPU time spent performing GC tasks on spare CPU resources that the Go scheduler couldn’t otherwise find a use for. This should be subtracted from the total GC CPU time to obtain a measure of compulsory GC CPU time. This metric is an overestimate and not directly comparable to system CPU time measurements. Compare only with other /cpu/classes metrics. Sourced from /cpu/classes/gc/mark/idle:cpu-seconds.

go_cpu_classes_gc_pause_cpu_seconds_total
counter

Estimated total CPU time spent with the application paused by the GC. Even if only one thread is running during the pause, this is computed as GOMAXPROCS multiplied by the pause latency, because nothing else can be executing. This is the exact sum of samples in /sched/pauses/total/gc:seconds if each sample is multiplied by GOMAXPROCS at the time it is taken. This metric is an overestimate and not directly comparable to system CPU time measurements. Compare only with other /cpu/classes metrics. Sourced from /cpu/classes/gc/pause:cpu-seconds.

go_cpu_classes_gc_total_cpu_seconds_total
counter

Estimated total CPU time spent performing GC tasks. This metric is an overestimate and not directly comparable to system CPU time measurements. Compare only with other /cpu/classes metrics. Sum of all metrics in /cpu/classes/gc. Sourced from /cpu/classes/gc/total:cpu-seconds.

go_cpu_classes_idle_cpu_seconds_total
counter

Estimated total available CPU time not spent executing any Go or Go runtime code. In other words, the part of /cpu/classes/total:cpu-seconds that was unused. This metric is an overestimate and not directly comparable to system CPU time measurements. Compare only with other /cpu/classes metrics. Sourced from /cpu/classes/idle:cpu-seconds.

go_cpu_classes_scavenge_assist_cpu_seconds_total
counter

Estimated total CPU time spent returning unused memory to the underlying platform in response eagerly to memory pressure. This metric is an overestimate and not directly comparable to system CPU time measurements. Compare only with other /cpu/classes metrics. Sourced from /cpu/classes/scavenge/assist:cpu-seconds.

go_cpu_classes_scavenge_background_cpu_seconds_total
counter

Estimated total CPU time spent performing background tasks to return unused memory to the underlying platform. This metric is an overestimate and not directly comparable to system CPU time measurements. Compare only with other /cpu/classes metrics. Sourced from /cpu/classes/scavenge/background:cpu-seconds.

go_cpu_classes_scavenge_total_cpu_seconds_total
counter

Estimated total CPU time spent performing tasks that return unused memory to the underlying platform. This metric is an overestimate and not directly comparable to system CPU time measurements. Compare only with other /cpu/classes metrics. Sum of all metrics in /cpu/classes/scavenge. Sourced from /cpu/classes/scavenge/total:cpu-seconds.

go_cpu_classes_total_cpu_seconds_total
counter

Estimated total available CPU time for user Go code or the Go runtime, as defined by GOMAXPROCS. In other words, GOMAXPROCS integrated over the wall-clock duration this process has been executing for. This metric is an overestimate and not directly comparable to system CPU time measurements. Compare only with other /cpu/classes metrics. Sum of all metrics in /cpu/classes. Sourced from /cpu/classes/total:cpu-seconds.

go_cpu_classes_user_cpu_seconds_total
counter

Estimated total CPU time spent running user Go code. This may also include some small amount of time spent in the Go runtime. This metric is an overestimate and not directly comparable to system CPU time measurements. Compare only with other /cpu/classes metrics. Sourced from /cpu/classes/user:cpu-seconds.

go_gc_cleanups_executed_cleanups_total
counter

Approximate total count of cleanup functions (created by runtime.AddCleanup) executed by the runtime. Subtract /gc/cleanups/queued:cleanups to approximate cleanup queue length. Useful for detecting slow cleanups holding up the queue. Sourced from /gc/cleanups/executed:cleanups.

go_gc_cleanups_queued_cleanups_total
counter

Approximate total count of cleanup functions (created by runtime.AddCleanup) queued by the runtime for execution. Subtract from /gc/cleanups/executed:cleanups to approximate cleanup queue length. Useful for detecting slow cleanups holding up the queue. Sourced from /gc/cleanups/queued:cleanups.

go_gc_cycles_automatic_gc_cycles_total
counter

Count of completed GC cycles generated by the Go runtime. Sourced from /gc/cycles/automatic:gc-cycles.

go_gc_cycles_forced_gc_cycles_total
counter

Count of completed GC cycles forced by the application. Sourced from /gc/cycles/forced:gc-cycles.

go_gc_cycles_total_gc_cycles_total
counter

Count of all completed GC cycles. Sourced from /gc/cycles/total:gc-cycles.

go_gc_duration_seconds
summary

A summary of the wall-time pause (stop-the-world) duration in garbage collection cycles. Time series: _sum, _count, and quantile=observation events of 0, 0.25, 0.5, 0.75, 1

go_gc_finalizers_executed_finalizers_total
counter

Total count of finalizer functions (created by runtime.SetFinalizer) executed by the runtime. Subtract /gc/finalizers/queued:finalizers to approximate finalizer queue length. Useful for detecting finalizers overwhelming the queue, either by being too slow, or by there being too many of them. Sourced from /gc/finalizers/executed:finalizers.

go_gc_finalizers_queued_finalizers_total
counter

Total count of finalizer functions (created by runtime.SetFinalizer) and queued by the runtime for execution. Subtract from /gc/finalizers/executed:finalizers to approximate finalizer queue length. Useful for detecting slow finalizers holding up the queue. Sourced from /gc/finalizers/queued:finalizers.

go_gc_gogc_percent
gauge

Heap size target percentage configured by the user, otherwise 100. This value is set by the GOGC environment variable, and the runtime/debug.SetGCPercent function. Sourced from /gc/gogc:percent.

go_gc_gomemlimit_bytes
gauge

Go runtime memory limit configured by the user, otherwise math.MaxInt64. This value is set by the GOMEMLIMIT environment variable, and the runtime/debug.SetMemoryLimit function. Sourced from /gc/gomemlimit:bytes.

go_gc_heap_allocs_by_size_bytes
histogram

Distribution of heap allocations by approximate size. Bucket counts increase monotonically. This doesn’t include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks. Sourced from /gc/heap/allocs-by-size:bytes. Time series: _bucket, _sum, _count. Labels: le= observation buckets of 8.999999999999998, 24.999999999999996, 64.99999999999999, 144.99999999999997, 320.99999999999994, 704.9999999999999, 1536.9999999999998, 3200.9999999999995, 6528.999999999999, 13568.999999999998 27264.999999999996, +Inf.

go_gc_heap_allocs_bytes_total
counter

Cumulative sum of memory allocated to the heap by the application. Sourced from /gc/heap/allocs:bytes.

go_gc_heap_allocs_objects_total
counter

Cumulative count of heap allocations triggered by the application; this doesn’t include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks. Sourced from /gc/heap/allocs:objects.

go_gc_heap_frees_by_size_bytes
histogram

Distribution of freed heap allocations by approximate size. Bucket counts increase monotonically. This doesn’t include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks. Sourced from /gc/heap/frees-by-size:bytes. Time series: _bucket, _sum, _count. Labels: le=observation buckets of 8.999999999999998, 24.999999999999996, 64.99999999999999, 144.99999999999997, 320.99999999999994, 704.9999999999999, 1536.9999999999998, 3200.9999999999995, 6528.999999999999, 13568.999999999998 27264.999999999996, +Inf.

go_gc_heap_frees_bytes_total
counter

Cumulative sum of heap memory freed by the garbage collector. Sourced from /gc/heap/frees:bytes.

go_gc_heap_frees_objects_total
counter

Cumulative count of heap allocations whose storage was freed by the garbage collector. This doesn’t include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks. Sourced from /gc/heap/frees:objects.

go_gc_heap_goal_bytes
gauge

Heap size target for the end of the GC cycle. Sourced from /gc/heap/goal:bytes.

go_gc_heap_live_bytes
gauge

Heap memory occupied by live objects that were marked by the previous GC. Sourced from /gc/heap/live:bytes.

go_gc_heap_objects_objects
gauge

Number of objects, live or unswept, occupying heap memory. Sourced from /gc/heap/objects:objects.

go_gc_heap_tiny_allocs_objects_total
counter

Count of small allocations that are packed together into blocks. These allocations are counted separately from other allocations because each individual allocation isn’t tracked by the runtime, only their block. Each block is already accounted for in allocs-by-size and frees-by-size. Sourced from /gc/heap/tiny/allocs:objects.

go_gc_limiter_last_enabled_gc_cycle
gauge

GC cycle the last time the GC CPU limiter was enabled. This metric is useful for diagnosing the root cause of an out-of-memory error, because the limiter trades memory for CPU time when the GC’s CPU time gets too high. This is most likely to occur with use of SetMemoryLimit. The first GC cycle is cycle 1, so a value of 0 indicates that it was never enabled. Sourced from /gc/limiter/last-enabled:gc-cycle.

go_gc_pauses_seconds
histogram

Deprecated. Prefer the identical /sched/pauses/total/gc:seconds. Sourced from /gc/pauses:seconds. Time series: _bucket, _sum, _count. Labels: le= observation buckets of 6.399999999999999e-08, 7.167999999999999e-06, 8.191999999999999e-05, 0.0009175039999999999, 0.010485759999999998, 0.11744051199999998, +Inf.

go_gc_scan_globals_bytes
gauge

The total amount of global variable space that is scannable. Sourced from /gc/scan/globals:bytes.

go_gc_scan_heap_bytes
gauge

The total amount of heap space that is scannable. Sourced from /gc/scan/heap:bytes.

go_gc_scan_stack_bytes
gauge

The number of bytes of stack that were scanned last GC cycle. Sourced from /gc/scan/stack:bytes.

go_gc_scan_total_bytes
gauge

The total amount of space that is scannable. Sum of all metrics in /gc/scan. Sourced from /gc/scan/total:bytes.

go_gc_stack_starting_size_bytes
gauge

The stack size of new goroutines. Sourced from /gc/stack/starting-size:bytes.

go_godebug_non_default_behavior_allowmultiplevcs_events_total
counter

The number of non-default behaviors executed by the cmd/go package due to a non-default GODEBUG=allowmultiplevcs=... setting. Sourced from /godebug/non-default-behavior/allowmultiplevcs:events.

go_godebug_non_default_behavior_asynctimerchan_events_total
counter

The number of non-default behaviors executed by the time package due to a non-default GODEBUG=asynctimerchan=... setting. Sourced from /godebug/non-default-behavior/asynctimerchan:events.

go_godebug_non_default_behavior_containermaxprocs_events_total
counter

The number of non-default behaviors executed by the runtime package due to a non-default GODEBUG=containermaxprocs=... setting. Sourced from /godebug/non-default-behavior/containermaxprocs:events.

go_godebug_non_default_behavior_cryptocustomrand_events_total
counter

The number of non-default behaviors executed by the crypto package due to a non-default GODEBUG=cryptocustomrand=... setting. Sourced from /godebug/non-default-behavior/cryptocustomrand:events.

go_godebug_non_default_behavior_embedfollowsymlinks_events_total
counter

The number of non-default behaviors executed by the cmd/go package due to a non-default GODEBUG=embedfollowsymlinks=... setting. Sourced from /godebug/non-default-behavior/embedfollowsymlinks:events.

go_godebug_non_default_behavior_execerrdot_events_total
counter

The number of non-default behaviors executed by the os/exec package due to a non-default GODEBUG=execerrdot=... setting. Sourced from /godebug/non-default-behavior/execerrdot:events.

go_godebug_non_default_behavior_gocachehash_events_total
counter

The number of non-default behaviors executed by the cmd/go package due to a non-default GODEBUG=gocachehash=... setting. Sourced from /godebug/non-default-behavior/gocachehash:events.

go_godebug_non_default_behavior_gocachetest_events_total
counter

The number of non-default behaviors executed by the cmd/go package due to a non-default GODEBUG=gocachetest=... setting. Sourced from /godebug/non-default-behavior/gocachetest:events.

go_godebug_non_default_behavior_gocacheverify_events_total
counter

The number of non-default behaviors executed by the cmd/go package due to a non-default GODEBUG=gocacheverify=... setting. Sourced from /godebug/non-default-behavior/gocacheverify:events.

go_godebug_non_default_behavior_gotestjsonbuildtext_events_total
counter

The number of non-default behaviors executed by the cmd/go package due to a non-default GODEBUG=gotestjsonbuildtext=... setting. Sourced from /godebug/non-default-behavior/gotestjsonbuildtext:events.

go_godebug_non_default_behavior_gotypesalias_events_total
counter

The number of non-default behaviors executed by the go/types package due to a non-default GODEBUG=gotypesalias=... setting. Sourced from /godebug/non-default-behavior/gotypesalias:events.

go_godebug_non_default_behavior_htmlmetacontenturlescape_events_total
counter

The number of non-default behaviors executed by the html/template package due to a non-default GODEBUG=htmlmetacontenturlescape=... setting. Sourced from /godebug/non-default-behavior/htmlmetacontenturlescape:events.

go_godebug_non_default_behavior_http2client_events_total
counter

The number of non-default behaviors executed by the net/http package due to a non-default GODEBUG=http2client=... setting. Sourced from /godebug/non-default-behavior/http2client:events.

go_godebug_non_default_behavior_http2server_events_total
counter

The number of non-default behaviors executed by the net/http package due to a non-default GODEBUG=http2server=... setting. Sourced from /godebug/non-default-behavior/http2server:events.

go_godebug_non_default_behavior_httpcookiemaxnum_events_total
counter

The number of non-default behaviors executed by the net/http package due to a non-default GODEBUG=httpcookiemaxnum=... setting. Sourced from /godebug/non-default-behavior/httpcookiemaxnum:events.

go_godebug_non_default_behavior_httplaxcontentlength_events_total
counter

The number of non-default behaviors executed by the net/http package due to a non-default GODEBUG=httplaxcontentlength=... setting. Sourced from /godebug/non-default-behavior/httplaxcontentlength:events.

go_godebug_non_default_behavior_httpmuxgo121_events_total
counter

The number of non-default behaviors executed by the net/http package due to a non-default GODEBUG=httpmuxgo121=... setting. Sourced from /godebug/non-default-behavior/httpmuxgo121:events.

go_godebug_non_default_behavior_httpservecontentkeepheaders_events_total
counter

The number of non-default behaviors executed by the net/http package due to a non-default GODEBUG=httpservecontentkeepheaders=... setting. Sourced from /godebug/non-default-behavior/httpservecontentkeepheaders:events.

go_godebug_non_default_behavior_installgoroot_events_total
counter

The number of non-default behaviors executed by the go/build package due to a non-default GODEBUG=installgoroot=... setting. Sourced from /godebug/non-default-behavior/installgoroot:events.

go_godebug_non_default_behavior_multipartmaxheaders_events_total
counter

The number of non-default behaviors executed by the mime/multipart package due to a non-default GODEBUG=multipartmaxheaders=... setting. Sourced from /godebug/non-default-behavior/multipartmaxheaders:events.

go_godebug_non_default_behavior_multipartmaxparts_events_total
counter

The number of non-default behaviors executed by the mime/multipart package due to a non-default GODEBUG=multipartmaxparts=... setting. Sourced from /godebug/non-default-behavior/multipartmaxparts:events.

go_godebug_non_default_behavior_multipathtcp_events_total
counter

The number of non-default behaviors executed by the net package due to a non-default GODEBUG=multipathtcp=... setting. Sourced from /godebug/non-default-behavior/multipathtcp:events.

go_godebug_non_default_behavior_netedns0_events_total
counter

The number of non-default behaviors executed by the net package due to a non-default GODEBUG=netedns0=... setting. Sourced from /godebug/non-default-behavior/netedns0:events.

go_godebug_non_default_behavior_panicnil_events_total
counter

The number of non-default behaviors executed by the runtime package due to a non-default GODEBUG=panicnil=... setting. Sourced from /godebug/non-default-behavior/panicnil:events.

go_godebug_non_default_behavior_randautoseed_events_total
counter

The number of non-default behaviors executed by the math/rand package due to a non-default GODEBUG=randautoseed=... setting. Sourced from /godebug/non-default-behavior/randautoseed:events.

go_godebug_non_default_behavior_randseednop_events_total
counter

The number of non-default behaviors executed by the math/rand package due to a non-default GODEBUG=randseednop=... setting. Sourced from /godebug/non-default-behavior/randseednop:events.

go_godebug_non_default_behavior_rsa1024min_events_total
counter

The number of non-default behaviors executed by the crypto/rsa package due to a non-default GODEBUG=rsa1024min=... setting. Sourced from /godebug/non-default-behavior/rsa1024min:events.

go_godebug_non_default_behavior_tarinsecurepath_events_total
counter

The number of non-default behaviors executed by the archive/tar package due to a non-default GODEBUG=tarinsecurepath=... setting. Sourced from /godebug/non-default-behavior/tarinsecurepath:events.

go_godebug_non_default_behavior_tls10server_events_total
counter

The number of non-default behaviors executed by the crypto/tls package due to a non-default GODEBUG=tls10server=... setting. Sourced from /godebug/non-default-behavior/tls10server:events.

go_godebug_non_default_behavior_tls3des_events_total
counter

The number of non-default behaviors executed by the crypto/tls package due to a non-default GODEBUG=tls3des=... setting. Sourced from /godebug/non-default-behavior/tls3des:events.

go_godebug_non_default_behavior_tlsmaxrsasize_events_total
counter

The number of non-default behaviors executed by the crypto/tls package due to a non-default GODEBUG=tlsmaxrsasize=... setting. Sourced from /godebug/non-default-behavior/tlsmaxrsasize:events.

go_godebug_non_default_behavior_tlsrsakex_events_total
counter

The number of non-default behaviors executed by the crypto/tls package due to a non-default GODEBUG=tlsrsakex=... setting. Sourced from /godebug/non-default-behavior/tlsrsakex:events.

go_godebug_non_default_behavior_tlssha1_events_total
counter

The number of non-default behaviors executed by the crypto/tls package due to a non-default GODEBUG=tlssha1=... setting. Sourced from /godebug/non-default-behavior/tlssha1:events.

go_godebug_non_default_behavior_tlsunsafeekm_events_total
counter

The number of non-default behaviors executed by the crypto/tls package due to a non-default GODEBUG=tlsunsafeekm=... setting. Sourced from /godebug/non-default-behavior/tlsunsafeekm:events.

go_godebug_non_default_behavior_updatemaxprocs_events_total
counter

The number of non-default behaviors executed by the runtime package due to a non-default GODEBUG=updatemaxprocs=... setting. Sourced from /godebug/non-default-behavior/updatemaxprocs:events.

go_godebug_non_default_behavior_urlmaxqueryparams_events_total
counter

The number of non-default behaviors executed by the net/url package due to a non-default GODEBUG=urlmaxqueryparams=... setting. Sourced from /godebug/non-default-behavior/urlmaxqueryparams:events.

go_godebug_non_default_behavior_urlstrictcolons_events_total
counter

The number of non-default behaviors executed by the net/url package due to a non-default GODEBUG=urlstrictcolons=... setting. Sourced from /godebug/non-default-behavior/urlstrictcolons:events.

go_godebug_non_default_behavior_winreadlinkvolume_events_total
counter

The number of non-default behaviors executed by the os package due to a non-default GODEBUG=winreadlinkvolume=... setting. Sourced from /godebug/non-default-behavior/winreadlinkvolume:events.

go_godebug_non_default_behavior_winsymlink_events_total
counter

The number of non-default behaviors executed by the os package due to a non-default GODEBUG=winsymlink=... setting. Sourced from /godebug/non-default-behavior/winsymlink:events.

go_godebug_non_default_behavior_x509keypairleaf_events_total
counter

The number of non-default behaviors executed by the crypto/tls package due to a non-default GODEBUG=x509keypairleaf=... setting. Sourced from /godebug/non-default-behavior/x509keypairleaf:events.

go_godebug_non_default_behavior_x509negativeserial_events_total
counter

The number of non-default behaviors executed by the crypto/x509 package due to a non-default GODEBUG=x509negativeserial=... setting. Sourced from /godebug/non-default-behavior/x509negativeserial:events.

go_godebug_non_default_behavior_x509rsacrt_events_total
counter

The number of non-default behaviors executed by the crypto/x509 package due to a non-default GODEBUG=x509rsacrt=... setting. Sourced from /godebug/non-default-behavior/x509rsacrt:events.

go_godebug_non_default_behavior_x509sha256skid_events_total
counter

The number of non-default behaviors executed by the crypto/x509 package due to a non-default GODEBUG=x509sha256skid=... setting. Sourced from /godebug/non-default-behavior/x509sha256skid:events.

go_godebug_non_default_behavior_x509usefallbackroots_events_total
counter

The number of non-default behaviors executed by the crypto/x509 package due to a non-default GODEBUG=x509usefallbackroots=... setting. Sourced from /godebug/non-default-behavior/x509usefallbackroots:events.

go_godebug_non_default_behavior_x509usepolicies_events_total
counter

The number of non-default behaviors executed by the crypto/x509 package due to a non-default GODEBUG=x509usepolicies=... setting. Sourced from /godebug/non-default-behavior/x509usepolicies:events.

go_godebug_non_default_behavior_zipinsecurepath_events_total
counter

The number of non-default behaviors executed by the archive/zip package due to a non-default GODEBUG=zipinsecurepath=... setting. Sourced from /godebug/non-default-behavior/zipinsecurepath:events.

go_goroutines
gauge

Number of goroutines that currently exist.

go_info
gauge

Information about the Go environment. Label: version=Go version.

go_memory_classes_heap_free_bytes
gauge

Memory that is completely free and eligible to be returned to the underlying system, but hasn’t been yet. This metric is the runtime’s estimate of free address space that is backed by physical memory. Sourced from /memory/classes/heap/free:bytes.

go_memory_classes_heap_objects_bytes
gauge

Memory occupied by live objects and dead objects that haven’t yet been marked free by the garbage collector. Sourced from /memory/classes/heap/objects:bytes.

go_memory_classes_heap_released_bytes
gauge

Memory that is completely free and has been returned to the underlying system. This metric is the runtime’s estimate of free address space that is still mapped into the process, but isn’t backed by physical memory. Sourced from /memory/classes/heap/released:bytes.

go_memory_classes_heap_stacks_bytes
gauge

Memory allocated from the heap that is reserved for stack space, whether or not it is currently in-use. Currently, this represents all stack memory for goroutines. It also includes all OS thread stacks in non-cgo programs. Note that stacks may be allocated differently in the future, and this may change. Sourced from /memory/classes/heap/stacks:bytes.

go_memory_classes_heap_unused_bytes
gauge

Memory that is reserved for heap objects but isn’t currently used to hold heap objects. Sourced from /memory/classes/heap/unused:bytes.

go_memory_classes_metadata_mcache_free_bytes
gauge

Memory that is reserved for runtime mcache structures but not in-use. Sourced from /memory/classes/metadata/mcache/free:bytes.

go_memory_classes_metadata_mcache_inuse_bytes
gauge

Memory that is occupied by runtime mcache structures that are currently being used. Sourced from /memory/classes/metadata/mcache/inuse:bytes.

go_memory_classes_metadata_mspan_free_bytes
gauge

Memory that is reserved for runtime mspan structures, but not in-use. Sourced from /memory/classes/metadata/mspan/free:bytes.

go_memory_classes_metadata_mspan_inuse_bytes
gauge

Memory that is occupied by runtime mspan structures that are currently being used. Sourced from /memory/classes/metadata/mspan/inuse:bytes.

go_memory_classes_metadata_other_bytes
gauge

Memory that is reserved for or used to hold runtime metadata. Sourced from /memory/classes/metadata/other:bytes.

go_memory_classes_os_stacks_bytes
gauge

Stack memory allocated by the underlying operating system. In non-cgo programs this metric is currently zero. This may change in the future. In cgo programs this metric includes OS thread stacks allocated directly from the OS. Currently, this only accounts for one stack in c-shared and c-archive build modes, and other sources of stacks from the OS aren’t measured. This too may change in the future. Sourced from /memory/classes/os-stacks:bytes.

go_memory_classes_other_bytes
gauge

Memory used by execution trace buffers, structures for debugging the runtime, finalizer and profiler specials, and more. Sourced from /memory/classes/other:bytes.

go_memory_classes_profiling_buckets_bytes
gauge

Memory that is used by the stack trace hash map used for profiling. Sourced from /memory/classes/profiling/buckets:bytes.

go_memory_classes_total_bytes
gauge

All memory mapped by the Go runtime into the current process as read-write. This doesn’t include memory mapped by code called via cgo or via the syscall package. Sum of all metrics in /memory/classes. Sourced from /memory/classes/total:bytes.

go_memstats_alloc_bytes
gauge

Number of bytes allocated in heap and currently in use. Equals to /memory/classes/heap/objects:bytes.

go_memstats_alloc_bytes_total
counter

Total number of bytes allocated in heap until now, even if released already. Equals to /gc/heap/allocs:bytes.

go_memstats_buck_hash_sys_bytes
gauge

Number of bytes used by the profiling bucket hash table. Equals to /memory/classes/profiling/buckets:bytes.

go_memstats_frees_total
counter

Total number of heap objects frees. Equals to /gc/heap/frees:objects + /gc/heap/tiny/allocs:objects.

go_memstats_gc_sys_bytes
gauge

Number of bytes used for garbage collection system metadata. Equals to /memory/classes/metadata/other:bytes.

go_memstats_heap_alloc_bytes
gauge

Number of heap bytes allocated and currently in use, same as go_memstats_alloc_bytes. Equals to /memory/classes/heap/objects:bytes.

go_memstats_heap_idle_bytes
gauge

Number of heap bytes waiting to be used. Equals to /memory/classes/heap/released:bytes + /memory/classes/heap/free:bytes.

go_memstats_heap_inuse_bytes
gauge

Number of heap bytes that are in use. Equals to /memory/classes/heap/objects:bytes + /memory/classes/heap/unused:bytes

go_memstats_heap_objects
gauge

Number of currently allocated objects. Equals to /gc/heap/objects:objects.

go_memstats_heap_released_bytes
gauge

Number of heap bytes released to OS. Equals to /memory/classes/heap/released:bytes.

go_memstats_heap_sys_bytes
gauge

Number of heap bytes obtained from system. Equals to /memory/classes/heap/objects:bytes + /memory/classes/heap/unused:bytes + /memory/classes/heap/released:bytes + /memory/classes/heap/free:bytes.

go_memstats_last_gc_time_seconds
gauge

Number of seconds since last garbage collection.

go_memstats_mallocs_total
counter

Total number of heap objects allocated, both live and gc-ed. Semantically a counter version for go_memstats_heap_objects gauge. Equals to /gc/heap/allocs:objects + /gc/heap/tiny/allocs:objects.

go_memstats_mcache_inuse_bytes
gauge

Number of bytes in use by mcache structures. Equals to /memory/classes/metadata/mcache/inuse:bytes.

go_memstats_mcache_sys_bytes
gauge

Number of bytes used for mcache structures obtained from system. Equals to /memory/classes/metadata/mcache/inuse:bytes + /memory/classes/metadata/mcache/free:bytes.

go_memstats_mspan_inuse_bytes
gauge

Number of bytes in use by mspan structures. Equals to /memory/classes/metadata/mspan/inuse:bytes.

go_memstats_mspan_sys_bytes
gauge

Number of bytes used for mspan structures obtained from system. Equals to /memory/classes/metadata/mspan/inuse:bytes + /memory/classes/metadata/mspan/free:bytes.

go_memstats_next_gc_bytes
gauge

Number of heap bytes when next garbage collection will take place. Equals to /gc/heap/goal:bytes.

go_memstats_other_sys_bytes
gauge

Number of bytes used for other system allocations. Equals to /memory/classes/other:bytes.

go_memstats_stack_inuse_bytes
gauge

Number of bytes obtained from system for stack allocator in non-CGO environments. Equals to /memory/classes/heap/stacks:bytes.

go_memstats_stack_sys_bytes
gauge

Number of bytes obtained from system for stack allocator. Equals to /memory/classes/heap/stacks:bytes + /memory/classes/os-stacks:bytes.

go_memstats_sys_bytes
gauge

Number of bytes obtained from system. Equals to /memory/classes/total:byte.

go_sched_gomaxprocs_threads
gauge

The current runtime.GOMAXPROCS setting, or the number of operating system threads that can execute user-level Go code simultaneously. Sourced from /sched/gomaxprocs:threads.

go_sched_goroutines_created_goroutines_total
counter

Count of goroutines created since program start. Sourced from /sched/goroutines-created:goroutines.

go_sched_goroutines_goroutines
gauge

Count of live goroutines. Sourced from /sched/goroutines:goroutines.

go_sched_goroutines_not_in_go_goroutines
gauge

Approximate count of goroutines running or blocked in a system call or cgo call. Not guaranteed to add up to /sched/goroutines:goroutines with other goroutine metrics. Sourced from /sched/goroutines/not-in-go:goroutines.

go_sched_goroutines_runnable_goroutines
gauge

Approximate count of goroutines ready to execute, but not executing. Not guaranteed to add up to /sched/goroutines:goroutines with other goroutine metrics. Sourced from /sched/goroutines/runnable:goroutines.

go_sched_goroutines_running_goroutines
gauge

Approximate count of goroutines executing. Always less than or equal to /sched/gomaxprocs:threads. Not guaranteed to add up to /sched/goroutines:goroutines with other goroutine metrics. Sourced from /sched/goroutines/running:goroutines.

go_sched_goroutines_waiting_goroutines
gauge

Approximate count of goroutines waiting on a resource (I/O or sync primitives). Not guaranteed to add up to /sched/goroutines:goroutines with other goroutine metrics. Sourced from /sched/goroutines/waiting:goroutines.

go_sched_latencies_seconds
histogram

Distribution of the time goroutines have spent in the scheduler in a runnable state before actually running. Bucket counts increase monotonically. Sourced from /sched/latencies:seconds. Time series: _bucket, _sum, _count. Label: le=observation buckets of 6.399999999999999e-08, 6.399999999999999e-07, 7.167999999999999e-06, 8.191999999999999e-05, 0.0009175039999999999, 0.010485759999999998, 0.11744051199999998, +Inf.

go_sched_pauses_stopping_gc_seconds
histogram

Distribution of individual GC-related stop-the-world stopping latencies. This is the time it takes from deciding to stop the world until all Ps are stopped. This is a subset of the total GC-related stop-the-world time (/sched/pauses/total/gc:seconds). During this time, some threads may be executing. Bucket counts increase monotonically. Sourced from /sched/pauses/stopping/gc:seconds. Time series: _bucket, _sum, _count. Label: le=observation buckets of 6.399999999999999e-08, 6.399999999999999e-07, 7.167999999999999e-06, 8.191999999999999e-05, 0.0009175039999999999, 0.010485759999999998, 0.11744051199999998, +Inf.

go_sched_pauses_stopping_other_seconds
histogram

Distribution of individual non-GC-related stop-the-world stopping latencies. This is the time it takes from deciding to stop the world until all pauses are stopped. This is a subset of the total non-GC-related stop-the-world time (/sched/pauses/total/other:seconds). During this time, some threads may be executing. Bucket counts increase monotonically. Sourced from /sched/pauses/stopping/other:seconds. Time series: _bucket, _sum, _count. Label: le=observation buckets of 6.399999999999999e-08, 6.399999999999999e-07, 7.167999999999999e-06, 8.191999999999999e-05, 0.0009175039999999999, 0.010485759999999998, 0.11744051199999998, +Inf.

go_sched_pauses_total_gc_seconds
histogram

Distribution of individual GC-related stop-the-world pause latencies. This is the time from deciding to stop the world until the world is started again. Some of this time is spent getting all threads to stop (this is measured directly in /sched/pauses/stopping/gc:seconds), during which some threads may still be running. Bucket counts increase monotonically. Sourced from /sched/pauses/total/gc:seconds. Time series: _bucket, _sum, _count. Label: le=observation buckets of 6.399999999999999e-08, 6.399999999999999e-07, 7.167999999999999e-06, 8.191999999999999e-05, 0.0009175039999999999, 0.010485759999999998, 0.11744051199999998, +Inf.

go_sched_pauses_total_other_seconds
histogram

Distribution of individual non-GC-related stop-the-world pause latencies. This is the time from deciding to stop the world until the world is started again. Some of this time is spent getting all threads to stop (measured directly in /sched/pauses/stopping/other:seconds). Bucket counts increase monotonically. Sourced from /sched/pauses/total/other:seconds. Time series: _bucket, _sum, _count. Label: le=observation buckets of 6.399999999999999e-08, 6.399999999999999e-07, 7.167999999999999e-06, 8.191999999999999e-05, 0.0009175039999999999, 0.010485759999999998, 0.11744051199999998, +Inf.

go_sched_threads_total_threads
gauge

The current count of live threads that are owned by the Go runtime. Sourced from /sched/threads/total:threads.

go_sync_mutex_wait_total_seconds_total
counter

Approximate cumulative time goroutines have spent blocked on a sync.Mutex, sync.RWMutex, or runtime-internal lock. This metric is useful for identifying global changes in lock contention. Collect a mutex or block profile using the runtime/pprof package for more detailed contention data. Sourced from /sync/mutex/wait/total:seconds.

go_threads
gauge

Number of OS threads created.

hug_config_diffs_total
counter

Total number of HAProxy configuration diffs by operation and resource type. Labels: operation=statuses created, deleted, or updated, resource=resource name.

hug_config_generation_duration_seconds
histogram

Duration of HAProxy configuration diff computation in seconds. Time series: _bucket, _sum, _count. Label: le=observation buckets of 0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, +Inf.

hug_config_transfer_duration_seconds
histogram

Duration of HAProxy configuration transfer and application in seconds. Time series: _bucket, _sum, _count. Label: le=observation buckets of 0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, +Inf.

hug_event_batch_duration_seconds
histogram

Duration of event batch processing in seconds. Time series: _bucket, _sum, _count. Label: le=observation buckets of 0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, +Inf.

hug_event_batch_errors_total
counter

Total number of event batch processing errors.

hug_event_batch_size
histogram

Number of events in a processed batch. Time series: _bucket, _sum, _count. Label: le=observation buckets of 1, 5, 10, 25, 50, 100, 250, 500, +Inf.

hug_event_batch_total
counter

Total number of event batches processed.

hug_events_processed_total
counter

Total number of Kubernetes resource events processed by event type. Label: event_type.

hug_haproxy_reload_total
counter

Total number of HAProxy configuration reloads.

process_cpu_seconds_total
counter

Total user and system CPU time spent in seconds.

process_max_fds
gauge

Maximum number of open file descriptors.

process_network_receive_bytes_total
counter

Number of bytes received by the process over the network.

process_network_transmit_bytes_total
counter

Number of bytes sent by the process over the network.

process_open_fds
gauge

Number of open file descriptors.

process_resident_memory_bytes
gauge

Resident memory size in bytes.

process_start_time_seconds
gauge

Start time of the process since unix epoch in seconds.

process_virtual_memory_bytes
gauge

Virtual memory size in bytes.

process_virtual_memory_max_bytes
gauge

Maximum amount of virtual memory available in bytes.

rest_client_requests_total
counter

Number of HTTP requests, partitioned by status code, method, and host. Labels: code, host, method.

workqueue_adds_total
counter

Total number of adds handled by workqueue. Labels: controller, name.

workqueue_depth
gauge

Current depth of workqueue by workqueue and priority. Labels: controller, name, priority.

workqueue_longest_running_processor_seconds
gauge

How many seconds has the longest running processor for workqueue been running. Labels: controller, name.

workqueue_queue_duration_seconds
histogram

How long in seconds an item stays in workqueue before being requested. Time series: _bucket, _sum, _count. Labels: controller, name, le=observation buckets of 1e-08, 1e-07, 1e-06, 9.999999999999999e-06, 9.999999999999999e-05 0.001, 0.01, 1, 10, 100, 1000, +Inf.

workqueue_retries_total
counter

Total number of retries handled by workqueue. Labels: controller, name.

workqueue_unfinished_work_seconds
gauge

How many seconds of work has been done that is in progress and hasn’t been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases. Labels: controller, name.

workqueue_work_duration_seconds
histogram

How long in seconds processing an item from workqueue takes. Time series: _bucket, _sum, _count. Labels: controller, name, le=observation buckets of 1e-08, 1e-07, 1e-06, 9.999999999999999e-06, 9.999999999999999e-05 0.001, 0.01, 1, 10, 100, 1000, +Inf.

Product Documentation

API Documentation

Configuration Documentation

Controller metrics

View controller metrics Jump to heading

Authentication modes Jump to heading

Enable none authentication mode Jump to heading

Enable kube-rbac authentication mode Jump to heading

Enable basic authentication mode Jump to heading

Troubleshooting Jump to heading

Controller metrics reference Jump to heading

See also Jump to heading

Product Documentation

API Documentation

Configuration Documentation

View controller metrics Jump to heading #

Authentication modes Jump to heading #

Enable none authentication mode Jump to heading #

Enable kube-rbac authentication mode Jump to heading #

Enable basic authentication mode Jump to heading #

Troubleshooting Jump to heading #

Controller metrics reference Jump to heading #

See also Jump to heading #

Privacy Settings

View controller metrics Jump to heading

Authentication modes Jump to heading

Enable none authentication mode Jump to heading

Enable kube-rbac authentication mode Jump to heading

Enable basic authentication mode Jump to heading

Troubleshooting Jump to heading

Controller metrics reference Jump to heading

See also Jump to heading