Administration

Troubleshoot

Several techniques exist for diagnosing problems with the ingress controller. They include viewing logs, viewing pod details, and gaining shell access into the pod to check files.

Enable debugging logs Jump to heading

Debugging logs give extra information that might help you troubleshoot. To enable debugging logs:

  • Modify your installation to set the ingress controller’s logging level to debug.

    bash
    helm upgrade haproxy-kubernetes-ingress haproxytech/kubernetes-ingress \
    --namespace haproxy-controller \
    --set controller.logging.level=debug
    bash
    helm upgrade haproxy-kubernetes-ingress haproxytech/kubernetes-ingress \
    --namespace haproxy-controller \
    --set controller.logging.level=debug

View pod details Jump to heading

  1. View the details of the ingress controller pod by using kubectl describe pod:

    bash
    kubectl describe pod -n haproxy-controller haproxy-kubernetes-ingress-7d84545b45-fpzm9
    bash
    kubectl describe pod -n haproxy-controller haproxy-kubernetes-ingress-7d84545b45-fpzm9
    output
    bash
    Name: haproxy-kubernetes-ingress-7d84545b45-fpzm9
    Namespace: haproxy-controller
    Priority: 0
    Node: minikube/192.168.99.116
    Start Time: Fri, 17 Jul 2020 13:06:37 -0400
    Labels: app.kubernetes.io/instance=kubernetes-ingress
    app.kubernetes.io/name=kubernetes-ingress
    pod-template-hash=7d84545b45
    output
    bash
    Name: haproxy-kubernetes-ingress-7d84545b45-fpzm9
    Namespace: haproxy-controller
    Priority: 0
    Node: minikube/192.168.99.116
    Start Time: Fri, 17 Jul 2020 13:06:37 -0400
    Labels: app.kubernetes.io/instance=kubernetes-ingress
    app.kubernetes.io/name=kubernetes-ingress
    pod-template-hash=7d84545b45
  2. You can also view the YAML that was used to deploy the ingress controller by using kubectl get deployment:

    bash
    kubectl get deployment haproxy-kubernetes-ingress -o yaml --namespace haproxy-controller
    bash
    kubectl get deployment haproxy-kubernetes-ingress -o yaml --namespace haproxy-controller
    output
    yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    annotations:
    deployment.kubernetes.io/revision: "1"
    creationTimestamp: "2020-07-17T17:06:37Z"
    generation: 1
    labels:
    app.kubernetes.io/instance: kubernetes-ingress
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kubernetes-ingress
    app.kubernetes.io/version: 1.4.5
    helm.sh/chart: kubernetes-ingress-1.4.2
    output
    yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    annotations:
    deployment.kubernetes.io/revision: "1"
    creationTimestamp: "2020-07-17T17:06:37Z"
    generation: 1
    labels:
    app.kubernetes.io/instance: kubernetes-ingress
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kubernetes-ingress
    app.kubernetes.io/version: 1.4.5
    helm.sh/chart: kubernetes-ingress-1.4.2

Gain shell access into the pod Jump to heading

Use kubectl exec to log into the ingress controller pod to inspect the files:

bash
kubectl exec -it haproxy-kubernetes-ingress-7d84545b45-fpzm9 -n haproxy-controller -- /bin/sh
cd /etc/haproxy
ls
bash
kubectl exec -it haproxy-kubernetes-ingress-7d84545b45-fpzm9 -n haproxy-controller -- /bin/sh
cd /etc/haproxy
ls
output
bash
certs haproxy.cfg maps modsec
output
bash
certs haproxy.cfg maps modsec

Enable core dumps for HAProxy Jump to heading

The HAProxy load balancer runs as a service within the HAProxy Ingress Controller Docker pod. In the rare event that the HAProxy load balancer process crashes or behaves abnormally, you can capture a core dump (also known as a crash dump). A core dump is a file that encapsulates the state of an application when it crashes and is useful in diagnosing and fixing potential issues. Core dumps are not enabled by default, so you must configure your OS settings to allow the collection of these files.

This diagnostic method can only be used for on-premises (non-cloud or otherwise managed) installations. It requires that you are able to restart your instance of Docker and that you can alter your Docker Systemd settings. It also requires that you are able to change the kernel settings of your nodes running your Kubernetes cluster.

Enable core dumps Jump to heading

Perform these steps on the nodes running your HAProxy Kubernetes Ingress Controller pods:

  1. Retrieve the names of the nodes running the HAProxy Kubernetes Ingress Controller by using the following command:

    bash
    kubectl get pod -o=custom-columns=NODE:.spec.nodeName,NAME:.metadata.name -n haproxy-controller
    bash
    kubectl get pod -o=custom-columns=NODE:.spec.nodeName,NAME:.metadata.name -n haproxy-controller
    output
    bash
    NODE NAME
    worker1 haproxy-kubernetes-ingress-765ccd48c9-5ggmh
    worker2 haproxy-kubernetes-ingress-765ccd48c9-kdb62
    output
    bash
    NODE NAME
    worker1 haproxy-kubernetes-ingress-765ccd48c9-5ggmh
    worker2 haproxy-kubernetes-ingress-765ccd48c9-kdb62
  2. Configure the kernel settings on your node to specify the location for saving core dumps. This location is communicated to all pods running on the instance.

    This command sets the kernel setting for core_pattern to specify that core dump files should be saved to /tmp:

    bash
    echo '/tmp/core.%P' | sudo tee /proc/sys/kernel/core_pattern
    bash
    echo '/tmp/core.%P' | sudo tee /proc/sys/kernel/core_pattern

    Enabling core dumps this way will prompt the system to produce a core file that will look something like core.257 where the numbers indicate the process ID, or PID.

  3. Edit the Docker Systemd service file /lib/systemd/system/docker.service on your node and add the following to the end of the line beginning with ExecStart:

    bash
    --default-ulimit core=-1
    bash
    --default-ulimit core=-1

    This sets the core dump file size limit to unlimited.

  4. Reload the docker.service Systemd configuration:

    bash
    sudo systemctl daemon-reload
    bash
    sudo systemctl daemon-reload
  5. Restart Docker:

    bash
    sudo systemctl restart docker
    bash
    sudo systemctl restart docker

Repeat for each node in your Kubernetes cluster. Core dumps will now be enabled for all pods.

Retrieve core dumps Jump to heading

To copy the core dump file from the pod where the HAProxy load balancer faulted, use the kubectl cp command.

bash
kubectl cp <namespace>/<pod>:/<pod_directory>/<corefile.pid> /<host_directory_destination>
bash
kubectl cp <namespace>/<pod>:/<pod_directory>/<corefile.pid> /<host_directory_destination>

For example, if you configured your node to place core files in /tmp, then you can expect core files to appear in the /tmp directory within the HAProxy Kubernetes Ingress Controller pod.

To locate and copy this file:

  1. Find the ID of the pod by calling kubectl get pods:

    bash
    kubectl get pods --namespace haproxy-controller
    bash
    kubectl get pods --namespace haproxy-controller
    output
    bash
    NAME READY STATUS RESTARTS AGE
    haproxy-kubernetes-ingress-56f55cf6d6-4tcbc 2/2 Running 0 24h
    output
    bash
    NAME READY STATUS RESTARTS AGE
    haproxy-kubernetes-ingress-56f55cf6d6-4tcbc 2/2 Running 0 24h
  2. Identify the core files by listing the directory contents with ls:

    bash
    kubectl exec -n haproxy-controller haproxy-kubernetes-ingress-56f55cf6d6-4tcbc -- ls -lrt /tmp
    bash
    kubectl exec -n haproxy-controller haproxy-kubernetes-ingress-56f55cf6d6-4tcbc -- ls -lrt /tmp
    output
    bash
    -rw------- 1 haproxy haproxy 6389760 Jul 25 21:25 core.257
    output
    bash
    -rw------- 1 haproxy haproxy 6389760 Jul 25 21:25 core.257
  3. Use kubectl cp to copy the file from the pod to your node:

    bash
    kubectl cp haproxy-controller/haproxy-kubernetes-ingress-56f55cf6d6-4tcbc:/tmp/core.257 /tmp/core.257
    bash
    kubectl cp haproxy-controller/haproxy-kubernetes-ingress-56f55cf6d6-4tcbc:/tmp/core.257 /tmp/core.257

    The core file will be copied to the directory you specified for the destination.

Enable techdump Jump to heading

Available since

version 1.10

Note: This feature is available only for the enterprise edition of HAProxy Kubernetes Ingress Controller.

The techdump tool allows you to collect diagnostic data for a running HAProxy Kubernetes Ingress Controller. A techdump creates a .zip file containing information about the state of the system that you can provide to the HAProxy Technologies support team. The .zip techdump file is named with the date and timezone in which the file was created, as well as the hostname of the node. For example:

  • techdump.2024-01-30T17:59:51Z.haproxy-ingress-6fb9847747-5z7gq.zip.

Its contents are as follows:

  • map directory: contains the HAProxy maps.
  • techdump-xxxxxxxxxx directory:
    • controller-owner-manifest.yaml: the controller owner manifest file (deployment/daemonset).
    • ingress-controller-manifest.yaml: the ingress controller manifest file.
    • node-manifest.yaml: node information where the ingress controller is running.
    • prometheus.txt: Prometheus metrics for the ingress controller.
    • stats.csv: load balancer stats.
    • data.yaml: contains the ingress controller information including: version, env, osargs, system information, kubernetes, and memory.
  • haproxy.cfg: load balancer configuration file.
  • haproxy-aux.cfg: load balancer auxiliary configuration file.

Enable with Helm Jump to heading

If you installed the ingress controller with Helm, you can run the helm upgrade command with an additional argument to enable the techdump feature. Be sure to specify version 1.10 or higher. Note that even if you have already installed version 1.10 or higher, you can still run helm upgrade. Replace [KEY] with your HAProxy Enterprise license key.

bash
helm upgrade haproxy-kubernetes-ingress haproxytech/kubernetes-ingress \
--create-namespace \
--namespace haproxy-controller \
--set controller.imageCredentials.registry=kubernetes-registry.haproxy.com \
--set controller.imageCredentials.username=[KEY] \
--set controller.imageCredentials.password=[KEY] \
--set controller.image.repository=kubernetes-registry.haproxy.com/hapee-ingress \
--set controller.image.tag=v1.10 \
--set controller.techdump.enabled=true
bash
helm upgrade haproxy-kubernetes-ingress haproxytech/kubernetes-ingress \
--create-namespace \
--namespace haproxy-controller \
--set controller.imageCredentials.registry=kubernetes-registry.haproxy.com \
--set controller.imageCredentials.username=[KEY] \
--set controller.imageCredentials.password=[KEY] \
--set controller.image.repository=kubernetes-registry.haproxy.com/hapee-ingress \
--set controller.image.tag=v1.10 \
--set controller.techdump.enabled=true

Setting controller.techdump.enabled=true enables the techdump feature.

Tip

If you have not yet installed the ingress controller, you can enable the techdump feature during the installation. Instead of helm upgrade use helm install and provide the --set controller.techdump.enabled=true argument.

Enable with kubectl Jump to heading

If you installed the ingress controller via kubectl with the deployment YAML file, you will use the same file again to enable the techdump feature. If you do not have the file you used for installation, you may need to download it again. Note that you will need to use version 1.10 of the file or higher.

To enable the techdump feature:

  1. Make a backup copy of the deployment YAML file (v1.10). You will need this to disable the techdump feature after you have collected your techdump archive. If you do not have the original file, you may need to download it again.

  2. Add the following permissions to the rules section of the ClusterRole resource named haproxy-ingress-cluster-role. These permissions allow the ingress controller to collect statistics about resources and to create the techdump archive.

    haproxy-ingress.hapee.yaml
    yaml
    - apiGroups:
    - "apps"
    resources:
    - replicasets
    - deployments
    - daemonsets
    verbs:
    - get
    - list
    haproxy-ingress.hapee.yaml
    yaml
    - apiGroups:
    - "apps"
    resources:
    - replicasets
    - deployments
    - daemonsets
    verbs:
    - get
    - list
  3. Use the kubectl apply command to deploy the changes:

    bash
    kubectl apply -f haproxy-ingress.hapee.yaml
    bash
    kubectl apply -f haproxy-ingress.hapee.yaml
    output
    bash
    namespace/haproxy-controller unchanged
    serviceaccount/haproxy-ingress-service-account unchanged
    clusterrole.rbac.authorization.k8s.io/haproxy-ingress-cluster-role configured
    [...]
    output
    bash
    namespace/haproxy-controller unchanged
    serviceaccount/haproxy-ingress-service-account unchanged
    clusterrole.rbac.authorization.k8s.io/haproxy-ingress-cluster-role configured
    [...]

Techdump is now enabled and you can create a techdump archive.

Create a techdump archive via the ingress controller runtime socket Jump to heading

To create a techdump archive via the ingress controller runtime socket:

  1. Find the ID of the pod by calling kubectl get pods:

    bash
    kubectl get pods --namespace haproxy-controller
    bash
    kubectl get pods --namespace haproxy-controller
    output
    bash
    NAME READY STATUS RESTARTS AGE
    haproxy-ingress-6fb9847747-5z7gq 1/1 Running 0 3h42m
    output
    bash
    NAME READY STATUS RESTARTS AGE
    haproxy-ingress-6fb9847747-5z7gq 1/1 Running 0 3h42m
  2. Issue the command to the ingress controller runtime socket to initiate creating the techdump archive. Note that there is no console output:

    bash
    kubectl exec -n haproxy-controller haproxy-ingress-6fb9847747-5z7gq -- bash -c 'echo "dump" | sudo socat - /var/run/ingress-controller-runtime.sock'
    bash
    kubectl exec -n haproxy-controller haproxy-ingress-6fb9847747-5z7gq -- bash -c 'echo "dump" | sudo socat - /var/run/ingress-controller-runtime.sock'

    Info

    When run in external mode, the runtime socket may instead be located at /tmp/haproxy-ingress/run/ingress-controller-runtime.sock. This is configurable using the --runtime-dir startup argument.

  3. Identify the name of the archive file with ls. The techdump archive will be located in /etc/haproxy/techdump:

    bash
    kubectl exec -n haproxy-controller haproxy-ingress-6fb9847747-5z7gq -- ls -lrt /etc/haproxy/techdump
    bash
    kubectl exec -n haproxy-controller haproxy-ingress-6fb9847747-5z7gq -- ls -lrt /etc/haproxy/techdump
    output
    bash
    -rw-r--r-- 1 root root 13213 Jan 30 20:53 techdump.2024-01-30T20:53:03Z.haproxy-ingress-6fb9847747-5z7gq.zip
    output
    bash
    -rw-r--r-- 1 root root 13213 Jan 30 20:53 techdump.2024-01-30T20:53:03Z.haproxy-ingress-6fb9847747-5z7gq.zip

    Info

    When run in external mode, the archive will be located at /tmp/haproxy-ingress/run by default. This is configurable using the --techdump-dir startup argument.

  4. Use kubectl cp to copy the techdump archive file from the pod to your node. Note that the destination filename cannot contain the : character, so here we have provided a shortened destination filename:

    bash
    kubectl cp haproxy-controller/haproxy-ingress-6fb9847747-5z7gq:/etc/haproxy/techdump/techdump.2024-01-30T20:53:03Z.haproxy-ingress-6fb9847747-5z7gq.zip ./techdump.haproxy-ingress-6fb9847747-5z7gq.zip
    bash
    kubectl cp haproxy-controller/haproxy-ingress-6fb9847747-5z7gq:/etc/haproxy/techdump/techdump.2024-01-30T20:53:03Z.haproxy-ingress-6fb9847747-5z7gq.zip ./techdump.haproxy-ingress-6fb9847747-5z7gq.zip

    The archive file will be copied to the directory you specified for the destination on your node.

Create a techdump archive via the HTTP endpoint Jump to heading

To create a techdump archive via an authenticated HTTP endpoint:

  1. Install the mkpasswd tool:

    bash
    # mkpasswd is included in the whois package
    sudo apt install whois
    bash
    # mkpasswd is included in the whois package
    sudo apt install whois
    bash
    sudo yum install mkpasswd
    bash
    sudo yum install mkpasswd
  2. Call mkpasswd with the SHA-256 algorithm to hash your password. Base64 encode the output:

    bash
    mkpasswd -m sha-256 mypassword123 | base64 | tr -d "\n"
    bash
    mkpasswd -m sha-256 mypassword123 | base64 | tr -d "\n"
    example output
    bash
    JDUkZU5pUjZvZ21QT2RnViRqVHU0alU1RXBEV0hZeUNwc3liWU1aNkw5dUQxUUhoN25Pa0R0Uk1IR3M4Cg==
    example output
    bash
    JDUkZU5pUjZvZ21QT2RnViRqVHU0alU1RXBEV0hZeUNwc3liWU1aNkw5dUQxUUhoN25Pa0R0Uk1IR3M4Cg==

    Warning

    This password (mypassword123) is for illustration purposes only. Be sure to use a more robust and secure password.

  3. Save the hashed password for the next step.

  4. In your YAML configuration, create a Secret. In this example, in a file named techdump-credentials.yaml, we declare a secret named techdump-credentials, since we will reference and use this Secret for connecting to the /techdump HTTP endpoint of our ingress controller. Note that the user name is techdumpuser and the password will be the base64-encoded SHA-256-encrypted password you created in the previous step. You can change the username to another value.

    techdump-credentials.yaml
    yaml
    apiVersion: v1
    kind: Secret
    metadata:
    name: techdump-credentials
    namespace: haproxy-controller
    type: Opaque
    data:
    techdumpuser: <your base64-encoded SHA-256-encrypted password>
    techdump-credentials.yaml
    yaml
    apiVersion: v1
    kind: Secret
    metadata:
    name: techdump-credentials
    namespace: haproxy-controller
    type: Opaque
    data:
    techdumpuser: <your base64-encoded SHA-256-encrypted password>
  5. Apply the change with the kubectl apply command:

    bash
    kubectl apply -f techdump-credentials.yaml
    bash
    kubectl apply -f techdump-credentials.yaml
    output
    bash
    secret/techdump-credentials created
    output
    bash
    secret/techdump-credentials created
  6. Reference the secret in your ConfigMap. The ingress controller looks for the presence of techdump-endpoint-auth-secret in data. Here we provide the name of the Secret, techdump-credentials:

    haproxy-ingress.hapee.yaml
    yaml
    apiVersion: v1
    kind: ConfigMap
    metadata:
    name: haproxy-configmap
    namespace: default
    data:
    techdump-endpoint-auth-secret: haproxy-controller/techdump-credentials
    haproxy-ingress.hapee.yaml
    yaml
    apiVersion: v1
    kind: ConfigMap
    metadata:
    name: haproxy-configmap
    namespace: default
    data:
    techdump-endpoint-auth-secret: haproxy-controller/techdump-credentials
  7. Apply the change with the kubectl apply command:

    bash
    kubectl apply -f haproxy-ingress.hapee.yaml
    bash
    kubectl apply -f haproxy-ingress.hapee.yaml
    output
    bash
    [...]
    configmap/haproxy-configmap configured
    [...]
    output
    bash
    [...]
    configmap/haproxy-configmap configured
    [...]

    Tip

    If you installed the ingress controller with Helm, you may not have the haproxy-ingress.hapee.yaml file (to edit the ConfigMap). To edit the ConfigMap that was installed automatically, use the following command:

    bash
    kubectl edit configmap haproxy-kubernetes-ingress --namespace haproxy-controller
    bash
    kubectl edit configmap haproxy-kubernetes-ingress --namespace haproxy-controller
  8. Use the kubectl apply command to apply the changes:

    bash
    kubectl apply -f haproxy-ingress.hapee.yaml
    bash
    kubectl apply -f haproxy-ingress.hapee.yaml
    output
    bash
    [...]
    secret/techdump-credentials created
    [...]
    output
    bash
    [...]
    secret/techdump-credentials created
    [...]
  9. Use curl to call the /techdump endpoint of the ingress controller, specifying the username and password you just created. Note that you may need to specify a different IP address or NodePort for your cluster depending on your setup and the instance or node from which you are connecting.

    bash
    curl -X POST -u techdumpuser:mypassword123 http://127.0.0.1:31634/techdump
    bash
    curl -X POST -u techdumpuser:mypassword123 http://127.0.0.1:31634/techdump

    Note that if there were no errors, the command returns no output. The techdump archive has been created.

  10. Find the ID of the pod by calling kubectl get pods:

    bash
    kubectl get pods --namespace haproxy-controller
    bash
    kubectl get pods --namespace haproxy-controller
    output
    bash
    NAME READY STATUS RESTARTS AGE
    haproxy-ingress-6fb9847747-5z7gq 1/1 Running 0 3h42m
    output
    bash
    NAME READY STATUS RESTARTS AGE
    haproxy-ingress-6fb9847747-5z7gq 1/1 Running 0 3h42m

    Tip

    If you have more than one ingress controller pod, the techdump archive may be on any one of them. Be sure to check them all to retrieve your techdump.

  11. Identify the name of the archive file with ls. The techdump archive will be located in /etc/haproxy/techdump:

    bash
    kubectl exec -n haproxy-controller haproxy-ingress-6fb9847747-5z7gq -- ls -lrt /etc/haproxy/techdump
    bash
    kubectl exec -n haproxy-controller haproxy-ingress-6fb9847747-5z7gq -- ls -lrt /etc/haproxy/techdump
    output
    bash
    -rw-r--r-- 1 root root 13213 Jan 30 20:53 techdump.2024-01-30T20:53:03Z.haproxy-ingress-6fb9847747-5z7gq.zip
    output
    bash
    -rw-r--r-- 1 root root 13213 Jan 30 20:53 techdump.2024-01-30T20:53:03Z.haproxy-ingress-6fb9847747-5z7gq.zip
  12. Use kubectl cp to copy the techdump archive file from the pod to your node. Note that the destination filename cannot contain the : character, so here we have provided a shortened destination filename:

    bash
    kubectl cp haproxy-controller/haproxy-ingress-6fb9847747-5z7gq:/etc/haproxy/techdump/techdump.2024-01-30T20:53:03Z.haproxy-ingress-6fb9847747-5z7gq.zip /tmp/techdump.haproxy-ingress-6fb9847747-5z7gq.zip
    bash
    kubectl cp haproxy-controller/haproxy-ingress-6fb9847747-5z7gq:/etc/haproxy/techdump/techdump.2024-01-30T20:53:03Z.haproxy-ingress-6fb9847747-5z7gq.zip /tmp/techdump.haproxy-ingress-6fb9847747-5z7gq.zip

    The archive file will be copied to the directory you specified for the destination on your node.

    Info

    When run in external mode, the archive will be located at /tmp/haproxy-ingress/run by default. This is configurable using the --techdump-dir startup argument.

Disable techdump Jump to heading

Once you have collected your techdump file, you can disable the techdump feature with either Helm or with kubectl, depending on how you installed the ingress controller.

Disable techdump with Helm Jump to heading

You can disable techdump by running helm upgrade again with the same arguments as before, but with -- set controller.techdump.enabled=false to disable techdump. Replace [KEY] with your HAProxy Enterprise license key.

bash
helm upgrade haproxy-kubernetes-ingress haproxytech/kubernetes-ingress \
--create-namespace \
--namespace haproxy-controller \
--set controller.imageCredentials.registry=kubernetes-registry.haproxy.com \
--set controller.imageCredentials.username=[KEY] \
--set controller.imageCredentials.password=[KEY] \
--set controller.image.repository=kubernetes-registry.haproxy.com/hapee-ingress \
--set controller.image.tag=v1.10 \
--set controller.techdump.enabled=false
bash
helm upgrade haproxy-kubernetes-ingress haproxytech/kubernetes-ingress \
--create-namespace \
--namespace haproxy-controller \
--set controller.imageCredentials.registry=kubernetes-registry.haproxy.com \
--set controller.imageCredentials.username=[KEY] \
--set controller.imageCredentials.password=[KEY] \
--set controller.image.repository=kubernetes-registry.haproxy.com/hapee-ingress \
--set controller.image.tag=v1.10 \
--set controller.techdump.enabled=false

Disable techdump with kubectl Jump to heading

You can disable techdump by re-applying your original deployment YAML file to remove the additional permissions (and ConfigMap / Secret changes if you used the HTTP endpoint to trigger the techdump):

  1. Use the kubectl apply command with your original deployment YAML file to remove the changes:

    bash
    kubectl apply -f haproxy-ingress.hapee.yaml
    bash
    kubectl apply -f haproxy-ingress.hapee.yaml

If this page was useful, please, Leave the feedback.