Troubleshoot

Several techniques exist for diagnosing problems with the ingress controller. They include viewing logs, viewing pod details, and gaining shell access into the pod to check files.

Enable debugging logs Jump to heading #

Debugging logs give extra information that might help you troubleshoot. To enable debugging logs:

• Modify your installation to set the ingress controller’s logging level to debug.

  nix
  helm upgrade haproxy-kubernetes-ingress haproxytech/kubernetes-ingress \
    --namespace haproxy-controller \
    --set controller.logging.level=debug 

  nix
  helm upgrade haproxy-kubernetes-ingress haproxytech/kubernetes-ingress \
    --namespace haproxy-controller \
    --set controller.logging.level=debug 

View pod details Jump to heading #

1. View the details of the ingress controller pod by using kubectl describe pod:

   nix
   kubectl describe pod -n haproxy-controller haproxy-kubernetes-ingress-7d84545b45-fpzm9

   nix
   kubectl describe pod -n haproxy-controller haproxy-kubernetes-ingress-7d84545b45-fpzm9

   output
   text
   Name:         haproxy-kubernetes-ingress-7d84545b45-fpzm9
   Namespace:    haproxy-controller
   Priority:     0
   Node:         minikube/192.168.99.116
   Start Time:   Fri, 17 Jul 2020 13:06:37 -0400
   Labels:       app.kubernetes.io/instance=kubernetes-ingress
                 app.kubernetes.io/name=kubernetes-ingress
                 pod-template-hash=7d84545b45

   output
   text
   Name:         haproxy-kubernetes-ingress-7d84545b45-fpzm9
   Namespace:    haproxy-controller
   Priority:     0
   Node:         minikube/192.168.99.116
   Start Time:   Fri, 17 Jul 2020 13:06:37 -0400
   Labels:       app.kubernetes.io/instance=kubernetes-ingress
                 app.kubernetes.io/name=kubernetes-ingress
                 pod-template-hash=7d84545b45

2. You can also view the YAML that was used to deploy the ingress controller by using kubectl get deployment:

   nix
   kubectl get deployment haproxy-kubernetes-ingress -o yaml --namespace haproxy-controller

   nix
   kubectl get deployment haproxy-kubernetes-ingress -o yaml --namespace haproxy-controller

   output
   yaml
   apiVersion: apps/v1
   kind: Deployment
   metadata:
   annotations:
     deployment.kubernetes.io/revision: "1"
   creationTimestamp: "2020-07-17T17:06:37Z"
   generation: 1
   labels:
     app.kubernetes.io/instance: kubernetes-ingress
     app.kubernetes.io/managed-by: Helm
     app.kubernetes.io/name: kubernetes-ingress
     app.kubernetes.io/version: 1.4.5
     helm.sh/chart: kubernetes-ingress-1.4.2

   output
   yaml
   apiVersion: apps/v1
   kind: Deployment
   metadata:
   annotations:
     deployment.kubernetes.io/revision: "1"
   creationTimestamp: "2020-07-17T17:06:37Z"
   generation: 1
   labels:
     app.kubernetes.io/instance: kubernetes-ingress
     app.kubernetes.io/managed-by: Helm
     app.kubernetes.io/name: kubernetes-ingress
     app.kubernetes.io/version: 1.4.5
     helm.sh/chart: kubernetes-ingress-1.4.2

Gain shell access into the pod Jump to heading #

Use kubectl exec to log into the ingress controller pod to inspect the files:

nix
kubectl exec -it haproxy-kubernetes-ingress-7d84545b45-fpzm9 -n haproxy-controller -- /bin/sh
cd /etc/haproxy
ls

nix
kubectl exec -it haproxy-kubernetes-ingress-7d84545b45-fpzm9 -n haproxy-controller -- /bin/sh
cd /etc/haproxy
ls

output
text
certs  haproxy.cfg  maps  modsec

output
text
certs  haproxy.cfg  maps  modsec

Enable core dumps for HAProxy Jump to heading #

The HAProxy load balancer runs as a service within the HAProxy Ingress Controller Docker pod. In the rare event that the HAProxy load balancer process crashes or behaves abnormally, you can capture a core dump (also known as a crash dump). A core dump is a file that encapsulates the state of an application when it crashes and is useful in diagnosing and fixing potential issues. Core dumps are not enabled by default, so you must configure your OS settings to allow the collection of these files.

This diagnostic method can only be used for on-premises (non-cloud or otherwise managed) installations. It requires that you are able to restart your instance of Docker and that you can alter your Docker Systemd settings. It also requires that you are able to change the kernel settings of your nodes running your Kubernetes cluster.

Enable core dumps Jump to heading #

Perform these steps on the nodes running your HAProxy Kubernetes Ingress Controller pods:

1. Retrieve the names of the nodes running the HAProxy Kubernetes Ingress Controller by using the following command:

   nix
   kubectl get pod -o=custom-columns=NODE:.spec.nodeName,NAME:.metadata.name -n haproxy-controller

   nix
   kubectl get pod -o=custom-columns=NODE:.spec.nodeName,NAME:.metadata.name -n haproxy-controller

   output
   text
   NODE     NAME
   worker1  haproxy-kubernetes-ingress-765ccd48c9-5ggmh
   worker2  haproxy-kubernetes-ingress-765ccd48c9-kdb62

   output
   text
   NODE     NAME
   worker1  haproxy-kubernetes-ingress-765ccd48c9-5ggmh
   worker2  haproxy-kubernetes-ingress-765ccd48c9-kdb62

2. Configure the kernel settings on your node to specify the location for saving core dumps. This location is communicated to all pods running on the instance.

   This command sets the kernel setting for core_pattern to specify that core dump files should be saved to /tmp:

   nix
   echo '/tmp/core.%P' | sudo tee /proc/sys/kernel/core_pattern

   nix
   echo '/tmp/core.%P' | sudo tee /proc/sys/kernel/core_pattern

   Enabling core dumps this way will prompt the system to produce a core file that will look something like core.257 where the numbers indicate the process ID, or PID.

3. Edit the Docker Systemd service file /lib/systemd/system/docker.service on your node and add the following to the end of the line beginning with ExecStart:

   nix
   --default-ulimit core=-1

   nix
   --default-ulimit core=-1

   This sets the core dump file size limit to unlimited.

4. Reload the docker.service Systemd configuration:

   nix
   sudo systemctl daemon-reload

   nix
   sudo systemctl daemon-reload

5. Restart Docker:

   nix
   sudo systemctl restart docker

   nix
   sudo systemctl restart docker

Repeat for each node in your Kubernetes cluster. Core dumps will now be enabled for all pods.

Retrieve core dumps Jump to heading #

To copy the core dump file from the pod where the HAProxy load balancer faulted, use the kubectl cp command.

nix
kubectl cp <namespace>/<pod>:/<pod_directory>/<corefile.pid> /<host_directory_destination>

nix
kubectl cp <namespace>/<pod>:/<pod_directory>/<corefile.pid> /<host_directory_destination>

For example, if you configured your node to place core files in /tmp, then you can expect core files to appear in the /tmp directory within the HAProxy Kubernetes Ingress Controller pod.

To locate and copy this file:

1. Find the ID of the pod by calling kubectl get pods:

   nix
   kubectl get pods --namespace haproxy-controller

   nix
   kubectl get pods --namespace haproxy-controller

   output
   text
   NAME                                          READY   STATUS    RESTARTS   AGE
   haproxy-kubernetes-ingress-56f55cf6d6-4tcbc   2/2     Running   0          24h

   output
   text
   NAME                                          READY   STATUS    RESTARTS   AGE
   haproxy-kubernetes-ingress-56f55cf6d6-4tcbc   2/2     Running   0          24h

2. Identify the core files by listing the directory contents with ls:

   nix
   kubectl exec -n haproxy-controller haproxy-kubernetes-ingress-56f55cf6d6-4tcbc -- ls -lrt /tmp

   nix
   kubectl exec -n haproxy-controller haproxy-kubernetes-ingress-56f55cf6d6-4tcbc -- ls -lrt /tmp

   output
   text
   -rw-------    1 haproxy  haproxy    6389760 Jul 25 21:25 core.257

   output
   text
   -rw-------    1 haproxy  haproxy    6389760 Jul 25 21:25 core.257

3. Use kubectl cp to copy the file from the pod to your node:

   nix
   kubectl cp haproxy-controller/haproxy-kubernetes-ingress-56f55cf6d6-4tcbc:/tmp/core.257 /tmp/core.257

   nix
   kubectl cp haproxy-controller/haproxy-kubernetes-ingress-56f55cf6d6-4tcbc:/tmp/core.257 /tmp/core.257

   The core file will be copied to the directory you specified for the destination.

Techdump Jump to heading #

The techdump tool allows you to collect diagnostic data for a running HAProxy Kubernetes Ingress Controller. A techdump creates a .zip file containing information about the state of the system that you can provide to the HAProxy Technologies support team. The .zip techdump file is named with the date and timezone in which the file was created, as well as the hostname of the node. For example:

• techdump.2024-01-30T17:59:51Z.haproxy-ingress-6fb9847747-5z7gq.zip

Its contents are as follows:

• map directory: contains the HAProxy maps.
• techdump-xxxxxxxxxx directory:
  • controller-owner-manifest.yaml: the controller owner manifest file (deployment/daemonset).
  • ingress-controller-manifest.yaml: the ingress controller manifest file.
  • node-manifest.yaml: node information where the ingress controller is running.
  • prometheus.txt: Prometheus metrics for the ingress controller.
  • stats.csv: load balancer stats.
  • data.yaml: contains the ingress controller information including: version, env, osargs, system information, kubernetes, and memory.
• haproxy.cfg: load balancer configuration file.
• haproxy-aux.cfg: load balancer auxiliary configuration file.

Note that techdumps and core dumps provide different diagnostic information, but may both be useful in diagnosing problems.

Enable Techdump Jump to heading #

Enable with Helm Jump to heading #

If you installed the ingress controller with Helm, you can run the helm upgrade command with an additional argument to enable the techdump feature. Be sure to specify version 1.10 or higher. Note that even if you have already installed version 1.10 or higher, you can still run helm upgrade. Replace [KEY] with your HAProxy Enterprise license key (retrieve license key).

nix
helm upgrade haproxy-kubernetes-ingress haproxytech/kubernetes-ingress \
  --create-namespace \
  --namespace haproxy-controller \
  --set controller.imageCredentials.registry=kubernetes-registry.haproxy.com \
  --set controller.imageCredentials.username=[KEY] \
  --set controller.imageCredentials.password=[KEY] \
  --set controller.image.repository=kubernetes-registry.haproxy.com/hapee-ingress \
  --set controller.image.tag=v3.0 \
  --set controller.techdump.enabled=true

nix
helm upgrade haproxy-kubernetes-ingress haproxytech/kubernetes-ingress \
  --create-namespace \
  --namespace haproxy-controller \
  --set controller.imageCredentials.registry=kubernetes-registry.haproxy.com \
  --set controller.imageCredentials.username=[KEY] \
  --set controller.imageCredentials.password=[KEY] \
  --set controller.image.repository=kubernetes-registry.haproxy.com/hapee-ingress \
  --set controller.image.tag=v3.0 \
  --set controller.techdump.enabled=true

Setting controller.techdump.enabled=true enables the techdump feature.

Enable with kubectl Jump to heading #

If you installed the ingress controller via kubectl with the deployment YAML file, you will use the same file again to enable the techdump feature. If you do not have the file you used for installation, you may need to download it again. Note that you will need to use version 1.10 of the file or higher.

To enable the techdump feature:

1. Make a backup copy of the deployment YAML file (v3.0). You will need this to disable the techdump feature after you have collected your techdump archive. If you do not have the original file, you may need to download it again.

2. Add the following permissions to the rules section of the ClusterRole resource named haproxy-ingress-cluster-role. These permissions allow the ingress controller to collect statistics about resources and to create the techdump archive.

   haproxy-ingress.hapee.yaml
   yaml
   - apiGroups:
     - "apps"
     resources:
     - replicasets
     - deployments
     - daemonsets
     verbs:
     - get
     - list

   haproxy-ingress.hapee.yaml
   yaml
   - apiGroups:
     - "apps"
     resources:
     - replicasets
     - deployments
     - daemonsets
     verbs:
     - get
     - list

3. Use the kubectl apply command to deploy the changes:

   nix
   kubectl apply -f haproxy-ingress.hapee.yaml

   nix
   kubectl apply -f haproxy-ingress.hapee.yaml

   output
   text
   namespace/haproxy-controller unchanged
   serviceaccount/haproxy-ingress-service-account unchanged
   clusterrole.rbac.authorization.k8s.io/haproxy-ingress-cluster-role configured
   [...]

   output
   text
   namespace/haproxy-controller unchanged
   serviceaccount/haproxy-ingress-service-account unchanged
   clusterrole.rbac.authorization.k8s.io/haproxy-ingress-cluster-role configured
   [...]

Techdump is now enabled and you can create a techdump archive.

Create a techdump archive Jump to heading #

When a problem occurs, create a techdump archive.

Create a techdump archive via the ingress controller runtime socket Jump to heading #

To create a techdump archive via the ingress controller runtime socket:

1. Find the ID of the pod by calling kubectl get pods:

   nix
   kubectl get pods --namespace haproxy-controller

   nix
   kubectl get pods --namespace haproxy-controller

   output
   text
   NAME                                       READY   STATUS    RESTARTS   AGE
   haproxy-ingress-6fb9847747-5z7gq           1/1     Running   0          3h42m

   output
   text
   NAME                                       READY   STATUS    RESTARTS   AGE
   haproxy-ingress-6fb9847747-5z7gq           1/1     Running   0          3h42m

2. Issue the command to the ingress controller runtime socket to initiate creating the techdump archive:

   nix
   kubectl exec -n haproxy-controller  haproxy-ingress-6fb9847747-5z7gq -- bash -c 'echo "dump" | socat - /var/run/ingress-controller-runtime.sock'

   nix
   kubectl exec -n haproxy-controller  haproxy-ingress-6fb9847747-5z7gq -- bash -c 'echo "dump" | socat - /var/run/ingress-controller-runtime.sock'

   Note that there is no console output. If a techdump archive already exists, a new techdump archive won’t be created; you’ll have to remove the old one before running the command above.

   Info

   When run in external mode, the runtime socket may instead be located at /tmp/haproxy-ingress/run/ingress-controller-runtime.sock. This is configurable using the --runtime-dir startup argument.

3. Identify the name of the archive file with ls. The techdump archive will be located in /etc/haproxy/techdump:

   nix
   kubectl exec -n haproxy-controller  haproxy-ingress-6fb9847747-5z7gq -- ls -lrt /etc/haproxy/techdump

   nix
   kubectl exec -n haproxy-controller  haproxy-ingress-6fb9847747-5z7gq -- ls -lrt /etc/haproxy/techdump

   output
   text
   -rw-r--r-- 1 root root 13213 Jan 30 20:53 techdump.2024-01-30T20:53:03Z.haproxy-ingress-6fb9847747-5z7gq.zip

   output
   text
   -rw-r--r-- 1 root root 13213 Jan 30 20:53 techdump.2024-01-30T20:53:03Z.haproxy-ingress-6fb9847747-5z7gq.zip

   Info

   When run in external mode, the archive will be located at /tmp/haproxy-ingress/run by default. This is configurable using the --techdump-dir startup argument.

4. Use kubectl cp to copy the techdump archive file from the pod to your node. Note that the destination filename cannot contain the : character, so here we have provided a shortened destination filename:

   nix
   kubectl cp haproxy-controller/haproxy-ingress-6fb9847747-5z7gq:/etc/haproxy/techdump/techdump.2024-01-30T20:53:03Z.haproxy-ingress-6fb9847747-5z7gq.zip ./techdump.haproxy-ingress-6fb9847747-5z7gq.zip

   nix
   kubectl cp haproxy-controller/haproxy-ingress-6fb9847747-5z7gq:/etc/haproxy/techdump/techdump.2024-01-30T20:53:03Z.haproxy-ingress-6fb9847747-5z7gq.zip ./techdump.haproxy-ingress-6fb9847747-5z7gq.zip

   The archive file will be copied to the directory you specified for the destination on your node.

Create a techdump archive via the HTTP endpoint Jump to heading #

To create a techdump archive via an authenticated HTTP endpoint:

1. Install the mkpasswd tool:

   apt yum

   nix
   # mkpasswd is included in the whois package
   sudo apt install whois

   nix
   # mkpasswd is included in the whois package
   sudo apt install whois

   nix
   sudo yum install mkpasswd

   nix
   sudo yum install mkpasswd

2. Call mkpasswd with the SHA-256 algorithm to hash your password. Base64 encode the output:

   nix
   mkpasswd -m sha-256 mypassword123 | base64 | tr -d "\n"

   nix
   mkpasswd -m sha-256 mypassword123 | base64 | tr -d "\n"

   example output
   nix
   JDUkZU5pUjZvZ21QT2RnViRqVHU0alU1RXBEV0hZeUNwc3liWU1aNkw5dUQxUUhoN25Pa0R0Uk1IR3M4Cg==

   example output
   nix
   JDUkZU5pUjZvZ21QT2RnViRqVHU0alU1RXBEV0hZeUNwc3liWU1aNkw5dUQxUUhoN25Pa0R0Uk1IR3M4Cg==

   Warning

   This password (mypassword123) is for illustration purposes only. Be sure to use a more robust and secure password.

3. Save the hashed password for the next step.

4. In your YAML configuration, create a Secret. In this example, in a file named techdump-credentials.yaml, we declare a secret named techdump-credentials, since we will reference and use this Secret for connecting to the /techdump HTTP endpoint of our ingress controller. Note that the user name is techdumpuser and the password will be the base64-encoded SHA-256-encrypted password you created in the previous step. You can change the username to another value.

   techdump-credentials.yaml
   yaml
   apiVersion: v1
   kind: Secret
   metadata:
     name: techdump-credentials
     namespace: haproxy-controller
   type: Opaque
   data:
     techdumpuser: <your base64-encoded SHA-256-encrypted password>

   techdump-credentials.yaml
   yaml
   apiVersion: v1
   kind: Secret
   metadata:
     name: techdump-credentials
     namespace: haproxy-controller
   type: Opaque
   data:
     techdumpuser: <your base64-encoded SHA-256-encrypted password>

5. Apply the change with the kubectl apply command:

   nix
   kubectl apply -f techdump-credentials.yaml

   nix
   kubectl apply -f techdump-credentials.yaml

   output
   text
   secret/techdump-credentials created

   output
   text
   secret/techdump-credentials created

6. Reference the secret in your ConfigMap. The ingress controller looks for the presence of techdump-endpoint-auth-secret in data. Here we provide the name of the Secret, techdump-credentials:

   haproxy-ingress.hapee.yaml
   yaml
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: haproxy-configmap
     namespace: default
   data:
     techdump-endpoint-auth-secret: haproxy-controller/techdump-credentials

   haproxy-ingress.hapee.yaml
   yaml
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: haproxy-configmap
     namespace: default
   data:
     techdump-endpoint-auth-secret: haproxy-controller/techdump-credentials

7. Apply the change with the kubectl apply command:

   nix
   kubectl apply -f haproxy-ingress.hapee.yaml

   nix
   kubectl apply -f haproxy-ingress.hapee.yaml

   output
   text
   [...]
   configmap/haproxy-configmap configured
   [...]

   output
   text
   [...]
   configmap/haproxy-configmap configured
   [...]

   Tip

   If you installed the ingress controller with Helm, you may not have the haproxy-ingress.hapee.yaml file (to edit the ConfigMap). To edit the ConfigMap that was installed automatically, use the following command:

   nix
   kubectl edit configmap haproxy-kubernetes-ingress --namespace haproxy-controller

   nix
   kubectl edit configmap haproxy-kubernetes-ingress --namespace haproxy-controller

8. Use the kubectl apply command to apply the changes:

   nix
   kubectl apply -f haproxy-ingress.hapee.yaml

   nix
   kubectl apply -f haproxy-ingress.hapee.yaml

   output
   text
   [...]
   secret/techdump-credentials created
   [...]

   output
   text
   [...]
   secret/techdump-credentials created
   [...]

9. Use curl to call the /techdump endpoint of the ingress controller, specifying the username and password you just created. Note that you may need to specify a different IP address or NodePort for your cluster depending on your setup and the instance or node from which you are connecting.

   nix
   curl -X POST -u techdumpuser:mypassword123 http://127.0.0.1:31634/techdump

   nix
   curl -X POST -u techdumpuser:mypassword123 http://127.0.0.1:31634/techdump

   Note that if there were no errors, the command returns no output. The techdump archive has been created.

10. Find the ID of the pod by calling kubectl get pods:

    nix
    kubectl get pods --namespace haproxy-controller

    nix
    kubectl get pods --namespace haproxy-controller

    output
    text
    NAME                                       READY   STATUS    RESTARTS   AGE
    haproxy-ingress-6fb9847747-5z7gq           1/1     Running   0          3h42m

    output
    text
    NAME                                       READY   STATUS    RESTARTS   AGE
    haproxy-ingress-6fb9847747-5z7gq           1/1     Running   0          3h42m

    Tip

    If you have more than one ingress controller pod, the techdump archive may be on any one of them. Be sure to check them all to retrieve your techdump.

11. Identify the name of the archive file with ls. The techdump archive will be located in /etc/haproxy/techdump:

    nix
    kubectl exec -n haproxy-controller  haproxy-ingress-6fb9847747-5z7gq -- ls -lrt /etc/haproxy/techdump

    nix
    kubectl exec -n haproxy-controller  haproxy-ingress-6fb9847747-5z7gq -- ls -lrt /etc/haproxy/techdump

    output
    text
    -rw-r--r-- 1 root root 13213 Jan 30 20:53 techdump.2024-01-30T20:53:03Z.haproxy-ingress-6fb9847747-5z7gq.zip

    output
    text
    -rw-r--r-- 1 root root 13213 Jan 30 20:53 techdump.2024-01-30T20:53:03Z.haproxy-ingress-6fb9847747-5z7gq.zip

12. Use kubectl cp to copy the techdump archive file from the pod to your node. Note that the destination filename cannot contain the : character, so here we have provided a shortened destination filename:

    nix
    kubectl cp haproxy-controller/haproxy-ingress-6fb9847747-5z7gq:/etc/haproxy/techdump/techdump.2024-01-30T20:53:03Z.haproxy-ingress-6fb9847747-5z7gq.zip /tmp/techdump.haproxy-ingress-6fb9847747-5z7gq.zip

    nix
    kubectl cp haproxy-controller/haproxy-ingress-6fb9847747-5z7gq:/etc/haproxy/techdump/techdump.2024-01-30T20:53:03Z.haproxy-ingress-6fb9847747-5z7gq.zip /tmp/techdump.haproxy-ingress-6fb9847747-5z7gq.zip

    The archive file will be copied to the directory you specified for the destination on your node.

    Info

    When run in external mode, the archive will be located at /tmp/haproxy-ingress/run by default. This is configurable using the --techdump-dir startup argument.