Troubleshoot

Several techniques exist for diagnosing problems with the ingress controller. They include viewing logs, viewing pod details, and gaining shell access into the pod to check files.

Enable debugging logs Jump to heading #

Debugging logs give extra information that might help you troubleshoot. To enable debugging logs:

• Modify your installation to set the ingress controller’s logging level to debug.

  bash
  helm upgrade haproxy-kubernetes-ingress haproxytech/kubernetes-ingress \
    --namespace haproxy-controller \
    --set controller.logging.level=debug 

  bash
  helm upgrade haproxy-kubernetes-ingress haproxytech/kubernetes-ingress \
    --namespace haproxy-controller \
    --set controller.logging.level=debug 

View pod details Jump to heading #

1. View the details of the ingress controller pod by using kubectl describe pod:

   bash
   kubectl describe pod -n haproxy-controller haproxy-kubernetes-ingress-7d84545b45-fpzm9

   bash
   kubectl describe pod -n haproxy-controller haproxy-kubernetes-ingress-7d84545b45-fpzm9

   output
   bash
   Name:         haproxy-kubernetes-ingress-7d84545b45-fpzm9
   Namespace:    haproxy-controller
   Priority:     0
   Node:         minikube/192.168.99.116
   Start Time:   Fri, 17 Jul 2020 13:06:37 -0400
   Labels:       app.kubernetes.io/instance=kubernetes-ingress
                 app.kubernetes.io/name=kubernetes-ingress
                 pod-template-hash=7d84545b45

   output
   bash
   Name:         haproxy-kubernetes-ingress-7d84545b45-fpzm9
   Namespace:    haproxy-controller
   Priority:     0
   Node:         minikube/192.168.99.116
   Start Time:   Fri, 17 Jul 2020 13:06:37 -0400
   Labels:       app.kubernetes.io/instance=kubernetes-ingress
                 app.kubernetes.io/name=kubernetes-ingress
                 pod-template-hash=7d84545b45

2. You can also view the YAML that was used to deploy the ingress controller by using kubectl get deployment:

   bash
   kubectl get deployment haproxy-kubernetes-ingress -o yaml --namespace haproxy-controller

   bash
   kubectl get deployment haproxy-kubernetes-ingress -o yaml --namespace haproxy-controller

   output
   yaml
   apiVersion: apps/v1
   kind: Deployment
   metadata:
   annotations:
     deployment.kubernetes.io/revision: "1"
   creationTimestamp: "2020-07-17T17:06:37Z"
   generation: 1
   labels:
     app.kubernetes.io/instance: kubernetes-ingress
     app.kubernetes.io/managed-by: Helm
     app.kubernetes.io/name: kubernetes-ingress
     app.kubernetes.io/version: 1.4.5
     helm.sh/chart: kubernetes-ingress-1.4.2

   output
   yaml
   apiVersion: apps/v1
   kind: Deployment
   metadata:
   annotations:
     deployment.kubernetes.io/revision: "1"
   creationTimestamp: "2020-07-17T17:06:37Z"
   generation: 1
   labels:
     app.kubernetes.io/instance: kubernetes-ingress
     app.kubernetes.io/managed-by: Helm
     app.kubernetes.io/name: kubernetes-ingress
     app.kubernetes.io/version: 1.4.5
     helm.sh/chart: kubernetes-ingress-1.4.2

Gain shell access into the pod Jump to heading #

Use kubectl exec to log into the ingress controller pod to inspect the files:

bash
kubectl exec -it haproxy-kubernetes-ingress-7d84545b45-fpzm9 -n haproxy-controller -- /bin/sh
cd /etc/haproxy
ls

bash
kubectl exec -it haproxy-kubernetes-ingress-7d84545b45-fpzm9 -n haproxy-controller -- /bin/sh
cd /etc/haproxy
ls

output
bash
certs  haproxy.cfg  maps  modsec

output
bash
certs  haproxy.cfg  maps  modsec

Enable core dumps for HAProxy Jump to heading #

The HAProxy load balancer runs as a service within the HAProxy Ingress Controller Docker pod. In the rare event that the HAProxy load balancer process crashes or behaves abnormally, you can capture a core dump (also known as a crash dump). A core dump is a file that encapsulates the state of an application when it crashes and is useful in diagnosing and fixing potential issues. Core dumps are not enabled by default, so you must configure your OS settings to allow the collection of these files.

This diagnostic method can only be used for on-premises (non-cloud or otherwise managed) installations. It requires that you are able to restart your instance of Docker and that you can alter your Docker Systemd settings. It also requires that you are able to change the kernel settings of your nodes running your Kubernetes cluster.

Enable core dumps Jump to heading #

Perform these steps on the nodes running your HAProxy Kubernetes Ingress Controller pods:

1. Retrieve the names of the nodes running the HAProxy Kubernetes Ingress Controller by using the following command:

   bash
   kubectl get pod -o=custom-columns=NODE:.spec.nodeName,NAME:.metadata.name -n haproxy-controller

   bash
   kubectl get pod -o=custom-columns=NODE:.spec.nodeName,NAME:.metadata.name -n haproxy-controller

   output
   bash
   NODE     NAME
   worker1  haproxy-kubernetes-ingress-765ccd48c9-5ggmh
   worker2  haproxy-kubernetes-ingress-765ccd48c9-kdb62

   output
   bash
   NODE     NAME
   worker1  haproxy-kubernetes-ingress-765ccd48c9-5ggmh
   worker2  haproxy-kubernetes-ingress-765ccd48c9-kdb62

2. Configure the kernel settings on your node to specify the location for saving core dumps. This location is communicated to all pods running on the instance.

   This command sets the kernel setting for core_pattern to specify that core dump files should be saved to /tmp:

   bash
   echo '/tmp/core.%P' | sudo tee /proc/sys/kernel/core_pattern

   bash
   echo '/tmp/core.%P' | sudo tee /proc/sys/kernel/core_pattern

   Enabling core dumps this way will prompt the system to produce a core file that will look something like core.257 where the numbers indicate the process ID, or PID.

3. Edit the Docker Systemd service file /lib/systemd/system/docker.service on your node and add the following to the end of the line beginning with ExecStart:

   bash
   --default-ulimit core=-1

   bash
   --default-ulimit core=-1

   This sets the core dump file size limit to unlimited.

4. Reload the docker.service Systemd configuration:

   bash
   sudo systemctl daemon-reload

   bash
   sudo systemctl daemon-reload

5. Restart Docker:

   bash
   sudo systemctl restart docker

   bash
   sudo systemctl restart docker

Repeat for each node in your Kubernetes cluster. Core dumps will now be enabled for all pods.

Retrieve core dumps Jump to heading #

To copy the core dump file from the pod where the HAProxy load balancer faulted, use the kubectl cp command.

bash
kubectl cp <namespace>/<pod>:/<pod_directory>/<corefile.pid> /<host_directory_destination>

bash
kubectl cp <namespace>/<pod>:/<pod_directory>/<corefile.pid> /<host_directory_destination>

For example, if you configured your node to place core files in /tmp, then you can expect core files to appear in the /tmp directory within the HAProxy Kubernetes Ingress Controller pod.

To locate and copy this file:

1. Find the ID of the pod by calling kubectl get pods:

   bash
   kubectl get pods --namespace haproxy-controller

   bash
   kubectl get pods --namespace haproxy-controller

   output
   bash
   NAME                                          READY   STATUS    RESTARTS   AGE
   haproxy-kubernetes-ingress-56f55cf6d6-4tcbc   2/2     Running   0          24h

   output
   bash
   NAME                                          READY   STATUS    RESTARTS   AGE
   haproxy-kubernetes-ingress-56f55cf6d6-4tcbc   2/2     Running   0          24h

2. Identify the core files by listing the directory contents with ls:

   bash
   kubectl exec -n haproxy-controller haproxy-kubernetes-ingress-56f55cf6d6-4tcbc -- ls -lrt /tmp

   bash
   kubectl exec -n haproxy-controller haproxy-kubernetes-ingress-56f55cf6d6-4tcbc -- ls -lrt /tmp

   output
   bash
   -rw-------    1 haproxy  haproxy    6389760 Jul 25 21:25 core.257

   output
   bash
   -rw-------    1 haproxy  haproxy    6389760 Jul 25 21:25 core.257

3. Use kubectl cp to copy the file from the pod to your node:

   bash
   kubectl cp haproxy-controller/haproxy-kubernetes-ingress-56f55cf6d6-4tcbc:/tmp/core.257 /tmp/core.257

   bash
   kubectl cp haproxy-controller/haproxy-kubernetes-ingress-56f55cf6d6-4tcbc:/tmp/core.257 /tmp/core.257

   The core file will be copied to the directory you specified for the destination.