HAProxy Kubernetes Ingress Controller Documentation 1.7

How to troubleshoot

Several techniques exist for diagnosing problems with the ingress controller. They include viewing logs, viewing pod details, and logging into the pod to check files.

View the logs

  1. Get the names of the haproxy-kubernetes-ingress pods:

    $ kubectl get pods --namespace haproxy-controller

    output

    NAME                                         READY   STATUS    RESTARTS   AGE
haproxy-kubernetes-ingress-7d84545b45-fpzm9  1/1     Running   0          3m49s
haproxy-kubernetes-ingress-7d84545b45-wq426  1/1     Running   0          3m49s

  2. Use kubectl logs to view the logs of a pod:

    $ kubectl logs haproxy-kubernetes-ingress-7d84545b45-fpzm9

    output

    2020/07/17 17:07:47
_   _    _    ____                        _____ _____
| | | |  / \  |  _ \ _ __ _____  ___   _  | ____| ____|
| |_| | / _ \ | |_) | '__/ _ \ \/ / | | | |  _| |  _|
|  _  |/ ___ \|  __/| | | (_) >  <| |_| | | |___| |___
|_| |_/_/   \_\_|   |_|  \___/_/\_\\__, | |_____|_____|
_  __     _                       |___/             ___ ____
| |/ /   _| |__   ___ _ __ _ __   ___| |_ ___  ___  |_ _/ ___|
| ' / | | | '_ \ / _ \ '__| '_ \ / _ \ __/ _ \/ __|  | | |
| . \ |_| | |_) |  __/ |  | | | |  __/ ||  __/\__ \  | | |___
|_|\_\__,_|_.__/ \___|_|  |_| |_|\___|\__\___||___/ |___\____|
   
   
2020/07/17 17:07:47 HAProxy Ingress Controller v1.4.5-ee1 c2b5156
   
2020/07/17 17:07:47 Build from: ssh://git@gitlab.int.haproxy.com/haproxy-controller/kubernetes-ingress-ee
2020/07/17 17:07:47 Build date: 2020-07-12T00:00:53
   
2020/07/17 17:07:47 ConfigMap: default/kubernetes-ingress
2020/07/17 17:07:47 Ingress class:
2020/07/17 17:07:47 Publish service:
2020/07/17 17:07:47 Default backend service: default/haproxy-kubernetes-ingress-default-backend
2020/07/17 17:07:47 Default ssl certificate: default/haproxy-kubernetes-ingress-default-cert
2020/07/17 17:07:47 Controller sync period: 5s
2020/07/17 17:07:47 controller.go:262 Running with HA-Proxy version 2.1.0-1.0.0-223.185 2020/06/18 - https://haproxy.org/
2020/07/17 17:07:47 INFO    controller.go:267 Starting HAProxy with /etc/haproxy/haproxy.cfg
2020/07/17 17:07:47 INFO    controller.go:272 Running on haproxy-kubernetes-ingress-7d84545b45-fpzm9
[NOTICE] 198/170747 (21) : New worker #1 (22) forked
2020/07/17 17:07:48 INFO    controller.go:95 Running on Kubernetes version: v1.18.3 linux/amd64
2020/07/17 17:07:48 INFO    monitor.go:35 executing syncPeriod every 5s
2020/07/17 17:07:54 INFO    controller.go:220 HAProxy reloaded

  3. For debug logs, upgrade the ingress controller with the log argument set to debug.

    $ helm upgrade haproxy-kubernetes-ingress haproxytech/kubernetes-ingress \
    --namespace haproxy-controller \
    --set controller.logging.level=debug

  4. By default, the ingress controller’s logs do not include traffic logs, but you can enable them by setting the syslog-server option to stdout.

View pod details

  1. View the details of the ingress controller pod by using kubectl describe pod:

    $ kubectl describe pod haproxy-kubernetes-ingress-7d84545b45-fpzm9

    output

    Name:         haproxy-kubernetes-ingress-7d84545b45-fpzm9
Namespace:    haproxy-controller
Priority:     0
Node:         minikube/192.168.99.116
Start Time:   Fri, 17 Jul 2020 13:06:37 -0400
Labels:       app.kubernetes.io/instance=kubernetes-ingress
              app.kubernetes.io/name=kubernetes-ingress
              pod-template-hash=7d84545b45

  2. You can also view the YAML that was used to deploy the ingress controller by using kubectl get deployment:

    $ kubectl get deployment haproxy-kubernetes-ingress -o yaml --namespace haproxy-controller

    output

    apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
  deployment.kubernetes.io/revision: "1"
creationTimestamp: "2020-07-17T17:06:37Z"
generation: 1
labels:
  app.kubernetes.io/instance: kubernetes-ingress
  app.kubernetes.io/managed-by: Helm
  app.kubernetes.io/name: kubernetes-ingress
  app.kubernetes.io/version: 1.4.5
  helm.sh/chart: kubernetes-ingress-1.4.2

Log into the pod

  1. Use kubectl exec to log into the ingress controller pod to inspect the HAProxy files:

    $ kubectl exec -it haproxy-kubernetes-ingress-7d84545b45-fpzm9 /bin/bash
$ cd /etc/haproxy
$ ls

    output

    certs  haproxy.cfg  maps  modsec

Enable core dumps for HAProxy

The HAProxy load balancer runs as a service within the HAProxy Ingress Controller Docker pod. In the rare event that the HAProxy load balancer process crashes or behaves abnormally, you can capture a core dump (also known as a crash dump). A core dump is a file that encapsulates the state of an application when it crashes and is useful in diagnosing and fixing potential issues. Core dumps are not enabled by default, so you must configure your OS settings to allow the collection of these files.

Note: This diagnostic method can only be used for on-premises (non-cloud or otherwise managed) installations. It requires that you are able to restart your instance of Docker and that you can alter your Docker Systemd settings. It also requires that you are able to change the kernel settings of your nodes running your Kubernetes cluster.

Enable core dumps

Perform these steps on the nodes running your HAProxy Kubernetes Ingress Controller pods. You can retrieve the IDs for the pods running the HAProxy Kubernetes Ingress Controller using the following command:

$ kubectl get pod -o=custom-columns=NODE:.spec.nodeName,NAME:.metadata.name -n haproxy-controller

output

NODE           NAME
worker1        haproxy-kubernetes-ingress-765ccd48c9-5ggmh
worker2        haproxy-kubernetes-ingress-765ccd48c9-kdb62

  1. Configure the kernel settings on your node to specify the location for saving core dumps. This location is communicated to all pods running on the instance.

    This command sets the kernel setting for core_pattern to specify that core dump files should be saved to /tmp.

    $ echo '/tmp/core.%P' | sudo tee /proc/sys/kernel/core_pattern

    Note: Enabling core dumps this way will prompt the system to produce a core file that will look something like core.257 where the numbers indicate the process ID, or PID.

  2. Edit the Docker Systemd service file /lib/systemd/system/docker.service on your node and add the following to the end of the line beginning with ExecStart:

    --default-ulimit core=-1

    This sets the core dump file size limit to unlimited.

  3. Reload the docker.service Systemd configuration.

    $ sudo systemctl daemon-reload

  4. Restart Docker.

    $ sudo systemctl restart docker

Repeat for each node in your Kubernetes cluster. Core dumps will now be enabled for all pods.

Retrieve core dumps

To copy the core dump file from the pod where the HAProxy load balancer faulted, use the kubectl cp command.

kubectl cp <namespace>/<pod>:/<pod_directory>/<corefile.pid> /<host_directory_destination>

For example, if you configured your node to place core files in /tmp, then you can expect core files to appear in the /tmp directory within the HAProxy Kubernetes Ingress Controller pod.

To locate and copy this file:

  1. Find the ID of the pod.

    kubectl get pods --namespace haproxy-controller

    output

    NAME                                          READY   STATUS    RESTARTS   AGE
haproxy-kubernetes-ingress-56f55cf6d6-4tcbc   2/2     Running   0          24h

  2. Identify the core files.

    kubectl exec -n haproxy-controller haproxy-kubernetes-ingress-56f55cf6d6-4tcbc -- ls -lrt /tmp

    output

    -rw-------    1 haproxy  haproxy    6389760 Jul 25 21:25 core.257

  3. Use kubectl cp to copy the file from the pod to your node.

    kubectl cp haproxy-controller/haproxy-kubernetes-ingress-56f55cf6d6-4tcbc:/tmp/core.257 /tmp/core.257

    The core file will be copied to the directory you specified for the destination.