Administration

Troubleshoot

Several techniques exist for diagnosing problems with the ingress controller. They include viewing logs, viewing pod details, and gaining shell access into the pod to check files.

Enable debugging logs Jump to heading

Debugging logs give extra information that might help you troubleshoot. To enable debugging logs:

  • Modify your installation to set the ingress controller’s logging level to debug.

    bash
    helm upgrade haproxy-kubernetes-ingress haproxytech/kubernetes-ingress \
    --namespace haproxy-controller \
    --set controller.logging.level=debug
    bash
    helm upgrade haproxy-kubernetes-ingress haproxytech/kubernetes-ingress \
    --namespace haproxy-controller \
    --set controller.logging.level=debug

View pod details Jump to heading

  1. View the details of the ingress controller pod by using kubectl describe pod:

    bash
    kubectl describe pod -n haproxy-controller haproxy-kubernetes-ingress-7d84545b45-fpzm9
    bash
    kubectl describe pod -n haproxy-controller haproxy-kubernetes-ingress-7d84545b45-fpzm9
    output
    bash
    Name: haproxy-kubernetes-ingress-7d84545b45-fpzm9
    Namespace: haproxy-controller
    Priority: 0
    Node: minikube/192.168.99.116
    Start Time: Fri, 17 Jul 2020 13:06:37 -0400
    Labels: app.kubernetes.io/instance=kubernetes-ingress
    app.kubernetes.io/name=kubernetes-ingress
    pod-template-hash=7d84545b45
    output
    bash
    Name: haproxy-kubernetes-ingress-7d84545b45-fpzm9
    Namespace: haproxy-controller
    Priority: 0
    Node: minikube/192.168.99.116
    Start Time: Fri, 17 Jul 2020 13:06:37 -0400
    Labels: app.kubernetes.io/instance=kubernetes-ingress
    app.kubernetes.io/name=kubernetes-ingress
    pod-template-hash=7d84545b45
  2. You can also view the YAML that was used to deploy the ingress controller by using kubectl get deployment:

    bash
    kubectl get deployment haproxy-kubernetes-ingress -o yaml --namespace haproxy-controller
    bash
    kubectl get deployment haproxy-kubernetes-ingress -o yaml --namespace haproxy-controller
    output
    yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    annotations:
    deployment.kubernetes.io/revision: "1"
    creationTimestamp: "2020-07-17T17:06:37Z"
    generation: 1
    labels:
    app.kubernetes.io/instance: kubernetes-ingress
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kubernetes-ingress
    app.kubernetes.io/version: 1.4.5
    helm.sh/chart: kubernetes-ingress-1.4.2
    output
    yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    annotations:
    deployment.kubernetes.io/revision: "1"
    creationTimestamp: "2020-07-17T17:06:37Z"
    generation: 1
    labels:
    app.kubernetes.io/instance: kubernetes-ingress
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kubernetes-ingress
    app.kubernetes.io/version: 1.4.5
    helm.sh/chart: kubernetes-ingress-1.4.2

Gain shell access into the pod Jump to heading

Use kubectl exec to log into the ingress controller pod to inspect the files:

bash
kubectl exec -it haproxy-kubernetes-ingress-7d84545b45-fpzm9 -n haproxy-controller -- /bin/sh
cd /etc/haproxy
ls
bash
kubectl exec -it haproxy-kubernetes-ingress-7d84545b45-fpzm9 -n haproxy-controller -- /bin/sh
cd /etc/haproxy
ls
output
bash
certs haproxy.cfg maps modsec
output
bash
certs haproxy.cfg maps modsec

Enable core dumps for HAProxy Jump to heading

The HAProxy load balancer runs as a service within the HAProxy Ingress Controller Docker pod. In the rare event that the HAProxy load balancer process crashes or behaves abnormally, you can capture a core dump (also known as a crash dump). A core dump is a file that encapsulates the state of an application when it crashes and is useful in diagnosing and fixing potential issues. Core dumps are not enabled by default, so you must configure your OS settings to allow the collection of these files.

This diagnostic method can only be used for on-premises (non-cloud or otherwise managed) installations. It requires that you are able to restart your instance of Docker and that you can alter your Docker Systemd settings. It also requires that you are able to change the kernel settings of your nodes running your Kubernetes cluster.

Enable core dumps Jump to heading

Perform these steps on the nodes running your HAProxy Kubernetes Ingress Controller pods:

  1. Retrieve the names of the nodes running the HAProxy Kubernetes Ingress Controller by using the following command:

    bash
    kubectl get pod -o=custom-columns=NODE:.spec.nodeName,NAME:.metadata.name -n haproxy-controller
    bash
    kubectl get pod -o=custom-columns=NODE:.spec.nodeName,NAME:.metadata.name -n haproxy-controller
    output
    bash
    NODE NAME
    worker1 haproxy-kubernetes-ingress-765ccd48c9-5ggmh
    worker2 haproxy-kubernetes-ingress-765ccd48c9-kdb62
    output
    bash
    NODE NAME
    worker1 haproxy-kubernetes-ingress-765ccd48c9-5ggmh
    worker2 haproxy-kubernetes-ingress-765ccd48c9-kdb62
  2. Configure the kernel settings on your node to specify the location for saving core dumps. This location is communicated to all pods running on the instance.

    This command sets the kernel setting for core_pattern to specify that core dump files should be saved to /tmp:

    bash
    echo '/tmp/core.%P' | sudo tee /proc/sys/kernel/core_pattern
    bash
    echo '/tmp/core.%P' | sudo tee /proc/sys/kernel/core_pattern

    Enabling core dumps this way will prompt the system to produce a core file that will look something like core.257 where the numbers indicate the process ID, or PID.

  3. Edit the Docker Systemd service file /lib/systemd/system/docker.service on your node and add the following to the end of the line beginning with ExecStart:

    bash
    --default-ulimit core=-1
    bash
    --default-ulimit core=-1

    This sets the core dump file size limit to unlimited.

  4. Reload the docker.service Systemd configuration:

    bash
    sudo systemctl daemon-reload
    bash
    sudo systemctl daemon-reload
  5. Restart Docker:

    bash
    sudo systemctl restart docker
    bash
    sudo systemctl restart docker

Repeat for each node in your Kubernetes cluster. Core dumps will now be enabled for all pods.

Retrieve core dumps Jump to heading

To copy the core dump file from the pod where the HAProxy load balancer faulted, use the kubectl cp command.

bash
kubectl cp <namespace>/<pod>:/<pod_directory>/<corefile.pid> /<host_directory_destination>
bash
kubectl cp <namespace>/<pod>:/<pod_directory>/<corefile.pid> /<host_directory_destination>

For example, if you configured your node to place core files in /tmp, then you can expect core files to appear in the /tmp directory within the HAProxy Kubernetes Ingress Controller pod.

To locate and copy this file:

  1. Find the ID of the pod by calling kubectl get pods:

    bash
    kubectl get pods --namespace haproxy-controller
    bash
    kubectl get pods --namespace haproxy-controller
    output
    bash
    NAME READY STATUS RESTARTS AGE
    haproxy-kubernetes-ingress-56f55cf6d6-4tcbc 2/2 Running 0 24h
    output
    bash
    NAME READY STATUS RESTARTS AGE
    haproxy-kubernetes-ingress-56f55cf6d6-4tcbc 2/2 Running 0 24h
  2. Identify the core files by listing the directory contents with ls:

    bash
    kubectl exec -n haproxy-controller haproxy-kubernetes-ingress-56f55cf6d6-4tcbc -- ls -lrt /tmp
    bash
    kubectl exec -n haproxy-controller haproxy-kubernetes-ingress-56f55cf6d6-4tcbc -- ls -lrt /tmp
    output
    bash
    -rw------- 1 haproxy haproxy 6389760 Jul 25 21:25 core.257
    output
    bash
    -rw------- 1 haproxy haproxy 6389760 Jul 25 21:25 core.257
  3. Use kubectl cp to copy the file from the pod to your node:

    bash
    kubectl cp haproxy-controller/haproxy-kubernetes-ingress-56f55cf6d6-4tcbc:/tmp/core.257 /tmp/core.257
    bash
    kubectl cp haproxy-controller/haproxy-kubernetes-ingress-56f55cf6d6-4tcbc:/tmp/core.257 /tmp/core.257

    The core file will be copied to the directory you specified for the destination.

If this page was useful, please, Leave the feedback.

© 2023 HAProxy Technologies, LLC. All Rights Reserved
Manage Cookie Preferences