Administration
Troubleshoot
Several techniques exist for diagnosing problems with the ingress controller. They include viewing logs, viewing pod details, and gaining shell access into the pod to check files.
Enable debugging logs Jump to heading
Debugging logs give extra information that might help you troubleshoot. To enable debugging logs:
-
Modify your installation to set the ingress controller’s logging level to
debug
.bashhelm upgrade haproxy-kubernetes-ingress haproxytech/kubernetes-ingress \--namespace haproxy-controller \--set controller.logging.level=debugbashhelm upgrade haproxy-kubernetes-ingress haproxytech/kubernetes-ingress \--namespace haproxy-controller \--set controller.logging.level=debug
View pod details Jump to heading
-
View the details of the ingress controller pod by using
kubectl describe pod
:bashkubectl describe pod -n haproxy-controller haproxy-kubernetes-ingress-7d84545b45-fpzm9bashkubectl describe pod -n haproxy-controller haproxy-kubernetes-ingress-7d84545b45-fpzm9outputbashName: haproxy-kubernetes-ingress-7d84545b45-fpzm9Namespace: haproxy-controllerPriority: 0Node: minikube/192.168.99.116Start Time: Fri, 17 Jul 2020 13:06:37 -0400Labels: app.kubernetes.io/instance=kubernetes-ingressapp.kubernetes.io/name=kubernetes-ingresspod-template-hash=7d84545b45outputbashName: haproxy-kubernetes-ingress-7d84545b45-fpzm9Namespace: haproxy-controllerPriority: 0Node: minikube/192.168.99.116Start Time: Fri, 17 Jul 2020 13:06:37 -0400Labels: app.kubernetes.io/instance=kubernetes-ingressapp.kubernetes.io/name=kubernetes-ingresspod-template-hash=7d84545b45 -
You can also view the YAML that was used to deploy the ingress controller by using
kubectl get deployment
:bashkubectl get deployment haproxy-kubernetes-ingress -o yaml --namespace haproxy-controllerbashkubectl get deployment haproxy-kubernetes-ingress -o yaml --namespace haproxy-controlleroutputyamlapiVersion: apps/v1kind: Deploymentmetadata:annotations:deployment.kubernetes.io/revision: "1"creationTimestamp: "2020-07-17T17:06:37Z"generation: 1labels:app.kubernetes.io/instance: kubernetes-ingressapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/name: kubernetes-ingressapp.kubernetes.io/version: 1.4.5helm.sh/chart: kubernetes-ingress-1.4.2outputyamlapiVersion: apps/v1kind: Deploymentmetadata:annotations:deployment.kubernetes.io/revision: "1"creationTimestamp: "2020-07-17T17:06:37Z"generation: 1labels:app.kubernetes.io/instance: kubernetes-ingressapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/name: kubernetes-ingressapp.kubernetes.io/version: 1.4.5helm.sh/chart: kubernetes-ingress-1.4.2
Gain shell access into the pod Jump to heading
Use kubectl exec
to log into the ingress controller pod to inspect the files:
bash
kubectl exec -it haproxy-kubernetes-ingress-7d84545b45-fpzm9 -n haproxy-controller -- /bin/shcd /etc/haproxyls
bash
kubectl exec -it haproxy-kubernetes-ingress-7d84545b45-fpzm9 -n haproxy-controller -- /bin/shcd /etc/haproxyls
outputbash
certs haproxy.cfg maps modsec
outputbash
certs haproxy.cfg maps modsec
Enable core dumps for HAProxy Jump to heading
The HAProxy load balancer runs as a service within the HAProxy Ingress Controller Docker pod. In the rare event that the HAProxy load balancer process crashes or behaves abnormally, you can capture a core dump (also known as a crash dump). A core dump is a file that encapsulates the state of an application when it crashes and is useful in diagnosing and fixing potential issues. Core dumps are not enabled by default, so you must configure your OS settings to allow the collection of these files.
This diagnostic method can only be used for on-premises (non-cloud or otherwise managed) installations. It requires that you are able to restart your instance of Docker and that you can alter your Docker Systemd settings. It also requires that you are able to change the kernel settings of your nodes running your Kubernetes cluster.
Enable core dumps Jump to heading
Perform these steps on the nodes running your HAProxy Kubernetes Ingress Controller pods:
-
Retrieve the names of the nodes running the HAProxy Kubernetes Ingress Controller by using the following command:
bashkubectl get pod -o=custom-columns=NODE:.spec.nodeName,NAME:.metadata.name -n haproxy-controllerbashkubectl get pod -o=custom-columns=NODE:.spec.nodeName,NAME:.metadata.name -n haproxy-controlleroutputbashNODE NAMEworker1 haproxy-kubernetes-ingress-765ccd48c9-5ggmhworker2 haproxy-kubernetes-ingress-765ccd48c9-kdb62outputbashNODE NAMEworker1 haproxy-kubernetes-ingress-765ccd48c9-5ggmhworker2 haproxy-kubernetes-ingress-765ccd48c9-kdb62 -
Configure the kernel settings on your node to specify the location for saving core dumps. This location is communicated to all pods running on the instance.
This command sets the kernel setting for
core_pattern
to specify that core dump files should be saved to/tmp
:bashecho '/tmp/core.%P' | sudo tee /proc/sys/kernel/core_patternbashecho '/tmp/core.%P' | sudo tee /proc/sys/kernel/core_patternEnabling core dumps this way will prompt the system to produce a core file that will look something like
core.257
where the numbers indicate the process ID, or PID. -
Edit the Docker Systemd service file
/lib/systemd/system/docker.service
on your node and add the following to the end of the line beginning withExecStart
:bash--default-ulimit core=-1bash--default-ulimit core=-1This sets the core dump file size limit to unlimited.
-
Reload the
docker.service
Systemd configuration:bashsudo systemctl daemon-reloadbashsudo systemctl daemon-reload -
Restart Docker:
bashsudo systemctl restart dockerbashsudo systemctl restart docker
Repeat for each node in your Kubernetes cluster. Core dumps will now be enabled for all pods.
Retrieve core dumps Jump to heading
To copy the core dump file from the pod where the HAProxy load balancer faulted, use the kubectl cp
command.
bash
kubectl cp <namespace>/<pod>:/<pod_directory>/<corefile.pid> /<host_directory_destination>
bash
kubectl cp <namespace>/<pod>:/<pod_directory>/<corefile.pid> /<host_directory_destination>
For example, if you configured your node to place core files in /tmp
, then you can expect core files to appear in the /tmp
directory within the HAProxy Kubernetes Ingress Controller pod.
To locate and copy this file:
-
Find the ID of the pod by calling
kubectl get pods
:bashkubectl get pods --namespace haproxy-controllerbashkubectl get pods --namespace haproxy-controlleroutputbashNAME READY STATUS RESTARTS AGEhaproxy-kubernetes-ingress-56f55cf6d6-4tcbc 2/2 Running 0 24houtputbashNAME READY STATUS RESTARTS AGEhaproxy-kubernetes-ingress-56f55cf6d6-4tcbc 2/2 Running 0 24h -
Identify the core files by listing the directory contents with
ls
:bashkubectl exec -n haproxy-controller haproxy-kubernetes-ingress-56f55cf6d6-4tcbc -- ls -lrt /tmpbashkubectl exec -n haproxy-controller haproxy-kubernetes-ingress-56f55cf6d6-4tcbc -- ls -lrt /tmpoutputbash-rw------- 1 haproxy haproxy 6389760 Jul 25 21:25 core.257outputbash-rw------- 1 haproxy haproxy 6389760 Jul 25 21:25 core.257 -
Use
kubectl cp
to copy the file from the pod to your node:bashkubectl cp haproxy-controller/haproxy-kubernetes-ingress-56f55cf6d6-4tcbc:/tmp/core.257 /tmp/core.257bashkubectl cp haproxy-controller/haproxy-kubernetes-ingress-56f55cf6d6-4tcbc:/tmp/core.257 /tmp/core.257The core file will be copied to the directory you specified for the destination.
If this page was useful, please, Leave the feedback.