Troubleshooting Traffic Mirroring

Confirm HAProxy Enterprise version

Make sure you are using a version of HAProxy Enterprise that supports traffic mirroring. The mirroring feature was introduced in version 1.9.0, so if you're using an older version, consider upgrading to a compatible release.

Check Network Connectivity

Ensure that the destination IP address and port specified for mirroring are reachable from the HAProxy Enterprise server. Verify the network connectivity between the two systems using tools like ping or telnet. If there are any firewalls, security groups, or access control lists in place, ensure they allow traffic between HAProxy Enterprise and the mirroring destination.

Monitor Resource Utilization

Traffic mirroring can be resource-intensive, especially if the mirrored traffic volume is significant. Monitor the resource utilization of the HAProxy Enterprise, including CPU, memory, and network usage. Ensure that the server has enough capacity to handle the additional load caused by mirroring.

Check SPOE mirror agent activity status

Check if the agent is running on the system:

$ ps aux | grep spoa | grep -v grep

hapee-m+   63734  0.0  0.2 756464  5900 ? Sl 12:24 0:00 /opt/hapee-extras/bin/hapee-spoa-mirror -D -r0 -uhttp://localhost:10100/ --logfile=a:/var/log/hapee-mirror.log -F /var/run/hapee-extras/hapee-spoa-mirror.pid

You can also check via the system status command:

$ systemctl status hapee-extras-spoa-mirror

● hapee-extras-spoa-mirror.service - LSB: HAPEE HTTP requests replicator
  Loaded: loaded (/etc/init.d/hapee-extras-spoa-mirror; generated)
  Active: active (running) since Thu 2023-06-01 19:48:27 UTC; 23h ago
    Docs: man:systemd-sysv-generator(8)
Main PID: 32783 (hapee-spoa-mirr)
    Tasks: 11 (limit: 9165)
  Memory: 2.9M

If the process is in an error state, you can run journalctl -xe| grep -A3 -B3 -i spoa for more details.

Check the logs

1. Search the /var/log/hapee-2.5/ logs for any issues. For example, st=0 means a successful response.

   May 31 16:59:51 ip-172-31-13-131 hapee-lb[713]:: SPOE: [mirror] <EVENT:on-frontend-http-request> sid=52 st=0 0/0/0/0/0 1/1 0/0 1/25

1. Enable logging for the mirroring agent:

   $ sudo touch /var/log/hapee-mirror.log
   $ sudo chown hapee-mirror:hapee /var/log/hapee-mirror.log

2. Use your editor to modify the startup script /etc/default/hapee-extras-spoa-mirror to add --logfile=a:. This option determines the mode of logging, which allows opening and writing at end-of-file. If a capital letter is used for the mode, then line buffering is used when writing to the log file.

   MIRROR_OPTIONS="-D -r0 -uhttp://10.0.1.4:8080/ --logfile=A:/var/log/hapee-mirror.log"