Transparent Proxy
HAProxy works has a reverse-proxy. Which means it maintains 2 connections when allowing a client to cross it:
– 1 connection between HAProxy and the client
– 1 connection between HAProxy and the server
HAProxy then manipulate buffers between these two connections.
One of the drawback of this mode is that HAProxy will let the kernel to establish the connection to the server. The kernel is going to use a local IP address to do this.
Because of this, HAProxy “hides” the client IP by its own one: this can be an issue in some cases.
Here comes the transparent proxy mode: HAProxy can be configured to spoof the client IP address when establishing the TCP connection to the server. That way, the server thinks the connection comes from the client directly (of course, the server must answer back to HAProxy and not to the client, otherwise it can’t work: the client will get an acknowledge from the server IP while it has established the connection on HAProxy‘s IP).
Transparent binding
By default, when one want HAProxy to get traffic, we have to tell it to bind an IP address and a port.
The IP address must exist on the operating system (unless you have setup the sysctl net.ipv4.ip_nonlocal_bind) and the OS must announce the availability to the other devices on the network through ARP protocol.
Well, in some cases we want HAProxy to be able to catch traffic on the fly without configuring any IP address or VRRP or whatever…
This is where transparent binding comes in: HAProxy can be configured to catch traffic on the fly even if the destination IP address is not configured on the server.
These IP addresses will never be pingable, but they’ll deliver the services configured in HAProxy.
HAProxy and the Linux Kernel
Unfortunately, HAProxy can’t do transparent binding or proxying alone. It must stand on a compiled and tuned Linux Kernel and operating system.
Below, I’ll explain how to do this in a standard Linux distribution.
Here is the check list to meet:
1. appropriate HAProxy compilation option
2. appropriate Linux Kernel compilation option
3. sysctl settings
4. iptables rules
5. ip route rules
6. HAProxy configuration
HAProxy compilation requirements
First of all, HAProxy must be compiled with the option TPROXY enabled.
It is enabled by default when you use the target LINUX26 or LINUX2628.
Linux Kernel requirements
You have to ensure your kernel has been compiled with the following options:
– CONFIG_NETFILTER_TPROXY
– CONFIG_NETFILTER_XT_TARGET_TPROXY
Of course, iptables must be enabled as well in your kernel 🙂
sysctl settings
The following sysctls must be enabled:
– net.ipv4.ip_forward
– net.ipv4.ip_nonlocal_bind
iptables rules
You must setup the following iptables rules:
iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT
Purpose is to mark packets which matches a socket bound locally (by HAProxy).
IP route rules
Then, tell the Operating System to forward packets marked by iptables to the loopback where HAProxy can catch them:
ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100
HAProxy configuration
Finally, you can configure HAProxy.
* Transparent binding can be configured like this:
[...] frontend ft_application bind 1.1.1.1:80 transparent [...]
* Transparent proxying can be configured like this:
[...] backend bk_application source 0.0.0.0 usesrc clientip [...]
Transparent mode in the ALOHA Load-Balancer
Now, the same steps in the ALOHA Load-balancer, which is an HAProxy based load-balancing appliance:
1-5. not required, the ALOHA kernel is deeply tuned for this purpose
6. HAProxy configuration
LB Admin tab (AKA click mode)
* Transparent binding can be configured like this, when editing a Frontend listener:
* Transparent proxying can be configured like this when editing a farm:
LB Layer 7 tab (vi in a browser mode)
* Transparent binding can be configured like this:
[...] frontend ft_application bind 1.1.1.1:80 transparent [...]
* Transparent proxying can be configured like this:
[...] backend bk_application source 0.0.0.0 usesrc clientip [...]
am having an issue with this. I have setup a galera cluster, 5 machines total 2 with haproxy and 3 with mariadb 10. These are all rhel 7 machines.
The kernel is version 3.10, with 1.5.14 of haproxy. Looking at the documentation this tells me this should work, as it is enabled by default in the kernel and the target system for haproxy should enable it.
when I add the line:
source 0.0.0.0 usersrc clientip
As well as the iptables and route rules I start to get the error :
ERROR 2013 (HY000): Lost connection to MySQL server at ‘reading initial communication packet’, system error: 0 “Internal error/check (Not system error)”
This is my haproxy configuration file
# This file managed by Puppet
global
chroot /var/lib/haproxy
daemon
group haproxy
log 10.76.10.99 local0
maxconn 4000
pidfile /var/run/haproxy.pid
stats socket /var/lib/haproxy/stats
# user haproxy
defaults
log global
maxconn 8000
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
stats enable
timeout queue 1m
timeout connect 10s
timeout client 10m
timeout server 10m
timeout check 10s
frontend mysqlcl
bind 10.76.10.105:3306 transparent
mode tcp
default_backend galera-cluster
backend galera-cluster
balance roundrobin
option httpchk
source 0.0.0.0 usesrc clientip
server mysqlcl00.lib.ncsu.edu 10.76.10.101:3306 check fall 2 inter 5000 maxconn 5000 port 9200 rise 2 weight 1
server mysqlcl01.lib.ncsu.edu 10.76.10.102:3306 check fall 2 inter 5000 maxconn 5000 port 9200 rise 2 weight 1
server mysqlcl02.lib.ncsu.edu 10.76.10.103:3306 check fall 2 inter 5000 maxconn 5000 port 9200 rise 2 weight 1
listen mysqlcl-web
bind 10.76.10.105:80
balance roundrobin
option tcplog
option ssl-hello-chk
Hi,
Can you confirm your galera clusters use HAProxy as their default gateway?
They are not.
It’s a requirement. It can’t work without it.
Do you happen to have a link to a document for the best way to do that? When I try and set that up I get ERROR 2013 (HY000): Lost connection to MySQL server at ‘reading initial communication packet’, system error: 0 “Internal error/check (Not system error)”, just in testing before I even set the gateway to the haproxy servers.
Hi Baptiste,
Our support manager Aaron just posted a blog about how to do single subnet TPROXY with HAPrxoy , which might be useful if people can’t change the default gateway for some reason?
I should point out that both haproxy machines and the galera cluster are on the same subnets.
How would I setup those iptables rules with firewalld?
No idea 🙂
# cat /etc/firewalld/direct.xml
-p tcp -m socket -j DIVERT
-j MARK –set-mark 1
-j ACCEPT
Does 1.1.1.1:80 represent a physical IP address or is this an IP like 0.0.0.0 (anywhere)?
I configured haproxy and kernel to work as TPROXY as described above.
everything is working well except that now the backend must be open to public ip since
it receives the client ip as source. Is there any haproxy option to show to the backend
that the real request come from the haproxy server?
thanks
Can you clarify this? If you want the mysql server to see the connection as from the haproxy server remove the mangled IPtables entrys or remove the clientIP from the back end, then it will see every connection as coming from the proxy.. but that introduces a security issue.
with the above method you should not need to “open up” the backend to the source IPs, it is just passed through the proxy server and seen a such, you just need to tell mysql the ‘user’@’source’ you want to be able to connect to it.
is there a way to make the backend to know that the real request come from the haproxy server and not open them to the public?