Client IP preservation

Enable transparent proxying

For scenarios where you’re proxying TCP traffic and cannot use the PROXY Protocol, you can use transparent proxying to preserve the client’s IP address when connecting to backend servers. This requires changing the default gateway on your backend servers so that return traffic passes through the load balancer, which may not be feasible in all cases.

To enable transparent proxying:

  1. On the load balancer, edit /etc/sysctl.conf and add:

    nix
    net.ipv4.ip_forward = 1
    net.ipv4.ip_nonlocal_bind = 1
    nix
    net.ipv4.ip_forward = 1
    net.ipv4.ip_nonlocal_bind = 1
  2. Restart the systemd-sysctl service.

    nix
    sudo systemctl restart systemd-sysctl
    nix
    sudo systemctl restart systemd-sysctl
  3. Add routing rules that intercept packets with a destination IP address matching a listening socket, which in this case is the client’s IP address. Also add policy-based routing rules to deliver the traffic locally.

    Create firewall and routing rules:

    nix
    sudo iptables -t mangle -N DIVERT
    sudo iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
    sudo iptables -t mangle -A PREROUTING -p udp -m socket -j DIVERT
    sudo iptables -t mangle -A DIVERT -j MARK --set-mark 1
    sudo iptables -t mangle -A DIVERT -j ACCEPT
    sudo ip rule add fwmark 1 lookup 100
    sudo ip route add local 0.0.0.0/0 dev lo table 100
    nix
    sudo iptables -t mangle -N DIVERT
    sudo iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
    sudo iptables -t mangle -A PREROUTING -p udp -m socket -j DIVERT
    sudo iptables -t mangle -A DIVERT -j MARK --set-mark 1
    sudo iptables -t mangle -A DIVERT -j ACCEPT
    sudo ip rule add fwmark 1 lookup 100
    sudo ip route add local 0.0.0.0/0 dev lo table 100

    To make the iptables changes persistent after reboot, use the iptables-save command. It saves the changes and configures the system to restore them at reboot.

    nix
    sudo apt install iptables-persistent
    sudo su -c 'iptables-save > /etc/iptables/rules.v4'
    nix
    sudo apt install iptables-persistent
    sudo su -c 'iptables-save > /etc/iptables/rules.v4'

    To verify the IP tables rules, use the iptables command. Note that the output may show tcp or the number 6, and udp or the number 17:

    nix
    sudo iptables -L -v -n -t mangle
    nix
    sudo iptables -L -v -n -t mangle
    output
    text
    Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
    pkts bytes target prot opt in out source destination
    1941 335K DIVERT tcp -- * * 0.0.0.0/0 0.0.0.0/0 socket
    0 0 DIVERT udp -- * * 0.0.0.0/0 0.0.0.0/0 socket
    ...
    Chain DIVERT (1 references)
    pkts bytes target prot opt in out source destination
    1941 335K MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK set 0x1
    1941 335K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
    output
    text
    Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
    pkts bytes target prot opt in out source destination
    1941 335K DIVERT tcp -- * * 0.0.0.0/0 0.0.0.0/0 socket
    0 0 DIVERT udp -- * * 0.0.0.0/0 0.0.0.0/0 socket
    ...
    Chain DIVERT (1 references)
    pkts bytes target prot opt in out source destination
    1941 335K MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK set 0x1
    1941 335K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0

    To make the policy route (ip rule) and route table (ip route) changes persist after reboot, your next step depends on whether or not your system uses netplan. Typically, Ubuntu uses it, but Debian doesn’t.

    • If your system uses netplan, persist the policy route (ip rule) and route table (ip route) changes in the netplan YAML configuration file located in /etc/netplan. The configuration file is probably the one having the lowest number and has a name like 00-installer-config.yaml or 01-netcfg.yaml.

      Edit the netplan YAML file, adding an lo section under the ethernets level:

      01-netcfg.yaml
      yaml
      network:
      ethernets:
      lo:
      routing-policy:
      - to: 0.0.0.0/0
      mark: 1
      table: 100
      routes:
      - to: 0.0.0.0/0
      type: local
      table: 100
      01-netcfg.yaml
      yaml
      network:
      ethernets:
      lo:
      routing-policy:
      - to: 0.0.0.0/0
      mark: 1
      table: 100
      routes:
      - to: 0.0.0.0/0
      type: local
      table: 100

      Then, use sudo netplan try and sudo netplan apply before rebooting to ensure the configuration is valid. Ignore warnings about Open vSwitch.

    • If your system doesn’t use netplan, persist the changes by creating a new service for loading them at boot. Create the systemd service file:

      nix
      sudo touch /etc/systemd/system/01-static-route.service
      sudo vi /etc/systemd/system/01-static-route.service
      nix
      sudo touch /etc/systemd/system/01-static-route.service
      sudo vi /etc/systemd/system/01-static-route.service

      Add the following lines to the service file:

      01-static-route.service
      text
      [Unit]
      Description=Add route table 100
      Wants=network-online.target
      After=network-online.target
      [Service]
      Type=oneshot
      # create the route table and rule
      ExecStart=-/usr/sbin/ip route add local 0.0.0.0/0 dev lo table 100
      ExecStart=-/usr/sbin/ip rule add fwmark 1 lookup 100
      [Install]
      WantedBy=multi-user.target
      01-static-route.service
      text
      [Unit]
      Description=Add route table 100
      Wants=network-online.target
      After=network-online.target
      [Service]
      Type=oneshot
      # create the route table and rule
      ExecStart=-/usr/sbin/ip route add local 0.0.0.0/0 dev lo table 100
      ExecStart=-/usr/sbin/ip rule add fwmark 1 lookup 100
      [Install]
      WantedBy=multi-user.target

      Set the service to start on boot:

      nix
      sudo systemctl enable 01-static-route.service
      nix
      sudo systemctl enable 01-static-route.service

      Restart the system. The saved settings will be restored after the restart.

    Create firewall rules:

    nix
    sudo firewall-cmd --permanent --direct --add-chain ipv4 mangle DIVERT
    sudo firewall-cmd --permanent --direct --add-rule ipv4 mangle PREROUTING 0 -p tcp -m socket -j DIVERT
    sudo firewall-cmd --permanent --direct --add-rule ipv4 mangle PREROUTING 0 -p udp -m socket -j DIVERT
    sudo firewall-cmd --permanent --direct --add-rule ipv4 mangle DIVERT 0 -j MARK --set-mark 1
    sudo firewall-cmd --permanent --direct --add-rule ipv4 mangle DIVERT 1 -j ACCEPT
    nix
    sudo firewall-cmd --permanent --direct --add-chain ipv4 mangle DIVERT
    sudo firewall-cmd --permanent --direct --add-rule ipv4 mangle PREROUTING 0 -p tcp -m socket -j DIVERT
    sudo firewall-cmd --permanent --direct --add-rule ipv4 mangle PREROUTING 0 -p udp -m socket -j DIVERT
    sudo firewall-cmd --permanent --direct --add-rule ipv4 mangle DIVERT 0 -j MARK --set-mark 1
    sudo firewall-cmd --permanent --direct --add-rule ipv4 mangle DIVERT 1 -j ACCEPT

    Reload the firewall:

    nix
    sudo firewall-cmd --reload
    nix
    sudo firewall-cmd --reload

    To persist the IP routing rules, create a new service for loading them at boot:

    nix
    sudo touch /etc/systemd/system/01-static-route.service
    sudo vi /etc/systemd/system/01-static-route.service
    nix
    sudo touch /etc/systemd/system/01-static-route.service
    sudo vi /etc/systemd/system/01-static-route.service

    Add the following lines to the service file:

    01-static-route.service
    text
    [Unit]
    Description=Add route table 100
    Wants=network-online.target
    After=network-online.target
    [Service]
    Type=oneshot
    # create the route table and rule
    ExecStart=-/usr/sbin/ip route add local 0.0.0.0/0 dev lo table 100
    ExecStart=-/usr/sbin/ip rule add fwmark 1 lookup 100
    [Install]
    WantedBy=multi-user.target
    01-static-route.service
    text
    [Unit]
    Description=Add route table 100
    Wants=network-online.target
    After=network-online.target
    [Service]
    Type=oneshot
    # create the route table and rule
    ExecStart=-/usr/sbin/ip route add local 0.0.0.0/0 dev lo table 100
    ExecStart=-/usr/sbin/ip rule add fwmark 1 lookup 100
    [Install]
    WantedBy=multi-user.target

    Set the service to start on boot:

    nix
    sudo systemctl enable 01-static-route.service
    nix
    sudo systemctl enable 01-static-route.service

    Restart the system.

    To verify the rule table, use the ip command:

    nix
    sudo ip rule ls
    nix
    sudo ip rule ls
    output
    text
    ...
    32765: from all fwmark 0x1 lookup 100
    ...
    output
    text
    ...
    32765: from all fwmark 0x1 lookup 100
    ...

    To verify the route table, use the ip command:

    nix
    sudo ip route ls table 100
    nix
    sudo ip route ls table 100
    output
    text
    local default dev lo scope host
    output
    text
    local default dev lo scope host
  4. Give the load balancer the capability to bind to an address with transparent proxying. You can either:

    • Add setcap cap_net_raw to the global section.

      haproxy
      global
      setcap cap_net_raw
      haproxy
      global
      setcap cap_net_raw
    • Or run as the root user by removing the user and group directives from the global section.

  5. Add a source directive to the backend section.

    • The first parameter sets the IP address to bind to before connecting to the server. You can set it to 0.0.0.0 for the load balancer to choose the appropriate address to reach the destination.
    • Set usesrc clientip to present the client’s IP address to the server.
    haproxy
    backend servers
    mode tcp
    source 0.0.0.0 usesrc clientip
    server web1 192.168.56.12:80 check
    haproxy
    backend servers
    mode tcp
    source 0.0.0.0 usesrc clientip
    server web1 192.168.56.12:80 check
  6. On the backend server, make the load balancer the default gateway. You can either:

    • Set a default gateway for traffic originating from an IP range. Below, we set the default gateway to 192.168.56.10, which is the load balancer’s address, for traffic originating on the 192.168.56.0/24 IP range. Change the address and network adapter, eth1, for your server.

      nix
      sudo ip rule add from 192.168.56.0/24 table 10
      sudo ip route add default via 192.168.56.10 dev eth1 table 10
      nix
      sudo ip rule add from 192.168.56.0/24 table 10
      sudo ip route add default via 192.168.56.10 dev eth1 table 10
    • Or change the default gateway for all traffic, which will affect SSH too.

      nix
      sudo ip route delete default
      sudo ip route add default via 192.168.56.10 dev eth1
      nix
      sudo ip route delete default
      sudo ip route add default via 192.168.56.10 dev eth1

Your backend servers should now be able to receive the traffic and see the source address as the client’s.

Do you have any suggestions on how we can improve the content of this page?