Scalable architecture
In order to make your architecture scalable, you may often want to use a load-balancer or an application delivery controller.
When using one of them (or a reverse-proxy), the client information is almost all the time hidden. Or if you want to get them, it requires huge modifications in the architecture.
Unfortunately, for troubleshooting and security purpose it would be interesting to know the client information (mainly source IP address)…
That’s where the proxy protocol comes in.
The proxy protocol???
As explained in a previous article“preserve source ip address despite reverse proxies”, the proxy protocol was developped to maintain client information when chaining proxies and reverse-proxies.
Two main advantages when using it:
- you can provide a downstream proxy or server (aka next hop) the client information (for now, mainly IP and port source)
- you can use servers in multiple datacenter without a complex network architecture (just need to provide routing for a TCP connection)
Why simply not using TPROXY (transparent proxy) mode?
TPROXY allows a load-balancer or reverse-proxy to open the TCP connection to the server using the client IP address.
One of the drawback of TPROXY is that the default gateway of the application servers must be the load-balancer.
Or you must do policy based routing on your network which could be painful.
Why Postfix and HAProxy?
HAProxy was the first software to implement the proxy protocol.
Note that you’ll have to use HAProxy 1.5 branch or patched HAProxy 1.4.
< advertisement >
An other solution would to use Aloha load-balancer which does everything for you in a box (from the OS to HAPrxoxy) with all the nice features you could expect. 😉
< /advertisement >
Lately, Postfix implemented it. It is available in Postfix 2.10.
It is the first application server first application server to ship with it: THANKS and CONGRATULATION!!!!
Hopefully other MTAs will implement it soon. It is simple and brings so many improvements to an architecture.
SMTP, spam and security
In SMTP, it is really important to know the client IP, since we use it most of the time through RBL to fight spam.
For security purpose as well: we may want to allow only some hosts to use our SMTP relays and block any other clients.
Without the proxy protocol, the load-balancer will hide the client IP with its own IP. You would have to maintain whitelists into the load-balancer (which is doable). Thanks to proxy protocol, Postscreen would be aware of the client IP, it means you could maintain lists directly into the MTA.
HAProxy and Postfix connection flow
The diagram below shows the protocols and the process in place in this kind of architecture:
smtp proxy-protocol + smtp (INTERNET) ---> 25 (HAPROXY) ---> srv1:10024 (Postscreen / smtpd) ---> srv2:10024 (Postscreen / smtpd)
Note that the default gateway of the MTA servers is not anymore the load-balancer.
Both servers migt be in the same LAN or datacenter. Any type of architecture is now doable.
Configuration
HAProxy
frontend ft_smtp bind 0.0.0.0:25 mode tcp timeout client 1m log global option tcplog default_backend bk_postfix backend bk_postfix mode tcp log global option tcplog timeout server 1m timeout connect 5s server postfix 127.0.0.1:10024 send-proxy
Postfix
Note: I installed postfix in /opt/postfix directory
main.cf
queue_directory = /opt/postfix/var/spool/postfix command_directory = /opt/postfix/usr/sbin daemon_directory = /opt/postfix/usr/libexec/postfix data_directory = /opt/postfix/var/lib/postfix mail_owner = postfix unknown_local_recipient_reject_code = 550 inet_interfaces = localhost sendmail_path = /opt/postfix/usr/sbin/sendmail newaliases_path = /opt/postfix/usr/bin/newaliases mailq_path = /opt/postfix/usr/bin/mailq setgid_group = postdrop html_directory = no manpage_directory = /opt/postfix/usr/local/man sample_directory = /opt/postfix/etc/postfix readme_directory = no inet_protocols = ipv4 postscreen_upstream_proxy_protocol = haproxy
master.cf
10024 inet n - n - 1 postscreen smtpd pass - - n - - smtpd
See the results in Postfix logs
No proxy protocol
Jun 30 01:18:14 sd-33932 postfix/postscreen[2375]: CONNECT from [127.0.0.1]:52841 to [127.0.0.1]:10024 Jun 30 01:18:22 sd-33932 postfix/smtpd[2376]: disconnect from localhost[127.0.0.1]
With proxy protocol
Jun 29 09:13:41 sd-33932 postfix/postscreen[30505]: CONNECT from [<client public IP>]:59338 to [<server IP>]:25 Jun 29 09:13:52 sd-33932 postfix/postscreen[30505]: DISCONNECT [<client public IP>]:59338
Hats off to you for this elaborated information about SMTP relay infrastructure with Postfix and load-balancers. It looks you have complete hold on this topic. Thanks!
I receive a “502 5.5.2 Error: command not recognized” error when adding ‘send-proxy’ to the HAPRoxy configuration. I am running HAProxy 1.5-dev12 and Postfix 2.9.0 on CentOS, as per the instructions above.
Any ideas?
Hi Jason,
Actually, proxy protocol has been implemented in postfix 2.10: http://blog.haproxy.com/proxy-protocol/
Let me update the article with this information.
cheers
Thanks for your speedy reply! After some time of research (only an hour ago), I worked out that it was implemented in Postfix 2.10 as per the article you linked and others online I did read somewhere that the proxy functionality was introduced in Postfix 2.8 – either it was mistaken or it was referring to something ever so different.
I built an RPM for Postfix 2.10 and all is working well! Thank you for your cool post 🙂
Hello jason,
Can you please provide me a complete configuration details for all three servers.
Hello,
thanks for your tutorial! I’m getting a problem when trying to add spamassassin and amavisd-new layer to my postfix cluster. I get: haproxy read: unexpected protocol header. It seems like I can’t anymore connect directly to a single postfix without using SMTP Proxy. Any ideas?
Thanks
Hi Alexio,
No idea, I’m sorry.
Please post your conifgurations to HAProxy ML, more suitable to get some help.
Baptiste
Is there any kind of health check possible for postfix, rather than just connect timeout?
PS: I don’t fully understand the setup as of now.. Hope you would help. Thanks
You guys are genius. I used the TCP Proxy mod for nginx before. For like 4 days. Then I found this, very nice! Thank you!
Hello, whether the proxy protocol easy support the UDP proxy ?
Thanks.
HAProxy does not support UDP, so this does not apply.
That said, technically, yes, proxy protocol could be implemented on UDP, but both client and server must be compatible.
Hi there, its grate to sounds like HAProxy can loadbalancing smtp servers. I have to install two data store server for example imap1 and imap2. Problem is users are stored A-M on imap1 and N-Z to imap2. It is possible to do with HAProxy?
Hi,
no, this is not possible with HAProxy.