How to Create Powerful, Customized Lua Mailers in HAProxy

SysOps teams know how crucial observability is to maintaining around-the-clock infrastructure health. And while HAProxy already provides reliable logging and stats (for diagnostics and monitoring), what if you could supplement those with a real-time alerting system? 

That idea inspired our initial inclusion of mailers in HAProxy. Since then, users have been asking for deeper mailer customizability—and we’re thrilled to deliver that with Lua-based email alerts in HAProxy 2.8. In this article, we’ll explore our new Lua implementation and some practical ways to experiment with mailers.

Why Should I Use HAProxy Mailers?

Servers form the backbone of modern services. It’s also inevitable that your servers will encounter problems, or experience state changes, throughout their lifetimes. Alternatively, there are times where you might purposefully alter server states for testing and maintenance purposes. How do we monitor all of these impactful changes as they occur?

Luckily, teams can use email alerts to highlight impactful backend issues in real time—while ignoring low-priority events—and restore functionality as quickly as possible. HAProxy and Lua help boost observability and provide yet another way to triage issues as they occur. That means better uptime and performance for your web applications while bringing you ever closer to that coveted 99.999% availability. 

Note that mailers aren’t a replacement for verbose logging. However, you can use mailers as an investigative starting point to reveal what’s wrong, and then parse your logs for deeper information. We believe that a well-rounded system can empower administrators across any environment. 

That said, mailers in HAProxy 2.8 (and later) work a little differently than those from 2015, onward. Let’s explore how mailers first started and how they’ve evolved.

How Did We Previously Implement HAProxy Mailers?

HAProxy’s previous implementation of mailers sends event-based log messages as email alerts when:

• An active server is marked as down and the configured alert level is alert or above

• A disabled server is marked as down and the configured alert level is notice or above

• A server is marked as up or enters the drain state and the configured alert level is notice or above

• The option logs-health-checks directive is enabled with the configured alert level as info or above, and a health check status update occurs

HAProxy attaches a Syslog severity level (in descending order from 0 to 7) to each alert which indicates an event’s seriousness and urgency. Accordingly, it’s likely that you’ll care most about events at or beyond a certain threshold. Administrators can assign various priority level filters to mailers—from emergency to debug. Focusing only on impactful notifications helps you reduce inbox clutter.

C-Based Mailers

Until recently, HAProxy mailers were exclusively written in C. While C is a capable and efficient programming language, it’s a compiled language—so any coding changes require recompilation before HAProxy can apply them. As a result, C-based mailers relied on code that was relatively rigid and difficult to maintain. 

Together with Simon Horman, a HAProxy community member who jump-started the initial C-based mailer implementation within HAProxy core, we recognized how email alerts could be a springboard towards better monitoring. However, our team needed to make some important improvements to help mailers evolve.

Improving Customization and User Experience

While C-based email alerts were fairly useful, they weren’t as user-friendly or programmable as we’d hoped for. Knowing how much HAProxy users value customization throughout our products, we took their feedback and extended that customization to email alerts. 

For example, email formats were limited and technically complex. A lack of verbosity, contextual value, and parsability undermined their usefulness to humans and bots. Administrators without solid HAProxy experience wouldn’t find these alerts very relevant. This was a consequence of automatically generating email alerts from HAProxy logs.  

It was clear that we needed to upgrade how HAProxy handled email alerts. The Lua event framework and scripting provide the flexibility we and our users were searching for.

How Do Mailers Now Work in HAProxy 2.8 & Later?

Mailers now exist as scripted Lua modules and leverage the Lua event framework versus being written in C. Plus, the Lua engine now handles dynamic server changes effectively with revamped Lua API integration in HAProxy 2.8. 

Previously, the HAProxy core itself would build alert messages and send them as emails over SMTP. Lua now generates these emails directly based on this readily accessible event data—with contextual help from proxy and server info. We also leverage a custom callback function to handle event data. Here’s how each component ties together:

It’s important to note that Lua mailers use HAProxy’s C-based mailer design as a foundation. Although they largely mimic that underlying functionality, the Lua implementation is much more flexible. Here’s a snapshot of what you can currently do by customizing the provided Lua mailers file: 

• Customize your alert email formats

• Include much more contextual information that coincides with specific server events

• Write, edit, and save your Lua scripts in a text editor, then automatically apply changes when HAProxy restarts

• Subscribe to certain events across all servers or only specific servers you choose

• Use queues (first-in, first-out storage) to fetch messages in the background and sequentially send them as alerts

We predict this list will change (and grow!) as more and more users experiment with the Lua event framework. Those familiar with Lua scripting will likely hit the ground running. And while Lua support opens the door for wider mailer usage, less fluent Lua users might need more time getting up to speed. Luckily, the default Lua mailers file doesn’t require much adaptation to get working. 

So, how do you get started with Lua email alerts? Let’s tackle the basic setup process.

How to Set Up Lua Mailers in HAProxy

Getting started with Lua mailers requires a few steps and is relatively quick. You’ll still need a HAProxy configuration file before continuing—which existing users should have readily available. You can also find an introduction and walkthrough on HAProxy mailers within our documentation.

Adding Basic Configurations

After creating your haproxy.cfg file, you’re ready to add mailer references. We’ll walk through how your starting mailers section might look.

First, have HAProxy load the provided Lua mailers script from the HAProxy global configuration section:

You can now add a “mailers” section (like you’d normally do for C-based mailers) to define mail servers that we’ll use as SMTP targets:

Configuring email-alerts

To enable email notifications, you’ll need to specify the following three directives at minimum:

1. email-alert mailers smtp_servers – Tells haproxy to use the mailer section previously configured to know which SMTP servers will be used as email targets.

2. email-alert from haproxy@example.com – Sets the email sender used in the MAIL-FROM SMTP command.

3. email-alert to helpdesk@example.com – Sets the email recipient used in RCPT-TO SMTP command.

You can specify these directives in the defaults section (to apply them globally). Alternatively, you can specify them directly within the backend or listen sections (to enable per-proxy configuration).

Aside from those three required directives, you can optionally use “email-alert level” to filter out email-notifications and “email-alert myhostname” to configure the domain name used for the HELO SMTP command.

Here’s a sample configuration:

Once you have these pieces in place, it’s time to extend the Lua module with your own customizations.

Start Experimenting With Lua Mailers

Our revamped mailers functionality is still evolving. However, you can do some pretty cool things already.

Email Templating Customization

A common complaint we received on our C-based mailers implementation centered on rigidity. Users couldn’t customize email alerts to their preferences, which limited readability. Not only that, but subject lines and email body text are often matched by default:

There’s no visible difference between C-based mailers and Lua mailers, initially. However, Lua mailers support deeper HTML customization for your emails—letting you structure alerts however you want. This makes it easier for Ops teams to scan and read every message. While flexibility is great, users can also employ stricter formatting for mailers that target scripts, bots, and automated subsystems. 

You can configure your subject lines for better context or easier filtering. Lua mailers also let you include supplemental information in the email body. This includes logs and links to helpful resources. Overall, mailers can include the following base info: 

• The event name

• The name of the server that’s experiencing the event

• The name of the backend that the server belongs to

• The severity (or priority level) of the problem(s) at hand

You can also leverage the event data itself and grab any relevant server or backend information right from Lua. These pieces will help give your emails deeper contextual value.

Preparing Your Email Body

There are many ways to customize Lua mailers. You can do this within the --prepare email body portion of the provided Lua script (shown below), which is part of the greater mail queue handling logic. Here’s a sample Lua snippet showing the current implementation:

You can pull the entire raw mailers.lua script from this HAProxy Git blob.

If you can access a piece of information with Lua, you can include it within your alert. For teams accustomed to Nagios-style alerts, Lua lets you emulate that structure by directly accessing the raw email body. 

Here are two alert email examples generated when server s2 goes DOWN and subsequently comes back UP:

To achieve something similar yourself, check out our demo on GitHub. Lastly, the Lua implementation lets you translate your messages into other languages for easier consumption.

Managing Your Alerting Scope With Lua

Server events are central to Lua mailer functionality. Your alerting configuration will depend on how granular your monitoring needs are, and the Lua event framework supports two unique approaches. It’s also important to note that you can employ these simultaneously.

The diagram above shows how Lua event monitoring functions. We’ll describe how you can leverage both alerting mechanisms to keep a watchful eye on your servers.

Monitoring Events Across All Servers

Starting broadly, you can use the core.event_sub() function to subscribe to events across all servers—generating the widest scope of mailers available. Depending on the priority rules you’ve established, HAProxy and Lua will generate related alerts across your entire server collection. 

Monitoring any meaningful server changes is important. However, a down server can wreak havoc on your web application—reducing overall performance or rendering services unreachable. It’s also helpful to know when new servers come online, as these events can alter backend performance. 

For example, you can add an event listener function for SERVER_ADD to track any new servers that spin up during your web application’s lifetime. 

The Lua event framework will recognize these additions and watch them for DOWN events, like any others. This paired with your other haproxy.cfg configurations is what enables Lua to send alert mailers to designated recipients. 

You can also create similar scripts and functions to fit any of Lua’s event types. However, customization isn’t always as simple as dropping in new event types without adapting your script. 

Plus, remember that global monitoring will theoretically generate more alert messages by default. If these become hard to manage—or if you believe you’re receiving faulty or irrelevant messages—you can leverage the EventSub.unsub() method to halt specific mailers.

Monitoring Events for One Server

While widespread monitoring is handy, sometimes we want to zero in on just one server. Lua mailers let you track important events on key servers while keeping alert message volumes to a minimum. 

You can do this by defining your Server.event_sub()function within your Lua script. It’s also possible to jointly leverage Server.event_sub()and core.event_sub()to apply blanket alerting across your servers for some events—then monitor other events on individual servers within that group. 

Single server Lua event tracking comes with other notable advantages: 

• You can write your Lua callback functions without any extra filtering.

• You avoid unnecessary wakeups.

• You can often perform monitoring and alerting more efficiently with numerous rules across individual servers, versus employing blanket mailer rules. 

Whether you’re monitoring one server or every server, the Lua event framework lets you establish as much oversight as you need. Users can even provision email settings via JSON file HTTP requests.

Substituting SMS Alerts for Email Alerts

What if your team needs text-based alerts alongside traditional email notifications? SMS alerts are ideal for those in areas with poor connectivity, or while away from work during off hours. Latency issues can sometimes prevent email alerts from arriving in a timely manner, and having a reliable SMS backup can reduce response times. That’s doubly true if your mailer inbox becomes disabled.  

HAProxy provides a native Lua HTTP client that lets you leverage APIs from major providers. You can seamlessly integrate your favorite solutions from PagerDuty, OpsGenie, and others to ensure you never miss an important alert. And you can also leverage your email-based mailers simultaneously if that’s what you prefer.

Similar Article: How to Extend HAProxy with Lua?

Set Up Your First Lua Mailers Today

We created our Lua mailers implementation based on years of feedback from HAProxy users. Lua provides numerous avenues to customize and add value to email alerts—beyond what C-based mailers previously offered. While the basic structure remains familiar, Lua scripting unlocks new configurations in a snap.

We’re not stopping here! We’d love for you to experiment with—and offer feedback on—our new Lua mailers features in HAProxy. Your customizations and use cases will help us expand upon the Lua event framework’s current capabilities. For more tips on working with Lua scripts, check out our documentation.

Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.