Extending HAProxy with Rust Beyond Lua Restrictions
According to the Stack Overflow survey, Rust is the most loved language for the 6th year in a row. Its code and memory safety, speed, and even non-blocking access to file operations are the features that make it so appealing. Have you ever thought about using Rust in HAProxy and what benefits it could give? Well, one of the HAProxyConf 2021 topics examined this question: the practical usage of the programming language Rust to extend HAProxy. Aleksandr Orlenko from Yelp described how to implement Rust in real-life use cases, transcending limitations people encounter when extending HAProxy with Lua.
HAProxy, by design, can be extended in any language using SPOE (or Stream Processing Offload Engine). However, this method requires running an external program (an agent) and establishing communication over TCP with HAProxy using a specific protocol named SPOP. Hosting an external agent comes with its own tradeoffs, such as the need to host the agent and learn the SPOE configuration syntax. Rust integration does not use SPOE, but instead runs within HAProxy and has access to all of HAProxy’s variables and functions.
Alex highlighted certain advantages of using Rust:
- The “mlua” crate, which Alex maintains, lets you easily call Lua functions from Rust or call Rust functions from Lua, plus it exposes Rust objects to Lua and even permits you to call methods,
- Another crate built at Yelp named “HAProxy-API” provides a high-level HAProxy API to use in Rust. You can define your own actions, converters, fetches, and tasks in Rust to use in your HAProxy configuration,
- Rust has lots of decent libraries (including the standard library) and this solution makes all of them available in HAProxy,
- Data sharing between HAProxy’s Lua workers. HAProxy 2.4 has a new feature to spawn multiple Lua instances, one per worker thread. However, it’s not simple to share data between them. All Lua global variables would have thread-local visibility. Thanks to Rust, now we can define global variables shared among all threads with safe access checked by the Rust compiler.
- The modern Rust ecosystem has many async libraries that are also available to use in HAProxy, with some limitations. You can execute non-blocking code that plays nice with the HAProxy scheduler. Or you can spawn Rust threads in HAProxy to do any CPU-intensive computations.
Yelp powers its edge load balancers using many features of HAProxy and Rust. For example, using Rust the team is able to send HAProxy logs directly to Amazon Kinesis using the Rusoto AWS SDK.
Alex demonstrated another example: creating a complex fetch named lookup_ip that takes advantage of Rust and the Lua-load-per-thread directive introduced in HAProxy 2.4. There is an important option insecure-fork-wanted that allows creating background threads in Rust, which allows you to avoid locking up HAProxy’s main process while doing a longer-running operation.
Aleksandr concedes that extending HAProxy with Rust has some limitations, such as:
- Yielding is not an option in Rust in any arbitrary place in a non-async context. It means that some HAProxy methods like “core.sleep” that yield by design, can not be called directly from Rust. They must be summoned from Lua. Async/.await is a certain case where yielding is allowed because of the implementation details.
- You can not have a REAL integration with the HAProxy event loop. It means that, for example, HAProxy can not poll networking sockets created in Rust. They just don’t know about them. In Rust, every async poll has a special associated context with the Waker struct that is used to mark async tasks as Ready. Whereas, in HAProxy, we can only poll Rust tasks hoping they are ready without getting feedback when they actually are.
- HAProxy can not suspend Rust calls, and you should be careful what to execute.
Orlenko created two HAProxy configurations to test out the new fetches: the first one uses native fetches, and the second one Rust-based fetches. They generated a file with 1-million IP subnets and loaded it into HAProxy. The results, which were measured using the Linux /usr/bin/time program, are telling – the Rust-based fetch was 67.1% faster and 68.4% lighter than the native ones.
If you are interested in extending HAProxy with Rust, watch the full HAProxyConf 2021 presentation video (embedded at the top of this page). We also recommend visiting our blog post about 5 ways to extend HAProxy with Lua.
Here you can view the slides used in this presentation if you’d like a quick overview of what was shown during the talk.
Software Engineer, Yelp
Alex is an enthusiastic software engineer who loves system programming and working on infrastructure projects. His favourite programming language is Rust. He has been working at Yelp in London since 2018 on the Edge team and their main focus is delivering users’ requests to Yelp services with low latency. Ex. perl developer, previously worked at Yandex.Direct. In his spare time he enjoys cycling and flying a small Cessna aircraft.
Organizations rapidly deploy HAProxy products to deliver websites and applications with the utmost performance, observability and security at any scale and in any environment. Looking for more stories?