How do we build robust, scalable systems in an ever-evolving tech landscape, and what role do human decisions play in this complex equation? This article captures a dynamic panel discussion from HAProxyConf featuring industry leaders Kelsey Hightower, Hussein Nasser, Andjelko Iharos, and Baptiste Assmann. They delve into critical aspects of modern infrastructure, from the nuances of dynamic configuration and the challenges of automation to the profound impact of human factors on system reliability and security.
This session explores how practices like clear documentation and understanding foundational protocols are more vital than ever, especially with the rise of new technologies like AI and LLMs. The panelists share insights on managing change, fostering curiosity in engineering teams, and the subtle ways abstractions can lead to unforeseen problems. You'll gain a deeper understanding of the human element in technical excellence and discover how prioritizing fundamental knowledge can lead to more resilient and efficient systems.
Iwan Price-Evans
Welcome back, everyone, and welcome to our guests. Change is inevitable, and the question for all of us is: what will you do when it comes for you?
Luckily, our panel features some of the brightest minds in our industry to help us navigate organizational change and turn disruption into opportunity.
On our panel today, we have Kelsey Hightower, returning from this morning. Kelsey is a distinguished software engineer and developer advocate, renowned for his contributions to Kubernetes, open-source software, and cloud computing, which have empowered developers worldwide.
Next, we have Hussein Nasser, an accomplished software engineer and popular host of the Backend Engineering Show on YouTube. He shares deep expertise in backend technologies, databases, and network protocols to help developers build robust and scalable systems.
You saw them from the HAProxy team this morning: Andjelko Iharos, VP of Architecture at HAProxy Technologies and a leading expert in large-scale systems, is the mastermind behind HAProxy Fusion.
And Baptiste Assmann, Director of Product Management at HAProxy Technologies, specializes in high-performance architectures and is now HAProxy's product visionary.
One thing we’ve definitely seen today is how many architectures are changing to become more dynamic, flexible, and scalable. This shift changes a lot, particularly how teams manage and observe application traffic. Security has been a major topic. My first question for the panel is: how is security changing as infrastructure becomes more dynamic?
Kelsey, perhaps we can start with you.
Kelsey Hightower:
I don't think security changes because infrastructure is dynamic; I think the impact of things being dynamic is much faster. Previously, if I had to check in code, write it to a file, and then restart a process for it to take effect, for many, that was already dynamic. The ability to even change it is where the dynamic aspect comes in.
I believe consistency comes from the implementation details. Before dynamic systems, we typically wrote many scripts and serialized them to files. Many in the audience are likely familiar with configuration management tools, where we would brute-force these changes from the outside.
Now, with dynamic infrastructure, the system we're trying to configure better understands our intent. Instead of working around the software, we can work with it. We now have an API for configuration. When I send a call, the system performs the correct checks, does things in the right order, and tracks in-flight changes. In some ways, automation has become easier because the intent is now more directly translated.
From a security standpoint, security as a practice remains the same. The core questions are: are you sending the correct values in the configuration, and how do you know when they are incorrect? Those fundamentals do not change due to the implementation of automation.
Andjelko Iharos:
I agree. One of the things that has become more important is observability. Due to the dynamic nature of changing infrastructure, evolving threats, or adapting to existing mitigations, observing the impact of any current or upcoming configuration changes is crucial. This is because we must adapt to situations more quickly and frequently.
Previously, when an environment was under duress, we might reconfigure some protection policies and likely be fine. However, now the environment might scale in response to increased load. Our protections need to be aware of how the infrastructure changes—whether we're accommodating more backend servers or if the load balancers have fewer resources because they need to handle more connections.
So, observability allows us to react to the environment or conditions and see the effect of those reactions. I believe that is even more important these days.
Kelsey Hightower:
We're not doing the corporate, one-question-at-a-time thing here, right? Great.
I'm not sure it's actually about observability. In 2010, when configuration management tools like Puppet emerged, most people didn't even know what they were looking at. We had logs, but if you don't understand what you're seeing, you won't do anything with them. You can have all the metrics you want from observability, but if you don’t know what you're looking at, it just doesn't matter.
I think the declarative approach of saying, "I want these 27 backends in HAProxy," expresses an intent. In a dynamic world where we don't know all servers beforehand, we'd declare, "I want every application with this specific label (key=value) to be in HAProxy." This declarative statement sets precisely what should be in HAProxy.
What I believe you mean by observability is, "How do I know when that's not true?" If I check HAProxy and it's sending traffic to things without that label, we have a problem. How do I get that notification? I think that's the linchpin for security policy.
So now, in a dynamic world, it's not about watching dashboards and thinking, "Oh, there are 71 backends in here." How would you know if it's supposed to be 71 or 70? You have no idea. What I want to know is: does it match my policy?
Anything with these labels should be there, and autoscaling can do whatever it wants as long as I have policy control and a declarative intent. I believe these two elements combine to create an auditable environment. Then, when you use observability, you genuinely know what you're looking at because when something turns red, it will point directly to the policy as the reason.
Andjelko Iharos:
That's a good take.
Baptiste Assmann:
Well, at the root of any security issue, there's often a human element—someone making a mistake. This could involve forgetting to configure something, opening a port, not closing it, and so on. The biggest problem with security is clearly people. As long as they aren't careful about what they're doing, you will always have security problems. Even the largest companies and banks, despite their efforts, there's always a way in.
Kelsey Hightower:
But remember, "people" can also include the vendor, who might write code in a way that, even if I did everything correctly, their error causes a problem like a buffer overflow.
Andjelko Iharos:
Right. It can be an omission or a combination of circumstances rather than a single person. For instance, an out-of-date specification or a port not updated in some configuration. I agree, but I guess there's a lot of nuance to it as well.
Hussein Nasser:
I would add that probably all the scripts and configurations we author ultimately come from us, humans. Mistakes will eventually happen. We can blame AI and machines all we want, but ultimately, humans have configured these things and written the scripts to do so.
These discussions remind me of an outage at Amazon in 2021, during the pandemic. Amazon AWS went down for, I think, 21 hours. I covered that on my YouTube show back then.
What truly happened was an engineer configured a policy to increase the auto-scaling policy, stating a number—I don't remember if it was 20%—because of the high load during the pandemic.
When that load was met, the auto-scaling policy scaled up, interestingly, 1,000 more backends than anticipated.
You might think, "Okay, that's good; a thousand more backends mean more instances to consume our requests." But consider that every backend spun up is a cold start. To define that, each backend needs to read its configuration.
At Amazon, specifically in that particular case—and this is all documented in their root cause analysis—they were querying a single-instance database. They thought, "This is a really simple configuration, in and out. I'll query it; I don't need another instance of that database." You always need more.
However, in that specific case, that "herd" of backends began initiating multiple queries to read the configuration, which then overloaded the database. There was a retry policy: "If you couldn't get your configuration within 500 milliseconds, try again."
Baptiste Assmann:
Try harder.
Hussein Nasser:
Try harder. And guess what?
This is a bit controversial, but you'd think a request would be canceled, but a request is almost never truly canceled. You, as the client, can give up, but the request might still be processing in the database.
So that request comes in, another request comes in, and then the database gets overloaded, eventually causing the entire backend infrastructure to fail.
As a result, CloudWatch, their monitoring platform, which uses the same database, went down because the database failed. The entire situation snowballed.
This whole thing brings me back to configuration and trust. We can discuss trust concerning different communication methods between clouds. I'd like to bring that up as our next topic. What do you guys think?
Kelsey Hightower:
But you're also describing how far away configuration is from the actual real state of the world. I want to scale this by 20%.
Hussein Nasser:
Yes.
Kelsey Hightower:
That's what I want. But what the rest of the system does isn't articulated in that statement. And that's the nature of reality.
So that's the practice. And I get back to the point that we don't have many tools that say, "The thing you just asked for caused these problems. Would you like to undo that?" The correlation isn't there.
When Kubernetes was released, the entire system was designed with a declarative interface throughout. When you ask for something and the system cannot do it, it will respond, "Hey, there isn't enough memory here to deploy all these things, so I cannot do it."
Then you can decide whether to remove that request, or the system will simply stay in this loop forever. Most of our systems are designed as a collective whole, where they feed back into each other.
Hussein Nasser:
What I would add is that even defining what a problem is, the system doesn't know what a problem is. We do, as humans, right? Because we think doing many requests simultaneously is a problem due to limited memory and CPU. That needs to be defined in the policy as well: "Don't exceed this much." And I think there's an unlimited number of configurations that could "box yourself in."
Kelsey Hightower:
What do you do when there's a problem? I think that's the other part of it. You can't know everything.
Hussein Nasser:
Yeah, exactly.
Kelsey Hightower:
So, in the case of a problem where it's like, "Hey, I can't do this," what is the failure state when you don't know? Five hundred errors aren't the best response, but at least I know that this isn't what I wanted.
You either fail open or you fail closed. And I think many of these systems just say, "Oh, I'm just going to keep trying forever, and who knows what state I'm in?"
Iwan Price-Evans:
You both raised the point that sometimes automation has unintended consequences. We seem to be under a lot of pressure to automate and scale, but automation can either scale your solution or scale your problems. So, how do we do automation correctly?
Andjelko Iharos:
Well, taking this example further, it's not about automation failing. It's about automation being incomplete or unaware of the entire system. The automation worked for the service consuming the database, but not on the database itself, or the automation wasn't aware of the database.
Perhaps that's a good separation of concerns: the database team worries about the database, and the application team worries about the application.
However, there has to be some knowledge sharing. When policies are put in place, they either need to be established with knowledge of the rest of the system or in coordination with other teams to ensure that this database doesn't randomly scale by 20%. There is an absolute limit there.
Alternatively, there needs to be a wider scope of coordination, where if the application scales, the database can scale as well. In that case, it works as a whole system rather than a loose collection of components.
Kelsey Hightower:
I would say my answer is simply correlation. You're never going to get these things perfectly right. We're so far along—what, 40 or 50 years into this? So, depending on the database, proxy, programming language, network drivers, Linux kernel, and system libraries you're using, I don't even know if you can ever manage it perfectly. So I think the industry's solution has been correlation.
If a change is made, there's a timestamp somewhere. Then, when errors start to generate, there's also a timestamp. That's a simple correlation: someone made a change, and these errors started happening around the same time. The naive approach is that these must be correlated. At least you have a starting point to try to figure out why everything else went wrong.
And I think sometimes, as we see in the HTTP world, there's that trace ID. I think that was one of the biggest revolutions when microservices emerged. If the proxy creates a trace ID from the very beginning, or even if the client does it, and we pass that trace ID all the way through the system.
When I was at Google, we used to put that trace ID in the database call. This way, we knew that a client higher up the stack started a series of events. The component making the query to the database also uses that trace ID as metadata. So that when the execution plan or we're doing a full table scan and we didn't intend to, we know the trace ID of where this problem originated, because I think it's just so random.
So I think the number one solution we're trying to achieve is simply the ability to correlate these things. And I think that's the foundation of what observability is meant to be: we may not have an answer to all these problems, but we can correlate them. We go from hours or days of troubleshooting to at least something that says, "Start here if you want to find out the root cause."
Baptiste Assmann:
That's how I explain the power of HAProxy observability, to bring it back to HAProxy a bit. We might not know precisely where the problem is, but you can look at the logs. HAProxy can log up to 11 times per request, including user termination state.
We might not know exactly what's happening, but we can tell you that when a specific client calls a particular URL, it takes 10 seconds, whereas when that same URL is probed from a different server, it takes only 10 milliseconds. This suggests the problem might be on the server itself.
People appreciate this because, typically, troubleshooting a problem can take an hour: 50 minutes to find the source and 10 minutes to fix it. If you can reduce the troubleshooting time, it's a win-win for everyone. That's my favorite feature of HAProxy.
Iwan Price-Evans:
Now, this being 2025, any discussion has to include AI and LLMs. We've had some mention of this earlier. We've seen people routing traffic to multiple LLM backends and trying to integrate with MCP.
What challenges and strategies are you seeing with these technologies, particularly regarding the management of traffic flows and automation of configuration?
Kelsey Hightower:
We're facing a similar situation again. Twenty years ago, we had the same discussion about databases. Queries were expensive due to limited server memory and storage. Every complex query reaching the database was costly.
Then, people discovered the "magical" technology called a cache. You would check if a query had been run before, retrieve the result from the cache, and give it to the client instead of burdening the database.
I think the current state—and it will improve—of performing inference on infrastructure is very expensive. It's almost one of the most expensive computational workloads we have today. So, do you really think you can use the same load balancing technology to send random requests?
We're now so accustomed to abundant memory and efficient general-purpose CPUs. We have instructions like AVX-512 that make many operations super fast, and most people simply say, "Scale out generic servers; it will be fine." But you are not auto-scaling $80,000 GPUs. In fact, you can't even buy all the ones you want right now.
So, we're again faced with the same challenge. We have a very expensive workload that you cannot simply send traffic to blindly. You'll need to become much more intimate with the low-level details: where is this preloaded? Do we cache this response?
I think we've come full circle. It's a different workload, certainly AI, but from a networking perspective, it's the same principle. If you have an expensive backend, you need to consider caches or other mitigation strategies until it becomes so efficient that you don't have to think about it.
Andjelko Iharos:
Since it's almost a new discovery period for the tech world, AI is still realizing its potential. It's on track to be a significant contributor in an unknown number of use cases that we need to consider, but we don't know the extent of the change or impact it's going to have.
So, our application developers are figuring things out. They're rapidly deploying LLM servers. We started with standard LLM servers; now, they're over the network.
The current list of LLM servers roughly maps to API calls. It exposes regular services that we already know and use, like databases or hosted APIs, to our LLMs. But I think that's probably only the first wave of integration we'll see. We'll likely have more as these LLMs grow in capability, which it seems they will continue to do.
There's a chance our application developers will come up with even newer ways to utilize this technology. We'll have to consider how to protect and load balance them. We're going to have to keep adapting and closely monitoring how things evolve because I think we're still in the early adoption phase.
Baptiste Assmann:
From a product standpoint, I looked into this because we saw the emergence of some AI gateways. But ultimately, what do you need in an AI gateway? You need caching, rate limiting, and authentication. I mean, that's HTTP; that's an API gateway. In the end, there's nothing magical.
So, yes, maybe they are going to reinvent the wheel. The only difference is that with a standard API call, you can analyze the payload, and you don't really care much beyond that; it's easy.
When you have an AI call, you have to understand the prompt. Two people can send different prompts that could yield the same response, so you might want to use a cache. But you can't do that with HAProxy today; for us, different prompts mean different requests.
So I guess it all depends on the type of use cases for LLMs and similar technologies. I assume when an agent is talking to an LLM, it will be easier because each agent will generate the same request, so the prompt and the answer can be cached easily. But, you know, that's definitely a new area of exploration for us, from a product and protocol perspective.
Hussein Nasser:
The way I understood the question was this: today, we have AI technology. We had AI before, but today it's a learning model. There's existing knowledge that can predict the next word. That's how I view it.
I have a proxy, a load balancer, and a rich dataset of all the requests that have passed through and all the responses that came back from the backend. So you can ask questions of this data using the language model, such as, "What is the trend?"
Then you can ask a prompt like, "What is the most expensive query or request, and what is the trend? Why did this particular request become more expensive than the others?"
It can also find interesting patterns that we might have missed or that are not easily calculated using existing monitoring tools.
Iwan Price-Evans:
Have you run any interesting experiments with AI that you'd like to share, and what challenges you've encountered?
Hussein Nasser:
I use AI for things I tend to forget, such as how to perform a TCP dump and the specific parameter for the port. I constantly forget those details. Things like that.
How do I filter based on the interface? Is it -i or --i? That's how I primarily use AI, for very basic tasks. Instead of Googling, I'd ask the AI. But sometimes, even that gets it wrong.
Kelsey Hightower:
One unique aspect here is why we are using the LLM in the first place. If you're using an LLM to build a deterministic system, I don't understand the motivation.
For example, in computing, when you calculate an authoritative answer to a question, you typically don't calculate it again for efficiency. When you build software, you compile it once, create a binary, distribute it a million times, and then auto-scale that. We don't auto-scale makefiles, and then when a request comes in, build the application first before responding. That's insane; you'd laugh at that idea.
So, when it comes to an LLM, its dynamic properties are one of its most valuable components. I'm receiving an unknown input, such as wanting to translate something, but I don't want to limit the input. So you can say whatever you want, and I'm going to send it to the LLM to attempt to answer this question. That's the beauty of this particular technique.
I think at the networking layer, we're dealing with a workload that, for the first time, we seem to be okay with being non-deterministic. So yes, if we get the exact same prompt, we probably want to be able to cache and retrieve the same answer. But at some point, you're never going to send everything to an LLM, like for a knowledge base.
You're probably still better off computing the knowledge base as a graph and then using search techniques, even if it's an LLM prompt, because it's simply more efficient.
So I think this is the new set of challenges: some people want to pass through, and some people genuinely want that dynamic lookup. However, there's a lot more negotiation that has to happen on the backend. Is the model loaded? Does the GPU have resources? What's the response time looking like?
But again, I just zoom all the way out and say, to me, this doesn't look any different than any other type of workload. You can call it AI, but you're still probably moving the same bits and bytes across the network. You just have different concerns in how you do it, and the same vocabulary tends to apply.
Baptiste Assmann:
That's an interesting point of view. So I guess you're using the nice UI they provide to ask your questions?
Kelsey Hightower:
Or APIs or SDKs.
Baptiste Assmann:
That's the point. I'm not sure about the SDK. But I discovered that when you use their UI or web frontend, you just have one front—and that's all. You don't have any settings.
I wanted to make a page in HAProxy to call an AI, just for fun. I discovered I have a nice API where I can change the temperature and have literally 100 parameters. I thought, "Wow, that's how I want to use the AI." I don't care about the UI where you can't choose what you want. And the first thing I said was "temperature zero." I want a predictable response. I don't want you to hallucinate. So maybe I'm not the right user for AI.
Kelsey Hightower:
For you, you want a predictable response. But I have 100 million customers, and they all want something different, and we're beyond the ability to do "if statements." The days of "if it's you, give you this response" are over. This new paradigm says we're going to move away from symbolic programming techniques and allow for the unpredictable.
Now, we may put constraints on it or try to limit what it says. But its absolute value is its generative properties, which happen to be somewhat unpredictable across your entire customer base.
So, the new thing here would be how observability gets used in this context. Instead of just capturing the request and the response, they need to be correlated again.
Why did you give this user that response? Well, the context was different. They were located in this particular country. The alignment parameters say they were not allowed to give those responses for political reasons, and this is why, collectively, that request received that response in this scenario.
So, it maybe amplifies the need for even more correlation so you can understand how the system is behaving.
Andjelko Iharos:
But the MCP throws even more complexity into the mix because the same prompt against the same LLM when calling an MCP tool, which is external to the LLM. The tool will run an API call or a database query.
Between one prompt and the next, the database query might return different results, and that API call might return different results. Therefore, the contribution from the MCP tool call will modify the LLM response.
Kelsey Hightower:
Let's be clear here. The MCP is going to return a predictable response here. This is the part about calling an MCP. It's an API with a predictable protocol for the calling side. It's not a net-new technology.
If you look at most code for an MCP server, it's using the same client libraries as before, calling the same database, calling the same Salesforce. If you call Salesforce with a client library, it's going to do the same whether it's an LLM calling it or you call it by yourself via cURL.
So, we know everything from MCP back to the other side is deterministic. It’s guaranteed. Let's not pretend we're talking about a whole new paradigm.
What's new here is the agent. The agent is unpredictable. It may call all of your MCP backends when it should only call one. That's a new thing you're going to open yourself up to.
The other thing we haven't learned how to do is scoped permissions. If I'm a user, and I go to a prompt and say, "delete all the users from the database." Well, some people still don't understand the fundamentals. So what they do is they've configured their MCP agents with root god tokens that can do anything.
So if the prompt makes it to the MCP server, it has no idea who it’s acting on behalf of. Why? Because if you read the protocol, there's nothing about permissions. There's nothing about headers. There's nothing about exchange and scope tokens we've been doing for 20 years.
People are starting to make the same mistakes because they forgot the fundamentals. We know how to make calls on behalf of other users. We do it in every SaaS product today. So, I think the fact that we say MCP as if it's some new magical construct is not accurate.
Baptiste Assmann:
So you see, you came back to my point: human errors.
Kelsey Hightower:
Human errors. Once again.
Andjelko Iharos:
New application developers are starting to enable the MCP protocol. This gives us an opportunity: if we proxy all the MCP calls, we get a known protocol with built-in discovery. That's how LLMs discover available tool calls, simply by getting a listing of all the tools.
From this, we can build in observability and generate access control lists (ACLs). We can also generate role-based access control (RBAC) by obtaining a listing from the backend server.
Kelsey Hightower:
The low-hanging fruit for you all is that you have a gift. It's like, in 10 years, they're going to discover everything you learned 20 years ago. Until then, customers will need something mature.
It's common for new technologies to be made to work the old way. That's what tends to happen. I saw it with Kubernetes. We added NFS. We added static IP addresses. We went and added all the things that we said were not cloud-native because it's reality.
If you're smart, you'll probably put a truly mature proxy in front of MCP to handle malformed HTTP requests. That's still a concern. There are TLS bugs and errors that will cost you your entire business. That's still a concern. And there are things like authorization. That’s still a concern.
Andjelko Iharos:
Exactly.
Kelsey Hightower:
MCP is the biggest gift to the proxy community in about a decade because everyone is excited. And if you try to use it as is, you're going to be in the news for the wrong reasons.
Andjelko Iharos:
It also brings value because every application that delivers an MCP server has a different set of rules, permissions, and users for configuration.
Kelsey Hightower:
And to state the same thing, the same is true for every API.
Andjelko Iharos:
It homogenizes the API calls from different databases and APIs into the MCP protocol. Now we are no longer dealing with query strings in one URL for one database and then JSON bodies in another. It's all homogenized to the MCP protocol, so we know what to expect.
Instead of trying to contend with each of your application developer groups or individual application developers about whether they are generating their API keys with limited or root permissions, we can bring that control into the load balancer. We can make checks there. These are the MCP servers your backend servers are exposing. You can limit access to them. You can prevent making calls through them. Or we can even authorize different LLMs differently and allow some of them to make more calls, and others to make fewer calls.
So there's a lot of flexibility we can expose.
Kelsey Hightower:
The thing I like that's new.
Before MCP came out, I think we hit a limit with JSON schema, where we only described the inputs and outputs, and then we stopped.
There was a whole movement around things like Backstage because no one knows what these APIs do. The documentation tends to not be great. You have a discovery problem to figure out what's even available. When you look at the MCP protocol, what makes it truly unique is a concept of a catalog: what is available? And then there's a very simple description: what should this thing do?
Humans could even benefit from that. You can imagine simply listing what calls are available, an example input, an example output, and a description on how to use it. Even someone using cURL could benefit from that. I can imagine a proxy saying, "Hey, based on this description, maybe we apply some security parameters, even though no one configured them ahead of time." So, I do think that's a new benefit, but it's still networking.
Hussein Nasser:
It's kind of back to the cycle, right? What you just said, Kelsey, the catalog idea, this reminds me of SOAP with the WSDL because you can query the endpoints. It says, "Okay, what do you have as a SOAP?" It's a Simple Object Access Protocol. So, yes, everything comes back.
Andjelko Iharos:
True, but in this case, there's a positive feedback loop. To make LLMs more effective using the MCP tools exposed to them, the descriptions need to be better for the LLMs to understand them better.
The trick is that LLMs understand human language just as well as we do. So, application developers are motivated to make their MCP tool descriptions more descriptive, more to the point, and to make their examples not just placeholders or to-dos, but actual examples that reflect their real-world tool usage. This is because if their examples are incomplete or out of date, those examples will propagate mistakes into the LLM tool calls.
Some LLMs can work through errors; others might not. Otherwise, they might make costly mistakes when descriptions are out of date or examples are incorrect. So, it motivates application developers to put more effort into the documentation side of things. Ultimately, that benefits human users as well.
Kelsey Hightower:
You know, there are many developers in the audience who are thinking, "No, do you know how much documentation sucks?" If you had good API documentation with good examples, then developers would not call your API incorrectly, they would respect the rate limits, and they would check the headers and the error body.
For some reason, we didn't do that for real intelligence, but maybe there's an incentive for artificial intelligence.
Iwan Price-Evans:
You guys have made the point that some things never change, and that the shiny new thing is not fundamentally different from what came before.
Andjelko Iharos:
There's nothing better to motivate engineers to do more work than the promise of having to do less work sometime in the future. So if we can promise that, we're good.
Baptiste Assmann:
Once, I applied for a job, and I was refused by HR because I said, "I'm lazy. I'm making my computer do my job." She said, "Oh, no, not you."
Iwan Price-Evans:
Here's what I'm curious about, guys. What are the things that have stayed the same and that drive you crazy? What are the things in this industry that you wish would change but haven't changed in 10 or 20 years?
Kelsey Hightower:
The best way to hide stuff from ignorant people is to write it down. The number of people who won't take the time to read about their craft—if you're a developer, you should read how your runtime works. Go to the language spec, read it once, and take a few notes.
How does it do memory management? Because when people don't do that, they show up, brute-force things, and complain on the internet. "This thing sucks" because, for example. Why are you running the JVM in desktop mode on the server? You spent $10 million on Amazon VMs because you want to run in desktop mode. That's a $10 million mistake, all because you didn't want to read the first two or three pages of how the JVM works. Those garbage collection (GC) flags make a big difference.
So, if you're going to be in this craft, it's an engineering discipline, or it's supposed to be. And the number of times I see people not even try to read the man page. If you don't like being "mansplained" to, make an alias to "mom." You listen to your mom; you can call them "mom pages."
But you should at least try to understand how this stuff works at an intimate level. It seems like most people just want to go straight to implementation details. You get the SDK, you import it, you just start calling things.
And you're not realizing, how expensive is that call? How much memory does that call add to the overall footprint of the application? Wow, you just downloaded the whole internet as a dependency so you can translate two letters from left to right.
Baptiste Assmann:
Do you have any programming language in mind with that?
Kelsey Hightower:
All of them can have this. No programming language is immune to abuse. The gaps we see are typically in the whole "What is my responsibility in terms of documentation?"
When I learn something, the reason why I think many developers gain a large following is that we explain things to people. I think the most recognizable people are those who take the time to explain.
If I show up to Slack and say, "Hey, why isn't this working?" that's a form of documentation. It doesn't have the reach of official documentation. This is why READMEs are so important.
But that's one thing where I think documentation is still treated as somewhat of an afterthought, in terms of how much productivity it can bring to the general population.
Baptiste Assmann:
But isn't this a business problem? Management or sales or marketing—whoever, non-engineering departments—just want things as soon as possible, as fast as possible. They don't care if the thing is efficient, works, or whatever. They just want it released tomorrow.
Kelsey Hightower:
Yeah, but I care.
When I was working on Kubernetes, there were many business reasons to keep producing products. I spent six months writing “Kubernetes the Hard Way.” It was painstakingly hard because I had to document everything, down to how to generate SSL certificates by hand and why Kubernetes reads them that way.
I remember in that discussion room, one group just wanted to build automation tools for a one-click deploy. But I thought, if we build these tools and there are errors, we're going to be answering questions forever, and our community won't understand.
I thought another approach was better. The more people who understood every nuance of Kubernetes, the more creative people could contribute in ways an automation tool hides from them.
So, I think you're right about the business side, but many engineers have political pull. Many of you actually have some social capital you can cash in. There have been many times in my career where I've chosen to cash in on the documentation side, and it had an industry-wide payoff for years to come. So you've got to make that call.
Baptiste Assmann:
I want to thank you for writing “Kubernetes the Hard Way.” That's how I started. That's interesting. We were having this discussion about using products, software, or whatever.
Going back to your talk this morning about the fundamentals, you have to understand the protocols exactly. If you know the protocols, whether you're using HAProxy, NGINX, Varnish, or whatever, HTTP is HTTP. A host header is a host header. Authentication, et cetera, an API is an API. If you understand this, then you can adapt, understand things, and make them better. And take your time as well.
If your marketing team comes and says, "Okay, we need this tomorrow," you know you can have it tomorrow. That's like giving me one hour to draw a horse, then one minute, then 10 seconds. You can have a horse, but one will be very beautiful, and the last one will be very ugly. It's up to you to choose, but there is engineering time needed to make things the right way.
Andjelko Iharos:
One thing that hasn’t changed in 20 years is whenever I open an older codebase, search for a bug fix, and find some crazy lines of code. I think, "Who is the crazy person who wrote this, and why is there no comment?" And then I check the blame, and it's me.
So why did I do that? Why did I not exercise more self-control and add those two lines of comments that would have meant the world to me and saved me three hours of racking my brains? That's one of the things that hasn’t changed in 20 years, and I'm pretty sure for a lot of us it's the same. I think the journey of self-improvement is still worth doing.
Baptiste Assmann:
Stop delaying. Don't say, "I will do it later." No, you will never.
Kelsey Hightower:
Well, I hope all these productivity gains from the promises of this new tooling don't just mean we create more mediocre stuff faster. I hope it means we actually have time to step back and do some of these things that require so much more time, like the things that only a human could do. Maybe we have more time to do it.
Hussein Nasser:
I want to add to this concept of productivity. Today, with the rise of AI, we can do things easier. And I want to define what easy means here. Easy is a single line of code instead of 5,000. But behind that single line of code is 50,000 lines of code. Whatever you did, the complexity, if we want to call it complexity, didn't go away. We just abstracted it away. We just made it pretty.
That's why I feel engineers these days gravitate towards "How can I be more productive?" by thinking, "Oh, I have this line of code; let me copy and paste it, and it did the work."
But behind it, as Kelsey said, there is so much behind the scenes. It's making all these calls; it's doing many TCP connections because it's not using connection pooling. All of this, to make it easy, you need to hide. And I think Baptiste mentioned that, right, when he talked about this UI with all these controls. He likes that stuff.
Baptiste Assmann:
No, I don't. I want the CLI.
Hussein Nasser:
You like the simple thing? So instead of a single prompt, you're given all these parameters and options. And these configurations are where you start.
That's why you have to go back and understand the fundamentals behind all these new technologies. Abstractions eventually will leak. They will leak as performance issues, 100% CPU usage, or lack of memory. They may work today, but in the future, you will see these problems, and you will not be armed with the tools to protect you because you don't have an understanding of what's going on behind the scenes.
Baptiste Assmann:
You're making a good point. I mean, I think for the last 10 years or so, we as an industry just built abstractions on top of abstractions on top of abstractions, and maybe another layer. And maybe that's something I would like to see disappear and get back, as you said, to your question.
Iwan Price-Evans:
You've talked a lot about how human mistakes impact security and the problems with MCP and agents. The things you wish would change, but haven't, are mostly human-related. How do you deal with your own personal development? Change isn't just about technology or our organizations; it's in us. So, how do you manage change in yourself?
Kelsey Hightower:
Man, I spent a long time. It was central to my retirement plan. I spent a lot of time learning to get into tech—what language, what skills to learn.
Then I spent even more time trying to get promoted. But there's a point of exit. Part of that analysis was, I was afraid of becoming a senior engineer but a junior human. Most people don't put this much effort into things outside of work. You're just trying to recover so you can work again.
When I think about all these tools and why we do what we do, I think humans are literally just trying to make decisions. They say writing is thinking when it comes to software. It's not just the tedious part of writing code and syntax for its own sake. Typically, you're thinking and making decisions: what protocol do I use? What endpoint do I call? How do I process this response? There are 10 fields coming back; I only need two. These are all the decisions you're making that, in many ways, train your own model and understanding of the system.
So when you write code, you're typically thinking and understanding the system. And then when there's a problem, that person is trained. They have full context of the scenario, and they do a better job because they understand how the system is put together and how it works. For me, I just try to find that balance of, "Am I also getting better as a person? Why am I doing this?” I just ask that simple question: "Why? Why am I doing it this way? Is there a better way?"
That forces me to go back and look at how people did things before. It forces me to ask other people, "Hey, I'm going to be a bit vulnerable. Here's how I'm doing this. Is there a better way?" Something as simple as a code review process. You know how much courage it takes to submit a PR? Because it's going to be judged by your peers, whether it's good or not. And I think that helps me recalibrate.
So, my goal is just to ensure that I'm progressively getting better, both on the tech side and as a person.
Iwan Price-Evans:
Andjelko, what's your approach?
Andjelko Iharos:
Well, I think up to a point, it's a personal direction or decision to keep learning, right? Or not. You might reach a point in a career or in your learning of technology or whatever, and say, "I'm good with this. This is my comfort level. I'm going to grow in a different direction from here on out." But even then, you're not stopping the learning process.
I think curiosity is part of human nature. We had it as kids, and I think we should keep it going. The technology landscape is an ideal place to keep that childhood curiosity alive because there are so many new things and new knowledge being created and generated, and new things to learn every other day. You just open the news pages, Hacker News, whatever, and there's new stuff. Somebody thought about something you never did.
And there's this creative whirlpool. Sparks of new knowledge are being generated all the time by humans, sometimes by LLMs, right? And that's where I think it just pays to be open-minded, to absorb more of the world, basically. And I think that's the best that we can do.
Baptiste Assmann:
Yeah, we are working in an industry where if I stopped learning the first thing I learned, I would still be developing in PHP 3 or something. Anyway, I was doing LAMP servers basically 25 years ago. I would be there today if I didn't want to learn and understand, "Okay, I have a web server, but how is the client getting connected to it?"
That's how I started getting deeper and deeper, spent many years in networking, and then switched to load balancing. You have to learn in this industry, that's it. You don't know where you're going in the end.
Andjelko Iharos:
We are a knowledge industry, I think. That's one of our benefits because if we were an industry making bricks, we would still be advancing, but the brick-baking toolset is evolving at a very steady pace. Romans built bricks, and we build bricks today, and the entire process isn't that much different.
Part of it relies on physical properties where you need to assemble materials and work on them, and you can't change the physical properties of atoms and materials. But knowledge builds upon itself, and by nature, being ephemeral, there's no limit to how much knowledge we can generate. Therefore, our pace of learning isn't limited by physical constraints.
We can keep learning and generating more knowledge, and that's why our industry is so fast and exciting to be a part of.
Kelsey Hightower:
There are probably leaders in the audience: team leads, directors of engineering, VP of engineering. One thing I advise a lot of people is to just think about the ways that you kill curiosity in your company, in your team.
Everyone complains about their company being slow or something not happening fast enough. And you're probably part of the problem if you don't really think intently about how you kill it.
A junior engineer comes in with what's perceived to be a dumb idea, but they're super excited, passionate, and fired up. They want to try this thing out, and then you crush them. You say we want to hire smart people, and then you just crush them. "Just follow the build guide. It's mature enough."
There's a way you manage that type of person. It's very different than "Do exactly A, B, and C." I actually want you to think. And sometimes I've got to give you a lane to swim in just so you can satisfy your own curiosity, and then I have to direct that energy back to something we can put into production.
But the way you manage that person, if you extinguish that, then you're just going to get somebody that comes in, checks for their assigned tickets in the morning, and does the bare minimum not to get fired. And then you try to repeat the process. We've got to find really good people who are motivated. Also, you can repeat the process when they join.
This is deliberate when you talk about the human interactions involved in all of this: What do you do specifically to cultivate someone who has that fire and avoid extinguishing it before they get going?
Iwan Price-Evans:
Baptiste, backstage, you gave me the real answer to this question, which is the way you change and stay up to date is by watching Hussein's YouTube channel. Hussein, managing a large YouTube channel, you probably have to be more adaptable than most. How do you stay on top of that?
Hussein Nasser:
Well, I have a bit of a controversial take on this, and that is four letters: FOMO, fear of missing out. I have this problem because it seems to me that every week we have a new technology, a new this and that. And the more it gets buzzed on social media, the more I feel, "Oh, I have to learn this." And I would be careful with this. Because if you follow random trends, eventually that thing will lead you nowhere.
So I tend to stick to the pure fundamentals. I'm talking TCP, Keep-Alive. I'm talking the basics of TLS. And you'll be surprised that by just trying to follow that flame inside you and trying to learn just that, you will then open doors and say, "Okay, I don't understand that." And then you will try to learn the next piece.
And this is what is called collateral knowledge. To understand something, you have to kind of unlock that other piece. And that, naturally, you're just moving towards what you're passionate about. And then you're learning things. Not consciously because you want to learn to get better, but naturally. And that's how my entire content is based.
You'll see that I'm excited about talking about something that was shipped 10 years ago. Or a post that Willy had posted four years ago, I would just think, "Oh, I really liked how HAProxy processed 2 million requests per second." I don't know how that is done. So I would just read it, and then I will talk about it. It's old, technically, right? On the internet, this is ages—four years ago. But I would just follow what I'm passionate about, and I think that's just general advice to everyone.
Andjelko Iharos:
Good advice.
Iwan Price-Evans:
I'd like to say a huge thank you to the panelists. I know we're out of time. I feel like we could keep on talking for ages, but we'll have to wrap it up. I think we're all going to take away something really useful from this discussion.
Let's give the panelists a big round of applause.
