What are large language models (LLMs)?

Large language models (LLMs) are machine-learning models designed to interpret natural human language and generate contextual responses that sound similarly human. LLMs accomplish this by consuming massive amounts of "training data" — either prepared intentionally or scraped from across the web — and continually improving based on exposure to different examples. This training data can range from simple to fairly complex. 

This reliance on big data, billions to trillions of internal operational parameters/variables, and demanding resource requirements collectively give LLMs their "large" moniker. 

At their core, most LLMs predict the next most likely word or token in a sequence — allowing them to create coherent and contextually relevant sentences, paragraphs, and even entire documents. This means they're effectively an advanced "autocomplete" program, able to guess the most likely next word or phrase based on the context they're provided with.

How do LLMs work?

LLMs thrive on the training data to formulate quality responses. Accuracy remains a key (yet hard to quantify) metric for LLM performance, and a model without access to targeted or filtered training data will have difficulty producing contextually relevant responses to prompts. Similarly, a model trained on data that's too specific may struggle to produce well-rounded responses and have limited adaptability. 

Plus, each LLM has a "context window," which is the maximum amount of text it can process at any given time. This functions like our short-term memory. The LLM can only hold a certain amount of information before it starts forgetting the earliest parts of the conversation. This limit is measured in tokens, briefly mentioned earlier.

Everything that happens in a single session (the initial prompt, the previous answers, and the latest question) must fit within this window. If the conversation gets too long, the model will "forget" what was discussed at the beginning. This is why a chatbot might lose track of an earlier instruction in a lengthy dialogue.

LLMs power generative AI, which ingests user inputs and produces text or image outputs in return. This is the use case that popular AI models such as ChatGPT, Google Gemini, Claude, and others support — and which has ultimately driven the democratization of LLMs, since these sites and tools are so accessible. Everyday users might be rate limited or token-limited, but they don't readily bear the cost. 

From a technical standpoint, here's how an LLM call typically works: 

1. Clients on a site such as chatgpt.com will send a question or task to the backend LLM via HTTP request. 

2. The LLM receives this request and processes it. This involves natural language processing (NLP), sentiment analysis, content creation, and other mechanisms via the model, which interprets the prompt and produces a contextual response. 

3. The LLM API sends this response back to the client. 

4. Clients may continue to make requests until they're rate limited, or as long as the LLM API key remains valid. 

Since LLMs also emulate human thinking, they're designed to mirror how our brains function. Many models rely on neural networks made of multiple nodes and layers to interpret, process, and serve requests. A system of thresholds and checkpoints determines how different portions of the LLM communicate and share information.

Additionally, LLMs use transformer models to better understand human language and the context behind it. The transformer mathematically identifies links between sequential phrases or parts of a sentence using a process called self-attention, and subsequently forms logical connections. Transformer models are especially helpful since prompt quality and clarity can widely vary. These mechanisms can help cut through the noise and deduce what users are trying to say when ambiguity strikes.

LLM responsiveness

LLM response times are typically lengthier than "normal" API response times, which are measured in milliseconds. By comparison, complex prompts can take multiple seconds or even minutes to complete (especially for image generation). Just like with other apps and services, increased demand can raise response times or cause the request to be temporarily rejected. 

That doesn't mean LLM responses can't be lightning fast, but the relative complexity of LLM prompts and their generative nature often slows things down. After all, there's a big difference between pinpointing data that's common knowledge and completing a multi-step operation. However, most LLM users seem to accept this tradeoff for various reasons.

What are some real-world applications of LLMs?

While many companies have created their own in-house LLMs to handle internal tasks or interact more readily with customers, others have adopted solutions from external vendors — typically using public APIs. 

Many of these APIs are free, but others from larger (or specialized) AI vendors are paid and subject to usage limitations based on demand. Many organizations have also adopted the Model Context Protocol (MCP) to more easily connect their AI/LLM-powered services with the data they need to function. 

As a result, core use cases and unique value propositions for each LLM can greatly vary. Organizations are using and designing LLMs for the following use cases: 

• Optimized search and discovery (as with Google's AI overview feature)

• Chatbot support (enabling 24/7 customer assistance) 

• Easier linting and code review, or automated code generation

• Content curation (as with Netflix, YouTube, and other preference-driven media platforms) 

• Text-based translation

• Data interpretation

• Risk assessment and fraud detection (mainly in the financial industry)

• Treatment planning and predictive medicine

• Enterprise automation and agentic workflows

This list is expected to expand as AI matures. Each LLM also excels at supporting different workflows — whether they're processing prompts from everyday users or large organizations. These organizations are also dipping their toes into retrieval-augmented generation (RAG) to enhance their enterprise LLMs with internal business data — leading to more accurate, contextual responses with cost savings.

What are some key considerations around LLM use?

Hallucinations, bias, and privacy 

It's important to mention that LLMs don't concretely "know" things, but instead generate statistically probable responses based on a blend of prompting and backend data. Consequently, LLMs have a tendency to “hallucinate” — or occasionally provide results that are factually incorrect, don't make sense, or are even fabricated. Fact checking remains essential for anything outside the realm of common knowledge. 

Bias is another consideration, both unintentional and intentional. First, human-designed pulling from faulty training data can amplify biases encompassing race, gender identity, and cultural background. Second, LLM organizations can introduce their own similar biases intentionally in response to political pressure or under the direction of leadership.

Lastly, users of public LLMs (and especially business users) should consider what data they're sending to these third parties. There's a risk that any data shared intentionally can be shared with (or leaked to) wider audiences thanks to bad actors. Users should consider limiting which sensitive or proprietary information they feed to public LLMs.

Cost and consumption

Questions around expense and demand are common in the AI/LLM space, yet aren't always easy to answer. For example, it's hard to determine how use will scale, how API vendor policies will change, or when teams will see measurable ROI. To dip their toes in the water, many app teams have slowly introduced AI features as a testbed. Many organizations with deeper pockets or a hunger to be on the bleeding edge, meanwhile, have adopted an all-in approach to LLM use. 

Vendors themselves face similar uncertainties tied to research and development, competitive pricing, and energy consumption from maintaining active, hardware-supported LLMs. This has been a crux of the ongoing discussion on the environmental sustainability of large-scale AI operations.

Ethics and fair use

While LLMs can be transformative, companies are still navigating some grey areas fairly new to the tech world. Because these models scrape the web for content, LLM responses are often compiled from existing works. Bots such as GPTBot, ClaudeBot, and others have raised new plagiarism concerns by scraping copyrighted or protected content — sometimes without consent or in defiance of robots.txt crawler rules. 

This doesn't mean that AI/LLM providers are inherently unethical. Some believe the AI race has led providers to pursue data collection without strong guardrails. However, companies such as Anthropic and Mistral claim to have taken humanistic, steerable approaches to AI development — those which consider inherent risks and "known unknowns" of LLM advancement. These providers and others have drafted copious messaging around building safe systems that are accessible and (sometimes) open source.

Does HAProxy support LLMs?

Yes! HAProxy One — the world's fastest application delivery and security platform — supports public and private LLM services running in any environment, whether they're hosted internally or externally. Our AI gateway solution helps your AI/LLM services scale massively while boosting reliability and security. This includes protection against prompt injection attacks, centralized API key and routing management, access control, end-to-end observability, and cost savings. 

To learn more about LLM support in HAProxy, check out our AI gateway solution and our blog post, Lessons learned in LLM prompt security: securing AI with AI.