Anand M.

Why do LLMs return different responses to the same prompt? 🤔👉 Interestingly enough the Transformer architecture underpinning Large Language Models (LLMs) is not inherently non-deterministic, given identical inputs it is capable of computing identical outputs! So why does the same prompt sometimes produce a different response in systems like ChatGPT or Claude? Non-determinism enters these system through two sources: design decisions and operational entropy. 1️⃣ By Design These systems don’t just "pick an answer", they generate text by sampling from a probability distribution of possible next tokens (small sequences of characters) and continually feed each selected token back through the system. So instead of always choosing the most probable token (known as greedy decoding), they intentionally sample to produce more natural human-like text. Factors such as: - Temperature settings control how varied the system can be when selecting the next token, higher temperature values yield more creative outputs - Top-k sampling limits reshape the probability space before selection which also constrains which can also help to increase or reduce perceived determinism balanced with diversity of responses We have to remember that these systems are designed to achieve their objectives as a human might, this design choice introduces intentional diversity in responses, non-determinism here is not a flaw. 2️⃣ Operational Entropy Even with temperature set to zero and a very small top-k, responses can still differ. That’s because floating-point arithmetic on GPUs and TPUs is non-associative, meaning the order of operations matters when the math is performed. When thousands of requests are processed concurrently, tiny timing or batching differences cause slight numerical shifts in the systems logits. These small deviations cascade producing subtle variations in all to be generated tokens. This of course compounds in very large responses. This is not due to inherent randomness in the base models themselves, but to the realities of distributed, parallel computation in a highly stateful system of this scale. 😬 So, non-determinism is both a feature and a by-product: - By design, it enables creativity and linguistic diversity - By circumstance, it reflects the unavoidable quirks of floating-point math and at scale inference Even “deterministic” configurations can still vary slightly which is why two identical prompts don’t always yield identical answers. (I’ve intentionally used the term "system" rather than "model" to reflect how modern tools like ChatGPT and Claude are no longer isolated models deployed within an inference stack, but complex, orchestrated systems in their own right, combining multiple models, retrieval mechanisms, control layers, and interaction policies to deliver coherent, adaptive behaviour.)

17

3 Comments

I’ve been thinking about a different way to design memory for AI agents: flashback memory. Humans rarely remember the past as isolated facts. If I lose my wallet, I don’t just recall, “I bought coffee.” I mentally replay a scene: I left home → walked to work → bought coffee → used my card → put the wallet back → took a taxi → reached the office → later noticed the wallet was gone. That one flashback restores a connected sequence of events and helps me infer something new: I probably left the wallet in the taxi. What if AI agents could remember in the same way? Instead of retrieving isolated chunks, an agent could reconstruct a past episode with the scenes around it: observation → context → hypothesis → failed reasoning → new evidence → action → outcome So if an agent sees a familiar production failure, it does not just retrieve: “Rollback worked last time.” It recalls the broader scene: What was happening then? What did I initially think? What evidence changed that view? Why did rollback work? Is today’s environment actually the same? That changes the memory flow from: retrieve → reuse to: remember → reconstruct the scene → compare → reason → decide The key idea is that one flashback can restore many connected memories at once, and those relationships can help the agent make a decision or even derive something that was never explicitly stored. I think this could be particularly useful for long-running agents in SRE, software engineering, cybersecurity, research, and personal assistance. The goal is not to make AI imitate humans for the sake of it. It is to let agents use past experience as perspective for understanding the present. I wrote a deeper architecture-focused article around this idea, including episodic reconstruction, memory routing, then-vs-now comparison, and counterfactual replay. #AI #AIAgents #LLM #AgentMemory #GenerativeAI #ArtificialIntelligence #AIArchitecture

9

I've been reviewing Stripe ML interview failures for the past six months. Most engineers show up ready to discuss standard algorithms and model architectures. That's not what Stripe cares about. They're evaluating systems thinking and product judgment. Yet candidates keep drilling LeetCode and memorizing hyperparameter definitions. That approach fails consistently. The reality is this: 🔹 Payment systems have unique constraints most ML roles never face 🔹 Fraud detection requires different thinking than recommendation engines 🔹 Stripe wants engineers who understand business impact, not just accuracy metrics Here's what actually matters: ✅ Addressing fairness and bias in financial decision systems ✅ Walking through feature engineering with payment data constraints ✅ Discussing model latency requirements when blocking transactions in real-time ✅ Explaining how you'd handle class imbalance in fraud detection (it's extreme at Stripe's scale) The engineers who succeed aren't the ones with the most impressive academic backgrounds. They're the ones who connect ML decisions to business outcomes and user experience. I've seen candidates with perfect technical answers get rejected because they couldn't explain why their solution mattered. And I've seen others advance because they framed everything around Stripe's actual challenges. Your Stripe prep should reflect what they actually evaluate. Study their products, understand their constraints, think like a Stripe engineer. 📘 Get the complete question breakdown: https://lnkd.in/gAz-r5PC

6

Many of the hard problems that LLMs struggle with are fundamentally not generative AI problems. They’re often statistical inference problems: how we structure and aggregate evidence, how we weigh competing signals, how sensitive we are to outliers, how uncertainty changes as we introduce more information. Even if large language models have large context windows in theory, outputs can get worse as inputs get longer. Your dataset might include everything needed to produce an answer in theory, but solving your data science question is just one of many things that models and agents are optimising for in practice, from prose readability to code run-ability. The OOLONG benchmark below shows just how much models can struggle to count and classify even simple, unambiguous datasets. (For context, 250k tokens would equate to about 2000 survey responses, each around 100 words.) Chunking up data to squeeze it inside the context window, or making models with longer context windows won't solve this problem. Because ultimately, these are statistical inference questions rather than just context problems.

27

1 Comment

LLMs don't hallucinate because they're "broken"—they do it because our entire training and evaluation ecosystem rewards guessing. A groundbreaking paper by Kalai et al. (2025) provides a rigorous statistical takedown of the problem. My latest blog post dives into their core arguments: 1️⃣ Statistical Origins: Hallucinations are a predictable result of pretraining, reducible to fundamental errors in binary classification. The math shows that if a model can't judge a fact, it's bound to generate fictions. 2️⃣ Misaligned Incentives: Our main benchmarks (MMLU, etc.) use 0-1 scoring, effectively penalizing models that admit uncertainty ("I don't know"). This creates an "epidemic" that trains models to bluff for a higher score. The paper offers a powerful, pragmatic solution: change the rules of the game. By modifying existing leaderboards to penalize wrong answers, we can incentivize the development of models that know what they don't know. This is a critical step towards building AI we can actually trust in the real world. Read my full analysis here: https://lnkd.in/eaSEnqvw #AI #LLM #Hallucination #MachineLearning #TrustworthyAI #Tech #Engineering

5

Will generative AI be for software what the printing press was for writing? The prospect is exciting, and there's a ton of hype and capital riding on the assumption that it is - or that it will be. If it pans out, the implications are potentially world-changing, in the same way that the printing press gave us revolutions in religion, politics, and science. But what would a printing press for software actually look like? It would be something that dramatically reduces the cost of software; from what I can tell, we're talking about several orders-of-magnitude cost reductions. The cost of a book went from multiple-weeks-of-wages to less-than-a-day's-wage over 200 years, according to a 2019 study. Relatedly, a software printing press will enable software production in scenarios which never made sense before. I spend a lot of time thinking about AI code generation in the enterprise, but that's not where we're going to see these kinds of transformative changes. Instead, we are likely to see the printing press effect emerge in bespoke software, written for a small group of people. Such software tackles personal finance management, productivity setups, vacation planning - and who knows what else. It stretches into small business settings, for inventory management, sales pipeline tracking, and a hundred different use cases that are either too quirky or too low-budget for big-business software. And let's not forget the constellation of small community groups that need membership directories and calendars, but find the SaaS offerings too constraining. There's a vast sea of software waiting to be written for these use cases. But who will serve this market? Does this market actually want to be served? Today there are offerings targeting small-scale bespoke software, such as Lovable and n8n, and they are preceded by a bunch of low-code/no-code platforms. Are they capable of dramatically lowering the cost of software engineering for these bespoke, small-scale use cases? There is reason to be hopeful: we are talking about relatively simple, greenfield software development, and in some cases short-lived software that doesn't need to be maintained for years. These are exactly the scenarios where generative AI shines. But even if the technology is up to the task, there are still questions about demand. Do individuals and small groups actually want bespoke software? In many cases, bespoke software could provide significant value. But will potential customers think to seek it out? If they do, will they be willing to wrangle the development process - such as it is? Will these customers be willing to pay for bespoke software, and if not - how will the underlying tools stay afloat? It's unclear whether this trend will take off. Supposing it does, what comes next? I don't think anyone who witnessed the invention of the printing press imagined that it would lead to a religious reformation 75 years later. What's going to happen if software becomes personal?

3

7 Comments

Vibecoding is going to crumble the whole edifice. but it doesn't have to be that way. AI-assisted coding actually frees up engineers to think more deeply about the underlying structures. Now, "slow" engineers can actually slow down and fix the deeper issues while AI agents do the mindless stuff.

3