Xuan Han

If you're at Anyscale's Ray Summit this week, three members of the Samsara team, Hao Yuan, Freeman Mak, and Eagle Lo, are sharing how we use Ray to run inference for physical AI. Their work is a major cornerstone of the amazing safety impact Samsara's technology brings to the world to the tune of 380,000 accidents prevented leading to about 4 lives saved per day over 100B miles last year. Check out their great work!

33

2 Comments

Here's an article I would have found useful when I was first trying to understand how the attention mechanism in LLMs / transformers actually works. There are already great explanations of transformers and attention out there. Jay Alammar's Illustrated Transformer is excellent, and Attention Is All You Need is more readable than people expect. This piece is my attempt to connect the dots between the math and prompting. The key idea is simple. Once you understand how attention computes relevance between tokens, prompting stops feeling like guesswork. You start to see why being specific helps, why chain-of-thought improves reasoning, and why instruction placement in long prompts matters. To make it more digestible, I built a few interactive visualizers so you can watch tokens get embedded, see attention scores form, and step through the prediction process yourself. If you've ever wondered what's really happening when you hit "send" on a prompt, this might be useful. https://lnkd.in/gApqTxAh

19

1 Comment

Why do LLMs return different responses to the same prompt? 🤔👉 Interestingly enough the Transformer architecture underpinning Large Language Models (LLMs) is not inherently non-deterministic, given identical inputs it is capable of computing identical outputs! So why does the same prompt sometimes produce a different response in systems like ChatGPT or Claude? Non-determinism enters these system through two sources: design decisions and operational entropy. 1️⃣ By Design These systems don’t just "pick an answer", they generate text by sampling from a probability distribution of possible next tokens (small sequences of characters) and continually feed each selected token back through the system. So instead of always choosing the most probable token (known as greedy decoding), they intentionally sample to produce more natural human-like text. Factors such as: - Temperature settings control how varied the system can be when selecting the next token, higher temperature values yield more creative outputs - Top-k sampling limits reshape the probability space before selection which also constrains which can also help to increase or reduce perceived determinism balanced with diversity of responses We have to remember that these systems are designed to achieve their objectives as a human might, this design choice introduces intentional diversity in responses, non-determinism here is not a flaw. 2️⃣ Operational Entropy Even with temperature set to zero and a very small top-k, responses can still differ. That’s because floating-point arithmetic on GPUs and TPUs is non-associative, meaning the order of operations matters when the math is performed. When thousands of requests are processed concurrently, tiny timing or batching differences cause slight numerical shifts in the systems logits. These small deviations cascade producing subtle variations in all to be generated tokens. This of course compounds in very large responses. This is not due to inherent randomness in the base models themselves, but to the realities of distributed, parallel computation in a highly stateful system of this scale. 😬 So, non-determinism is both a feature and a by-product: - By design, it enables creativity and linguistic diversity - By circumstance, it reflects the unavoidable quirks of floating-point math and at scale inference Even “deterministic” configurations can still vary slightly which is why two identical prompts don’t always yield identical answers. (I’ve intentionally used the term "system" rather than "model" to reflect how modern tools like ChatGPT and Claude are no longer isolated models deployed within an inference stack, but complex, orchestrated systems in their own right, combining multiple models, retrieval mechanisms, control layers, and interaction policies to deliver coherent, adaptive behaviour.)

17

3 Comments

I read a fascinating paper over the weekend that continues to convince me neuroscience and AI model research are eventually related tracks: https://lnkd.in/eTGYtG4x. Researchers discovered they can identify specific neurons in LLMs that predict when the model is about to hallucinate. Even more striking: less than 0.1% of neurons are responsible for these errors. Think about that. Just like neuroscientists can pinpoint which parts of the human brain light up during specific activities, we can now see which AI neurons are firing when a model generates false information. The team traced these neurons back to pre-training, showing they emerge early in the model's development. This is very similar to how certain neural pathways form in our own brains. The research reveals these neurons are causally linked to over-compliance behaviors, where models prioritize generating plausible-sounding responses over admitting uncertainty. What excites me most is the practical potential. We can develop better training approaches and even real-time detection systems. The researchers expanded on previous work to identify multiple categories of hallucination causes, giving us a more nuanced view of why models fail. Really looking forward to how the research evolves on this one!

12

1 Comment

I got tired of manually comparing LLMs… so I published this, rankthellmmodel.info (Rank the LLM model), which I built for my own use. based on my requirements. While working with LLMs, I kept hitting the same problem: • Checking multiple sites for benchmarks   • Looking up pricing separately   • Comparing everything manually   • Still guessing which model is actually “better.” It was slow and confusing. 📊 A simple leaderboard that ranks models based on: ✔ Quality   ✔ Cost   ✔ Speed  (Data sourced from artificialanalysis.ai , will keep adding more benchmarks) ⚙️ How it works Quality → GPQA, MMLU, coding scores   Cost → price per million tokens   Speed → tokens/sec, latency  Overall = 0.5 × Quality + 0.3 × Cost + 0.2 × Speed   👉 Good + Fast + Affordable 💡 What I realized • The smartest model is not always the most useful   • Faster models are often “good enough.”   • Some top models are too expensive for production  🚧 Next: • Custom weights   • Real API latency testing   • More benchmark data  Feel free to check it out and share your feedback #AI #LLM #GenAI #MachineLearning #OpenAI #Gemini #Claude #Engineering

54

1 Comment

Are LLMs a dead end? In a recent interview (https://lnkd.in/gcFcFk7P), Richard Sutton — often called the father of reinforcement learning — said he believes large language models are a dead end. That statement struck a chord. At UCLA, researchers once described a contrasting idea: Small Data, Big Intelligence — the notion that human-level reasoning may arise not from scale, but from abstraction, generalization, and structure. These perspectives aren’t mutually exclusive. They reveal the growing divide between two camps: those chasing scale, and those chasing understanding. And yet, regardless of where AGI research goes, the current generation of AI systems already lets us build extraordinary tools. They may not fascinate theorists, but they are transforming how products are built, how physicians work, and how knowledge spreads. The frontier of intelligence might not be where we expect — but millions of builders won't wait for that. Do you think scale alone will be driving the next breakthrough?

5

1 Comment

Many are heralding large language models (LLMs) as transformative forces in our world. While I acknowledge the significant advancements they represent, I believe LLMs are not the ultimate goal for artificial general intelligence (AGI). For years, we have relied on neural networks, which serve as the foundation for LLMs. We have developed architectures that predict outcomes based on data to support decision-making. Additionally, we have refined datasets to provide context for various problems. However, the ability of LLMs to generate text and code does not equate to true intelligence. My skepticism stems not from a lack of potential in these models, but from the understanding that they are fundamentally similar to the algorithms that preceded them. AGI will not and does not stem from the LLMs we see today. LLMs are simply machines that have been trained on large datasets to determine the best word to place in front of the string of words they have already generated. Please do not be mistaken in thinking your LLM is somehow intelligent beyond making some significant mistakes.

9