Yuntao Jia

Post-training is now driving most improvements in foundation models, and billions are being poured into RL. But while RL might look like the next obvious infrastructure category, I’m not convinced it is. Inference scaled because it was an engineering problem. RL and post-training are different. The hard parts (environments, reward design, and evaluation) require domain expertise in the task and trial and error. A viable post-training infra company likely needs to offer something meaningfully new: continual, on-policy learning as a service. The service would need to provide: 1. Pipelines that synthesize RL environments directly from model outputs 2. Systems that construct (or at least propose) reward signals and verification 3. Background updates to model weights with minimal human intervention Full post in the comments.

19

2 Comments

Week 1 of #OlapOnTap covered: 1. Typing: how much it can matter in OLAP contexts: https://lnkd.in/gCQnNTNt 2. A deep dive into Cardinality, how it affects performance, and how using enums and LowCardinality() can help improve your performance: https://lnkd.in/g2d_pQJZ 3. Informed by this, how to efficiently design your table's ORDER BY to maximize your database's performance: https://lnkd.in/g2BnBCFF 4. How denormalization, and wider, flatter tables, are a better fit for OLAP database engines and query patterns https://lnkd.in/gyQQ62Gr The TL;DR for Week 1 is: 🤏 More restrictive types drastically improve performance, and low cardinality types can improve that even more 🔭 Structure your data based on your consumption patterns (both in terms of how you order your data, and how you structure your tables). Next topics coming up are: ➜ Modeling JSON & semi-structured data (noting how much heavy lifting the word "semi" is doing in this title) ➜ Nullability and default values ➜ Compression and encoding (and how you can take the reins more than you think here) ➜ Precomputing with materialized views The starting theme is modeling / setting up your OLAP database (if that wasn't obvious!). I am just kinda exploring topics I find interesting in the OLAP space, and I want to write LLM documents for my agents in respect of. But if any of you have particular topics you want me to cover, holler! Also, I'm focusing on ClickHouse implementations, but let me know if you want me to cover other databases. #DataEngineering #OLAP #MooseStack

11

LLMs are amazing at generating code. However, one of the biggest limitation of this code isn’t intelligence - it’s lack of context. Things get interesting (and messy) when it comes to large codebases. It falls apart and it's painfully obvious why. LLMs can generate working implementations from a high-quality prompt. At their best, they can produce a correct feature in one shot (“one-shotting”). This helps dramatically speed up development for clearly scoped tasks. This helps with: ✅ Productivity boost for implementation-heavy tasks - For repetitive CRUD operations or isolated scripts, LLMs can collapse hours of work to minutes. ✅ Enhanced prototyping — They help quickly explore solutions, enabling engineers to iterate design ideas faster. ✅ Lower entry barrier — Less context switching (e.g., from editor to docs), faster feedback loops. But these benefits fade when tasks require deep architectural knowledge, modular context, or long dependency chains, especially in large codebases. In this blog post Kieran Gill from Blueberry Pediatrics highlights some of the reasons it fails and how it can be improved. Here are a few reasons why LLMs fail with large codebases: ❌ Lack of global context: LLMs don’t “understand” the whole system - they only see what’s in the prompt. ❌ Hallucinations in unfamiliar parts: When missing architectural knowledge, LLMs may generate incorrect code, leading to bugs or regressions. ❌ Rework becomes more expensive than doing it manually: Failing to one-shot means repeated rounds — which erodes the time savings. Here's how you can solve the challenges: ▶️ Guide LLM using Prompt libraries that show Architecture, best practices, domain knowledge, etc. ▶️ Improve the codebase by making code modular, consistency in naming, and keep code clean. ▶️ Invest in review & automation for verification of LLM's design choices and end product (code, unit tests, etc.) In large codebases, the real productivity is gained by guidance + oversight + modular architecture. #ai #llm #aiengineering #engineering https://lnkd.in/gFzFunEV

18

Why do LLMs seem so hell-bent on providing fallbacks? Even with rules like "no fallbacks" and "fail fast by default", they bend over backwards to avoid failure to the point where they conflate domain concepts to avoid throwing an exception.

10

18 Comments

Everyone says LLMs are getting better. True. But shipping an LLM that users actually love still requires significant craft, meticulous engineering, and a focus that goes far beyond the model itself. ​The gap between a "cool demo" and a "reliable product" is where the real work happens. ​Here are the key areas that demand attention for building high-quality, production-ready LLM applications: 1. Prompting still matters. Models have improved, but they can’t read your mind. A well-crafted prompt can make results shine, while a poor one can make them fall flat. 2. Design for prompt agility. Build a deployment path where you can update prompts/configs without a full release. (Feature flags, versioned prompts, audit trails.) 3. Language mixing is real and sneaky. Sometimes the LLM itself slips in the wrong language mid-sentence. Without careful handling, your output ends up a confusing mix instead of what the user expects. 4. Multilingual isn’t free. Don’t assume the LLM will magically handle every language. You still need to evaluate results carefully and when things break, be ready to fix them with prompt engineering or fine-tuning. 5. 80% quality is “demo-easy.” 95% is war. The last 15% is where you pay: edge cases, latency tails, safety filters, caching strategy, and human UX. 6. Evaluation is annoying but non-negotiable. Do both: - Small, focused human evals for truth, nuance, and unexpected failure modes. - Large, automated runs (LLM judges + task metrics) for coverage and speed. Ship a repeatable eval pipeline before you ship features. 7. Integrate with the actual app early. LLMs don’t live in notebooks. Think auth, rate limits, retries, drafts/confirm flows, offline states, and analytics from day one. 8. Write failures into the UX. Hallucinations, timeouts, and “I don’t know” need graceful fallbacks. 9. Latency is a product feature. Users forgive a small mistake; they don’t forgive lag. Token streaming, caching, and truncation matter. 10. Guardrails are not just one filter. Stack them: prompt hardening, retrieval constraints, post-filters, and red-team tests. Defense in depth. 11. Training is expensive. Inference is more expensive. Training burns a hole in your budget once. Inference burns it forever - every request, every user, every day. Optimize throughput and cost-per-request early. Treat prompts like code, eval like CI, and the LLM like one component in a system not the whole system.

18

Everyone is debating the latest model releases and benchmark scores. But the real competition isn’t at the LLM layer anymore, it’s shifting to the application layer that sits on top of chat interfaces. Models will keep improving, but they’re converging. The moat is shifting from raw model intelligence to context and memory. Platforms that accumulate user intent, preferences, and behavioural context fastest will hit escape velocity. That’s why ChatGPT Apps matter so much. 🚀 They create a value exchange: third-party developers get organic distribution, and the platform gains more usage, more intent signals, and more personalized context to improve the experience for every app More apps → more usage → more user context → better app experiences → more apps. And round it goes. A full flywheel. ⚙️ But here’s the subtlety people miss: Apps and products become more powerful in this model, not less. They plug directly into an interface that already understands the user. Authentication, preference history, behavioural patterns, and previous queries all become inputs to your app’s experience. In every platform shift, visibility goes to the early movers. AI will be no different. Build your AI-app with Listo

43

2 Comments

On Tuesday we hosted a live product demo for Anyshift in San Francisco! 🔥 Thanks to Hilary Brennan-Marquez from MotherDuck for being part of it! Two new features shown for the first time: - Annie can now generate architecture diagrams on the fly from the full infrastructure graph - Proactive Annie continuously monitors your production, surfacing weak signals and monitoring gaps before they become incidents Hilary joined us from New York 🏙️ to share how her team uses Annie at MotherDuck. With a team that tripled in two years, Annie became the go-to partner for junior engineers during on-call and onboarding. The team runs everything through Slack and now has an "Ask Annie" channel where even non-infra engineers drop in with questions. More demos coming soon 👊 Reach out if you want to see Annie in action.

16

I read a fascinating paper over the weekend that continues to convince me neuroscience and AI model research are eventually related tracks: https://lnkd.in/eTGYtG4x. Researchers discovered they can identify specific neurons in LLMs that predict when the model is about to hallucinate. Even more striking: less than 0.1% of neurons are responsible for these errors. Think about that. Just like neuroscientists can pinpoint which parts of the human brain light up during specific activities, we can now see which AI neurons are firing when a model generates false information. The team traced these neurons back to pre-training, showing they emerge early in the model's development. This is very similar to how certain neural pathways form in our own brains. The research reveals these neurons are causally linked to over-compliance behaviors, where models prioritize generating plausible-sounding responses over admitting uncertainty. What excites me most is the practical potential. We can develop better training approaches and even real-time detection systems. The researchers expanded on previous work to identify multiple categories of hallucination causes, giving us a more nuanced view of why models fail. Really looking forward to how the research evolves on this one!

12

1 Comment

Weekly arXiv Scan: 3 Papers that caught my eye (and my Agent’s) My custom Gemini x OpenClaw agent scanned recent papers on arXiv this week in Audio, NLP, and GenAI. It filtered for novelty and engineering rigor. I filtered for practical application and "human" impact. Here are the winners that made the cut during the baby nap time :) 1. Scaling Open Discrete Audio Foundation Models (SODA) https://lnkd.in/eXYnjfpd The Agent's TL;DR: Introduces SODA, a 4B parameter native audio model trained on interleaved semantic, acoustic, and text tokens, establishing new scaling laws for non-cascaded speech tasks. My Take: Human conversation relies on tone, pauses, and emotion. Those are the things that get stripped away when we convert Speech-to-Text for an LLM. By processing audio tokens natively, we aren't just processing data; we are preserving the humanity of the interaction. Native audio-in architecture is what we need for empathetic AI. 2. Reverso: Efficient Time Series Foundation Models https://lnkd.in/eyAgsN_h The Agent's TL;DR: Proposes a hybrid architecture interleaving linear RNNs and long convolutions that matches Transformer zero-shot forecasting performance while being 100x smaller in parameter count. My Take: Time-series analysis and prediction might be the problem where the sledgehammer solution is an overkill. A model that is 100x smaller is a model one can actually use for the personal gains (like some extra signals for investments? maybe?). It’s faster to train, cheaper to serve, and easier to debug. 3. A Generative-First Neural Audio Autoencoder https://lnkd.in/eeFtDpkB The Agent's TL;DR: Proposes an architecture with a massive 3360x temporal downsampling factor, allowing 60 seconds of audio to be represented by fewer than 800 tokens for extremely fast generation. My Take: High fidelity is great, but latency is the user experience killer. Compressing a minute of audio into <800 tokens is a massive engineering win. This reduces the context burden on the LLM significantly, meaning faster, cheaper, and more responsive voice agents. This is a great step towards the feeling of a snappy real-time conversations. The trend this week is “Efficiency over Excess”, finding architectures that do the same work with a fraction of the compute. #NLP #AudioAI #MachineLearning #HumanInTheLoop #arXivScan

41

8 Comments

While I’m over here crafting "𝙘𝙖𝙩𝙘𝙝𝙮" LinkedIn hooks and fluffy strategy pieces, the actual geniuses at Pinch AI are busy engineering 𝟵𝟬% 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗴𝗮𝗶𝗻𝘀. They just moved our feature runs from hours to minutes. 🚀🚀 🦀 I’ll stick to the emojis. 🦀 If you want to see what real technical heavy lifting looks like (and why I leave the data architecture to people far smarter than me), check out Mandy's (Mandeep Kaur) breakdown on our switch to DuckDB.

12

80.3% on SWE-Bench Pro. Anthropic held that number back from public release for three months, judging it too disruptive. As of June 9, any team with an API key can run it. Claude Fable 5 is the first Mythos-class model opened to general developers. The key capability: full codebase-wide migrations completed in a single day. If you've been estimating migration work in weeks, that math has changed. Worth knowing before you ship anything with it. Anthropic routes sensitive queries - biosecurity, cyberweapons - to a weaker model rather than refusing outright. The company retains all traffic data for 30 days under its own security policy. The higher-capability Mythos 5 tier went to existing Project Glasswing partners at the same time - public Fable 5 is not the ceiling. Separately, the White House executive order signed June 2 requires a mandatory 30-day pre-release government testing window for AI models going forward. If you're building on this stack long-term, that review layer matters for release timeline planning. At SOKIL we run Java/React delivery teams with AI-augmented workflows. Honest read: Fable 5 is the first model where we'd scope a migration task as an agent task, not a human task with AI assistance. That changes how we write estimates and how we talk to clients about timelines. If your competitors start running autonomous agents on migration work, when does that start showing up in their proposals to your customers? #AI #EngineeringLeadership #DeveloperTools #Anthropic #LLM

3