Rahul Mitra

'If OpenAI or Anthropic were given twice the inference compute today, they would double their revenue within a month.' The compute bottleneck, beautifully illustrated by Jonathan Ross, Groq, when asked by Harry Stebbings: Answer: These companies are compute-limited, not demand-limited. Three quick reasons: 1. Rate limits → lost sales. Anthropic/OpenAI cap tokens & QPS. More inference = more tokens served = more billable usage, immediately. 2. Latency drives engagement. Faster responses grow session length, task completion, and repeat usage—directly raising ARPU. 3. Quality scales with compute. More tokens/ensembles/tool calls per query = better answers = higher willingness to pay. The CPG analogy (John's example): In consumer packaged goods, the highest margins correlate with speed of effect (think nicotine vs. water). Faster dopamine loop → stronger brand affinity → better economics. AI is similar: shorter latency = stickier product = higher revenue. We’ve seen this movie before: - Google obsessing over page load speed. - Facebook shaving 100ms and seeing conversion lift (order-of-single-digits percent per 100ms). Speed compounds engagement. Engagement compounds revenue. Bottom line: Give OpenAI or Anthropic 2× inference and, within weeks, you unlock: more tokens served, higher conversion, better outputs per query, and premium-tier users actually buy. In a compute-constrained market, throughput and latency are the growth levers. Congrats to Groq on their latest raise at a $7B valuation and the new partnership with McLaren Racing — pushing the frontier of compute, speed, and scale. Mohsen Moazami | Ian Andrews | Chelsey Susin Kantor https://lnkd.in/es53UTkR

27

Why are we still renting GPUs 24/7? I looked at real telemetry. Most AI infrastructure is sitting idle, burning money. Even NVIDIA has pointed this out: enterprise GPUs often run single models at ~30% memory utilization. We keep engines running at red lights because we’re afraid of cold starts. Somewhere along the way, we normalized waste. I wrote about why this needs to change and what serverless inference actually looks like. Full article: https://lnkd.in/g6-vreTx

29

2 Comments

Process is key! For anyone who has taken my Data Modeling course (formally, "Intro to Data Science"), I make students use the cross-industry standard process for data mining (CrISP-DM). Andrew Ng's post helps us see that the CrISP-DM process also fits the deployment of AI. His focus on the "Evaluation" step of the CrISP-DM showcases the robustness of the process (all the other steps are important too).

9

Ram Rajya is back. This year at NVIDIA GTC, after all the flashy launches, one thing became very clear to me: We’re no longer paying for compute. We’re paying a heavy premium for memory. GPUs are becoming commoditized faster than we expected. TFLOPs are everywhere. But RAM is the real bottleneck. LLMs don’t choke because of lack of compute. They choke because context needs memory. Agents need memory. Inference at scale needs memory. The future isn’t CPU-bound. It isn’t even GPU-bound. It’s RAM-bound. Whoever controls: high-bandwidth memory larger context windows cheaper, scalable RAM access …controls the next wave of AI. We’re quietly moving into a world where: Memory is the new oil. Welcome to Ram Rajya 😅 And this time, it’s not a mythological one.

205

7 Comments

AWS spent years building its own AI chip. Then it announced it needs someone else's. Last Thursday, AWS split inference across two companies' silicon — Trainium for prefill, Cerebras for decode. No hyperscaler has done this before. Amazon Web Services (AWS). Cerebras. NVIDIA. Groq. Huawei. AMD. Google. Broadcom. Eight companies. Three continents. One architectural conclusion. The GPU isn't dying. It's splitting, because reasoning models broke the economics. Prefill on one chip. Decode on another. And the company that holds the pieces together captures the margin. New piece on The AI Realist: "AWS Built Its Own AI Chip. Now It Needs Someone Else’s" - Link in comments 👇

83

7 Comments