LLM Trends

22 Sep 2025, 5:00 pm IST · Chennai
Anand S · LLM Psychologist · Straive
Transcript · CC0 - Public Domain

https://talks.s-anand.net/

LLM are getting better and cheaper

LLM context windows are increasing

So policy documents can be the code / rules

LLMs read & write with more attention

Dynamic code is a new modality

Demos:

  • Topic Trends: an AI-coded app that shows trends in documents (design-to-deploy in 2 hours).
  • SchemaForge: automate data engineering; schema + DBT tests
  • HypoForge: automate data science; generate + test hypotheses
  • DataGen: automating synthetic data generation for stress testing

LLMs think for longer without errors

This lets us automate innovation

LLMs evaluate LLMs as well as humans

So we can evaluate quality at scale

  • Double-checking. Using multiple LLMs-as-a-judge with consensus reduces hallucinations dramatically
  • Prompt effectiveness. Using an LLM-as-a-judge automation framework systematically highlights effective prompts
  • Prompt improvements. Use LLMs to write and improve prompts by evaluating the effectiveness of each

LLMs are far more multi-modal

That lets us generate far more than text

  • On-device AI surges: NPUs in phones/laptops (Apple Intelligence PCC; Copilot+ PCs) make private, low-latency inference mainstream. (arXiv)
  • Open-weight models are catching closed models on key benchmarks; costs per task are falling fast (AI Index 2025). (Stanford HAI)
  • Inference throughput jumps via open stacks (vLLM, TensorRT-LLM/SGLang): paged-KV caching, continuous batching, and quantization. (VLLM)
  • Data licensing + provenance mature: major publisher deals (e.g., News Corp) and C2PA Content Credentials rollout. (OpenAI)
  • Regulation + power reshape roadmaps: EU AI Act GPAI duties from 2 Aug 2025; data-centre electricity demand set to ~double by 2030. (Digital Strategy)
  • “OpenAI-compatible” APIs become a de-facto portability layer across clouds/runtimes; tooling lock-in still varies. (SemiAnalysis)
  • Synthetic data’s share in training rises; benefits vs. model-collapse risks remain debated. (WIRED)
  • Graph-RAG/hybrid retrieval outperforms naive vector search; early wins, but long-context advances may narrow the gap. (Google Research)
  • Browser-native LLMs via WebGPU (WebLLM/Transformers.js) enable private/offline features; fragmented support and bugs slow scale-up. (blog.mlc.ai)
  • Non-NVIDIA accelerators (AMD MI300, Gaudi 3) gain inference share; CUDA ecosystem advantage clouds outlook.

LLM Trends

22 Sep 2025, 5:00 pm IST · Chennai
Anand S · LLM Psychologist · Straive
Transcript · CC0 - Public Domain

https://talks.s-anand.net/

Quiz

  1. What’s the key difference between prompt caching and normal caching in LLM APIs?
  2. Why can long context windows still miss relevant info (“lost in the middle”) and how would you mitigate it?
  3. In this workflow, where should analysis run on full data and where is sampling OK?
  4. When is “LLM-as-judge” reliable, and what biases should you watch for?
  5. Name two reasons code sandboxes avoid direct internet access and how to work around it.

Errata

  • Prompt caching ≠ a “tiny copy of the model.” It stores/charges cached prompt segments to speed/cheaply reuse, without changing the model. (Medium)
  • Advanced Data Analysis (code interpreter) has no open internet. Browsing is a separate tool; the sandbox itself blocks external calls. (OpenAI)
  • “LLMs evaluate LLMs as well as humans” — nuanced. LLM judges correlate with humans but show biases; human calibration still needed. (16x Prompt)
  • Context window ≠ long-term memory. Bigger windows allow longer inputs, but retrieval can drop for middle content. App “memory” is separate. (ar5iv)
  • “Gemini is free & good at photos/video/audio.” There is a free web app; advanced multimodal features are via Gemini 1.5/API tiers. (OpenAI)
  • “Whisper is behind ChatGPT voice.” OpenAI documents Whisper as its speech-to-text used in ChatGPT voice features. (OpenAI)
  • “LLMs are getting cheaper” — evidence. Example prices (per 1M tokens): GPT-4o $5 in/$15 out (May 2024); vendors have introduced price cuts/cheaper tiers. (arXiv)
  • Model capabilities & windows. Current public windows: Gemini 1.5 up to 1M; Claude 3.7 Sonnet 200k; OpenAI o-series context sizes are model-specific. (Claude Docs)

Counterpoints

  • “LLM context windows are increasing” → But… Longer windows can hurt recall; retrieval/RAG or windowing often beats naive full-context dumps. Consider chunk-and-vote or salient-first prompts. (ar5iv)
  • “LLM-as-judge scales quality” → But… Judges show systematic biases (position, verbosity, self-enhancement). Use blinded, shuffled, multi-judge panels + human spot-checks and gold standards. (16x Prompt)
  • “More structure (JSON/XML) now” → But… Plain “JSON mode” is not bullet-proof. Prefer structured outputs/function calling with schemas for reliability. (OpenAI)
  • “Far more multimodal” → But… Vision-language models still hallucinate objects and captions. Keep human-in-the-loop for safety-critical use. (ACL Anthology)
  • “Policy docs can be the code/rules” → But… Governance needs documented controls. Follow NIST AI RMF (Govern/Map/Measure/Manage) when automating compliance. (NIST AI Resource Center)
  • “LLMs think for longer without errors” → But… Long-horizon agency is far from solved; MAUD/METR find persistent failures on multi-step tasks. Use decomposition & verifiable subgoals. (Vals AI)

Feedback

  1. Label claims vs. opinions. Add “Claim/Why/Limitations” badges on slides with bold assertions (e.g., LLM-as-judge, policy-as-code).
  2. Show the why behind choices. One slide comparing ChatGPT/Claude/Gemini/Jules/Codex by task (EDA, front-end, agents, cost), with 1-line rationale each.
  3. Make failure modes visible. Add a “Bugs I expect” checklist (net-blocked sandbox, path issues, file limits) and the quickest CLI/browser fixes.
  4. Quantify effects. When stating “longer books rate higher,” show effect sizes + CI, then the same chart after outlier clipping.
  5. Tighten the demo loop. Pre-bake a tiny dataset and a repo branch that always builds; live-switch to bigger data only if time permits.