Mahesh Chulet

Tech companies are quietly moving away from classic MCP tool calling. Here's what changed: MCP is the standard that lets AI agents use tools. Ask your agent: "Find this customer and show me their orders." • It calls tool 1 to find the customer • The answer goes back to the model • It calls tool 2 for the orders • Back to the model again Bigger jobs? Again. And again. And again. Every trip costs tokens. And with thousands of tools, the tool list alone floods the context. The fix: Code Mode. The model writes code. The code calls the tools. Everything runs in one go. Why it works: LLMs have read millions of lines of real code. They've seen very few tool-call examples. Cloudflare's API has 2,500+ endpoints. With Code Mode, the agent only needs 2 tools. Stop making agents call tools. Let them write code.

Got an AI engineer interview coming up? These 25 questions come up again and again. Prep them and you'll walk in ready: • What's the difference between a chatbot and an agent? • How does an agent decide which tool to use? • Explain the reasoning loop: think, act, observe, repeat • When would you use RAG instead of fine-tuning? • How do embeddings and vector databases work together? • How do you handle a context window that's too small? • Short-term vs long-term memory: how would you design both? • What is MCP and why does it matter? • Single agent or multi-agent: when do you pick each? • How does an orchestrator split work between sub-agents? • How do you stop an agent from looping forever? • What guardrails would you add before giving it real tools? • Where should a human approve actions? • How do you evaluate whether an agent actually works? • What metrics would you track in production? • How do you debug an agent that gave a wrong answer? • How do you reduce hallucinations? • How do you cut token cost without hurting quality? • How do you reduce latency in a multi-step agent? • How would you handle a tool call that fails? • How do you protect against prompt injection? • How would you design an agent that books flights? • How would you design a customer support agent? • What breaks first when you scale agents to 1M users? • Tell me about an agent you built and what failed Most candidates can explain an LLM. Very few can design a system around one. Bookmark this and send it to someone with an interview this month.

50 terms every AI builder should know: 1.) LLM: a large language model trained to predict the next word 2.) Token: a small chunk of text the model reads and writes 3.) Context window: how much text the model can see at once 4.) Prompt: the instruction you give the model 5.) System prompt: hidden instructions that shape every response 6.) Temperature: how random or creative the output is 7.) Inference: running a trained model to get an answer 8.) Latency: how long you wait for a response 9.) Hallucination: when the model confidently makes things up 10.) Fine-tuning: training a model further on your own data 11.) RAG: fetching relevant documents and feeding them to the model 12.) Embedding: turning text into numbers that capture meaning 13.) Vector database: storage built to search embeddings fast 14.) Chunking: splitting documents into pieces for RAG 15.) Reranking: sorting search results so the best ones go first 16.) Agent: a model that plans and takes actions on its own 17.) Tool calling: letting the model use APIs, code or search 18.) MCP: a standard way to connect models to tools and data 19.) Agent loop: think, act, observe, repeat 20.) Subagent: a smaller agent handed one piece of the job 21.) Orchestration: coordinating multiple agents or steps 22.) Memory: what an agent remembers between sessions 23.) Guardrails: rules that block unsafe or off-topic output 24.) Prompt injection: hidden text that tricks a model into misbehaving 25.) Jailbreak: a prompt that gets around a model's safety rules 26.) Evals: tests that measure how well your AI performs 27.) Benchmark: a standard test used to compare models 28.) LLM-as-judge: using one model to grade another's output 29.) Ground truth: the correct answer you test against 30.) Structured output: forcing the model to reply in JSON or a set format 31.) Function schema: the definition of a tool the model can call 32.) Streaming: showing the response word by word as it's generated 33.) Caching: reusing past results to save time and money 34.) Prompt caching: reusing a repeated prompt prefix to cut cost 35.) Rate limit: how many requests you're allowed per minute 36.) Open weights: a model you can download and run yourself 37.) Distillation: training a small model to copy a big one 38.) Quantization: shrinking a model so it runs faster and cheaper 39.) LoRA: a cheap way to fine-tune only a small part of a model 40.) GPU: the chip that runs AI models 41.) Multimodal: a model that handles text, images, audio or video 42.) Reasoning model: a model that thinks step by step before answering 43.) Chain of thought: the model writing out its reasoning first 44.) Few-shot: giving the model examples in the prompt 45.) Zero-shot: asking with no examples at all 46.) Vibe coding: building software by describing it to AI 47.) Harness: the setup of tools and rules around a coding agent 48.) CLAUDE.md: a file of project rules Claude Code reads every session 49.) Skills: reusable instructions an agent loads when it needs them 50.) Human in the loop: a person approving key steps before they run Bookmark this. You'll need it in your next AI meeting.

It’s almost impossible to comprehend just how massive the gap has become between people working at the AI frontier and everyone else. What’s even crazier is that this gap isn’t closing. It’s getting wider. If AI progress had started to plateau, most people and companies would have had time to catch up, integrate the technology, and standardize their workflows. Eventually, AI would have become a level playing field, with everyone using roughly the same tools and capabilities. The competitive advantage would have diminished. But we’re seeing the exact opposite. AI progress is accelerating at an almost vertical rate. That means the workflows you carefully built and standardized 3–6 months ago could already be outdated. Now think about companies that started adopting AI a year ago, assumed they had figured it out, and moved on. They’re potentially even further behind. I’ve never experienced a technology in my life or career that evolves this quickly. And that means our expectations need to evolve just as fast. Our brains aren’t wired to process this pace of change, and neither are traditional organizational change management processes. When you introduce recursive self-improvement into the equation, things get even more interesting. We’re entering a period where you constantly need to revisit problems you couldn’t solve a few months ago and ask whether AI can solve them today. Solutions that were impractical yesterday might suddenly become viable. AI-native companies clearly have a significant head start in this environment. The real question now is whether larger, more established organizations can adapt quickly enough to keep pace. Wild times.

Tired of Claude Code asking "Allow this command?" 50 times a day: ~/.claude/settings.json { "permissions": { "allow": [ "Bash(npm run test:*)", "Bash(git status)", "Bash(git diff:*)" ] } } Safe commands run. Everything else still asks.

Here’s how to truly master LLM evals, step by step, with real projects to demonstrate your skills: 1. Build something that can fail first Don’t study evals in theory alone. Build a small LLM application with real failure points: RAG over a documentation site, a text-to-SQL bot, or an agent that uses tools. It doesn’t need to look good. You need traces, not polish. 2. Manually inspect 100 traces Export inputs, outputs, retrieved chunks, and tool calls into a spreadsheet. Describe each failure in one plain-English sentence. Group similar failures, then count them. Most people skip this and end up building metrics for problems they don’t actually have. Project: Publish your failure taxonomy with counts for each category. 3. Create your first golden dataset Start with 50 cases covering your top 3 failure categories, 20 straightforward happy-path cases, and 10 adversarial examples. Include columns for input, expected output or rubric, category, difficulty, and segment. Commit the dataset to git and tag every prompt change with the dataset version used to evaluate it. 4. Generate synthetic data the right way Define your dimensions (persona × intent × complexity × edge case), generate combinations with a script, and use an LLM to turn each combination into a natural-language query. Review and discard low-quality examples. Use embeddings to identify near-duplicates. Project: Build a 300-case synthetic dataset with a coverage matrix showing which combinations you’ve tested. 5. Write deterministic checks before using LLM judges Use pytest and pydantic for schema validation, regex checks, required fields, and length constraints. Execute generated SQL against a real SQLite test database. Run generated code inside a sandbox. Aim to make at least half of your failure categories testable without an LLM. 6. Build an LLM judge and align it with human judgments Manually label 100+ examples as pass or fail. Split them into train, dev, and test sets. Use binary verdicts, a written rubric, and a critique-before-verdict approach. Iterate on the judge prompt using the dev set until both TPR and TNR are high, then evaluate on the test set once. Lock the judge model version. Project: Create a judge-alignment notebook containing a confusion matrix and every prompt iteration you tested. 7. Evaluate RAG retrieval and generation separately Label the relevant chunks for 50 queries. Calculate recall@5, MRR, and nDCG. Then measure faithfulness by breaking each answer into claims and verifying each claim against the retrieved context. Change the chunk size or embedding model and observe which metrics improve or decline. 8. Evaluate agent trajectories Log the tool name, arguments, results, and token usage at every step. Create reference paths for 30 tasks. Measure tool precision and recall, argument correctness, step count, and error recovery. Intentionally make a tool fail 20% of the time and see whether your agent recovers or gets stuck in a loop. 9. Simulate users for multi-turn conversations Give a simulator LLM a persona, hidden goal, and constraints (for example, “won’t share the order number unless asked twice”). Run 50 conversations with a 15-turn limit. Evaluate goal completion, turns to resolution, and policy violations. 10. Measure consistency, not just one lucky run Run every test case 5 times at your production temperature. Report pass@1 and pass^5 side by side. Cases that produce inconsistent results are the flaky behaviors you need to fix before production. 11. Learn enough statistics to avoid fooling yourself Bootstrap your confidence intervals. Compare models on the same examples using paired tests (McNemar’s test works for binary pass/fail outcomes). If your test cannot distinguish the two runs, you haven’t demonstrated an improvement. You may have just gotten lucky. Further reading: Hamel Husain and Shreya Shankar on evals, Eugene Yan’s posts on LLM evaluation, and the τ-bench paper (the source of pass^k).

Evals are the most underrated skill in AI engineering. 1. Start with error analysis, not metrics Sample 100+ production traces. Annotate each failure in free text (open coding), then cluster them into failure categories (axial coding). Count frequency per category. Your top 3 categories are your first metrics. 2. Build a versioned golden dataset Store inputs, expected outputs or rubrics, and metadata (intent, difficulty, user segment) in a versioned dataset. Stratify it so rare but critical cases aren't drowned out by easy ones. Version it alongside your prompts so every score is reproducible. 3. Generate synthetic data along explicit dimensions Don't just ask an LLM for "test questions." Define dimensions (persona × intent × complexity × edge case), generate combinations as tuples, then turn each tuple into a natural input. You get coverage instead of 200 near-identical examples. 4. Deterministic assertions first JSON schema validation, regex checks, required fields, length limits, code execution, SQL that actually runs against a test DB. These are cheap, fast, and have zero variance. Reserve model-based grading for what code genuinely can't check. 5. Evaluate retrieval separately from generation For RAG, measure retrieval with recall@k, MRR, or nDCG against labeled relevant chunks. Measure generation with faithfulness (is every claim supported by retrieved context?). If you only score the final answer, you can't tell which half is broken. 6. Trajectory evals for agents Log every step: tool name, arguments, result, tokens. Score tool-call precision and recall against a reference path, argument correctness, step count, and error recovery. An agent that reaches the right answer in 14 calls when 3 would do is a cost and latency bug. 7. pass@k vs pass^k pass@k = succeeded at least once in k runs. pass^k = succeeded in all k runs. At 80% per-run success, pass^5 drops to roughly 33%. Production reliability lives in pass^k, so run each case multiple times at your real temperature. 8. Simulated users for multi-turn Give an LLM a persona, a hidden goal, and constraints, then let it converse with your system until the goal is met or a turn limit hits. Grade goal completion, turns-to-resolution, and policy violations. Single-turn evals miss most state and memory bugs. 9. Align your LLM judge like a classifier Hand-label 100+ examples as pass/fail. Split into train/dev/test. Iterate on the judge prompt using dev, then report TPR and TNR on test. Use binary outputs with a written rubric, require a short critique before the verdict, and pin the judge model version. 10. Control judge bias in pairwise comparisons LLM judges have position bias and often prefer longer answers. Run every A/B comparison twice with swapped order and only count consistent verdicts. Normalize or cap length if verbosity is skewing results. 11. Respect statistics With 100 examples at 80% accuracy, your 95% confidence interval is about ±8 points. A jump from 80% to 83% is probably noise. Use bootstrapped confidence intervals, or a paired test on the same examples, before declaring a win. 12. Gate deploys in CI Run the eval suite on every PR that touches prompts, models, tools, or retrieval config. Set per-category thresholds instead of one global score, so a big win in one area can't hide a regression in another. Fail the build on a regression. 13. Online evals in production Sample a percentage of live traffic, run your judges asynchronously, and alert on drift per failure category. Use shadow deployments to run a new prompt or model on real traffic without showing users the output. 14. Make cost and latency first-class metrics Track tokens per request, cost per successful task (not per call), and p50/p95 latency next to quality. A change that keeps quality flat while cutting cost 40% is a real win and deserves to ship. Prompts are disposable. Models get replaced every few months. Your eval infrastructure is what compounds.

Looking to connect with more builders on X 🤝 If you're into: → SaaS & startups → AI & coding → Building in public → Growing on X → Learning new things Let's connect! 🚀 Introduce yourself, tell me what you're building, and what you're excited about right now. Always happy to meet people who love building things. 🙌🏻

The quietest gold rush in AI right now isn't building models. It's shrinking them. It's called distillation: you take a giant model, have it teach a tiny one and end up with something close to frontier quality that runs at a fraction of the cost. Why everyone wants it: • Frontier APIs get expensive fast at scale • Small models are faster and can run on your own servers • Your data never leaves your company • You own the model instead of renting it So a new business is forming around it: "Send us your use case. We'll hand you a small model that does that one job really well." Call it Distillation as a Service. The big labs build the giant brains. The next wave of startups will sell the cheap, fast, specialized ones. Every company paying a big AI bill is a potential customer.

AI founders I have a lot of respect for (in no particular order): - @sama (OpenAI) - @DarioAmodei (Anthropic) - @demishassabis (Google DeepMind) - @ilyasut (Safe Superintelligence) - @miramurati (Thinking Machines) - @karpathy (Eureka Labs) - @drfeifei (World Labs) - @arthurmensch (Mistral) - @aidangomez (Cohere) - @ClementDelangue (Hugging Face) - @AravSrinivas (Perplexity) - @alexandr_wang (Scale AI) - @mustafasuleyman (Inflection) - @mntruell (Cursor) - @ScottWu46 (Cognition) - @amasad (Replit) - @rauchg (Vercel) - @hwchase17 (LangChain) - @jerryjliu0 (LlamaIndex) - @c_valenzuelab (Runway) - @DavidSHolz (Midjourney) - @Suhail (Mixpanel) - @EMostaque (Stability AI) - @gdb (OpenAI) Who's missing?