Ship a Reliability Win

Capstone, aur mission ka success criterion: ek lever lo, use ek aise agent par apply karo jo aap sach mein chalate ho, aur measure karo ki wo zyada reliable ho gaya.

Ab aapke paas poora map hai: chhe-move turn, workflow⇄agent dial aur uske paanch patterns (L2), aur wo chaar levers jinme hum deep gaye — context (L3), stop & budget (L4), recovery & state (L5). Bina kisi shipped change ke knowledge sirf fluency hai. Yeh lesson use storage strength mein badalta hai us ek tareeke se jo tikta hai: TradingAgents par real kaam karke (ya aapke jo bhi agent sabse shaky hai us par).

Step 1 — Touch karne se pehle measure karo (ek tiny eval banao)

Jo cheez aap measure nahi karte usse improve nahi kar sakte, aur bina eval set ke measure nahi kar sakte. Anthropic se good news: aap chhote se shuru karte ho.

Begin with the manual checks you run during development — the behaviors you verify before each release and common tasks end users try. … Start with 20–50 simple tasks drawn from real failures. — Anthropic, Demystifying Evals for AI Agents

Apne existing manual spot-checks aur past incidents ko eval tasks mein badlo. Ek accha task wo hai jahan “two domain experts would independently reach the same pass/fail verdict” — unambiguous, aur provably solvable.1 Grade karo outcomes, na ki path: yeh check karna ki agent ne tool calls ka exact sequence liya, bahut rigid hai aur valid solutions ko penalise karta hai.1

def grade(task, output):                 # OUTCOME grade karo, tool path nahi
    return task.check(output)            # code / model-judge / human

def pass_hat_k(task, k=5):               # pass^k: SAARE k runs succeed hone chahiye
    runs = [grade(task, agent(task, env=fresh_env())) for _ in range(k)]
    return all(runs)                     # consistency, ek lucky run nahi

score = mean(pass_hat_k(t) for t in eval_set)   # <- jo number beat karna hai

Step 2 — Diagnose karo ki kaunsa lever sabse weak hai

Failing transcripts padho — jab tak padhoge nahi, pata nahi chalega ki kya galat hai.1 Phir har failure ko us lever se map karo jo uska owner hai:

Step 3 — Ek lever change karo, re-measure karo, graduate karo

Ek lever ka fix apply karo — paanch ek saath nahi, warna pata nahi chalega ki number kisne move kiya. Eval dobara chalao. Agar pass^k bina kisi regression ke improve hua, toh aapne ek measurable reliability win ship kar di. Phir use lock in karo:

After an agent is launched and optimized, capability evals with high pass rates can “graduate” to become a regression suite that is run continuously to catch any drift. — Anthropic, Demystifying Evals for AI Agents

Wo graduated regression suite hi is poore course ka quiet payoff hai: yeh wo feedback loop hai jo loop par hone wale har future change ko honest rakhta hai.

Aapki real-world checklist

Yeh lesson ka feedback loop hai — quiz nahi, balki actual moves. Jaise-jaise aap apne agent par karte jaao, har ek ko tick karte jaao.

• Collect karo 20–50 tasks real TradingAgents failures & pre-release checks se.
• Outcomes grade karo, tool-call paths nahi; per task code / model / human chuno.
• Baseline pass^k nikaalo clean, isolated runs se — number likh lo.
• Read karo failing transcripts; dominant failure ko ek lever se map karo.
• Change karo bilkul ek lever ka knob (context, budget, recovery, ya control flow).
• Re-measure karo pass^k; confirm karo ki wo bina regressions ke badha.
• Graduate karo passing tasks ko ek continuous regression suite mein.

Recall check

Poora course, interleaved. Memory se retrieve karo.

Sources

1. Anthropic — Demystifying Evals for AI Agents. Start with 20–50 tasks from real failures; two-expert agreement; grade outcomes not paths; code / model / human graders; pass@k vs pass^k; isolated trials; read transcripts; graduate capability evals into regression suites.
2. HumanLayer — 12-Factor Agents. Small, focused agents as the unit you evaluate and improve.