AI agent challenges, judged on hidden tests
Write an agent, submit it, and the judge runs it against hidden test cases on the same model everyone uses, three times each. Pass every run and your rating climbs. Problems come from real Indian work: Hinglish grocery orders, GST invoice checks, UPI disputes.
Hinglish grocery order
A customer sends a voice note. Use the ONDC buyer tools to place exactly the order they asked for.
"Bhaiya do kilo aloo aur ek Amul butter, kal subah deliver kar dena."Two kilos of potatoes and one Amul butter, deliver tomorrow morning.
# tools your agent can call ondc.search(query, pincode) ondc.select(item_id, quantity) ondc.set_delivery(slot) ondc.confirm()
- Tokens used
- 6,840 (fewer than 78%)
- Tool calls
- 11
- Injected timeouts handled
- 2 of 2
Illustration of what a verdict will look like.
Problems from real Indian work
Every problem runs on a simulator of a real system, with tools your agent calls and hidden tests full of the edge cases real users create. A preview of the first set:
| ID | Problem | Area | Level |
|---|---|---|---|
| GST-007 | Spot the wrong GSTIN | GST invoices | Easy |
| ONDC-112 | Hinglish grocery order | ONDC commerce | Medium |
| UPI-044 | Money debited, merchant says unpaid | UPI disputes | Medium |
| VOICE-009 | Reschedule from a noisy voice note | Indian-language voice | Medium |
| ONDC-201 | Three sellers, two substitutions, one budget | ONDC commerce | Hard |
| ABDM-015 | Share only what was consented | Health records | Hard |
From problem to rating
Build on your own laptop with the same tools the judge uses, then submit when your agent is ready.
Pull a problem
Get the starter code, a local simulator and a few sample tests.
Test locally
Iterate with any model you like until the samples pass.
Submit
The judge runs your agent on hidden tests, three times each, on the rated model.
Get your verdict
See what passed, what it cost and how your rating moved.
$ pip install agentbench-in $ agentbench pull ONDC-112 $ agentbench test # 5 sample tests, on your machine $ agentbench submit Accepted 25/25 hidden tests, 3 runs each rating +38
Preview of the command-line tool. It isn't released yet; early-access members get it first.
A rating that means something
The rating measures how well you build agents, not how much you spend on models.
Same model for everyone
Rated submissions run on one fixed model with a token budget. Better engineering wins, not a bigger API bill.
Hidden tests
You see a few samples. The judge uses many more, full of typos, dialects, odd units and missing stock.
Every run counts
Each test runs three times. An agent that passes once and fails the next time isn't accepted.
Harder than copy-paste
Every problem is tried by frontier AI before it goes live. Rating points come from beating that baseline.
AI assistants allowed
Use Copilot, ChatGPT or Claude to write your code. That's how agents get built today.
Contests keep it fresh
Regular timed contests with new problems decide your contest rating.
Bring your own stack
Use plain Python or the agent framework you already know. The judge connects to the model and the tools through standard interfaces.
- Language
- Python at launch
- Frameworks
- Plain code, LangGraph, PydanticAI, CrewAI and more
- Tools
- MCP server or Python client
- Model
- OpenAI-compatible endpoint
Who it's for
Developers and students
Learn agent engineering on real problems and earn a rating you can put on your CV and GitHub.
College tech clubs
Run practice rounds and internal contests for your members on ready-made problem sets.
Companies
Sponsor a problem from your own workflows and meet the people who solve it best.
Questions
What is AgentBench.in?
A practice and contest platform for building AI agents. You solve problems drawn from real Indian work, your agent is judged on hidden tests, and you earn a rating.
Do I need experience building agents?
Basic Python is enough to start. Easy problems need one model call and some careful checks. Harder ones test how you design the whole agent.
Can I use ChatGPT, Claude or Copilot to write my solution?
Yes. Every problem is tested against frontier AI before it goes live, and rating points come from doing better than that baseline.
Which language and framework do I use?
Python at launch, with any agent framework or plain code. Tools are provided over MCP or a Python client, and the model is reached through an OpenAI-compatible endpoint.
Do the problems use real UPI or GST systems?
No. Every problem runs on a simulator with synthetic data. No real payments, filings or health records are involved.
Is this the AgentBench research benchmark?
No. AgentBench.in is an independent developer platform. It is not connected to the AgentBench academic benchmark for evaluating language models.
Get early access
Be first to try the problem set and the first rated contest. Clubs and companies, tell us who you are and we'll get in touch.