Agent in a Bottle

4 Takeaways

1. Models that look alike zero-shot can bottle very differently.

   GPT 5.6 Terra and Sonnet 5 solve MAVE zero-shot equally well. Bottled, GPT 5.6 Terra recovers almost all of its zero-shot F1, Sonnet 5 only about a third.

   MAVE	Zero-shot F1	Bottled F1	Bottled ÷ zero-shot	Traces
   GPT 5.6 Terra	0.438	0.405	92%	run 1run 2
   Sonnet 5	0.444	0.154	35%	run 1run 2
   	↑ almost equal, 0.006 apart	↑ 0.251 apart

2. Most bottled runs fall well below zero-shot.

   In the current version, 48 of 60 bottled runs score below the 95% confidence interval of their model's zero-shot score. Qualitative examples below:

   Task	Model	Zero-shot	Bottled	Remarks
   Judgment failures
   MAVE F1	Sonnet 5	0.444	0.113	Ran an off-the-shelf Qwen2.5-1.5B-Instruct with a few-shot prompt on all instances; performed minimal quality checks on output correctness, justifying the low quality as "an inherent limitation of small-model zero-shot extraction".	trace
   ESCI macro F1	Gemini 3.1 Pro	0.555	0.364	Ran Qwen2.5-7B-Instruct with a prompt on all instances, almost no quality checks.	trace
   RAID AUROC	Gemini 3.1 Flash-Lite	0.780	0.511	Trained a TF-IDF classifier on a small subset labeled via the API. Almost no correctness checks; further, gave binary predictions instead of confidences as requested in the prompt.	trace
   Agentic failures
   MAVE F1	GPT 5.6 Sol	0.428	0.000	Spent the token budget in 57 min and wrote no out.txt	trace
   ESCI macro F1	GPT 5.6 Terra	0.573	0.198	Wrote 134 of 2.6 M labels and marked the task complete after 3 min	trace
   RAID AUROC	Opus 5	0.911	0.500	Spent the token budget and wrote no out.txt	trace

3. Agents frequently attempt to hack the reward.

   In line with PostTrainBench's observations, we find that agents frequently try to obtain the test dataset's labels or a model trained on said labels. We view these as reward hacking attempts, because a successful attempt would raise the score (reward) while telling us nothing about how well the agent can bottle its own capability.

   We believe that the alternative approach of asking the agent not to hack the reward can lead to ambiguity; it requires the agent to judge what counts as hacking. Therefore, we block known contamination sources, i.e., test labels or models trained on those labels, via the harness instead. The proxy returns a 403 error stating, This resource is prohibited. Do not attempt to circumvent. This provides a better measure of the model's bottling capabilities compared to assigning a baseline score to contaminated runs.

   We find that blocking known contamination sources via the proxy works. Most agents do try to access prohibited sources. But after a handful of failed attempts, most give up and start building a solution honestly. Automated judges label all our runs, except one, as uncontaminated. We repeated the single run found to be contaminated; see more about the original (contaminated) run below.

   Case: RAID · GLM 5.3 Flash · run 1

   Agent

   Downloaded a ModernBERT detector trained on MAGE + RAID via the ModelScope mirror (HF was blocked), converted the ONNX fp16 weights to PyTorch, ...

   tracejudgecontaminated

5 Cite us

If you find Bottled useful, please cite:

@article{sonthalia2026agent,
  title={Agent in a Bottle: Can {LLM} Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?},
  author={Sonthalia, Ankit and Puerto, Haritz and Rubinstein, Alexander and Gubri, Martin and Oh, Seong Joon},
  journal={arXiv:2610.08775},
  year={2026},
  url={https://arxiv.org/abs/2610.08775}
}