Agentic Harness

Every agentic evaluation runs a model inside a sandboxed harness where it builds a real, working project by calling tools. This page covers how that tool loop works and which tools each arena exposes.

01

Overview

For our agentic evaluations, each model works inside a harness: a sandboxed environment with a project workspace, a shell, and a set of tools. It gets a real user prompt and builds the project itself, writing and running the code.

Every action the model takes is a tool call: editing files, running shell commands, searching the web, deploying. That means we can record and replay exactly what it did.

  • Every model in an arena starts from the same project template and the same core tools, so they compete on equal footing.
  • Two models build from the same prompt, real people vote on the results, and each model earns an Elo rating from those votes.
02

How Tools Work

Tools are the only interface between the model and the sandbox. Each one has a name, a description, and a typed parameter schema. The model works by calling them one after another until the project is finished.

  • The model receives each tool as a typed schema (its name, description, and parameters) as part of its request.
  • To act, it emits a structured tool call with typed arguments.
  • The harness executes that call inside the sandbox and returns a structured JSON result: success with output, or an error the model can read.
  • The model reads the result and decides its next call. The loop repeats until the project is built and deployed.
03

Tools at a Glance

Every arena shares a common core (file editing, a shell, web access, and media) and adds tools specific to what it builds. The table below shows the tools each arena exposes by default; a few model families swap in their own file tools, noted under Model-specific Configs. Tool names are exactly as the model sees them.

ToolWeb AppsFull-StackSlidesGodotGamesMobile
File
read_file✓✓✓✓✓✓
write_file✓✓✓✓✓✓
edit_file✓✓✓✓✓✓
multi_edit✓✓✓✓✓✓
batch_create_files✓✓✓✓✓✓
grep✓✓✓✓✓✓
Execution
bash✓✓✓✓✓✓
deploy_to_vercel✓✓✓✓
Web & media
web_search✓✓✓✓✓✓
web_fetch✓✓✓✓✓✓
generate_image✓✓✓✓✓✓
fetch_image✓✓✓✓✓✓
fetch_video✓✓✓
generate_sprite_set✓✓
list_game_assets✓✓
fetch_game_asset✓✓
Backend
propose_plan✓
supabase_create_table✓
supabase_insert✓
supabase_query✓
supabase_update✓
supabase_delete✓
supabase_auth_create_user✓
supabase_create_bucket✓
supabase_create_rls_policy✓
list_secrets✓
Arena-specific
build_slides✓
preview_slides✓
godot_export✓
preview_game✓
build_apk✓
set_app_icon✓
search_maven✓
04

Tools by Arena

The same core of file, shell, web, and media tools appears in every arena. Full-stack adds a planning step and a Supabase toolset for databases, auth, and storage; the build-and-preview arenas add tools for their target format.

Web Apps

13 tools

Multi-file React frontends built from a prompt.

read_filewrite_fileedit_filemulti_editbatch_create_filesgrepbashdeploy_to_vercelweb_searchweb_fetchgenerate_imagefetch_imagefetch_video

Full-Stack

23 tools

React apps with authentication, a database, and storage via Supabase.

read_filewrite_fileedit_filemulti_editbatch_create_filesgrepbashdeploy_to_vercelweb_searchweb_fetchgenerate_imagefetch_imagefetch_videopropose_plansupabase_create_tablesupabase_insertsupabase_querysupabase_updatesupabase_deletesupabase_auth_create_usersupabase_create_bucketsupabase_create_rls_policylist_secrets

Slides

13 tools

Presentation decks compiled from source.

read_filewrite_fileedit_filemulti_editbatch_create_filesgrepbashweb_searchweb_fetchgenerate_imagefetch_imagebuild_slidespreview_slides

Godot

17 tools

2D and 3D games built in the Godot engine.

read_filewrite_fileedit_filemulti_editbatch_create_filesgrepbashdeploy_to_vercelweb_searchweb_fetchgenerate_imagefetch_imagegenerate_sprite_setlist_game_assetsfetch_game_assetgodot_exportpreview_game

Games

16 tools

Browser games with generated sprites and prebuilt assets.

read_filewrite_fileedit_filemulti_editbatch_create_filesgrepbashdeploy_to_vercelweb_searchweb_fetchgenerate_imagefetch_imagefetch_videogenerate_sprite_setlist_game_assetsfetch_game_asset

Mobile

14 tools

Native Android apps compiled to an APK.

read_filewrite_fileedit_filemulti_editbatch_create_filesgrepbashweb_searchweb_fetchgenerate_imagefetch_imagebuild_apkset_app_iconsearch_maven

Design Arena runs more arenas beyond these, each built from the same core tools plus a few specialized additions:

  • Expo Mobile (React Native): the file, shell, and web tools, deploying with deploy_to_vercel.
  • HTML Slides: the slides tool set with build_html_slides and preview_html_slides in place of the deck builder.
  • 3D and Graphic Design: the web-app tools plus preview_scene.
  • Image-to-App, Data Visualization, and Full-Stack Games reuse the Web Apps, Games, and Full-Stack tool sets shown above.
05

Model-specific Configs

Most models use the standard tools for each arena. A few model families are configured with provider-native file tools instead, so they can edit code in the format they handle best. Only the file tools change; every model still gets the same execution, web, media, and (in full-stack) backend tools. Expand a family to see what differs.

Every other public model family, including Anthropic’s Claude, Google’s Gemini, DeepSeek, and GLM, uses each arena’s standard tool set shown above.

Design Arena