Published execution records

Run traces

Request, provider attempt, extraction, and verifier events for every published model-task outcome.

Task

sv_001 · hard

The amber signal has gone dark, the train door is sitting too far to the right, and the overhead cable has an awkward sag. Put those three details back in place and leave the rest of the station alone.

34 traces

Model	Outcome	Progress	Attempts	Retention
Claude Sonnet 5 anthropic/claude-sonnet-5	partial	33.3%	1	legacy
KAT Coder Air V2.5 kwaipilot/kat-coder-air-v2.5	partial	33.3%	1	legacy
MiniMax M3 minimax/minimax-m3	partial	66.7%	1	legacy
HY 3 tencent/hy3	partial	54.5%	1	legacy
Gemma 4 31B IT google/gemma-4-31b-it	partial	57.6%	1	legacy
Qwen3.5 397B A17B qwen/qwen3.5-397b-a17b	partial	31.1%	1	legacy
DeepSeek V4 Pro deepseek/deepseek-v4-pro	partial	66.7%	1	legacy
Kimi K2.6 moonshotai/kimi-k2.6	invalid	0.0%	1	legacy
Qwen3 Coder qwen/qwen3-coder	partial	63.6%	1	legacy
Gemini 3.1 Flash Lite google/gemini-3.1-flash-lite-preview	partial	52.7%	1	legacy
Devstral 2512 mistralai/devstral-2512	partial	54.5%	1	legacy
Inkling thinkingmachines/inkling	partial	54.5%	1	legacy
GPT-5 openai/gpt-5	partial	54.5%	1	legacy
Nemotron 3 Ultra nvidia/nemotron-3-ultra-550b-a55b	partial	36.4%	1	legacy
Claude Haiku 4.5 anthropic/claude-haiku-4.5	partial	52.3%	1	legacy
Seed 2.0 Mini bytedance-seed/seed-2.0-mini	partial	28.1%	1	legacy
GPT-OSS 120B openai/gpt-oss-120b	partial	49.3%	1	legacy
Llama 4 Maverick meta-llama/llama-4-maverick	partial	39.4%	1	legacy
Mistral Small 2603 mistralai/mistral-small-2603	fail	40.5%	1	legacy
GPT-5.4 Nano openai/gpt-5.4-nano	partial	57.6%	1	legacy
GPT-OSS 20B openai/gpt-oss-20b	partial	46.5%	1	legacy
Phi-4 microsoft/phi-4	invalid	0.0%	1	legacy
Laguna M.1 poolside/laguna-m.1	partial	33.3%	1	legacy
Ling 2.6 Flash inclusionai/ling-2.6-flash	partial	33.3%	1	legacy
Granite 4.1 8B ibm-granite/granite-4.1-8b	invalid	0.0%	1	legacy
GLM 4.7 z-ai/glm-4.7	partial	60.6%	1	legacy
MiMo V2.5 Pro xiaomi/mimo-v2.5-pro	invalid	0.0%	1	legacy
Qwen3.6 35B A3B qwen/qwen3.6-35b-a3b	invalid	0.0%	1	legacy
Trinity Large Thinking arcee-ai/trinity-large-thinking	invalid	0.0%	1	legacy
Gemini 3.1 Pro Preview google/gemini-3.1-pro-preview	invalid	0.0%	1	legacy
Nemotron 3 Nano nvidia/nemotron-3-nano-30b-a3b	invalid	0.0%	1	legacy
OLMo 3 32B Think allenai/olmo-3-32b-think	err	0.0%	1	legacy
Reka Flash 3 rekaai/reka-flash-3	invalid	0.0%	1	legacy
Step 3.7 Flash stepfun/step-3.7-flash	invalid	0.0%	1	legacy