Vector-Bench · a thetalab benchmark

theta labs / task

sv_003hard

scenic_multi

instructionnatural language

The station clock is missing its hour hand and has an extra minute hand. The blue poster has also shifted sideways, and a crack along the platform has been bent out of shape. Repair only those details.

initial · brokeninput

target · expected fixcanonical

published model outputs

What each model produced.

34 endpoints

model result explorerKAT Coder Air V2.5

KAT Coder Air V2.5

kwaipilot/kat-coder-air-v2.5

TracePASS

requested repairs

4/4

semantic preservation

clean

SVG validity

valid

source fidelity

unchanged

target match

different

Full pass requires the first three gates. Source fidelity and canonical target match are diagnostics only.

outcome

pass

progress

74.1%

UCR

0.0%

preserve

100.0%

elapsed

3141.5s

cost

$0.0029

requested repairs4/4

platform-crack.dpassed

expected M18 108 C32 104 42 111 56 108 S82 106 96 110 / produced M18 112 C32 111 42 113 56 112 S82 111 96 112

path-coordinate-rms: 3.082 <= 3.5 coordinate RMS / 39.2% repaired

poster-blue.xpassed

expected 178 / produced 181

numeric-delta: 3 <= 3.5 user units / 57.1% repaired

clock-minute-dup.existspassed

expected false / produced false

required presence matches

clock-hour.existspassed

expected true / produced true

inserted-subtree: 0.857 <= 1 normalized tolerance / 100.0% repaired

Specification outcome

All required gates passed.

Unexpected changes

None

Semantic preservation failures

None

Source fidelity diagnostics

None

prompt 3,339 tokoutput 4,073 tokreasoning 837 tokceiling 4,115 tokfinish stopcheap control

resolved kwaipilot/kat-coder-air-v2.5

response gen-1784548011-YnN4vDXMxod90TTKCtqH

expected diff4 edits

Part	Attr	Before	After
platform-crack	d	M18 112 C32 116 42 103 56 112 S82 114 96 108	M18 108 C32 104 42 111 56 108 S82 106 96 110
poster-blue	x	185	178
clock-minute-dup	exists	true	false
clock-hour	exists	false	true

partsscope

Target (4)

platform-crack

poster-blue

clock-minute-dup

clock-hour

Should preserve (63)

sky-band

overhead-wire

station-wall

platform

platform-edge

rail-back

rail-front

track-tie-1

track-tie-2

track-tie-3

track-tie-4

track-tie-5

train-body

train-roof

window-left

window-center

window-right

train-door

door-handle

wheel-left

wheel-right

signal-post

signal-box

signal-red

signal-amber

signal-green

clock-face

clock-minute

poster-pink

bench-seat

bench-leg-left

bench-leg-right

passenger-head

passenger-body

passenger-bag

skin-03-pink-tree-trunk

skin-03-pink-tree-crown

skin-03-pink-tree-glow

identity-03-blossom-trunk

identity-03-blossom-crown

identity-03-blossom-light

variant-03-dot

variant-03-rule

variant-03-chip-01

variant-03-chip-02

variant-03-chip-03

variant-03-chip-04

variant-03-chip-05

variant-03-chip-06

variant-03-chip-07

variant-03-chip-08

variant-03-chip-09

variant-03-chip-10

variant-03-chip-11

variant-03-chip-12

variant-03-pin-01

variant-03-pin-02

variant-03-pin-03

variant-03-pin-04

variant-03-pin-05

variant-03-pin-06

variant-03-pin-07

variant-03-pin-08

initial svgsource

<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 220 140" width="220" height="140">
<rect id="__bg" x="0" y="0" width="220" height="140" fill="#faf8ff" />
  <rect id="sky-band" fill="#ede9fe" x="0" y="0" width="220" height="64" />
  <path id="overhead-wire" fill="none" stroke="#334155" stroke-width="1" d="M8 18 C42 10 74 10 108 18 S176 26 212 18" />
  <rect id="station-wall" fill="#ffe4e6" x="0" y="64" width="220" height="30" />
  <rect id="platform" fill="#d8b4fe" x="0" y="94" width="220" height="46" />
  <rect id="platform-edge" fill="#f9a8d4" x="0" y="94" width="220" height="5" />
  <path id="platform-crack" fill="none" stroke="#94a3b8" stroke-width="1" d="M18 112 C32 116 42 103 56 112 S82 114 96 108" />
  <line id="rail-back" stroke="#64748b" stroke-width="3" x1="0" y1="118" x2="220" y2="118" />
  <line id="rail-front" stroke="#475569" stroke-width="3" x1="0" y1="130" x2="220" y2="130" />
  <rect id="track-tie-1" fill="#7c2d12" x="18" y="116" width="7" height="18" />
  <rect id="track-tie-2" fill="#7c2d12" x="56" y="116" width="7" height="18" />
  <rect id="track-tie-3" fill="#7c2d12" x="94" y="116" width="7" height="18" />
  <rect id="track-tie-4" fill="#7c2d12" x="132" y="116" width="7" height="18" />
  <rect id="track-tie-5" fill="#7c2d12" x="170" y="116" width="7" height="18" />
  <rect id="train-body" fill="#f43f5e" stroke="#222" stroke-width="1.5" x="26" y="38" width="126" height="46" rx="6" ry="6" />
  <rect id="train-roof" fill="#be123c" stroke="#222" stroke-width="1" x="34" y="30" width="110" height="12" rx="4" ry="4" />
  <rect id="window-left" fill="#bfdbfe" stroke="#222" stroke-width="1" x="38" y="46" width="24" height="16" />
  <rect id="window-center" fill="#bfdbfe" stroke="#222" stroke-width="1" x="68" y="46" width="24" height="16" />
  <rect id="window-right" fill="#bfdbfe" stroke="#222" stroke-width="1" x="98" y="46" width="24" height="16" />
  <rect id="train-door" fill="#fef3c7" stroke="#222" stroke-width="1" x="128" y="44" width="16" height="34" />
  <circle id="door-handle" fill="#222" cx="140" cy="61" r="1.6" />
  <circle id="wheel-left" fill="#111827" cx="48" cy="86" r="6" />
  <circle id="wheel-right" fill="#111827" cx="126" cy="86" r="6" />
  <line id="signal-post" stroke="#334155" stroke-width="4" x1="170" y1="32" x2="170" y2="94" />
  <rect id="signal-box" fill="#111827" x="162" y="22" width="16" height="34" rx="3" ry="3" />
  <circle id="signal-red" fill="#ef4444" cx="170" cy="30" r="3" />
  <circle id="signal-amber" fill="#f59e0b" cx="170" cy="39" r="3" />
  <circle id="signal-green" fill="#22c55e" cx="170" cy="48" r="3" />
  <circle id="clock-face" fill="#ffffff" stroke="#222" stroke-width="1.5" cx="196" cy="31" r="12" />
  <line id="clock-minute" stroke="#222" stroke-width="1" x1="196" y1="31" x2="203" y2="31" />
  <rect id="poster-blue" fill="#7c3aed" stroke="#222" stroke-width="0.8" x="185" y="62" width="14" height="20" />
  <rect id="poster-pink" fill="#8b5cf6" stroke="#222" stroke-width="0.8" x="198" y="64" width="12" height="18" />
  <rect id="bench-seat" fill="#92400e" x="38" y="100" width="42" height="5" />
  <line id="bench-leg-left" stroke="#451a03" stroke-width="2" x1="46" y1="105" x2="43" y2="114" />
  <line id="bench-leg-right" stroke="#451a03" stroke-width="2" x1="72" y1="105" x2="75" y2="114" />
  <circle id="passenger-head" fill="#f2c078" cx="98" cy="96" r="4" />
  <rect id="passenger-body" fill="#2563eb" x="94" y="100" width="8" height="16" />
  <rect id="passenger-bag" fill="#a16207" x="104" y="108" width="8" height="7" />
  <rect id="skin-03-pink-tree-trunk" fill="#7c2d12" opacity="0.82" x="4" y="54" width="12" height="62" />
  <circle id="skin-03-pink-tree-crown" fill="#f9a8d4" opacity="0.72" cx="15" cy="48" r="33" />
  <circle id="skin-03-pink-tree-glow" fill="#fce7f3" opacity="0.8" cx="38" cy="35" r="18" />
  <rect id="identity-03-blossom-trunk" fill="#7c2d12" x="17" y="58" width="5" height="31" />
  <circle id="identity-03-blossom-crown" fill="#f9a8d4" opacity="0.85" cx="20" cy="50" r="18" />
  <circle id="identity-03-blossom-light" fill="#fce7f3" opacity="0.9" cx="29" cy="43" r="9" />
  <circle id="variant-03-dot" fill="#8b5cf6" opacity="0.72" cx="13" cy="15" r="2.2" />
  <line id="variant-03-rule" stroke="#8b5cf6" stroke-width="1.2" opacity="0.58" x1="19" y1="18" x2="35" y2="16" />
  <rect id="variant-03-chip-01" fill="#84cc16" opacity="0.68" x="11" y="4" width="7" height="3" rx="1" />
  <rect id="variant-03-chip-02" fill="#22c55e" opacity="0.68" x="42" y="5" width="8" height="4" rx="1" />
  <rect id="variant-03-chip-03" fill="#14b8a6" opacity="0.68" x="79" y="6" width="9" height="3" rx="1" />
  <rect id="variant-03-chip-04" fill="#06b6d4" opacity="0.68" x="110" y="4" width="7" height="4" rx="1" />
  <rect id="variant-03-chip-05" fill="#3b82f6" opacity="0.68" x="147" y="5" width="8" height="3" rx="1" />
  <rect id="variant-03-chip-06" fill="#6366f1" opacity="0.68" x="178" y="6" width="9" height="4" rx="1" />
  <rect id="variant-03-chip-07" fill="#8b5cf6" opacity="0.68" x="11" y="141" width="7" height="3" rx="1" />
  <rect id="variant-03-chip-08" fill="#d946ef" opacity="0.68" x="42" y="140" width="8" height="4" rx="1" />
  <rect id="variant-03-chip-09" fill="#ec4899" opacity="0.68" x="79" y="139" width="9" height="3" rx="1" />
  <rect id="variant-03-chip-10" fill="#ef4444" opacity="0.68" x="110" y="141" width="7" height="4" rx="1" />
  <rect id="variant-03-chip-11" fill="#f97316" opacity="0.68" x="147" y="140" width="8" height="3" rx="1" />
  <rect id="variant-03-chip-12" fill="#f59e0b" opacity="0.68" x="178" y="139" width="9" height="4" rx="1" />
  <circle id="variant-03-pin-01" fill="#06b6d4" opacity="0.66" cx="29" cy="25" r="2.55" />
  <circle id="variant-03-pin-02" fill="#3b82f6" opacity="0.66" cx="190" cy="39" r="1.8" />
  <circle id="variant-03-pin-03" fill="#6366f1" opacity="0.66" cx="15" cy="53" r="2.05" />
  <circle id="variant-03-pin-04" fill="#8b5cf6" opacity="0.66" cx="200" cy="67" r="2.3" />
  <circle id="variant-03-pin-05" fill="#d946ef" opacity="0.66" cx="25" cy="81" r="2.55" />
  <circle id="variant-03-pin-06" fill="#ec4899" opacity="0.66" cx="210" cy="95" r="1.8" />
  <circle id="variant-03-pin-07" fill="#ef4444" opacity="0.66" cx="11" cy="109" r="2.05" />
  <circle id="variant-03-pin-08" fill="#f97316" opacity="0.66" cx="196" cy="123" r="2.3" />
  <line id="clock-minute-dup" stroke="#222" stroke-width="1" x1="191" y1="38" x2="198" y2="38" />
</svg>

target svgsource

<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 220 140" width="220" height="140">
<rect id="__bg" x="0" y="0" width="220" height="140" fill="#faf8ff" />
  <rect id="sky-band" fill="#ede9fe" x="0" y="0" width="220" height="64" />
  <path id="overhead-wire" fill="none" stroke="#334155" stroke-width="1" d="M8 18 C42 10 74 10 108 18 S176 26 212 18" />
  <rect id="station-wall" fill="#ffe4e6" x="0" y="64" width="220" height="30" />
  <rect id="platform" fill="#d8b4fe" x="0" y="94" width="220" height="46" />
  <rect id="platform-edge" fill="#f9a8d4" x="0" y="94" width="220" height="5" />
  <path id="platform-crack" fill="none" stroke="#94a3b8" stroke-width="1" d="M18 108 C32 104 42 111 56 108 S82 106 96 110" />
  <line id="rail-back" stroke="#64748b" stroke-width="3" x1="0" y1="118" x2="220" y2="118" />
  <line id="rail-front" stroke="#475569" stroke-width="3" x1="0" y1="130" x2="220" y2="130" />
  <rect id="track-tie-1" fill="#7c2d12" x="18" y="116" width="7" height="18" />
  <rect id="track-tie-2" fill="#7c2d12" x="56" y="116" width="7" height="18" />
  <rect id="track-tie-3" fill="#7c2d12" x="94" y="116" width="7" height="18" />
  <rect id="track-tie-4" fill="#7c2d12" x="132" y="116" width="7" height="18" />
  <rect id="track-tie-5" fill="#7c2d12" x="170" y="116" width="7" height="18" />
  <rect id="train-body" fill="#f43f5e" stroke="#222" stroke-width="1.5" x="26" y="38" width="126" height="46" rx="6" ry="6" />
  <rect id="train-roof" fill="#be123c" stroke="#222" stroke-width="1" x="34" y="30" width="110" height="12" rx="4" ry="4" />
  <rect id="window-left" fill="#bfdbfe" stroke="#222" stroke-width="1" x="38" y="46" width="24" height="16" />
  <rect id="window-center" fill="#bfdbfe" stroke="#222" stroke-width="1" x="68" y="46" width="24" height="16" />
  <rect id="window-right" fill="#bfdbfe" stroke="#222" stroke-width="1" x="98" y="46" width="24" height="16" />
  <rect id="train-door" fill="#fef3c7" stroke="#222" stroke-width="1" x="128" y="44" width="16" height="34" />
  <circle id="door-handle" fill="#222" cx="140" cy="61" r="1.6" />
  <circle id="wheel-left" fill="#111827" cx="48" cy="86" r="6" />
  <circle id="wheel-right" fill="#111827" cx="126" cy="86" r="6" />
  <line id="signal-post" stroke="#334155" stroke-width="4" x1="170" y1="32" x2="170" y2="94" />
  <rect id="signal-box" fill="#111827" x="162" y="22" width="16" height="34" rx="3" ry="3" />
  <circle id="signal-red" fill="#ef4444" cx="170" cy="30" r="3" />
  <circle id="signal-amber" fill="#f59e0b" cx="170" cy="39" r="3" />
  <circle id="signal-green" fill="#22c55e" cx="170" cy="48" r="3" />
  <circle id="clock-face" fill="#ffffff" stroke="#222" stroke-width="1.5" cx="196" cy="31" r="12" />
  <line id="clock-minute" stroke="#222" stroke-width="1" x1="196" y1="31" x2="203" y2="31" />
  <rect id="poster-blue" fill="#7c3aed" stroke="#222" stroke-width="0.8" x="178" y="62" width="14" height="20" />
  <rect id="poster-pink" fill="#8b5cf6" stroke="#222" stroke-width="0.8" x="198" y="64" width="12" height="18" />
  <rect id="bench-seat" fill="#92400e" x="38" y="100" width="42" height="5" />
  <line id="bench-leg-left" stroke="#451a03" stroke-width="2" x1="46" y1="105" x2="43" y2="114" />
  <line id="bench-leg-right" stroke="#451a03" stroke-width="2" x1="72" y1="105" x2="75" y2="114" />
  <circle id="passenger-head" fill="#f2c078" cx="98" cy="96" r="4" />
  <rect id="passenger-body" fill="#2563eb" x="94" y="100" width="8" height="16" />
  <rect id="passenger-bag" fill="#a16207" x="104" y="108" width="8" height="7" />
  <rect id="skin-03-pink-tree-trunk" fill="#7c2d12" opacity="0.82" x="4" y="54" width="12" height="62" />
  <circle id="skin-03-pink-tree-crown" fill="#f9a8d4" opacity="0.72" cx="15" cy="48" r="33" />
  <circle id="skin-03-pink-tree-glow" fill="#fce7f3" opacity="0.8" cx="38" cy="35" r="18" />
  <rect id="identity-03-blossom-trunk" fill="#7c2d12" x="17" y="58" width="5" height="31" />
  <circle id="identity-03-blossom-crown" fill="#f9a8d4" opacity="0.85" cx="20" cy="50" r="18" />
  <circle id="identity-03-blossom-light" fill="#fce7f3" opacity="0.9" cx="29" cy="43" r="9" />
  <circle id="variant-03-dot" fill="#8b5cf6" opacity="0.72" cx="13" cy="15" r="2.2" />
  <line id="variant-03-rule" stroke="#8b5cf6" stroke-width="1.2" opacity="0.58" x1="19" y1="18" x2="35" y2="16" />
  <rect id="variant-03-chip-01" fill="#84cc16" opacity="0.68" x="11" y="4" width="7" height="3" rx="1" />
  <rect id="variant-03-chip-02" fill="#22c55e" opacity="0.68" x="42" y="5" width="8" height="4" rx="1" />
  <rect id="variant-03-chip-03" fill="#14b8a6" opacity="0.68" x="79" y="6" width="9" height="3" rx="1" />
  <rect id="variant-03-chip-04" fill="#06b6d4" opacity="0.68" x="110" y="4" width="7" height="4" rx="1" />
  <rect id="variant-03-chip-05" fill="#3b82f6" opacity="0.68" x="147" y="5" width="8" height="3" rx="1" />
  <rect id="variant-03-chip-06" fill="#6366f1" opacity="0.68" x="178" y="6" width="9" height="4" rx="1" />
  <rect id="variant-03-chip-07" fill="#8b5cf6" opacity="0.68" x="11" y="141" width="7" height="3" rx="1" />
  <rect id="variant-03-chip-08" fill="#d946ef" opacity="0.68" x="42" y="140" width="8" height="4" rx="1" />
  <rect id="variant-03-chip-09" fill="#ec4899" opacity="0.68" x="79" y="139" width="9" height="3" rx="1" />
  <rect id="variant-03-chip-10" fill="#ef4444" opacity="0.68" x="110" y="141" width="7" height="4" rx="1" />
  <rect id="variant-03-chip-11" fill="#f97316" opacity="0.68" x="147" y="140" width="8" height="3" rx="1" />
  <rect id="variant-03-chip-12" fill="#f59e0b" opacity="0.68" x="178" y="139" width="9" height="4" rx="1" />
  <circle id="variant-03-pin-01" fill="#06b6d4" opacity="0.66" cx="29" cy="25" r="2.55" />
  <circle id="variant-03-pin-02" fill="#3b82f6" opacity="0.66" cx="190" cy="39" r="1.8" />
  <circle id="variant-03-pin-03" fill="#6366f1" opacity="0.66" cx="15" cy="53" r="2.05" />
  <circle id="variant-03-pin-04" fill="#8b5cf6" opacity="0.66" cx="200" cy="67" r="2.3" />
  <circle id="variant-03-pin-05" fill="#d946ef" opacity="0.66" cx="25" cy="81" r="2.55" />
  <circle id="variant-03-pin-06" fill="#ec4899" opacity="0.66" cx="210" cy="95" r="1.8" />
  <circle id="variant-03-pin-07" fill="#ef4444" opacity="0.66" cx="11" cy="109" r="2.05" />
  <circle id="variant-03-pin-08" fill="#f97316" opacity="0.66" cx="196" cy="123" r="2.3" />
  <line id="clock-hour" stroke="#222" stroke-width="1.5" x1="196" y1="31" x2="196" y2="23" />
</svg>