Vector-Bench · a thetalab benchmark

theta labs / task

sv_014very_hard

scenic_very_hard

instructionnatural language

Turn the right treetop back around, return the moon to the upper-right sky, and replace the missing star nearby. The campfire smoke should curl naturally, and the right tent seam should be thin again. Keep all other campsite details as they are.

initial · brokeninput

target · expected fixcanonical

published model outputs

What each model produced.

34 endpoints

model result explorerKAT Coder Air V2.5

KAT Coder Air V2.5

kwaipilot/kat-coder-air-v2.5

TracePARTIAL

requested repairs

4/6

semantic preservation

clean

SVG validity

valid

source fidelity

unchanged

target match

different

Full pass requires the first three gates. Source fidelity and canonical target match are diagnostics only.

outcome

partial

progress

48.2%

UCR

0.0%

preserve

100.0%

elapsed

3711.4s

cost

$0.0027

requested repairs4/6

tent-line-right.stroke-widthpassed

expected 1 / produced 1

line-weight-delta: 0 <= 1 user units / 100.0% repaired

smoke-curl.dpassed

expected M168 80 C154 72 178 64 164 56 C154 50 174 46 166 38 / produced M168 80 C156 72 182 64 164 56 C152 48 178 40 162 32

path-shape: 2.391 <= 3.945 sampled geometry / 57.6% repaired

star-4.existsmissed

expected true / produced true

inserted-subtree: 2 <= 1 normalized tolerance / 0.0% repaired

moon.cxpassed

expected 184 / produced 188

numeric-delta: 4 <= 5 user units / 60.0% repaired

moon.cypassed

expected 24 / produced 22

numeric-delta: 2 <= 3.5 user units / 71.4% repaired

tree-right-crown.pointsmissed

expected [[181.99999999999994,40],[163.99999999999994,82],[175.99999999999994,76],[169.99999999999994,94],[197.99999999999994,... / produced [[182,38],[162,80],[202,80]]

numeric shape differs

Why it did not pass

requested repairs incomplete

Unexpected changes

None

Semantic preservation failures

None

Source fidelity diagnostics

None

prompt 3,109 tokoutput 3,775 tokreasoning 885 tokceiling 4,096 tokfinish stopcheap control

resolved kwaipilot/kat-coder-air-v2.5

response gen-1784548598-uqCstQx1SIdPADqB5V0a

expected diff6 edits

Part	Attr	Before	After
tent-line-right	stroke-width	3	1
smoke-curl	d	M168 80 C152 88 186 62 158 58 C146 52 180 44 160 38	M168 80 C154 72 178 64 164 56 C154 50 174 46 166 38
star-4	exists	false	true
moon	cx	194	184
moon	cy	17	24
tree-right-crown	points	[[184.28571428571428,40],[202.28571428571428,82],[190.28571428571428,76],[196.28571428571428,94],[168.28571428571428,92],[176.28571428571428,76],[164.28571428571428,82]]	[[181.99999999999994,40],[163.99999999999994,82],[175.99999999999994,76],[169.99999999999994,94],[197.99999999999994,92],[189.99999999999994,76],[201.99999999999994,82]]

partsscope

Target (5)

tent-line-right

smoke-curl

star-4

moon

tree-right-crown

Should preserve (51)

night-sky

ground

far-hill

mid-hill

star-1

star-2

star-3

tent-body

tent-side

tent-flap

tent-line-left

tree-left-trunk

tree-left-crown

tree-right-trunk

log-left

log-right

flame-back

flame-front

smoke-1

smoke-2

backpack

path

trail-curve

skin-14-mountain-wide-left

skin-14-mountain-wide-snow

skin-14-mountain-wide-right

identity-14-ridge-left

identity-14-ridge-snow

identity-14-ridge-right

variant-14-dot

variant-14-rule

variant-14-chip-01

variant-14-chip-02

variant-14-chip-03

variant-14-chip-04

variant-14-chip-05

variant-14-chip-06

variant-14-chip-07

variant-14-chip-08

variant-14-chip-09

variant-14-chip-10

variant-14-chip-11

variant-14-chip-12

variant-14-pin-01

variant-14-pin-02

variant-14-pin-03

variant-14-pin-04

variant-14-pin-05

variant-14-pin-06

variant-14-pin-07

variant-14-pin-08

initial svgsource

<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 220 140" width="220" height="140">
<rect id="__bg" x="0" y="0" width="220" height="140" fill="#f8f9ff" />
  <rect id="night-sky" fill="#1e293b" x="0" y="0" width="220" height="88" />
  <rect id="ground" fill="#334155" x="0" y="88" width="220" height="52" />
  <polygon id="far-hill" fill="#64748b" points="0,88 42,50 88,88" />
  <polygon id="mid-hill" fill="#475569" points="46,88 104,42 166,88" />
  <circle id="moon" fill="#fef3c7" cx="194" cy="17" r="11" />
  <circle id="star-1" fill="#fde68a" cx="28" cy="20" r="2" />
  <circle id="star-2" fill="#fde68a" cx="72" cy="16" r="1.8" />
  <circle id="star-3" fill="#fde68a" cx="130" cy="22" r="2.2" />
  <polygon id="tent-body" fill="#f97316" stroke="#222" stroke-width="1" points="46,96 92,48 138,96" />
  <polygon id="tent-side" fill="#64748b" stroke="#222" stroke-width="1" points="92,48 138,96 112,96" />
  <polygon id="tent-flap" fill="#7c2d12" stroke="#222" stroke-width="1" points="82,96 96,66 110,96" />
  <line id="tent-line-left" stroke="#f8fafc" stroke-width="1" x1="54" y1="92" x2="30" y2="110" />
  <line id="tent-line-right" stroke="#f8fafc" stroke-width="3" x1="130" y1="92" x2="154" y2="110" />
  <rect id="tree-left-trunk" fill="#854d0e" x="20" y="74" width="8" height="34" />
  <polygon id="tree-left-crown" fill="#475569" points="24,42 8,80 40,80" />
  <rect id="tree-right-trunk" fill="#854d0e" x="178" y="72" width="8" height="36" />
  <polygon id="tree-right-crown" fill="#475569" points="184.28571428571428,40 202.28571428571428,82 190.28571428571428,76 196.28571428571428,94 168.28571428571428,92 176.28571428571428,76 164.28571428571428,82" />
  <line id="log-left" stroke="#92400e" stroke-width="5" x1="150" y1="112" x2="176" y2="122" />
  <line id="log-right" stroke="#92400e" stroke-width="5" x1="176" y1="112" x2="150" y2="122" />
  <polygon id="flame-back" fill="#f97316" points="163,110 152,88 172,101 181,86 180,114" />
  <polygon id="flame-front" fill="#fde047" points="166,111 160,96 170,104 175,96 174,114" />
  <ellipse id="smoke-1" fill="#94a3b8" opacity="0.65" cx="160" cy="80" rx="6" ry="3" />
  <ellipse id="smoke-2" fill="#94a3b8" opacity="0.5" cx="174" cy="72" rx="7" ry="4" />
  <path id="smoke-curl" fill="none" stroke="#cbd5e1" stroke-width="1.4" d="M168 80 C152 88 186 62 158 58 C146 52 180 44 160 38" />
  <rect id="backpack" fill="#2563eb" stroke="#111827" stroke-width="1" x="36" y="104" width="14" height="18" rx="3" ry="3" />
  <polyline id="path" fill="none" stroke="#a3e635" stroke-width="5" points="96,100 108,114 118,140" />
  <path id="trail-curve" fill="none" stroke="#bef264" stroke-width="2" d="M86 104 C100 112 98 126 112 140" />
  <polygon id="skin-14-mountain-wide-left" fill="#64748b" opacity="0.74" points="-8,112 44,34 104,112" />
  <polygon id="skin-14-mountain-wide-snow" fill="#e2e8f0" opacity="0.96" points="44,34 25,62 50,55 66,78" />
  <polygon id="skin-14-mountain-wide-right" fill="#475569" opacity="0.58" points="116,112 176,38 230,112" />
  <polygon id="identity-14-ridge-left" fill="#64748b" opacity="0.75" points="2,92 38,38 80,92" />
  <polygon id="identity-14-ridge-snow" fill="#e2e8f0" opacity="0.95" points="38,38 26,56 44,52 54,66" />
  <polygon id="identity-14-ridge-right" fill="#475569" opacity="0.6" points="124,92 170,42 218,92" />
  <circle id="variant-14-dot" fill="#6366f1" opacity="0.72" cx="56" cy="24" r="3" />
  <line id="variant-14-rule" stroke="#6366f1" stroke-width="1.2" opacity="0.58" x1="62" y1="27" x2="78" y2="30" />
  <rect id="variant-14-chip-01" fill="#f59e0b" opacity="0.68" x="10" y="6" width="9" height="3" rx="1" />
  <rect id="variant-14-chip-02" fill="#84cc16" opacity="0.68" x="47" y="4" width="7" height="4" rx="1" />
  <rect id="variant-14-chip-03" fill="#22c55e" opacity="0.68" x="78" y="5" width="8" height="3" rx="1" />
  <rect id="variant-14-chip-04" fill="#14b8a6" opacity="0.68" x="115" y="6" width="9" height="4" rx="1" />
  <rect id="variant-14-chip-05" fill="#06b6d4" opacity="0.68" x="146" y="4" width="7" height="3" rx="1" />
  <rect id="variant-14-chip-06" fill="#3b82f6" opacity="0.68" x="183" y="5" width="8" height="4" rx="1" />
  <rect id="variant-14-chip-07" fill="#6366f1" opacity="0.68" x="10" y="139" width="9" height="3" rx="1" />
  <rect id="variant-14-chip-08" fill="#8b5cf6" opacity="0.68" x="47" y="141" width="7" height="4" rx="1" />
  <rect id="variant-14-chip-09" fill="#d946ef" opacity="0.68" x="78" y="140" width="8" height="3" rx="1" />
  <rect id="variant-14-chip-10" fill="#ec4899" opacity="0.68" x="115" y="139" width="9" height="4" rx="1" />
  <rect id="variant-14-chip-11" fill="#ef4444" opacity="0.68" x="146" y="141" width="7" height="3" rx="1" />
  <rect id="variant-14-chip-12" fill="#f97316" opacity="0.68" x="183" y="140" width="8" height="4" rx="1" />
  <circle id="variant-14-pin-01" fill="#22c55e" opacity="0.66" cx="10" cy="26" r="2.3" />
  <circle id="variant-14-pin-02" fill="#14b8a6" opacity="0.66" cx="207" cy="40" r="2.55" />
  <circle id="variant-14-pin-03" fill="#06b6d4" opacity="0.66" cx="20" cy="54" r="1.8" />
  <circle id="variant-14-pin-04" fill="#3b82f6" opacity="0.66" cx="193" cy="68" r="2.05" />
  <circle id="variant-14-pin-05" fill="#6366f1" opacity="0.66" cx="30" cy="82" r="2.3" />
  <circle id="variant-14-pin-06" fill="#8b5cf6" opacity="0.66" cx="203" cy="96" r="2.55" />
  <circle id="variant-14-pin-07" fill="#d946ef" opacity="0.66" cx="16" cy="110" r="1.8" />
  <circle id="variant-14-pin-08" fill="#ec4899" opacity="0.66" cx="189" cy="124" r="2.05" />
</svg>

target svgsource

<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 220 140" width="220" height="140">
<rect id="__bg" x="0" y="0" width="220" height="140" fill="#f8f9ff" />
  <rect id="night-sky" fill="#1e293b" x="0" y="0" width="220" height="88" />
  <rect id="ground" fill="#334155" x="0" y="88" width="220" height="52" />
  <polygon id="far-hill" fill="#64748b" points="0,88 42,50 88,88" />
  <polygon id="mid-hill" fill="#475569" points="46,88 104,42 166,88" />
  <circle id="moon" fill="#fef3c7" cx="184" cy="24" r="11" />
  <circle id="star-1" fill="#fde68a" cx="28" cy="20" r="2" />
  <circle id="star-2" fill="#fde68a" cx="72" cy="16" r="1.8" />
  <circle id="star-3" fill="#fde68a" cx="130" cy="22" r="2.2" />
  <polygon id="tent-body" fill="#f97316" stroke="#222" stroke-width="1" points="46,96 92,48 138,96" />
  <polygon id="tent-side" fill="#64748b" stroke="#222" stroke-width="1" points="92,48 138,96 112,96" />
  <polygon id="tent-flap" fill="#7c2d12" stroke="#222" stroke-width="1" points="82,96 96,66 110,96" />
  <line id="tent-line-left" stroke="#f8fafc" stroke-width="1" x1="54" y1="92" x2="30" y2="110" />
  <line id="tent-line-right" stroke="#f8fafc" stroke-width="1" x1="130" y1="92" x2="154" y2="110" />
  <rect id="tree-left-trunk" fill="#854d0e" x="20" y="74" width="8" height="34" />
  <polygon id="tree-left-crown" fill="#475569" points="24,42 8,80 40,80" />
  <rect id="tree-right-trunk" fill="#854d0e" x="178" y="72" width="8" height="36" />
  <polygon id="tree-right-crown" fill="#475569" points="181.99999999999994,40 163.99999999999994,82 175.99999999999994,76 169.99999999999994,94 197.99999999999994,92 189.99999999999994,76 201.99999999999994,82" />
  <line id="log-left" stroke="#92400e" stroke-width="5" x1="150" y1="112" x2="176" y2="122" />
  <line id="log-right" stroke="#92400e" stroke-width="5" x1="176" y1="112" x2="150" y2="122" />
  <polygon id="flame-back" fill="#f97316" points="163,110 152,88 172,101 181,86 180,114" />
  <polygon id="flame-front" fill="#fde047" points="166,111 160,96 170,104 175,96 174,114" />
  <ellipse id="smoke-1" fill="#94a3b8" opacity="0.65" cx="160" cy="80" rx="6" ry="3" />
  <ellipse id="smoke-2" fill="#94a3b8" opacity="0.5" cx="174" cy="72" rx="7" ry="4" />
  <path id="smoke-curl" fill="none" stroke="#cbd5e1" stroke-width="1.4" d="M168 80 C154 72 178 64 164 56 C154 50 174 46 166 38" />
  <rect id="backpack" fill="#2563eb" stroke="#111827" stroke-width="1" x="36" y="104" width="14" height="18" rx="3" ry="3" />
  <polyline id="path" fill="none" stroke="#a3e635" stroke-width="5" points="96,100 108,114 118,140" />
  <path id="trail-curve" fill="none" stroke="#bef264" stroke-width="2" d="M86 104 C100 112 98 126 112 140" />
  <polygon id="skin-14-mountain-wide-left" fill="#64748b" opacity="0.74" points="-8,112 44,34 104,112" />
  <polygon id="skin-14-mountain-wide-snow" fill="#e2e8f0" opacity="0.96" points="44,34 25,62 50,55 66,78" />
  <polygon id="skin-14-mountain-wide-right" fill="#475569" opacity="0.58" points="116,112 176,38 230,112" />
  <polygon id="identity-14-ridge-left" fill="#64748b" opacity="0.75" points="2,92 38,38 80,92" />
  <polygon id="identity-14-ridge-snow" fill="#e2e8f0" opacity="0.95" points="38,38 26,56 44,52 54,66" />
  <polygon id="identity-14-ridge-right" fill="#475569" opacity="0.6" points="124,92 170,42 218,92" />
  <circle id="variant-14-dot" fill="#6366f1" opacity="0.72" cx="56" cy="24" r="3" />
  <line id="variant-14-rule" stroke="#6366f1" stroke-width="1.2" opacity="0.58" x1="62" y1="27" x2="78" y2="30" />
  <rect id="variant-14-chip-01" fill="#f59e0b" opacity="0.68" x="10" y="6" width="9" height="3" rx="1" />
  <rect id="variant-14-chip-02" fill="#84cc16" opacity="0.68" x="47" y="4" width="7" height="4" rx="1" />
  <rect id="variant-14-chip-03" fill="#22c55e" opacity="0.68" x="78" y="5" width="8" height="3" rx="1" />
  <rect id="variant-14-chip-04" fill="#14b8a6" opacity="0.68" x="115" y="6" width="9" height="4" rx="1" />
  <rect id="variant-14-chip-05" fill="#06b6d4" opacity="0.68" x="146" y="4" width="7" height="3" rx="1" />
  <rect id="variant-14-chip-06" fill="#3b82f6" opacity="0.68" x="183" y="5" width="8" height="4" rx="1" />
  <rect id="variant-14-chip-07" fill="#6366f1" opacity="0.68" x="10" y="139" width="9" height="3" rx="1" />
  <rect id="variant-14-chip-08" fill="#8b5cf6" opacity="0.68" x="47" y="141" width="7" height="4" rx="1" />
  <rect id="variant-14-chip-09" fill="#d946ef" opacity="0.68" x="78" y="140" width="8" height="3" rx="1" />
  <rect id="variant-14-chip-10" fill="#ec4899" opacity="0.68" x="115" y="139" width="9" height="4" rx="1" />
  <rect id="variant-14-chip-11" fill="#ef4444" opacity="0.68" x="146" y="141" width="7" height="3" rx="1" />
  <rect id="variant-14-chip-12" fill="#f97316" opacity="0.68" x="183" y="140" width="8" height="4" rx="1" />
  <circle id="variant-14-pin-01" fill="#22c55e" opacity="0.66" cx="10" cy="26" r="2.3" />
  <circle id="variant-14-pin-02" fill="#14b8a6" opacity="0.66" cx="207" cy="40" r="2.55" />
  <circle id="variant-14-pin-03" fill="#06b6d4" opacity="0.66" cx="20" cy="54" r="1.8" />
  <circle id="variant-14-pin-04" fill="#3b82f6" opacity="0.66" cx="193" cy="68" r="2.05" />
  <circle id="variant-14-pin-05" fill="#6366f1" opacity="0.66" cx="30" cy="82" r="2.3" />
  <circle id="variant-14-pin-06" fill="#8b5cf6" opacity="0.66" cx="203" cy="96" r="2.55" />
  <circle id="variant-14-pin-07" fill="#d946ef" opacity="0.66" cx="16" cy="110" r="1.8" />
  <circle id="variant-14-pin-08" fill="#ec4899" opacity="0.66" cx="189" cy="124" r="2.05" />
  <circle id="star-4" fill="#fde68a" cx="204" cy="45" r="1.8" />
</svg>