Vector-Bench · a thetalab benchmark

theta labs / task

sv_011very_hard

scenic_very_hard

instructionnatural language

The harbor mast and the outline of the long wave are too heavy and misshapen, the lighthouse window is blue instead of glowing yellow, and its beam has shifted. The crate on the right also needs to sit back with the other cargo. Repair only these visible mistakes.

initial · brokeninput

target · expected fixcanonical

published model outputs

What each model produced.

34 endpoints

model result explorerClaude Sonnet 5

Claude Sonnet 5

anthropic/claude-sonnet-5

TracePARTIAL

requested repairs

5/6

semantic preservation

changed

SVG validity

valid

source fidelity

different

target match

different

Full pass requires the first three gates. Source fidelity and canonical target match are diagnostics only.

outcome

partial

progress

74.3%

UCR

0.0%

preserve

100.0%

elapsed

1033.6s

cost

$0.0466

requested repairs5/6

crate-right.xpassed

expected 28 / produced 27

numeric-delta: 1 <= 5 user units / 90.0% repaired

crate-right.ypassed

expected 76 / produced 76

numeric-delta: 0 <= 2 user units / 100.0% repaired

harbor-wave-long.dpassed

expected M82 118 C98 112 112 124 128 118 S158 112 174 118 / produced M82 122 C98 118 112 126 128 122 S158 118 174 122

path-coordinate-rms: 3.215 <= 3.5 coordinate RMS / 55.8% repaired

light-beam.pointsmissed

expected [[180,30],[132,18],[132,42]] / produced [[191,38],[140,16],[140,60]]

points-rms: 10.336 <= 3.479 coordinate RMS / 0.0% repaired

lighthouse-window.fillpassed

expected #fde047 / produced #fde047

color-lab: 0 <= 18 Delta E76 / 100.0% repaired

mast.stroke-widthpassed

expected 3 / produced 3

line-weight-delta: 0 <= 1.2 user units / 100.0% repaired

Why it did not pass

requested repairs incompleteunintended document changes

Unexpected changes

harbor-wave-long

Semantic preservation failures

None

Source fidelity diagnostics

None

prompt 3,483 tokoutput 3,959 tokreasoning 0 tokceiling 4,096 tokfinish stopfrontier

resolved anthropic/claude-sonnet-5

response gen-1784563523-EZMeERFD0i6SR4OeHiNB

expected diff6 edits

Part	Attr	Before	After
crate-right	x	38	28
crate-right	y	80	76
harbor-wave-long	d	M82 122 C98 126 112 110 128 122 S158 126 174 122	M82 118 C98 112 112 124 128 118 S158 112 174 118
light-beam	points	[[188,26],[140,14],[140,38]]	[[180,30],[132,18],[132,42]]
lighthouse-window	fill	#38bdf8	#fde047
mast	stroke-width	6	3

partsscope

Target (5)

crate-right

harbor-wave-long

light-beam

lighthouse-window

mast

Should preserve (56)

sky

water

horizon

sun

cloud-left

cloud-right

pier-deck

rope-coil

pier-leg-1

pier-leg-2

boat-hull

boat-cabin

cabin-window

sail-main

sail-small

flag

lighthouse-body

lighthouse-stripe-1

lighthouse-stripe-2

lighthouse-cap

buoy-red

buoy-red-stripe

buoy-green

buoy-green-stripe

wave-1

wave-2

crate-left

skin-11-beacon-beam

skin-11-bird-wide-a

skin-11-bird-wide-b

identity-11-beam-wide

identity-11-gull-a

identity-11-gull-b

identity-11-gull-c

variant-11-dot

variant-11-rule

variant-11-chip-01

variant-11-chip-02

variant-11-chip-03

variant-11-chip-04

variant-11-chip-05

variant-11-chip-06

variant-11-chip-07

variant-11-chip-08

variant-11-chip-09

variant-11-chip-10

variant-11-chip-11

variant-11-chip-12

variant-11-pin-01

variant-11-pin-02

variant-11-pin-03

variant-11-pin-04

variant-11-pin-05

variant-11-pin-06

variant-11-pin-07

variant-11-pin-08

initial svgsource

<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 220 140" width="220" height="140">
<rect id="__bg" x="0" y="0" width="220" height="140" fill="#f8fbff" />
  <rect id="sky" fill="#dbeafe" x="0" y="0" width="220" height="70" />
  <rect id="water" fill="#38bdf8" x="0" y="70" width="220" height="70" />
  <line id="horizon" stroke="#0284c7" stroke-width="2" x1="0" y1="70" x2="220" y2="70" />
  <circle id="sun" fill="#fde047" cx="190" cy="22" r="12" />
  <ellipse id="cloud-left" fill="#ffffff" cx="36" cy="26" rx="18" ry="7" />
  <ellipse id="cloud-right" fill="#ffffff" cx="56" cy="25" rx="14" ry="6" />
  <rect id="pier-deck" fill="#92400e" x="0" y="88" width="72" height="10" />
  <path id="rope-coil" fill="none" stroke="#f8fafc" stroke-width="2" d="M48 82 C58 72 70 74 70 84 C70 94 54 96 50 88 C46 80 60 78 62 86" />
  <line id="pier-leg-1" stroke="#451a03" stroke-width="4" x1="16" y1="96" x2="13" y2="132" />
  <line id="pier-leg-2" stroke="#451a03" stroke-width="4" x1="56" y1="96" x2="60" y2="132" />
  <polygon id="boat-hull" fill="#be123c" stroke="#222" stroke-width="1" points="84,84 158,84 146,104 96,104" />
  <rect id="boat-cabin" fill="#f8fafc" stroke="#222" stroke-width="1" x="106" y="62" width="30" height="20" />
  <rect id="cabin-window" fill="#93c5fd" stroke="#222" stroke-width="0.8" x="114" y="67" width="14" height="8" />
  <line id="mast" stroke="#222" stroke-width="6" x1="126" y1="84" x2="126" y2="26" />
  <polygon id="sail-main" fill="#fef9c3" stroke="#222" stroke-width="1" points="126,30 154,76 126,76" />
  <polygon id="sail-small" fill="#e0f2fe" stroke="#222" stroke-width="1" points="124,34 100,76 124,76" />
  <polygon id="flag" fill="#3b82f6" points="126,25 140,29 126,33" />
  <polygon id="lighthouse-body" fill="#f8fafc" stroke="#222" stroke-width="1" points="176,82 206,82 200,28 182,28" />
  <rect id="lighthouse-stripe-1" fill="#ef4444" x="181" y="48" width="20" height="7" />
  <rect id="lighthouse-stripe-2" fill="#ef4444" x="179" y="64" width="24" height="7" />
  <rect id="lighthouse-cap" fill="#0f172a" x="180" y="22" width="22" height="8" />
  <rect id="lighthouse-window" fill="#38bdf8" stroke="#222" stroke-width="0.7" x="188" y="34" width="6" height="8" />
  <polygon id="light-beam" fill="#fde68a" opacity="0.45" points="188,26 140,14 140,38" />
  <circle id="buoy-red" fill="#f8fafc" stroke="#222" stroke-width="1" cx="42" cy="112" r="8" />
  <rect id="buoy-red-stripe" fill="#ef4444" x="35" y="110" width="14" height="4" />
  <circle id="buoy-green" fill="#f8fafc" stroke="#222" stroke-width="1" cx="184" cy="112" r="7" />
  <rect id="buoy-green-stripe" fill="#22c55e" x="178" y="110" width="12" height="4" />
  <polyline id="wave-1" fill="none" stroke="#0ea5e9" stroke-width="2" points="6,122 22,118 38,122 54,118 70,122" />
  <path id="harbor-wave-long" fill="none" stroke="#0ea5e9" stroke-width="2" d="M82 122 C98 126 112 110 128 122 S158 126 174 122" />
  <polyline id="wave-2" fill="none" stroke="#0ea5e9" stroke-width="2" points="128,120 146,116 164,120 182,116 200,120" />
  <rect id="crate-left" fill="#a16207" stroke="#451a03" stroke-width="1" x="12" y="76" width="14" height="12" />
  <rect id="crate-right" fill="#b45309" stroke="#451a03" stroke-width="1" x="38" y="80" width="14" height="12" />
  <polygon id="skin-11-beacon-beam" fill="#fde68a" opacity="0.58" points="150,48 220,10 220,78" />
  <polyline id="skin-11-bird-wide-a" fill="none" stroke="#334155" stroke-width="3" opacity="0.92" points="16,40 28,30 40,40" />
  <polyline id="skin-11-bird-wide-b" fill="none" stroke="#334155" stroke-width="3" opacity="0.92" points="54,28 68,18 82,28" />
  <polygon id="identity-11-beam-wide" fill="#fde68a" opacity="0.45" points="161,32 212,12 212,48" />
  <polyline id="identity-11-gull-a" fill="none" stroke="#334155" stroke-width="1.7" points="24,36 30,31 36,36" />
  <polyline id="identity-11-gull-b" fill="none" stroke="#334155" stroke-width="1.7" points="48,25 55,20 62,25" />
  <polyline id="identity-11-gull-c" fill="none" stroke="#334155" stroke-width="1.7" points="76,41 82,36 88,41" />
  <circle id="variant-11-dot" fill="#3b82f6" opacity="0.72" cx="53" cy="19" r="3" />
  <line id="variant-11-rule" stroke="#3b82f6" stroke-width="1.2" opacity="0.58" x1="59" y1="22" x2="75" y2="20" />
  <rect id="variant-11-chip-01" fill="#ec4899" opacity="0.68" x="13" y="6" width="9" height="3" rx="1" />
  <rect id="variant-11-chip-02" fill="#ef4444" opacity="0.68" x="44" y="4" width="7" height="4" rx="1" />
  <rect id="variant-11-chip-03" fill="#f97316" opacity="0.68" x="81" y="5" width="8" height="3" rx="1" />
  <rect id="variant-11-chip-04" fill="#f59e0b" opacity="0.68" x="112" y="6" width="9" height="4" rx="1" />
  <rect id="variant-11-chip-05" fill="#84cc16" opacity="0.68" x="149" y="4" width="7" height="3" rx="1" />
  <rect id="variant-11-chip-06" fill="#22c55e" opacity="0.68" x="180" y="5" width="8" height="4" rx="1" />
  <rect id="variant-11-chip-07" fill="#14b8a6" opacity="0.68" x="13" y="139" width="9" height="3" rx="1" />
  <rect id="variant-11-chip-08" fill="#06b6d4" opacity="0.68" x="44" y="141" width="7" height="4" rx="1" />
  <rect id="variant-11-chip-09" fill="#3b82f6" opacity="0.68" x="81" y="140" width="8" height="3" rx="1" />
  <rect id="variant-11-chip-10" fill="#6366f1" opacity="0.68" x="112" y="139" width="9" height="4" rx="1" />
  <rect id="variant-11-chip-11" fill="#8b5cf6" opacity="0.68" x="149" y="141" width="7" height="3" rx="1" />
  <rect id="variant-11-chip-12" fill="#d946ef" opacity="0.68" x="180" y="140" width="8" height="4" rx="1" />
  <circle id="variant-11-pin-01" fill="#d946ef" opacity="0.66" cx="13" cy="23" r="2.55" />
  <circle id="variant-11-pin-02" fill="#ec4899" opacity="0.66" cx="198" cy="37" r="1.8" />
  <circle id="variant-11-pin-03" fill="#ef4444" opacity="0.66" cx="23" cy="51" r="2.05" />
  <circle id="variant-11-pin-04" fill="#f97316" opacity="0.66" cx="208" cy="65" r="2.3" />
  <circle id="variant-11-pin-05" fill="#f59e0b" opacity="0.66" cx="9" cy="79" r="2.55" />
  <circle id="variant-11-pin-06" fill="#84cc16" opacity="0.66" cx="194" cy="93" r="1.8" />
  <circle id="variant-11-pin-07" fill="#22c55e" opacity="0.66" cx="19" cy="107" r="2.05" />
  <circle id="variant-11-pin-08" fill="#14b8a6" opacity="0.66" cx="204" cy="121" r="2.3" />
</svg>

target svgsource

<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 220 140" width="220" height="140">
<rect id="__bg" x="0" y="0" width="220" height="140" fill="#f8fbff" />
  <rect id="sky" fill="#dbeafe" x="0" y="0" width="220" height="70" />
  <rect id="water" fill="#38bdf8" x="0" y="70" width="220" height="70" />
  <line id="horizon" stroke="#0284c7" stroke-width="2" x1="0" y1="70" x2="220" y2="70" />
  <circle id="sun" fill="#fde047" cx="190" cy="22" r="12" />
  <ellipse id="cloud-left" fill="#ffffff" cx="36" cy="26" rx="18" ry="7" />
  <ellipse id="cloud-right" fill="#ffffff" cx="56" cy="25" rx="14" ry="6" />
  <rect id="pier-deck" fill="#92400e" x="0" y="88" width="72" height="10" />
  <path id="rope-coil" fill="none" stroke="#f8fafc" stroke-width="2" d="M48 82 C58 72 70 74 70 84 C70 94 54 96 50 88 C46 80 60 78 62 86" />
  <line id="pier-leg-1" stroke="#451a03" stroke-width="4" x1="16" y1="96" x2="13" y2="132" />
  <line id="pier-leg-2" stroke="#451a03" stroke-width="4" x1="56" y1="96" x2="60" y2="132" />
  <polygon id="boat-hull" fill="#be123c" stroke="#222" stroke-width="1" points="84,84 158,84 146,104 96,104" />
  <rect id="boat-cabin" fill="#f8fafc" stroke="#222" stroke-width="1" x="106" y="62" width="30" height="20" />
  <rect id="cabin-window" fill="#93c5fd" stroke="#222" stroke-width="0.8" x="114" y="67" width="14" height="8" />
  <line id="mast" stroke="#222" stroke-width="3" x1="126" y1="84" x2="126" y2="26" />
  <polygon id="sail-main" fill="#fef9c3" stroke="#222" stroke-width="1" points="126,30 154,76 126,76" />
  <polygon id="sail-small" fill="#e0f2fe" stroke="#222" stroke-width="1" points="124,34 100,76 124,76" />
  <polygon id="flag" fill="#3b82f6" points="126,25 140,29 126,33" />
  <polygon id="lighthouse-body" fill="#f8fafc" stroke="#222" stroke-width="1" points="176,82 206,82 200,28 182,28" />
  <rect id="lighthouse-stripe-1" fill="#ef4444" x="181" y="48" width="20" height="7" />
  <rect id="lighthouse-stripe-2" fill="#ef4444" x="179" y="64" width="24" height="7" />
  <rect id="lighthouse-cap" fill="#0f172a" x="180" y="22" width="22" height="8" />
  <rect id="lighthouse-window" fill="#fde047" stroke="#222" stroke-width="0.7" x="188" y="34" width="6" height="8" />
  <polygon id="light-beam" fill="#fde68a" opacity="0.45" points="180,30 132,18 132,42" />
  <circle id="buoy-red" fill="#f8fafc" stroke="#222" stroke-width="1" cx="42" cy="112" r="8" />
  <rect id="buoy-red-stripe" fill="#ef4444" x="35" y="110" width="14" height="4" />
  <circle id="buoy-green" fill="#f8fafc" stroke="#222" stroke-width="1" cx="184" cy="112" r="7" />
  <rect id="buoy-green-stripe" fill="#22c55e" x="178" y="110" width="12" height="4" />
  <polyline id="wave-1" fill="none" stroke="#0ea5e9" stroke-width="2" points="6,122 22,118 38,122 54,118 70,122" />
  <path id="harbor-wave-long" fill="none" stroke="#0ea5e9" stroke-width="2" d="M82 118 C98 112 112 124 128 118 S158 112 174 118" />
  <polyline id="wave-2" fill="none" stroke="#0ea5e9" stroke-width="2" points="128,120 146,116 164,120 182,116 200,120" />
  <rect id="crate-left" fill="#a16207" stroke="#451a03" stroke-width="1" x="12" y="76" width="14" height="12" />
  <rect id="crate-right" fill="#b45309" stroke="#451a03" stroke-width="1" x="28" y="76" width="14" height="12" />
  <polygon id="skin-11-beacon-beam" fill="#fde68a" opacity="0.58" points="150,48 220,10 220,78" />
  <polyline id="skin-11-bird-wide-a" fill="none" stroke="#334155" stroke-width="3" opacity="0.92" points="16,40 28,30 40,40" />
  <polyline id="skin-11-bird-wide-b" fill="none" stroke="#334155" stroke-width="3" opacity="0.92" points="54,28 68,18 82,28" />
  <polygon id="identity-11-beam-wide" fill="#fde68a" opacity="0.45" points="161,32 212,12 212,48" />
  <polyline id="identity-11-gull-a" fill="none" stroke="#334155" stroke-width="1.7" points="24,36 30,31 36,36" />
  <polyline id="identity-11-gull-b" fill="none" stroke="#334155" stroke-width="1.7" points="48,25 55,20 62,25" />
  <polyline id="identity-11-gull-c" fill="none" stroke="#334155" stroke-width="1.7" points="76,41 82,36 88,41" />
  <circle id="variant-11-dot" fill="#3b82f6" opacity="0.72" cx="53" cy="19" r="3" />
  <line id="variant-11-rule" stroke="#3b82f6" stroke-width="1.2" opacity="0.58" x1="59" y1="22" x2="75" y2="20" />
  <rect id="variant-11-chip-01" fill="#ec4899" opacity="0.68" x="13" y="6" width="9" height="3" rx="1" />
  <rect id="variant-11-chip-02" fill="#ef4444" opacity="0.68" x="44" y="4" width="7" height="4" rx="1" />
  <rect id="variant-11-chip-03" fill="#f97316" opacity="0.68" x="81" y="5" width="8" height="3" rx="1" />
  <rect id="variant-11-chip-04" fill="#f59e0b" opacity="0.68" x="112" y="6" width="9" height="4" rx="1" />
  <rect id="variant-11-chip-05" fill="#84cc16" opacity="0.68" x="149" y="4" width="7" height="3" rx="1" />
  <rect id="variant-11-chip-06" fill="#22c55e" opacity="0.68" x="180" y="5" width="8" height="4" rx="1" />
  <rect id="variant-11-chip-07" fill="#14b8a6" opacity="0.68" x="13" y="139" width="9" height="3" rx="1" />
  <rect id="variant-11-chip-08" fill="#06b6d4" opacity="0.68" x="44" y="141" width="7" height="4" rx="1" />
  <rect id="variant-11-chip-09" fill="#3b82f6" opacity="0.68" x="81" y="140" width="8" height="3" rx="1" />
  <rect id="variant-11-chip-10" fill="#6366f1" opacity="0.68" x="112" y="139" width="9" height="4" rx="1" />
  <rect id="variant-11-chip-11" fill="#8b5cf6" opacity="0.68" x="149" y="141" width="7" height="3" rx="1" />
  <rect id="variant-11-chip-12" fill="#d946ef" opacity="0.68" x="180" y="140" width="8" height="4" rx="1" />
  <circle id="variant-11-pin-01" fill="#d946ef" opacity="0.66" cx="13" cy="23" r="2.55" />
  <circle id="variant-11-pin-02" fill="#ec4899" opacity="0.66" cx="198" cy="37" r="1.8" />
  <circle id="variant-11-pin-03" fill="#ef4444" opacity="0.66" cx="23" cy="51" r="2.05" />
  <circle id="variant-11-pin-04" fill="#f97316" opacity="0.66" cx="208" cy="65" r="2.3" />
  <circle id="variant-11-pin-05" fill="#f59e0b" opacity="0.66" cx="9" cy="79" r="2.55" />
  <circle id="variant-11-pin-06" fill="#84cc16" opacity="0.66" cx="194" cy="93" r="1.8" />
  <circle id="variant-11-pin-07" fill="#22c55e" opacity="0.66" cx="19" cy="107" r="2.05" />
  <circle id="variant-11-pin-08" fill="#14b8a6" opacity="0.66" cx="204" cy="121" r="2.3" />
</svg>