Vector-Bench · a thetalab benchmark

theta labs / task

sv_004hard

scenic_multi

instructionnatural language

The tent flap is out of alignment, one star has disappeared, the front of the campfire is the wrong color, and the smoke curls the wrong way. Restore the campsite without touching the trees or other objects.

initial · brokeninput

target · expected fixcanonical

published model outputs

What each model produced.

34 endpoints

model result explorerClaude Haiku 4.5

Claude Haiku 4.5

anthropic/claude-haiku-4.5

TracePARTIAL

requested repairs

2/4

semantic preservation

clean

SVG validity

valid

source fidelity

unchanged

target match

different

Full pass requires the first three gates. Source fidelity and canonical target match are diagnostics only.

outcome

partial

progress

56.1%

UCR

0.0%

preserve

100.0%

elapsed

738.4s

cost

$0.0149

requested repairs2/4

smoke-curl.dmissed

expected M168 80 C154 72 178 64 164 56 C154 50 174 46 166 38 / produced M168 80 C154 72 182 66 168 58 C156 50 186 46 170 38

path-shape: 6.374 <= 4.2 sampled geometry / 27.6% repaired

flame-front.fillpassed

expected #fde047 / produced #fbbf24

color-lab: 17.403 <= 18 Delta E76 / 70.5% repaired

star-2.existspassed

expected true / produced true

inserted-subtree: 0.571 <= 1 normalized tolerance / 100.0% repaired

tent-flap.pointsmissed

expected [[82,96],[96,66],[110,96]] / produced [[83,101],[92,71],[101,101]]

points-rms: 5.37 <= 4.004 coordinate RMS / 26.2% repaired

Why it did not pass

requested repairs incomplete

Unexpected changes

None

Semantic preservation failures

None

Source fidelity diagnostics

None

prompt 2,540 tokoutput 2,464 tokreasoning 0 tokceiling 4,096 tokfinish stopfrontier

resolved anthropic/claude-haiku-4.5

response gen-1784563245-2HDIxMLPq24ZQSrNLFx9

expected diff4 edits

Part	Attr	Before	After
smoke-curl	d	M168 80 C182 72 150 66 172 58 C184 50 154 46 170 38	M168 80 C154 72 178 64 164 56 C154 50 174 46 166 38
flame-front	fill	#f97316	#fde047
star-2	exists	false	true
tent-flap	points	[[73,101],[87,71],[101,101]]	[[82,96],[96,66],[110,96]]

partsscope

Target (4)

smoke-curl

flame-front

star-2

tent-flap

Should preserve (50)

night-sky

ground

far-hill

mid-hill

moon

star-1

star-3

star-4

tent-body

tent-side

tent-line-left

tent-line-right

tree-left-trunk

tree-left-crown

tree-right-trunk

tree-right-crown

log-left

log-right

flame-back

smoke-1

smoke-2

backpack

path

trail-curve

skin-04-aurora-wide-a

skin-04-aurora-wide-b

identity-04-aurora-a

identity-04-aurora-b

variant-04-dot

variant-04-rule

variant-04-chip-01

variant-04-chip-02

variant-04-chip-03

variant-04-chip-04

variant-04-chip-05

variant-04-chip-06

variant-04-chip-07

variant-04-chip-08

variant-04-chip-09

variant-04-chip-10

variant-04-chip-11

variant-04-chip-12

variant-04-pin-01

variant-04-pin-02

variant-04-pin-03

variant-04-pin-04

variant-04-pin-05

variant-04-pin-06

variant-04-pin-07

variant-04-pin-08

initial svgsource

<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 220 140" width="220" height="140">
<rect id="__bg" x="0" y="0" width="220" height="140" fill="#fffaf5" />
  <rect id="night-sky" fill="#0f172a" x="0" y="0" width="220" height="88" />
  <rect id="ground" fill="#0f766e" x="0" y="88" width="220" height="52" />
  <polygon id="far-hill" fill="#14532d" points="0,88 42,50 88,88" />
  <polygon id="mid-hill" fill="#166534" points="46,88 104,42 166,88" />
  <circle id="moon" fill="#fef3c7" cx="184" cy="24" r="11" />
  <circle id="star-1" fill="#fde68a" cx="28" cy="20" r="2" />
  <circle id="star-3" fill="#fde68a" cx="130" cy="22" r="2.2" />
  <circle id="star-4" fill="#fde68a" cx="204" cy="45" r="1.8" />
  <polygon id="tent-body" fill="#f97316" stroke="#222" stroke-width="1" points="46,96 92,48 138,96" />
  <polygon id="tent-side" fill="#ea580c" stroke="#222" stroke-width="1" points="92,48 138,96 112,96" />
  <polygon id="tent-flap" fill="#7c2d12" stroke="#222" stroke-width="1" points="73,101 87,71 101,101" />
  <line id="tent-line-left" stroke="#f8fafc" stroke-width="1" x1="54" y1="92" x2="30" y2="110" />
  <line id="tent-line-right" stroke="#f8fafc" stroke-width="1" x1="130" y1="92" x2="154" y2="110" />
  <rect id="tree-left-trunk" fill="#854d0e" x="20" y="74" width="8" height="34" />
  <polygon id="tree-left-crown" fill="#15803d" points="24,42 8,80 40,80" />
  <rect id="tree-right-trunk" fill="#854d0e" x="178" y="72" width="8" height="36" />
  <polygon id="tree-right-crown" fill="#15803d" points="182,40 164,82 176,76 170,94 198,92 190,76 202,82" />
  <line id="log-left" stroke="#92400e" stroke-width="5" x1="150" y1="112" x2="176" y2="122" />
  <line id="log-right" stroke="#92400e" stroke-width="5" x1="176" y1="112" x2="150" y2="122" />
  <polygon id="flame-back" fill="#f97316" points="163,110 152,88 172,101 181,86 180,114" />
  <polygon id="flame-front" fill="#f97316" points="166,111 160,96 170,104 175,96 174,114" />
  <ellipse id="smoke-1" fill="#94a3b8" opacity="0.65" cx="160" cy="80" rx="6" ry="3" />
  <ellipse id="smoke-2" fill="#94a3b8" opacity="0.5" cx="174" cy="72" rx="7" ry="4" />
  <path id="smoke-curl" fill="none" stroke="#cbd5e1" stroke-width="1.4" d="M168 80 C182 72 150 66 172 58 C184 50 154 46 170 38" />
  <rect id="backpack" fill="#2563eb" stroke="#111827" stroke-width="1" x="36" y="104" width="14" height="18" rx="3" ry="3" />
  <polyline id="path" fill="none" stroke="#a3e635" stroke-width="5" points="96,100 108,114 118,140" />
  <path id="trail-curve" fill="none" stroke="#bef264" stroke-width="2" d="M86 104 C100 112 98 126 112 140" />
  <path id="skin-04-aurora-wide-a" fill="none" stroke="#22d3ee" stroke-width="10" opacity="0.48" d="M4 24 C42 0 70 50 108 24 S170 4 216 30" />
  <path id="skin-04-aurora-wide-b" fill="none" stroke="#a7f3d0" stroke-width="8" opacity="0.42" d="M0 48 C54 16 92 68 138 38 S182 24 220 54" />
  <path id="identity-04-aurora-a" fill="none" stroke="#22d3ee" stroke-width="5" opacity="0.42" d="M10 24 C38 6 62 42 90 22 S144 4 196 26" />
  <path id="identity-04-aurora-b" fill="none" stroke="#a7f3d0" stroke-width="4" opacity="0.4" d="M12 38 C48 18 78 52 114 34 S166 22 208 42" />
  <circle id="variant-04-dot" fill="#f97316" opacity="0.72" cx="30" cy="26" r="2.6" />
  <line id="variant-04-rule" stroke="#f97316" stroke-width="1.2" opacity="0.58" x1="36" y1="29" x2="52" y2="32" />
  <rect id="variant-04-chip-01" fill="#22c55e" opacity="0.68" x="12" y="5" width="8" height="3" rx="1" />
  <rect id="variant-04-chip-02" fill="#14b8a6" opacity="0.68" x="43" y="6" width="9" height="4" rx="1" />
  <rect id="variant-04-chip-03" fill="#06b6d4" opacity="0.68" x="80" y="4" width="7" height="3" rx="1" />
  <rect id="variant-04-chip-04" fill="#3b82f6" opacity="0.68" x="111" y="5" width="8" height="4" rx="1" />
  <rect id="variant-04-chip-05" fill="#6366f1" opacity="0.68" x="148" y="6" width="9" height="3" rx="1" />
  <rect id="variant-04-chip-06" fill="#8b5cf6" opacity="0.68" x="179" y="4" width="7" height="4" rx="1" />
  <rect id="variant-04-chip-07" fill="#d946ef" opacity="0.68" x="12" y="140" width="8" height="3" rx="1" />
  <rect id="variant-04-chip-08" fill="#ec4899" opacity="0.68" x="43" y="139" width="9" height="4" rx="1" />
  <rect id="variant-04-chip-09" fill="#ef4444" opacity="0.68" x="80" y="141" width="7" height="3" rx="1" />
  <rect id="variant-04-chip-10" fill="#f97316" opacity="0.68" x="111" y="140" width="8" height="4" rx="1" />
  <rect id="variant-04-chip-11" fill="#f59e0b" opacity="0.68" x="148" y="139" width="9" height="3" rx="1" />
  <rect id="variant-04-chip-12" fill="#84cc16" opacity="0.68" x="179" y="141" width="7" height="4" rx="1" />
  <circle id="variant-04-pin-01" fill="#6366f1" opacity="0.66" cx="12" cy="26" r="1.8" />
  <circle id="variant-04-pin-02" fill="#8b5cf6" opacity="0.66" cx="209" cy="40" r="2.05" />
  <circle id="variant-04-pin-03" fill="#d946ef" opacity="0.66" cx="22" cy="54" r="2.3" />
  <circle id="variant-04-pin-04" fill="#ec4899" opacity="0.66" cx="195" cy="68" r="2.55" />
  <circle id="variant-04-pin-05" fill="#ef4444" opacity="0.66" cx="8" cy="82" r="1.8" />
  <circle id="variant-04-pin-06" fill="#f97316" opacity="0.66" cx="205" cy="96" r="2.05" />
  <circle id="variant-04-pin-07" fill="#f59e0b" opacity="0.66" cx="18" cy="110" r="2.3" />
  <circle id="variant-04-pin-08" fill="#84cc16" opacity="0.66" cx="191" cy="124" r="2.55" />
</svg>

target svgsource

<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 220 140" width="220" height="140">
<rect id="__bg" x="0" y="0" width="220" height="140" fill="#fffaf5" />
  <rect id="night-sky" fill="#0f172a" x="0" y="0" width="220" height="88" />
  <rect id="ground" fill="#0f766e" x="0" y="88" width="220" height="52" />
  <polygon id="far-hill" fill="#14532d" points="0,88 42,50 88,88" />
  <polygon id="mid-hill" fill="#166534" points="46,88 104,42 166,88" />
  <circle id="moon" fill="#fef3c7" cx="184" cy="24" r="11" />
  <circle id="star-1" fill="#fde68a" cx="28" cy="20" r="2" />
  <circle id="star-3" fill="#fde68a" cx="130" cy="22" r="2.2" />
  <circle id="star-4" fill="#fde68a" cx="204" cy="45" r="1.8" />
  <polygon id="tent-body" fill="#f97316" stroke="#222" stroke-width="1" points="46,96 92,48 138,96" />
  <polygon id="tent-side" fill="#ea580c" stroke="#222" stroke-width="1" points="92,48 138,96 112,96" />
  <polygon id="tent-flap" fill="#7c2d12" stroke="#222" stroke-width="1" points="82,96 96,66 110,96" />
  <line id="tent-line-left" stroke="#f8fafc" stroke-width="1" x1="54" y1="92" x2="30" y2="110" />
  <line id="tent-line-right" stroke="#f8fafc" stroke-width="1" x1="130" y1="92" x2="154" y2="110" />
  <rect id="tree-left-trunk" fill="#854d0e" x="20" y="74" width="8" height="34" />
  <polygon id="tree-left-crown" fill="#15803d" points="24,42 8,80 40,80" />
  <rect id="tree-right-trunk" fill="#854d0e" x="178" y="72" width="8" height="36" />
  <polygon id="tree-right-crown" fill="#15803d" points="182,40 164,82 176,76 170,94 198,92 190,76 202,82" />
  <line id="log-left" stroke="#92400e" stroke-width="5" x1="150" y1="112" x2="176" y2="122" />
  <line id="log-right" stroke="#92400e" stroke-width="5" x1="176" y1="112" x2="150" y2="122" />
  <polygon id="flame-back" fill="#f97316" points="163,110 152,88 172,101 181,86 180,114" />
  <polygon id="flame-front" fill="#fde047" points="166,111 160,96 170,104 175,96 174,114" />
  <ellipse id="smoke-1" fill="#94a3b8" opacity="0.65" cx="160" cy="80" rx="6" ry="3" />
  <ellipse id="smoke-2" fill="#94a3b8" opacity="0.5" cx="174" cy="72" rx="7" ry="4" />
  <path id="smoke-curl" fill="none" stroke="#cbd5e1" stroke-width="1.4" d="M168 80 C154 72 178 64 164 56 C154 50 174 46 166 38" />
  <rect id="backpack" fill="#2563eb" stroke="#111827" stroke-width="1" x="36" y="104" width="14" height="18" rx="3" ry="3" />
  <polyline id="path" fill="none" stroke="#a3e635" stroke-width="5" points="96,100 108,114 118,140" />
  <path id="trail-curve" fill="none" stroke="#bef264" stroke-width="2" d="M86 104 C100 112 98 126 112 140" />
  <path id="skin-04-aurora-wide-a" fill="none" stroke="#22d3ee" stroke-width="10" opacity="0.48" d="M4 24 C42 0 70 50 108 24 S170 4 216 30" />
  <path id="skin-04-aurora-wide-b" fill="none" stroke="#a7f3d0" stroke-width="8" opacity="0.42" d="M0 48 C54 16 92 68 138 38 S182 24 220 54" />
  <path id="identity-04-aurora-a" fill="none" stroke="#22d3ee" stroke-width="5" opacity="0.42" d="M10 24 C38 6 62 42 90 22 S144 4 196 26" />
  <path id="identity-04-aurora-b" fill="none" stroke="#a7f3d0" stroke-width="4" opacity="0.4" d="M12 38 C48 18 78 52 114 34 S166 22 208 42" />
  <circle id="variant-04-dot" fill="#f97316" opacity="0.72" cx="30" cy="26" r="2.6" />
  <line id="variant-04-rule" stroke="#f97316" stroke-width="1.2" opacity="0.58" x1="36" y1="29" x2="52" y2="32" />
  <rect id="variant-04-chip-01" fill="#22c55e" opacity="0.68" x="12" y="5" width="8" height="3" rx="1" />
  <rect id="variant-04-chip-02" fill="#14b8a6" opacity="0.68" x="43" y="6" width="9" height="4" rx="1" />
  <rect id="variant-04-chip-03" fill="#06b6d4" opacity="0.68" x="80" y="4" width="7" height="3" rx="1" />
  <rect id="variant-04-chip-04" fill="#3b82f6" opacity="0.68" x="111" y="5" width="8" height="4" rx="1" />
  <rect id="variant-04-chip-05" fill="#6366f1" opacity="0.68" x="148" y="6" width="9" height="3" rx="1" />
  <rect id="variant-04-chip-06" fill="#8b5cf6" opacity="0.68" x="179" y="4" width="7" height="4" rx="1" />
  <rect id="variant-04-chip-07" fill="#d946ef" opacity="0.68" x="12" y="140" width="8" height="3" rx="1" />
  <rect id="variant-04-chip-08" fill="#ec4899" opacity="0.68" x="43" y="139" width="9" height="4" rx="1" />
  <rect id="variant-04-chip-09" fill="#ef4444" opacity="0.68" x="80" y="141" width="7" height="3" rx="1" />
  <rect id="variant-04-chip-10" fill="#f97316" opacity="0.68" x="111" y="140" width="8" height="4" rx="1" />
  <rect id="variant-04-chip-11" fill="#f59e0b" opacity="0.68" x="148" y="139" width="9" height="3" rx="1" />
  <rect id="variant-04-chip-12" fill="#84cc16" opacity="0.68" x="179" y="141" width="7" height="4" rx="1" />
  <circle id="variant-04-pin-01" fill="#6366f1" opacity="0.66" cx="12" cy="26" r="1.8" />
  <circle id="variant-04-pin-02" fill="#8b5cf6" opacity="0.66" cx="209" cy="40" r="2.05" />
  <circle id="variant-04-pin-03" fill="#d946ef" opacity="0.66" cx="22" cy="54" r="2.3" />
  <circle id="variant-04-pin-04" fill="#ec4899" opacity="0.66" cx="195" cy="68" r="2.55" />
  <circle id="variant-04-pin-05" fill="#ef4444" opacity="0.66" cx="8" cy="82" r="1.8" />
  <circle id="variant-04-pin-06" fill="#f97316" opacity="0.66" cx="205" cy="96" r="2.05" />
  <circle id="variant-04-pin-07" fill="#f59e0b" opacity="0.66" cx="18" cy="110" r="2.3" />
  <circle id="variant-04-pin-08" fill="#84cc16" opacity="0.66" cx="191" cy="124" r="2.55" />
  <circle id="star-2" fill="#fde68a" cx="72" cy="16" r="1.8" />
</svg>