Vector-Bench · a thetalab benchmark

theta labs / task

sv_010hard

scenic_multi

instructionnatural language

Remove the duplicate smoke puff, turn the right-hand treetop back the correct way, make the front flame yellow again, and move the backpack back beside the tent. Preserve the rest of the campsite.

initial · brokeninput

target · expected fixcanonical

published model outputs

What each model produced.

34 endpoints

model result explorerDeepSeek V4 Pro

DeepSeek V4 Pro

deepseek/deepseek-v4-pro

TracePARTIAL

requested repairs

3/4

semantic preservation

changed

SVG validity

valid

source fidelity

different

target match

different

Full pass requires the first three gates. Source fidelity and canonical target match are diagnostics only.

outcome

partial

progress

73.4%

UCR

0.0%

preserve

100.0%

elapsed

511.4s

cost

$0.0129

requested repairs3/4

backpack.xmissed

expected 36 / produced 16

numeric-delta: 20 <= 5.5 user units / 0.0% repaired

flame-front.fillpassed

expected #fde047 / produced #fde047

color-lab: 0 <= 18 Delta E76 / 100.0% repaired

tree-right-crown.pointspassed

expected [[181.99999999999994,40],[163.99999999999994,82],[175.99999999999994,76],[169.99999999999994,94],[197.99999999999994,... / produced [[182,40],[164,82],[176,76],[168,92],[196,92],[188,76],[200,82]]

points-rms: 1.195 <= 4.2 coordinate RMS / 93.6% repaired

smoke-2-dup.existspassed

expected false / produced false

required presence matches

Why it did not pass

requested repairs incompleteunintended document changes

Unexpected changes

backpack

Semantic preservation failures

None

Source fidelity diagnostics

None

prompt 2,553 tokoutput 2,425 tokreasoning 0 tokceiling 4,096 tokfinish stopopen weight

resolved deepseek/deepseek-v4-pro

response gen-1784544046-hQGOE05q504WvvQu75b5

expected diff4 edits

Part	Attr	Before	After
backpack	x	50	36
flame-front	fill	#f97316	#fde047
tree-right-crown	points	[[184.28571428571428,40],[202.28571428571428,82],[190.28571428571428,76],[196.28571428571428,94],[168.28571428571428,92],[176.28571428571428,76],[164.28571428571428,82]]	[[181.99999999999994,40],[163.99999999999994,82],[175.99999999999994,76],[169.99999999999994,94],[197.99999999999994,92],[189.99999999999994,76],[201.99999999999994,82]]
smoke-2-dup	exists	true	false

partsscope

Target (4)

backpack

flame-front

tree-right-crown

smoke-2-dup

Should preserve (53)

night-sky

ground

far-hill

mid-hill

moon

star-1

star-2

star-3

star-4

tent-body

tent-side

tent-flap

tent-line-left

tent-line-right

tree-left-trunk

tree-left-crown

tree-right-trunk

log-left

log-right

flame-back

smoke-1

smoke-2

smoke-curl

path

trail-curve

skin-10-comet-wide-trail

skin-10-comet-wide-core

skin-10-comet-wide-glow

identity-10-comet-trail

identity-10-comet-core

identity-10-comet-glow

variant-10-dot

variant-10-rule

variant-10-chip-01

variant-10-chip-02

variant-10-chip-03

variant-10-chip-04

variant-10-chip-05

variant-10-chip-06

variant-10-chip-07

variant-10-chip-08

variant-10-chip-09

variant-10-chip-10

variant-10-chip-11

variant-10-chip-12

variant-10-pin-01

variant-10-pin-02

variant-10-pin-03

variant-10-pin-04

variant-10-pin-05

variant-10-pin-06

variant-10-pin-07

variant-10-pin-08

initial svgsource

<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 220 140" width="220" height="140">
<rect id="__bg" x="0" y="0" width="220" height="140" fill="#f6fffb" />
  <rect id="night-sky" fill="#111827" x="0" y="0" width="220" height="88" />
  <rect id="ground" fill="#0f766e" x="0" y="88" width="220" height="52" />
  <polygon id="far-hill" fill="#14532d" points="0,88 42,50 88,88" />
  <polygon id="mid-hill" fill="#166534" points="46,88 104,42 166,88" />
  <circle id="moon" fill="#fef3c7" cx="184" cy="24" r="11" />
  <circle id="star-1" fill="#fde68a" cx="28" cy="20" r="2" />
  <circle id="star-2" fill="#fde68a" cx="72" cy="16" r="1.8" />
  <circle id="star-3" fill="#fde68a" cx="130" cy="22" r="2.2" />
  <circle id="star-4" fill="#fde68a" cx="204" cy="45" r="1.8" />
  <polygon id="tent-body" fill="#94a3b8" stroke="#222" stroke-width="1" points="46,96 92,48 138,96" />
  <polygon id="tent-side" fill="#475569" stroke="#222" stroke-width="1" points="92,48 138,96 112,96" />
  <polygon id="tent-flap" fill="#7c2d12" stroke="#222" stroke-width="1" points="82,96 96,66 110,96" />
  <line id="tent-line-left" stroke="#f8fafc" stroke-width="1" x1="54" y1="92" x2="30" y2="110" />
  <line id="tent-line-right" stroke="#f8fafc" stroke-width="1" x1="130" y1="92" x2="154" y2="110" />
  <rect id="tree-left-trunk" fill="#854d0e" x="20" y="74" width="8" height="34" />
  <polygon id="tree-left-crown" fill="#15803d" points="24,42 8,80 40,80" />
  <rect id="tree-right-trunk" fill="#854d0e" x="178" y="72" width="8" height="36" />
  <polygon id="tree-right-crown" fill="#15803d" points="184.28571428571428,40 202.28571428571428,82 190.28571428571428,76 196.28571428571428,94 168.28571428571428,92 176.28571428571428,76 164.28571428571428,82" />
  <line id="log-left" stroke="#92400e" stroke-width="5" x1="150" y1="112" x2="176" y2="122" />
  <line id="log-right" stroke="#92400e" stroke-width="5" x1="176" y1="112" x2="150" y2="122" />
  <polygon id="flame-back" fill="#f97316" points="163,110 152,88 172,101 181,86 180,114" />
  <polygon id="flame-front" fill="#f97316" points="166,111 160,96 170,104 175,96 174,114" />
  <ellipse id="smoke-1" fill="#94a3b8" opacity="0.65" cx="160" cy="80" rx="6" ry="3" />
  <ellipse id="smoke-2" fill="#94a3b8" opacity="0.5" cx="174" cy="72" rx="7" ry="4" />
  <path id="smoke-curl" fill="none" stroke="#cbd5e1" stroke-width="1.4" d="M168 80 C154 72 178 64 164 56 C154 50 174 46 166 38" />
  <rect id="backpack" fill="#2563eb" stroke="#111827" stroke-width="1" x="50" y="104" width="14" height="18" rx="3" ry="3" />
  <polyline id="path" fill="none" stroke="#a3e635" stroke-width="5" points="96,100 108,114 118,140" />
  <path id="trail-curve" fill="none" stroke="#bef264" stroke-width="2" d="M86 104 C100 112 98 126 112 140" />
  <path id="skin-10-comet-wide-trail" fill="none" stroke="#fef3c7" stroke-width="10" opacity="0.72" d="M-2 40 C54 14 94 30 150 8" />
  <circle id="skin-10-comet-wide-core" fill="#fde047" opacity="0.95" cx="163" cy="6" r="14" />
  <circle id="skin-10-comet-wide-glow" fill="#fef08a" opacity="0.24" cx="163" cy="6" r="25" />
  <path id="identity-10-comet-trail" fill="none" stroke="#fef3c7" stroke-width="4" opacity="0.8" d="M20 28 C58 14 86 20 120 10" />
  <circle id="identity-10-comet-core" fill="#fde047" cx="128" cy="8" r="7" />
  <circle id="identity-10-comet-glow" fill="#fef08a" opacity="0.28" cx="128" cy="8" r="12" />
  <circle id="variant-10-dot" fill="#10b981" opacity="0.72" cx="36" cy="36" r="2.6" />
  <line id="variant-10-rule" stroke="#10b981" stroke-width="1.2" opacity="0.58" x1="42" y1="39" x2="58" y2="42" />
  <rect id="variant-10-chip-01" fill="#d946ef" opacity="0.68" x="12" y="5" width="8" height="3" rx="1" />
  <rect id="variant-10-chip-02" fill="#ec4899" opacity="0.68" x="43" y="6" width="9" height="4" rx="1" />
  <rect id="variant-10-chip-03" fill="#ef4444" opacity="0.68" x="80" y="4" width="7" height="3" rx="1" />
  <rect id="variant-10-chip-04" fill="#f97316" opacity="0.68" x="111" y="5" width="8" height="4" rx="1" />
  <rect id="variant-10-chip-05" fill="#f59e0b" opacity="0.68" x="148" y="6" width="9" height="3" rx="1" />
  <rect id="variant-10-chip-06" fill="#84cc16" opacity="0.68" x="179" y="4" width="7" height="4" rx="1" />
  <rect id="variant-10-chip-07" fill="#22c55e" opacity="0.68" x="12" y="140" width="8" height="3" rx="1" />
  <rect id="variant-10-chip-08" fill="#14b8a6" opacity="0.68" x="43" y="139" width="9" height="4" rx="1" />
  <rect id="variant-10-chip-09" fill="#06b6d4" opacity="0.68" x="80" y="141" width="7" height="3" rx="1" />
  <rect id="variant-10-chip-10" fill="#3b82f6" opacity="0.68" x="111" y="140" width="8" height="4" rx="1" />
  <rect id="variant-10-chip-11" fill="#6366f1" opacity="0.68" x="148" y="139" width="9" height="3" rx="1" />
  <rect id="variant-10-chip-12" fill="#8b5cf6" opacity="0.68" x="179" y="141" width="7" height="4" rx="1" />
  <circle id="variant-10-pin-01" fill="#6366f1" opacity="0.66" cx="30" cy="22" r="2.3" />
  <circle id="variant-10-pin-02" fill="#8b5cf6" opacity="0.66" cx="203" cy="36" r="2.55" />
  <circle id="variant-10-pin-03" fill="#d946ef" opacity="0.66" cx="16" cy="50" r="1.8" />
  <circle id="variant-10-pin-04" fill="#ec4899" opacity="0.66" cx="189" cy="64" r="2.05" />
  <circle id="variant-10-pin-05" fill="#ef4444" opacity="0.66" cx="26" cy="78" r="2.3" />
  <circle id="variant-10-pin-06" fill="#f97316" opacity="0.66" cx="199" cy="92" r="2.55" />
  <circle id="variant-10-pin-07" fill="#f59e0b" opacity="0.66" cx="12" cy="106" r="1.8" />
  <circle id="variant-10-pin-08" fill="#84cc16" opacity="0.66" cx="209" cy="120" r="2.05" />
  <ellipse id="smoke-2-dup" fill="#94a3b8" opacity="0.5" cx="164" cy="66" rx="7" ry="4" />
</svg>

target svgsource

<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 220 140" width="220" height="140">
<rect id="__bg" x="0" y="0" width="220" height="140" fill="#f6fffb" />
  <rect id="night-sky" fill="#111827" x="0" y="0" width="220" height="88" />
  <rect id="ground" fill="#0f766e" x="0" y="88" width="220" height="52" />
  <polygon id="far-hill" fill="#14532d" points="0,88 42,50 88,88" />
  <polygon id="mid-hill" fill="#166534" points="46,88 104,42 166,88" />
  <circle id="moon" fill="#fef3c7" cx="184" cy="24" r="11" />
  <circle id="star-1" fill="#fde68a" cx="28" cy="20" r="2" />
  <circle id="star-2" fill="#fde68a" cx="72" cy="16" r="1.8" />
  <circle id="star-3" fill="#fde68a" cx="130" cy="22" r="2.2" />
  <circle id="star-4" fill="#fde68a" cx="204" cy="45" r="1.8" />
  <polygon id="tent-body" fill="#94a3b8" stroke="#222" stroke-width="1" points="46,96 92,48 138,96" />
  <polygon id="tent-side" fill="#475569" stroke="#222" stroke-width="1" points="92,48 138,96 112,96" />
  <polygon id="tent-flap" fill="#7c2d12" stroke="#222" stroke-width="1" points="82,96 96,66 110,96" />
  <line id="tent-line-left" stroke="#f8fafc" stroke-width="1" x1="54" y1="92" x2="30" y2="110" />
  <line id="tent-line-right" stroke="#f8fafc" stroke-width="1" x1="130" y1="92" x2="154" y2="110" />
  <rect id="tree-left-trunk" fill="#854d0e" x="20" y="74" width="8" height="34" />
  <polygon id="tree-left-crown" fill="#15803d" points="24,42 8,80 40,80" />
  <rect id="tree-right-trunk" fill="#854d0e" x="178" y="72" width="8" height="36" />
  <polygon id="tree-right-crown" fill="#15803d" points="181.99999999999994,40 163.99999999999994,82 175.99999999999994,76 169.99999999999994,94 197.99999999999994,92 189.99999999999994,76 201.99999999999994,82" />
  <line id="log-left" stroke="#92400e" stroke-width="5" x1="150" y1="112" x2="176" y2="122" />
  <line id="log-right" stroke="#92400e" stroke-width="5" x1="176" y1="112" x2="150" y2="122" />
  <polygon id="flame-back" fill="#f97316" points="163,110 152,88 172,101 181,86 180,114" />
  <polygon id="flame-front" fill="#fde047" points="166,111 160,96 170,104 175,96 174,114" />
  <ellipse id="smoke-1" fill="#94a3b8" opacity="0.65" cx="160" cy="80" rx="6" ry="3" />
  <ellipse id="smoke-2" fill="#94a3b8" opacity="0.5" cx="174" cy="72" rx="7" ry="4" />
  <path id="smoke-curl" fill="none" stroke="#cbd5e1" stroke-width="1.4" d="M168 80 C154 72 178 64 164 56 C154 50 174 46 166 38" />
  <rect id="backpack" fill="#2563eb" stroke="#111827" stroke-width="1" x="36" y="104" width="14" height="18" rx="3" ry="3" />
  <polyline id="path" fill="none" stroke="#a3e635" stroke-width="5" points="96,100 108,114 118,140" />
  <path id="trail-curve" fill="none" stroke="#bef264" stroke-width="2" d="M86 104 C100 112 98 126 112 140" />
  <path id="skin-10-comet-wide-trail" fill="none" stroke="#fef3c7" stroke-width="10" opacity="0.72" d="M-2 40 C54 14 94 30 150 8" />
  <circle id="skin-10-comet-wide-core" fill="#fde047" opacity="0.95" cx="163" cy="6" r="14" />
  <circle id="skin-10-comet-wide-glow" fill="#fef08a" opacity="0.24" cx="163" cy="6" r="25" />
  <path id="identity-10-comet-trail" fill="none" stroke="#fef3c7" stroke-width="4" opacity="0.8" d="M20 28 C58 14 86 20 120 10" />
  <circle id="identity-10-comet-core" fill="#fde047" cx="128" cy="8" r="7" />
  <circle id="identity-10-comet-glow" fill="#fef08a" opacity="0.28" cx="128" cy="8" r="12" />
  <circle id="variant-10-dot" fill="#10b981" opacity="0.72" cx="36" cy="36" r="2.6" />
  <line id="variant-10-rule" stroke="#10b981" stroke-width="1.2" opacity="0.58" x1="42" y1="39" x2="58" y2="42" />
  <rect id="variant-10-chip-01" fill="#d946ef" opacity="0.68" x="12" y="5" width="8" height="3" rx="1" />
  <rect id="variant-10-chip-02" fill="#ec4899" opacity="0.68" x="43" y="6" width="9" height="4" rx="1" />
  <rect id="variant-10-chip-03" fill="#ef4444" opacity="0.68" x="80" y="4" width="7" height="3" rx="1" />
  <rect id="variant-10-chip-04" fill="#f97316" opacity="0.68" x="111" y="5" width="8" height="4" rx="1" />
  <rect id="variant-10-chip-05" fill="#f59e0b" opacity="0.68" x="148" y="6" width="9" height="3" rx="1" />
  <rect id="variant-10-chip-06" fill="#84cc16" opacity="0.68" x="179" y="4" width="7" height="4" rx="1" />
  <rect id="variant-10-chip-07" fill="#22c55e" opacity="0.68" x="12" y="140" width="8" height="3" rx="1" />
  <rect id="variant-10-chip-08" fill="#14b8a6" opacity="0.68" x="43" y="139" width="9" height="4" rx="1" />
  <rect id="variant-10-chip-09" fill="#06b6d4" opacity="0.68" x="80" y="141" width="7" height="3" rx="1" />
  <rect id="variant-10-chip-10" fill="#3b82f6" opacity="0.68" x="111" y="140" width="8" height="4" rx="1" />
  <rect id="variant-10-chip-11" fill="#6366f1" opacity="0.68" x="148" y="139" width="9" height="3" rx="1" />
  <rect id="variant-10-chip-12" fill="#8b5cf6" opacity="0.68" x="179" y="141" width="7" height="4" rx="1" />
  <circle id="variant-10-pin-01" fill="#6366f1" opacity="0.66" cx="30" cy="22" r="2.3" />
  <circle id="variant-10-pin-02" fill="#8b5cf6" opacity="0.66" cx="203" cy="36" r="2.55" />
  <circle id="variant-10-pin-03" fill="#d946ef" opacity="0.66" cx="16" cy="50" r="1.8" />
  <circle id="variant-10-pin-04" fill="#ec4899" opacity="0.66" cx="189" cy="64" r="2.05" />
  <circle id="variant-10-pin-05" fill="#ef4444" opacity="0.66" cx="26" cy="78" r="2.3" />
  <circle id="variant-10-pin-06" fill="#f97316" opacity="0.66" cx="199" cy="92" r="2.55" />
  <circle id="variant-10-pin-07" fill="#f59e0b" opacity="0.66" cx="12" cy="106" r="1.8" />
  <circle id="variant-10-pin-08" fill="#84cc16" opacity="0.66" cx="209" cy="120" r="2.05" />
</svg>