Vector-Bench · a thetalab benchmark

theta labs / task

sv_009hard

scenic_multi

instructionnatural language

The market bicycle has gained a second right wheel, one red awning panel has turned white, and the shopper's bag hangs too low. Restore the wavy edge of the left awning as well, leaving everything else intact.

initial · brokeninput

target · expected fixcanonical

published model outputs

What each model produced.

34 endpoints

model result explorerLlama 4 Maverick

Llama 4 Maverick

meta-llama/llama-4-maverick

TracePARTIAL

requested repairs

2/4

semantic preservation

clean

SVG validity

valid

source fidelity

unchanged

target match

different

Full pass requires the first three gates. Source fidelity and canonical target match are diagnostics only.

outcome

partial

progress

75.9%

UCR

0.0%

preserve

100.0%

elapsed

323.9s

cost

$0.0042

requested repairs2/4

left-awning-scallop.dmissed

expected M14 60 C22 68 30 52 38 60 S54 68 62 60 S78 52 86 60 S94 68 96 60 / produced M14 60 C26 48 38 60 50 60 C62 60 70 48 82 60 C94 72 98 60 96 60

path-shape: 6.001 <= 4.2 sampled geometry / 48.1% repaired

shopper-bag.ymissed

expected 94 / produced 98

numeric-delta: 4 <= 3.5 user units / 55.6% repaired

left-awning-red-2.fillpassed

expected #f97316 / produced #f97316

color-lab: 0 <= 18 Delta E76 / 100.0% repaired

bike-wheel-right-dup.existspassed

expected false / produced false

required presence matches

Why it did not pass

requested repairs incomplete

Unexpected changes

None

Semantic preservation failures

None

Source fidelity diagnostics

None

prompt 2,891 tokoutput 2,738 tokreasoning 0 tokceiling 4,234 tokfinish stopopen weight

resolved meta-llama/llama-4-maverick

response gen-1784544205-ALav8LlpXaMFcOOfpmFQ

expected diff4 edits

Part	Attr	Before	After
left-awning-scallop	d	M14 60 C24 52 30 68 38 60 S54 52 62 60 S78 68 86 60 S94 52 96 60	M14 60 C22 68 30 52 38 60 S54 68 62 60 S78 52 86 60 S94 68 96 60
shopper-bag	y	103	94
left-awning-red-2	fill	#f8fafc	#f97316
bike-wheel-right-dup	exists	true	false

partsscope

Target (4)

left-awning-scallop

shopper-bag

left-awning-red-2

bike-wheel-right-dup

Should preserve (63)

back-wall

street

left-stall-counter

left-awning-base

left-awning-red-1

left-awning-red-3

left-post

left-post-right

right-stall-counter

right-awning-base

right-awning-blue-1

right-awning-blue-2

right-awning-blue-3

right-post-left

right-post-right

apple-1

apple-2

apple-3

pear-1

pear-2

pear-3

crate-apples

crate-pears

hanging-sign

sign-pin-left

sign-pin-right

lamp

bike-frame

bike-handlebar

bike-wheel-left

bike-wheel-right

shopper-head

shopper-body

skin-09-festival-wire

skin-09-festival-light-a

skin-09-festival-light-b

skin-09-festival-light-c

identity-09-light-wire

identity-09-light-a

identity-09-light-b

identity-09-light-c

variant-09-dot

variant-09-rule

variant-09-chip-01

variant-09-chip-02

variant-09-chip-03

variant-09-chip-04

variant-09-chip-05

variant-09-chip-06

variant-09-chip-07

variant-09-chip-08

variant-09-chip-09

variant-09-chip-10

variant-09-chip-11

variant-09-chip-12

variant-09-pin-01

variant-09-pin-02

variant-09-pin-03

variant-09-pin-04

variant-09-pin-05

variant-09-pin-06

variant-09-pin-07

variant-09-pin-08

initial svgsource

<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 220 140" width="220" height="140">
<rect id="__bg" x="0" y="0" width="220" height="140" fill="#fffdf4" />
  <rect id="back-wall" fill="#fef3c7" x="0" y="0" width="220" height="82" />
  <rect id="street" fill="#e5e7eb" x="0" y="82" width="220" height="58" />
  <rect id="left-stall-counter" fill="#78350f" stroke="#451a03" stroke-width="1" x="18" y="76" width="74" height="34" />
  <rect id="left-awning-base" fill="#f8fafc" stroke="#222" stroke-width="1" x="14" y="42" width="82" height="18" />
  <rect id="left-awning-red-1" fill="#f97316" x="14" y="42" width="14" height="18" />
  <rect id="left-awning-red-2" fill="#f8fafc" x="42" y="42" width="14" height="18" />
  <rect id="left-awning-red-3" fill="#f97316" x="70" y="42" width="14" height="18" />
  <path id="left-awning-scallop" fill="none" stroke="#222" stroke-width="1" d="M14 60 C24 52 30 68 38 60 S54 52 62 60 S78 68 86 60 S94 52 96 60" />
  <line id="left-post" stroke="#451a03" stroke-width="3" x1="22" y1="60" x2="22" y2="118" />
  <line id="left-post-right" stroke="#451a03" stroke-width="3" x1="88" y1="60" x2="88" y2="118" />
  <rect id="right-stall-counter" fill="#78350f" stroke="#451a03" stroke-width="1" x="128" y="78" width="72" height="32" />
  <rect id="right-awning-base" fill="#f8fafc" stroke="#222" stroke-width="1" x="124" y="44" width="80" height="18" />
  <rect id="right-awning-blue-1" fill="#22c55e" x="124" y="44" width="16" height="18" />
  <rect id="right-awning-blue-2" fill="#22c55e" x="156" y="44" width="16" height="18" />
  <rect id="right-awning-blue-3" fill="#22c55e" x="188" y="44" width="16" height="18" />
  <line id="right-post-left" stroke="#451a03" stroke-width="3" x1="134" y1="62" x2="134" y2="118" />
  <line id="right-post-right" stroke="#451a03" stroke-width="3" x1="196" y1="62" x2="196" y2="118" />
  <circle id="apple-1" fill="#dc2626" cx="34" cy="72" r="5" />
  <circle id="apple-2" fill="#dc2626" cx="48" cy="70" r="5" />
  <circle id="apple-3" fill="#dc2626" cx="62" cy="72" r="5" />
  <ellipse id="pear-1" fill="#a3e635" cx="148" cy="72" rx="5" ry="7" />
  <ellipse id="pear-2" fill="#a3e635" cx="164" cy="70" rx="5" ry="7" />
  <ellipse id="pear-3" fill="#a3e635" cx="180" cy="72" rx="5" ry="7" />
  <rect id="crate-apples" fill="#a16207" stroke="#451a03" stroke-width="1" x="28" y="76" width="44" height="12" />
  <rect id="crate-pears" fill="#a16207" stroke="#451a03" stroke-width="1" x="140" y="76" width="48" height="12" />
  <rect id="hanging-sign" fill="#fef3c7" stroke="#222" stroke-width="1" x="92" y="18" width="36" height="18" />
  <line id="sign-pin-left" stroke="#222" stroke-width="1" x1="100" y1="18" x2="100" y2="8" />
  <line id="sign-pin-right" stroke="#222" stroke-width="1" x1="120" y1="18" x2="120" y2="8" />
  <circle id="lamp" fill="#facc15" stroke="#222" stroke-width="1" cx="110" cy="10" r="5" />
  <polyline id="bike-frame" fill="none" stroke="#334155" stroke-width="2" points="84,114 104,96 122,114 96,114 104,96" />
  <path id="bike-handlebar" fill="none" stroke="#334155" stroke-width="2" d="M122 96 C130 88 136 92 136 98" />
  <circle id="bike-wheel-left" fill="none" stroke="#334155" stroke-width="2" cx="84" cy="114" r="10" />
  <circle id="bike-wheel-right" fill="none" stroke="#334155" stroke-width="2" cx="122" cy="114" r="10" />
  <circle id="shopper-head" fill="#f2c078" cx="112" cy="78" r="4" />
  <rect id="shopper-body" fill="#16a34a" x="108" y="82" width="8" height="18" />
  <rect id="shopper-bag" fill="#f59e0b" x="118" y="103" width="8" height="10" />
  <polyline id="skin-09-festival-wire" fill="none" stroke="#475569" stroke-width="2.4" opacity="0.9" points="0,34 42,18 84,34 126,18 168,34 220,18" />
  <circle id="skin-09-festival-light-a" fill="#fde047" opacity="0.95" cx="42" cy="19" r="8" />
  <circle id="skin-09-festival-light-b" fill="#f97316" opacity="0.95" cx="126" cy="19" r="8" />
  <circle id="skin-09-festival-light-c" fill="#22c55e" opacity="0.95" cx="205" cy="22" r="8" />
  <polyline id="identity-09-light-wire" fill="none" stroke="#475569" stroke-width="1.3" points="16,26 48,18 82,26 118,18 154,26 196,18" />
  <circle id="identity-09-light-a" fill="#fde047" opacity="0.95" cx="48" cy="19" r="4" />
  <circle id="identity-09-light-b" fill="#f97316" opacity="0.95" cx="118" cy="19" r="4" />
  <circle id="identity-09-light-c" fill="#22c55e" opacity="0.95" cx="196" cy="19" r="4" />
  <circle id="variant-09-dot" fill="#f59e0b" opacity="0.72" cx="19" cy="25" r="2.2" />
  <line id="variant-09-rule" stroke="#f59e0b" stroke-width="1.2" opacity="0.58" x1="25" y1="28" x2="41" y2="26" />
  <rect id="variant-09-chip-01" fill="#8b5cf6" opacity="0.68" x="11" y="4" width="7" height="3" rx="1" />
  <rect id="variant-09-chip-02" fill="#d946ef" opacity="0.68" x="42" y="5" width="8" height="4" rx="1" />
  <rect id="variant-09-chip-03" fill="#ec4899" opacity="0.68" x="79" y="6" width="9" height="3" rx="1" />
  <rect id="variant-09-chip-04" fill="#ef4444" opacity="0.68" x="110" y="4" width="7" height="4" rx="1" />
  <rect id="variant-09-chip-05" fill="#f97316" opacity="0.68" x="147" y="5" width="8" height="3" rx="1" />
  <rect id="variant-09-chip-06" fill="#f59e0b" opacity="0.68" x="178" y="6" width="9" height="4" rx="1" />
  <rect id="variant-09-chip-07" fill="#84cc16" opacity="0.68" x="11" y="141" width="7" height="3" rx="1" />
  <rect id="variant-09-chip-08" fill="#22c55e" opacity="0.68" x="42" y="140" width="8" height="4" rx="1" />
  <rect id="variant-09-chip-09" fill="#14b8a6" opacity="0.68" x="79" y="139" width="9" height="3" rx="1" />
  <rect id="variant-09-chip-10" fill="#06b6d4" opacity="0.68" x="110" y="141" width="7" height="4" rx="1" />
  <rect id="variant-09-chip-11" fill="#3b82f6" opacity="0.68" x="147" y="140" width="8" height="3" rx="1" />
  <rect id="variant-09-chip-12" fill="#6366f1" opacity="0.68" x="178" y="139" width="9" height="4" rx="1" />
  <circle id="variant-09-pin-01" fill="#06b6d4" opacity="0.66" cx="23" cy="26" r="2.05" />
  <circle id="variant-09-pin-02" fill="#3b82f6" opacity="0.66" cx="208" cy="40" r="2.3" />
  <circle id="variant-09-pin-03" fill="#6366f1" opacity="0.66" cx="9" cy="54" r="2.55" />
  <circle id="variant-09-pin-04" fill="#8b5cf6" opacity="0.66" cx="194" cy="68" r="1.8" />
  <circle id="variant-09-pin-05" fill="#d946ef" opacity="0.66" cx="19" cy="82" r="2.05" />
  <circle id="variant-09-pin-06" fill="#ec4899" opacity="0.66" cx="204" cy="96" r="2.3" />
  <circle id="variant-09-pin-07" fill="#ef4444" opacity="0.66" cx="29" cy="110" r="2.55" />
  <circle id="variant-09-pin-08" fill="#f97316" opacity="0.66" cx="190" cy="124" r="1.8" />
  <circle id="bike-wheel-right-dup" fill="none" stroke="#334155" stroke-width="2" cx="134" cy="110" r="10" />
</svg>

target svgsource

<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 220 140" width="220" height="140">
<rect id="__bg" x="0" y="0" width="220" height="140" fill="#fffdf4" />
  <rect id="back-wall" fill="#fef3c7" x="0" y="0" width="220" height="82" />
  <rect id="street" fill="#e5e7eb" x="0" y="82" width="220" height="58" />
  <rect id="left-stall-counter" fill="#78350f" stroke="#451a03" stroke-width="1" x="18" y="76" width="74" height="34" />
  <rect id="left-awning-base" fill="#f8fafc" stroke="#222" stroke-width="1" x="14" y="42" width="82" height="18" />
  <rect id="left-awning-red-1" fill="#f97316" x="14" y="42" width="14" height="18" />
  <rect id="left-awning-red-2" fill="#f97316" x="42" y="42" width="14" height="18" />
  <rect id="left-awning-red-3" fill="#f97316" x="70" y="42" width="14" height="18" />
  <path id="left-awning-scallop" fill="none" stroke="#222" stroke-width="1" d="M14 60 C22 68 30 52 38 60 S54 68 62 60 S78 52 86 60 S94 68 96 60" />
  <line id="left-post" stroke="#451a03" stroke-width="3" x1="22" y1="60" x2="22" y2="118" />
  <line id="left-post-right" stroke="#451a03" stroke-width="3" x1="88" y1="60" x2="88" y2="118" />
  <rect id="right-stall-counter" fill="#78350f" stroke="#451a03" stroke-width="1" x="128" y="78" width="72" height="32" />
  <rect id="right-awning-base" fill="#f8fafc" stroke="#222" stroke-width="1" x="124" y="44" width="80" height="18" />
  <rect id="right-awning-blue-1" fill="#22c55e" x="124" y="44" width="16" height="18" />
  <rect id="right-awning-blue-2" fill="#22c55e" x="156" y="44" width="16" height="18" />
  <rect id="right-awning-blue-3" fill="#22c55e" x="188" y="44" width="16" height="18" />
  <line id="right-post-left" stroke="#451a03" stroke-width="3" x1="134" y1="62" x2="134" y2="118" />
  <line id="right-post-right" stroke="#451a03" stroke-width="3" x1="196" y1="62" x2="196" y2="118" />
  <circle id="apple-1" fill="#dc2626" cx="34" cy="72" r="5" />
  <circle id="apple-2" fill="#dc2626" cx="48" cy="70" r="5" />
  <circle id="apple-3" fill="#dc2626" cx="62" cy="72" r="5" />
  <ellipse id="pear-1" fill="#a3e635" cx="148" cy="72" rx="5" ry="7" />
  <ellipse id="pear-2" fill="#a3e635" cx="164" cy="70" rx="5" ry="7" />
  <ellipse id="pear-3" fill="#a3e635" cx="180" cy="72" rx="5" ry="7" />
  <rect id="crate-apples" fill="#a16207" stroke="#451a03" stroke-width="1" x="28" y="76" width="44" height="12" />
  <rect id="crate-pears" fill="#a16207" stroke="#451a03" stroke-width="1" x="140" y="76" width="48" height="12" />
  <rect id="hanging-sign" fill="#fef3c7" stroke="#222" stroke-width="1" x="92" y="18" width="36" height="18" />
  <line id="sign-pin-left" stroke="#222" stroke-width="1" x1="100" y1="18" x2="100" y2="8" />
  <line id="sign-pin-right" stroke="#222" stroke-width="1" x1="120" y1="18" x2="120" y2="8" />
  <circle id="lamp" fill="#facc15" stroke="#222" stroke-width="1" cx="110" cy="10" r="5" />
  <polyline id="bike-frame" fill="none" stroke="#334155" stroke-width="2" points="84,114 104,96 122,114 96,114 104,96" />
  <path id="bike-handlebar" fill="none" stroke="#334155" stroke-width="2" d="M122 96 C130 88 136 92 136 98" />
  <circle id="bike-wheel-left" fill="none" stroke="#334155" stroke-width="2" cx="84" cy="114" r="10" />
  <circle id="bike-wheel-right" fill="none" stroke="#334155" stroke-width="2" cx="122" cy="114" r="10" />
  <circle id="shopper-head" fill="#f2c078" cx="112" cy="78" r="4" />
  <rect id="shopper-body" fill="#16a34a" x="108" y="82" width="8" height="18" />
  <rect id="shopper-bag" fill="#f59e0b" x="118" y="94" width="8" height="10" />
  <polyline id="skin-09-festival-wire" fill="none" stroke="#475569" stroke-width="2.4" opacity="0.9" points="0,34 42,18 84,34 126,18 168,34 220,18" />
  <circle id="skin-09-festival-light-a" fill="#fde047" opacity="0.95" cx="42" cy="19" r="8" />
  <circle id="skin-09-festival-light-b" fill="#f97316" opacity="0.95" cx="126" cy="19" r="8" />
  <circle id="skin-09-festival-light-c" fill="#22c55e" opacity="0.95" cx="205" cy="22" r="8" />
  <polyline id="identity-09-light-wire" fill="none" stroke="#475569" stroke-width="1.3" points="16,26 48,18 82,26 118,18 154,26 196,18" />
  <circle id="identity-09-light-a" fill="#fde047" opacity="0.95" cx="48" cy="19" r="4" />
  <circle id="identity-09-light-b" fill="#f97316" opacity="0.95" cx="118" cy="19" r="4" />
  <circle id="identity-09-light-c" fill="#22c55e" opacity="0.95" cx="196" cy="19" r="4" />
  <circle id="variant-09-dot" fill="#f59e0b" opacity="0.72" cx="19" cy="25" r="2.2" />
  <line id="variant-09-rule" stroke="#f59e0b" stroke-width="1.2" opacity="0.58" x1="25" y1="28" x2="41" y2="26" />
  <rect id="variant-09-chip-01" fill="#8b5cf6" opacity="0.68" x="11" y="4" width="7" height="3" rx="1" />
  <rect id="variant-09-chip-02" fill="#d946ef" opacity="0.68" x="42" y="5" width="8" height="4" rx="1" />
  <rect id="variant-09-chip-03" fill="#ec4899" opacity="0.68" x="79" y="6" width="9" height="3" rx="1" />
  <rect id="variant-09-chip-04" fill="#ef4444" opacity="0.68" x="110" y="4" width="7" height="4" rx="1" />
  <rect id="variant-09-chip-05" fill="#f97316" opacity="0.68" x="147" y="5" width="8" height="3" rx="1" />
  <rect id="variant-09-chip-06" fill="#f59e0b" opacity="0.68" x="178" y="6" width="9" height="4" rx="1" />
  <rect id="variant-09-chip-07" fill="#84cc16" opacity="0.68" x="11" y="141" width="7" height="3" rx="1" />
  <rect id="variant-09-chip-08" fill="#22c55e" opacity="0.68" x="42" y="140" width="8" height="4" rx="1" />
  <rect id="variant-09-chip-09" fill="#14b8a6" opacity="0.68" x="79" y="139" width="9" height="3" rx="1" />
  <rect id="variant-09-chip-10" fill="#06b6d4" opacity="0.68" x="110" y="141" width="7" height="4" rx="1" />
  <rect id="variant-09-chip-11" fill="#3b82f6" opacity="0.68" x="147" y="140" width="8" height="3" rx="1" />
  <rect id="variant-09-chip-12" fill="#6366f1" opacity="0.68" x="178" y="139" width="9" height="4" rx="1" />
  <circle id="variant-09-pin-01" fill="#06b6d4" opacity="0.66" cx="23" cy="26" r="2.05" />
  <circle id="variant-09-pin-02" fill="#3b82f6" opacity="0.66" cx="208" cy="40" r="2.3" />
  <circle id="variant-09-pin-03" fill="#6366f1" opacity="0.66" cx="9" cy="54" r="2.55" />
  <circle id="variant-09-pin-04" fill="#8b5cf6" opacity="0.66" cx="194" cy="68" r="1.8" />
  <circle id="variant-09-pin-05" fill="#d946ef" opacity="0.66" cx="19" cy="82" r="2.05" />
  <circle id="variant-09-pin-06" fill="#ec4899" opacity="0.66" cx="204" cy="96" r="2.3" />
  <circle id="variant-09-pin-07" fill="#ef4444" opacity="0.66" cx="29" cy="110" r="2.55" />
  <circle id="variant-09-pin-08" fill="#f97316" opacity="0.66" cx="190" cy="124" r="1.8" />
</svg>