ytz009/image_pot_bench

Image-PoT Bench (v2)

28 code-with-image tasks where models must write and run code against the image to reach the answer — pure visual inspection ("just look") is insufficient by design: answers require pixel-level precision (±px coordinates, exact counts, sub-degree angles, per-channel color recovery).

Each task is a config; each config has three splits:

split	size	source photos	role
train	60 (15/tier)	COCO train2017	reflection / skill-learning (tr60)
validation	50 (13/13/12/12)	COCO train2017	selection set (sel50, the exact stratified slice used in runs)
test	100 (25/tier)	COCO val2017	final evaluation (test_mini)

Zero photo overlap between train/validation (train2017) and test (val2017). Difficulty tiers: easy / medium / hard / expert in the difficulty column.

from datasets import load_dataset
d = load_dataset("ytz009/image_pot_bench", "alignment")
item = d["test"][0]   # columns: id, image, image_2, prompt, answer, difficulty, meta

image_2 is non-null only for two-image tasks (e.g. congruent_pair). meta is a JSON string with generation parameters (constructive ground truth). Scoring rules per task follow SCORING_CRITERIA.md in the companion code repo: https://github.com/YangTianze009/LVLM_POT (includes full reflection trajectories via LFS).

Tasks (28)

Transformation & rearrangement: alignment, circular_shift, shift_rot_align, rot_scale, transform_chain, jigsaw, rotated_jigsaw, unshuffle2d, restore_then_count, restore_then_measure Perception & counting: object_count, peak_count, grating_period, trace_length, congruent_pair, anomaly_v2, occlusion_order Localization: crop_provenance, vanishing_point, brightest_blob, sub_rect_swap, retouch_region, circle_fit Color: max_sat_cell2, film_color, swatch_rgb, patch_mean_dots, color_count

Note: trace_length uses the revised (v2fix) generator with a reference-solver fairness screen.