A VLM answers every question at temperature 0 with reasoning enabled. Its answer is compared with the official ground truth by the verifier described below; matches go to CoSynMath_RS_think and CoSynMath_RS_nothink, non-matches go to CoSynMath_rejected with the ground truth preserved.

The question field is byte-identical to the prompt the model actually received, including any appended instruction, so the stored training text and the inference-time input cannot drift apart.

How a match is decided

Every rule below is stated with its exact constants and a runnable reference implementation, so any accept/reject decision in this dataset can be recomputed. All snippets assume:

def levenshtein(a, b):
    if a == b: return 0
    if not a: return len(b)
    if not b: return len(a)
    prev = list(range(len(b) + 1))
    for i, ca in enumerate(a, 1):
        cur = [i] + [0] * len(b)
        for j, cb in enumerate(b, 1):
            cur[j] = min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (ca != cb))
        prev = cur
    return prev[len(b)]

`anls` — Average Normalized Levenshtein Similarity

The official DocVQA/InfoVQA metric. Both strings are lowercased and stripped, the normalized Levenshtein distance NL = lev(pred, gt) / max(len(pred), len(gt)) is computed, and the score is 1 - NL when NL < 0.5, otherwise 0. A pair is accepted when the score is above zero and both guards below pass.

def anls_pair(pred, gt, tau=0.5):
    p, q = pred.lower().strip(), gt.lower().strip()
    if not p and not q: return 1.0
    m = max(len(p), len(q))
    if m == 0: return 0.0
    nl = levenshtein(p, q) / m
    return (1.0 - nl) if nl < tau else 0.0

Multiple choice. When the question enumerates its own lettered options, the only correct target is the letter, so a single-letter reference is also matched against the first standalone letter in the prediction — C, (C), The answer is C and **Answer: C** all match C.

Two guards that override a positive score

Numeric guard. ANLS was designed for OCR text and is far too permissive on numbers: 99 degrees vs 80 degrees differs by two characters out of ten and scores 0.8. When both sides state numbers and those numbers disagree, the pair is rejected regardless of string similarity. Thousand separators are removed first, so 1,329.87 and 1329.87 are the same number.

_DIGITS = re.compile(r"\d+(?:\.\d+)?")
_THOUS  = re.compile(r"(?<=\d),(?=\d\d\d\b)")

def num_conflict(a, b):
    a, b = _THOUS.sub("", a or ""), _THOUS.sub("", b or "")
    na, nb = _DIGITS.findall(a), _DIGITS.findall(b)
    if not na or not nb:
        return False
    fa, fb = [float(x) for x in na], [float(x) for x in nb]
    if len(fa) == 1 and len(fb) == 1:
        return abs(fa[0] - fb[0]) > 1e-6 * max(1.0, abs(fb[0]))
    return set(na) != set(nb)

Edit budget. ANLS's fixed tau=0.5 lets a 7-character answer differ by 3 characters, which accepts SBOXCAR against a reference of boxcar at a score of 0.857. That threshold is meant for evaluating a model, where OCR noise is expected; selecting TRAINING data needs precision, because an accepted row is shipped as ground truth. The tolerated number of character edits scales with length, and is measured after stripping case, punctuation, whitespace and a leading article — otherwise stop. vs stop would burn the whole budget on a period.

reference length	edits allowed
≤ 8	0 (must be exact)
9–16	1
17–32	2
> 32	length / 12

_ART = re.compile(r"^(?:a|an|the)\s+")

def budget_norm(x):
    return re.sub(r"[^a-z0-9]", "", _ART.sub("", (x or "").strip().lower()))

def edit_budget(gt):
    n = len(gt or "")
    return 0 if n <= 8 else 1 if n <= 16 else 2 if n <= 32 else max(3, n // 12)

def within_budget(pred, gt):
    p, g = budget_norm(pred), budget_norm(gt)
    return p == g or levenshtein(p, g) <= edit_budget(g)

Shared pre-processing (applies to every verifier below)

Both sides are expanded into a small set of candidate strings and the best-scoring pair decides. The prediction contributes: the raw string; its first line; its first paragraph; the text after the last Answer: marker; the span between Answer: and Rationale:; the stated final answer of a worked solution (recognised from \boxed{…}, Answer:, the final answer is …, the answer is …, so the answer is …, 故选X); and a markdown-stripped variant. The reference contributes the same forms.

When the reference is a wrapper around the real answer (The answer is Red), the raw string is replaced by the extracted answer rather than kept beside it: the wrapper text is shared by every wrong answer too, so The answer is Blue would otherwise score 0.78 against it.

import re

def final_answer(gt):
    """Stated answer of a worked solution, or None."""
    pats = [r"\\boxed\{([^{}]{1,80})\}",
            r"(?is)\banswer\s*[:：]\s*(.{1,80}?)\s*$",
            r"(?is)\b(?:the\s+)?final\s+answer\s+is\s*[:：]?\s*(.{1,80}?)\s*[.。]?\s*$",
            r"(?is)\b(?:the\s+)?answer\s+is\s*[:：]?\s*(.{1,80}?)\s*[.。]?\s*$",
            r"(?s)故\s*选\s*[:：]?\s*\(?([A-H])\)?"]
    for p in pats:
        ms = list(re.finditer(p, (gt or "").strip()))
        if ms:
            v = re.sub(r"\s+", " ", ms[-1].group(1).strip().strip("$ \t.。，,;:"))
            if v and len(v) <= 80:
                return v
    return None

Second pass: model-as-judge

String verifiers cannot judge free-form answers at all, and they under-credit answers that are correct but differently phrased. Every item the verifier rejected is therefore re-examined by the same VLM in a text-only pass that sees the question, the reference and the candidate — no image, no re-generation, it only grades text that was already produced.

The judge is instructed to compare against the reference and never to solve the question itself, so an answer is not promoted merely because the model believes it is true. It answers CORRECT, WRONG, or UNSURE; numbers must agree, and UNSURE is reserved for references that are themselves unusable or ambiguous. UNSURE is never counted as correct — only a CORRECT verdict moves a row into the accepted splits. WRONG and UNSURE both remain in the rejected split, so an unresolved case is never silently promoted to a pass.

Calibration: run over items the string verifier had already accepted, the judge agreed on 59/60. Spot-checking its overturns shows it discriminates rather than rubber-stamps — it keeps \frac{L}{n} rejected against a reference of \frac{L}{n-1}, and returns UNSURE when the model argues no finite answer exists against a numeric reference.

What `answer` holds

In the accepted splits answer is the model's own verified output — the text that passed the verifier — so it agrees with the reasoning in think. The original reference annotation is kept beside it as org_answer. Train on answer; use org_answer when you need the source label.

The two differ more often than one would guess: the verifier accepts on meaning, not on characters, so casing, spacing and formatting routinely differ (bridgestone vs Bridgestone, ( n x m ) vs (n x m)). Storing the reference in the answer slot would pair a model's reasoning with someone else's wording.

In the rejected split the roles are different: answer is the official ground truth and wrong_vlm is what the model said instead.

Schema

image — HF Image() — renders directly in the dataset viewer
qa — list of {question, answer, note}
clean_meta — per-row provenance and the cleaning policy that admitted it
image_sha256 — content hash of the image

Examples

Three rows taken straight from this split. Images are the original files as stored in the dataset, and long fields are shown in full.

Example 1

field	value
`question`	Could you help me with this problem? End your solution with the final answer inside \boxed{}.
`answer`	To solve this problem, we need to understand the properties of cyclic groups and their subgroups: 1) A cyclic group of order $n$, denoted as $C_n$, has subgroups for each divisor of $n$. 2) The order of each subgroup is equal to the corresponding divisor. 3) There is exactly one subgroup for each divisor of $n$. Let's approach this step-by-step: 1) First, we need to find the divisors of 8: The divisors of 8 are 1, 2, 4, and 8. 2) Now, we can determine the subgroups: - There is one subgroup of order 1 (the trivial subgroup) - There is one subgroup of order 2 - There is one subgroup of order 4 - There is one subgroup of order 8 (the entire group) 3) To count the total number of subgroups, we simply add these up: 1 + 1 + 1 + 1 = 4 Therefore, the cyclic group $C_8$ has 4 subgroups in total. In the context of the urban planning committee, this means: - The full committee of 8 members can make decisions as a whole. - There are also subcommittees of 4 members and 2 members that can make certain decisions. - The trivial subgroup (1 member) represents individual decision-making power, which might be relevant for certain minor decisions or emergency situations. This structure allows for flexible decision-making at various levels, supporting the urban planning official's belief in collaborative and empowered community decision-making. Answer: The committee structure, modeled as $C_8$, has 4 subgroups with orders 1, 2, 4, and 8.
`note`	long-form: excluded from rejection sampling

Example 2

field	value
`question`	Provide a solution to this math problem. End your solution with the final answer inside \boxed{}.
`answer`	To solve this problem, we'll follow these steps: 1) The infinitesimal generator $X$ is found by differentiating $A(s)$ with respect to $s$ at $s=0$: $$X = \frac{d}{ds}A(s)\bigg\|_{s=0}$$ 2) Let's differentiate $A(s)$: $$\frac{d}{ds}A(s) = \frac{d}{ds}\left(\begin{pmatrix} \cos s & -\sin s \ \sin s & \cos s \end{pmatrix} \begin{pmatrix} e^s & 0 \ 0 & e^s \end{pmatrix}\right)$$ $$= \begin{pmatrix} -\sin s & -\cos s \ \cos s & -\sin s \end{pmatrix} \begin{pmatrix} e^s & 0 \ 0 & e^s \end{pmatrix} + \begin{pmatrix} \cos s & -\sin s \ \sin s & \cos s \end{pmatrix} \begin{pmatrix} e^s & 0 \ 0 & e^s \end{pmatrix}$$ 3) Evaluating at $s=0$: $$X = \begin{pmatrix} -\sin 0 & -\cos 0 \ \cos 0 & -\sin 0 \end{pmatrix} \begin{pmatrix} 1 & 0 \ 0 & 1 \end{pmatrix} + \begin{pmatrix} \cos 0 & -\sin 0 \ \sin 0 & \cos 0 \end{pmatrix} \begin{pmatrix} 1 & 0 \ 0 & 1 \end{pmatrix}$$ $$= \begin{pmatrix} 0 & -1 \ 1 & 0 \end{pmatrix} + \begin{pmatrix} 1 & 0 \ 0 & 1 \end{pmatrix} = \begin{pmatrix} 1 & -1 \ 1 & 1 \end{pmatrix}$$ 4) Interpretation: The infinitesimal generator $X$ represents the instantaneous change in the disease spread pattern under the action of the Lie group. The matrix $X$ can be decomposed into two parts: $$X = \begin{pmatrix} 0 & -1 \ 1 & 0 \end{pmatrix} + \begin{pmatrix} 1 & 0 \ 0 & 1 \end{pmatrix}$$ The first matrix represents rotation, while the second represents scaling. In terms of disease spread, this suggests that the pattern undergoes both a rotation (changing the direction of spread) and an expansion (increasing the overall affected area) simultaneously. The rotation could represent changes in population movement patterns, while the scaling could represent an increase in the rate of spread due to social factors like increased population density or mobility. Answer: The infinitesimal generator $X$ is: $$X = \begin{pmatrix} 1 & -1 \ 1 & 1 \end{pmatrix}$$ This represents a combination of rotation and scaling in the disease spread pattern, indicating simultaneous changes in the direction and rate of disease propagation due to social factors.
`note`	long-form: excluded from rejection sampling

Example 3

field	value
`question`	Can you solve this math problem? End your solution with the final answer inside \boxed{}.
`answer`	To solve this problem, we'll use the Central Limit Theorem and approximate the distribution of the total cost as a normal distribution. 1. First, let's calculate the mean and variance of the cost for a single prop: Mean: $\mu = \frac{100 + 500}{2} = 300$ Variance: $\sigma^2 = \frac{(500 - 100)^2}{12} = \frac{400^2}{12} = 13333.33$ 2. For 20 props, the mean and variance of the total cost will be: Mean of total: $\mu_{total} = 20 \times 300 = 6000$ Variance of total: $\sigma_{total}^2 = 20 \times 13333.33 = 266666.67$ Standard deviation of total: $\sigma_{total} = \sqrt{266666.67} = 516.40$ 3. We want to find $P(X \leq 5000)$ where $X$ is the total cost. 4. Standardizing the random variable: $Z = \frac{X - \mu_{total}}{\sigma_{total}} = \frac{5000 - 6000}{516.40} = -1.94$ 5. Using the standard normal distribution table or calculator, we can find: $P(Z \leq -1.94) = 0.0262$ This means the probability that the total cost is less than or equal to $5000 is approximately 0.0262 or 2.62%. Answer: The probability that the total cost of all 20 props will be within the budget of $5000 is approximately 0.0262 or 2.62%.
`note`	long-form: excluded from rejection sampling

Cleaning applied before rejection sampling

Provenance check. Each QA item records where it came from; items whose recorded source does not resolve to the image they are attached to are dropped.
Task purity. The pool's InfinityMM slice assigns samples to a family by image match rather than by dataset origin, which pulls unrelated general-VQA questions into benchmark families. Families where that content measured off-task keep only their genuine benchmark QA.
Multiple-choice normalisation. Where options had been flattened into prose, they are re-rendered as a lettered block and the reference is rewritten to the bare letter, so the task is well posed and the answer is unambiguous.

Downloads last month: 9

Total file size:

6 GB

You need to agree to share your contact information to access this dataset

CoSynMath — CoSynMath_longform

How the data was produced