AgroCoT

AgroCoT is a multimodal Chain-of-Thought benchmark and instruction dataset for agriculture, covering crop and pest identification, disease diagnosis, plant/insect species recognition, and counting/detection tasks. Each example pairs an image with a multiple-choice or open-ended question, a final answer, and a step-by-step natural-language reasoning trace written by domain experts, spanning 5 task dimensions and 11 question types.

This dataset is indexed on https://project-agml.github.io/ as part of the AgML python library. Standardized to the HF image_text_to_text format: one conversational messages schema, imagefolder-native images, and (if present) a text_only parquet config. All images live in a single flat images/ directory (no subfolders), following the HF imagefolder convention. Each file is named <source-subdir>_<original-filename>, sanitized so only A-Za-z0-9._- remain — this makes every filename unique across the original nested source folders. The file_names field in metadata.jsonl points at these flat paths (e.g. images/Apple_Leaf_Rust_1.jpg).

Stats

Total QA pairs	4,759
Unique images	4,628
Multi-image rows	504
Task dimensions (dimension_id 1-5)	5
Question format types (type_id 1-11)	11

Rows per type_id:
1 → 1,453
2 → 684
3 → 728
4 → 354
5 → 97
6 → 1,065
7 → 97
8 → 53
9 → 86
10 → 7
11 → 135

The source paper organizes the benchmark into 5 first-level task dimensions — Object Detection, Quantitative Analysis, Disease Monitoring, Spatial Understanding, and Environmental Management — further split into 15 second-level sub-dimensions (e.g. Organism Identification, Organism Counting, Disease Diagnosis, Boundary Analysis, Agri-Tools). The exact mapping from dimension_id/sub_dimension_id to these names is not published in the source VQRA.json; counts above are reported by raw numeric ID, preserved verbatim in raw_metadata.

Usage

from datasets import load_dataset, concatenate_datasets

# Single image folder -> one default config
ds = load_dataset("Project-AgML/AgroCoT")

# Stream without downloading
ds = load_dataset("Project-AgML/AgroCoT", streaming=True)

Every record shares the SAME columns so heterogeneous datasets concatenate cleanly: id, file_names (1..N images), messages, origin_dataset, and raw_metadata. raw_metadata is a JSON-encoded string holding every original source field that was NOT already folded into messages/file_names (the question, answer, options, and image paths are omitted to avoid duplication), preserved verbatim, or {} if none remain; restore them with json.loads(row["raw_metadata"]). Using a JSON string (not a native struct) is what lets concatenate_datasets([...]) work across datasets whose raw fields differ in type. Multi-image rows return images as a list aligned to the {"type": "image"} placeholders in messages.

Optional preprocessing: Chain-of-Thought reasoning

By default the assistant turn in messages contains only the final short answer (e.g. A single MCQ letter), matching the rest of the AgML multimodal corpus. The source AgroCoT paper additionally provides a step-by-step reasoning trace for every example, which the authors report improves training when combined with the final answer.

This trace is preserved verbatim and is available per-row inside raw_metadata. To build a chain-of-thought assistant response, parse raw_metadata and prepend the reasoning to the existing answer text:

import json
from datasets import load_dataset

ds = load_dataset("Project-AgML/AgroCoT")

def add_reasoning(example):
    raw = json.loads(example["raw_metadata"])
    reasoning = raw.get("reasoning", "")
    answer = example["messages"][-1]["content"][0]["text"]
    example["messages"][-1]["content"][0]["text"] = f"{reasoning}\n\nFinal Answer: {answer}"
    return example

ds_with_reasoning = ds.map(add_reasoning)

This step is optional and not applied by default, to keep the on-disk format identical to the zero-conversion imagefolder contract used across the rest of the AgML multimodal datasets.

Citation

@misc{wen2025agrocot,
      title={AgroCoT: A Chain-of-Thought Benchmark for Evaluating Reasoning in Vision-Language Models for Agriculture},
      author={Wen, Yibin and Li, Qingmei and Ye, Zi and Zhang, Jiarui and Fan, Xiaoya and Mai, Zurong and Wu, Jing and Lou, Shuohong and Chen, Yuhang and Huang, Henglian and Zhang, Yang and Gu, Defeng and Zhao, Lingyuan and Lu, Yutong and Fu, Haohuan and Huang, Jianxi and Zheng, Juepeng},
      year={2025},
      eprint={2511.23253},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2511.23253}
}

Wen, Yibin; Li, Qingmei; Ye, Zi; Zhang, Jiarui; Fan, Xiaoya; Mai, Zurong; Wu, Jing; Lou, Shuohong; Chen, Yuhang; Huang, Henglian; Zhang, Yang; Gu, Defeng; Zhao, Lingyuan; Lu, Yutong; Fu, Haohuan; Huang, Jianxi; Zheng, Juepeng (2025), "AgroCoT: A Chain-of-Thought Benchmark for Evaluating Reasoning in Vision-Language Models for Agriculture", arXiv:2511.23253