Screen2AX-Element is part of the Screen2AX dataset suite, a research-driven collection for advancing accessibility in macOS applications using computer vision and deep learning.

This dataset focuses on UI element detection from macOS application screenshots and is designed for training and evaluating object detection models.

🧠 Dataset Summary

Each sample in the dataset consists of:

An application screenshot (image)
A dictionary of UI elements (objects) with 2 keys:
- bbox (List[List[int]]): List of bounding box coordinates in [x_min, y_min, x_max, y_max] format
- category (List[str]): List of labels indicating the UI element type (AXButton, AXDisclosureTriangle, AXLink, AXTextArea, AXImage)

The dataset supports training deep learning models for object detection tasks specifically tuned for graphical user interfaces (GUIs) on macOS.

Splits:

train
valid
test

Task Category:

object-detection

📚 Usage

Load with `datasets` library

from datasets import load_dataset

dataset = load_dataset("macpaw-research/Screen2AX-Element")

Example structure

sample = dataset["train"][0]
print(sample.keys())
# dict_keys(['image', 'objects'])

print(sample["objects"])
# {'bbox': [[x_min, y_min, x_max, y_max], ...], 'category': ['AXButton', ...]}

📜 License

This dataset is licensed under the Apache 2.0 License.

🔗 Related Projects

✍️ Citation

If you use this dataset, please cite the Screen2AX paper:

@misc{muryn2025screen2axvisionbasedapproachautomatic,
      title={Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation}, 
      author={Viktor Muryn and Marta Sumyk and Mariya Hirna and Sofiya Garkot and Maksym Shamrai},
      year={2025},
      eprint={2507.16704},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2507.16704}, 
}