|
Download README.md from math-ai/AutoMathText: direct link, hf CLI and curl.
- Browser
- Download file 18.8 kB
-
https://huggingface.co/datasets/math-ai/AutoMathText/resolve/main/README.md
- Command line
-
hf download hf://datasets/math-ai/AutoMathText/README.md
-
curl -L -o README.md https://huggingface.co/datasets/math-ai/AutoMathText/resolve/main/README.md
18.8 kB
| language: | |
| - en | |
| license: cc-by-sa-4.0 | |
| size_categories: | |
| - 10B<n<100B | |
| task_categories: | |
| - text-generation | |
| - question-answering | |
| pretty_name: AutoMathText | |
| configs: | |
| - config_name: web-0.50-to-1.00 | |
| data_files: | |
| - split: train | |
| path: | |
| - data/web/0.95-1.00.jsonl | |
| - data/web/0.90-0.95.jsonl | |
| - data/web/0.85-0.90.jsonl | |
| - data/web/0.80-0.85.jsonl | |
| - data/web/0.75-0.80.jsonl | |
| - data/web/0.70-0.75.jsonl | |
| - data/web/0.65-0.70.jsonl | |
| - data/web/0.60-0.65.jsonl | |
| - data/web/0.55-0.60.jsonl | |
| - data/web/0.50-0.55.jsonl | |
| default: true | |
| - config_name: web-0.60-to-1.00 | |
| data_files: | |
| - split: train | |
| path: | |
| - data/web/0.95-1.00.jsonl | |
| - data/web/0.90-0.95.jsonl | |
| - data/web/0.85-0.90.jsonl | |
| - data/web/0.80-0.85.jsonl | |
| - data/web/0.75-0.80.jsonl | |
| - data/web/0.70-0.75.jsonl | |
| - data/web/0.65-0.70.jsonl | |
| - data/web/0.60-0.65.jsonl | |
| - config_name: web-0.70-to-1.00 | |
| data_files: | |
| - split: train | |
| path: | |
| - data/web/0.95-1.00.jsonl | |
| - data/web/0.90-0.95.jsonl | |
| - data/web/0.85-0.90.jsonl | |
| - data/web/0.80-0.85.jsonl | |
| - data/web/0.75-0.80.jsonl | |
| - data/web/0.70-0.75.jsonl | |
| - data/web/0.65-0.70.jsonl | |
| - data/web/0.60-0.65.jsonl | |
| - config_name: web-0.80-to-1.00 | |
| data_files: | |
| - split: train | |
| path: | |
| - data/web/0.95-1.00.jsonl | |
| - data/web/0.90-0.95.jsonl | |
| - data/web/0.85-0.90.jsonl | |
| - data/web/0.80-0.85.jsonl | |
| - config_name: web-full | |
| data_files: data/web/*.jsonl | |
| - config_name: arxiv-0.50-to-1.00 | |
| data_files: | |
| - split: train | |
| path: | |
| - data/arxiv/0.90-1.00/*.jsonl | |
| - data/arxiv/0.80-0.90/*.jsonl | |
| - data/arxiv/0.70-0.80/*.jsonl | |
| - data/arxiv/0.60-0.70/*.jsonl | |
| - data/arxiv/0.50-0.60/*.jsonl | |
| - config_name: arxiv-0.60-to-1.00 | |
| data_files: | |
| - split: train | |
| path: | |
| - data/arxiv/0.90-1.00/*.jsonl | |
| - data/arxiv/0.80-0.90/*.jsonl | |
| - data/arxiv/0.70-0.80/*.jsonl | |
| - data/arxiv/0.60-0.70/*.jsonl | |
| - config_name: arxiv-0.70-to-1.00 | |
| data_files: | |
| - split: train | |
| path: | |
| - data/arxiv/0.90-1.00/*.jsonl | |
| - data/arxiv/0.80-0.90/*.jsonl | |
| - data/arxiv/0.70-0.80/*.jsonl | |
| - config_name: arxiv-0.80-to-1.00 | |
| data_files: | |
| - split: train | |
| path: | |
| - data/arxiv/0.90-1.00/*.jsonl | |
| - data/arxiv/0.80-0.90/*.jsonl | |
| - config_name: arxiv-full | |
| data_files: | |
| - split: train | |
| path: | |
| - data/arxiv/0.90-1.00/*.jsonl | |
| - data/arxiv/0.80-0.90/*.jsonl | |
| - data/arxiv/0.70-0.80/*.jsonl | |
| - data/arxiv/0.60-0.70/*.jsonl | |
| - data/arxiv/0.50-0.60/*.jsonl | |
| - data/arxiv/0.00-0.50/*.jsonl | |
| - config_name: code-0.50-to-1.00 | |
| data_files: | |
| - split: train | |
| path: | |
| - data/code/agda/0.95-1.00.jsonl | |
| - data/code/agda/0.90-0.95.jsonl | |
| - data/code/agda/0.85-0.90.jsonl | |
| - data/code/agda/0.80-0.85.jsonl | |
| - data/code/agda/0.75-0.80.jsonl | |
| - data/code/agda/0.70-0.75.jsonl | |
| - data/code/agda/0.65-0.70.jsonl | |
| - data/code/agda/0.60-0.65.jsonl | |
| - data/code/agda/0.55-0.60.jsonl | |
| - data/code/agda/0.50-0.55.jsonl | |
| - data/code/c/0.95-1.00.jsonl | |
| - data/code/c/0.90-0.95.jsonl | |
| - data/code/c/0.85-0.90.jsonl | |
| - data/code/c/0.80-0.85.jsonl | |
| - data/code/c/0.75-0.80.jsonl | |
| - data/code/c/0.70-0.75.jsonl | |
| - data/code/c/0.65-0.70.jsonl | |
| - data/code/c/0.60-0.65.jsonl | |
| - data/code/c/0.55-0.60.jsonl | |
| - data/code/c/0.50-0.55.jsonl | |
| - data/code/cpp/0.95-1.00.jsonl | |
| - data/code/cpp/0.90-0.95.jsonl | |
| - data/code/cpp/0.85-0.90.jsonl | |
| - data/code/cpp/0.80-0.85.jsonl | |
| - data/code/cpp/0.75-0.80.jsonl | |
| - data/code/cpp/0.70-0.75.jsonl | |
| - data/code/cpp/0.65-0.70.jsonl | |
| - data/code/cpp/0.60-0.65.jsonl | |
| - data/code/cpp/0.55-0.60.jsonl | |
| - data/code/cpp/0.50-0.55.jsonl | |
| - data/code/fortran/0.95-1.00.jsonl | |
| - data/code/fortran/0.90-0.95.jsonl | |
| - data/code/fortran/0.85-0.90.jsonl | |
| - data/code/fortran/0.80-0.85.jsonl | |
| - data/code/fortran/0.75-0.80.jsonl | |
| - data/code/fortran/0.70-0.75.jsonl | |
| - data/code/fortran/0.65-0.70.jsonl | |
| - data/code/fortran/0.60-0.65.jsonl | |
| - data/code/fortran/0.55-0.60.jsonl | |
| - data/code/fortran/0.50-0.55.jsonl | |
| - data/code/gap/0.95-1.00.jsonl | |
| - data/code/gap/0.90-0.95.jsonl | |
| - data/code/gap/0.85-0.90.jsonl | |
| - data/code/gap/0.80-0.85.jsonl | |
| - data/code/gap/0.75-0.80.jsonl | |
| - data/code/gap/0.70-0.75.jsonl | |
| - data/code/gap/0.65-0.70.jsonl | |
| - data/code/gap/0.60-0.65.jsonl | |
| - data/code/gap/0.55-0.60.jsonl | |
| - data/code/gap/0.50-0.55.jsonl | |
| - data/code/github-coq-train/0.95-1.00.jsonl | |
| - data/code/github-coq-train/0.90-0.95.jsonl | |
| - data/code/github-coq-train/0.85-0.90.jsonl | |
| - data/code/github-coq-train/0.80-0.85.jsonl | |
| - data/code/github-coq-train/0.75-0.80.jsonl | |
| - data/code/github-coq-train/0.70-0.75.jsonl | |
| - data/code/github-coq-train/0.65-0.70.jsonl | |
| - data/code/github-coq-train/0.60-0.65.jsonl | |
| - data/code/github-coq-train/0.55-0.60.jsonl | |
| - data/code/github-coq-train/0.50-0.55.jsonl | |
| - data/code/github-isabelle-train/0.95-1.00.jsonl | |
| - data/code/github-isabelle-train/0.90-0.95.jsonl | |
| - data/code/github-isabelle-train/0.85-0.90.jsonl | |
| - data/code/github-isabelle-train/0.80-0.85.jsonl | |
| - data/code/github-isabelle-train/0.75-0.80.jsonl | |
| - data/code/github-isabelle-train/0.70-0.75.jsonl | |
| - data/code/github-isabelle-train/0.65-0.70.jsonl | |
| - data/code/github-isabelle-train/0.60-0.65.jsonl | |
| - data/code/github-isabelle-train/0.55-0.60.jsonl | |
| - data/code/github-isabelle-train/0.50-0.55.jsonl | |
| - data/code/github-lean-train/0.95-1.00.jsonl | |
| - data/code/github-lean-train/0.90-0.95.jsonl | |
| - data/code/github-lean-train/0.85-0.90.jsonl | |
| - data/code/github-lean-train/0.80-0.85.jsonl | |
| - data/code/github-lean-train/0.75-0.80.jsonl | |
| - data/code/github-lean-train/0.70-0.75.jsonl | |
| - data/code/github-lean-train/0.65-0.70.jsonl | |
| - data/code/github-lean-train/0.60-0.65.jsonl | |
| - data/code/github-lean-train/0.55-0.60.jsonl | |
| - data/code/github-lean-train/0.50-0.55.jsonl | |
| - data/code/github-MATLAB-train/0.95-1.00.jsonl | |
| - data/code/github-MATLAB-train/0.90-0.95.jsonl | |
| - data/code/github-MATLAB-train/0.85-0.90.jsonl | |
| - data/code/github-MATLAB-train/0.80-0.85.jsonl | |
| - data/code/github-MATLAB-train/0.75-0.80.jsonl | |
| - data/code/github-MATLAB-train/0.70-0.75.jsonl | |
| - data/code/github-MATLAB-train/0.65-0.70.jsonl | |
| - data/code/github-MATLAB-train/0.60-0.65.jsonl | |
| - data/code/github-MATLAB-train/0.55-0.60.jsonl | |
| - data/code/github-MATLAB-train/0.50-0.55.jsonl | |
| - data/code/haskell/0.95-1.00.jsonl | |
| - data/code/haskell/0.90-0.95.jsonl | |
| - data/code/haskell/0.85-0.90.jsonl | |
| - data/code/haskell/0.80-0.85.jsonl | |
| - data/code/haskell/0.75-0.80.jsonl | |
| - data/code/haskell/0.70-0.75.jsonl | |
| - data/code/haskell/0.65-0.70.jsonl | |
| - data/code/haskell/0.60-0.65.jsonl | |
| - data/code/haskell/0.55-0.60.jsonl | |
| - data/code/haskell/0.50-0.55.jsonl | |
| - data/code/idris/0.95-1.00.jsonl | |
| - data/code/idris/0.90-0.95.jsonl | |
| - data/code/idris/0.85-0.90.jsonl | |
| - data/code/idris/0.80-0.85.jsonl | |
| - data/code/idris/0.75-0.80.jsonl | |
| - data/code/idris/0.70-0.75.jsonl | |
| - data/code/idris/0.65-0.70.jsonl | |
| - data/code/idris/0.60-0.65.jsonl | |
| - data/code/idris/0.55-0.60.jsonl | |
| - data/code/idris/0.50-0.55.jsonl | |
| - data/code/isa_proofsteps/0.95-1.00.jsonl | |
| - data/code/isa_proofsteps/0.90-0.95.jsonl | |
| - data/code/isa_proofsteps/0.85-0.90.jsonl | |
| - data/code/isa_proofsteps/0.80-0.85.jsonl | |
| - data/code/isa_proofsteps/0.75-0.80.jsonl | |
| - data/code/isa_proofsteps/0.70-0.75.jsonl | |
| - data/code/isa_proofsteps/0.65-0.70.jsonl | |
| - data/code/isa_proofsteps/0.60-0.65.jsonl | |
| - data/code/isa_proofsteps/0.55-0.60.jsonl | |
| - data/code/isa_proofsteps/0.50-0.55.jsonl | |
| - data/code/julia/0.95-1.00.jsonl | |
| - data/code/julia/0.90-0.95.jsonl | |
| - data/code/julia/0.85-0.90.jsonl | |
| - data/code/julia/0.80-0.85.jsonl | |
| - data/code/julia/0.75-0.80.jsonl | |
| - data/code/julia/0.70-0.75.jsonl | |
| - data/code/julia/0.65-0.70.jsonl | |
| - data/code/julia/0.60-0.65.jsonl | |
| - data/code/julia/0.55-0.60.jsonl | |
| - data/code/julia/0.50-0.55.jsonl | |
| - data/code/jupyter-notebook/0.95-1.00.jsonl | |
| - data/code/jupyter-notebook/0.90-0.95.jsonl | |
| - data/code/jupyter-notebook/0.85-0.90.jsonl | |
| - data/code/jupyter-notebook/0.80-0.85.jsonl | |
| - data/code/jupyter-notebook/0.75-0.80.jsonl | |
| - data/code/jupyter-notebook/0.70-0.75.jsonl | |
| - data/code/jupyter-notebook/0.65-0.70.jsonl | |
| - data/code/jupyter-notebook/0.60-0.65.jsonl | |
| - data/code/jupyter-notebook/0.55-0.60.jsonl | |
| - data/code/jupyter-notebook/0.50-0.55.jsonl | |
| - data/code/lean_proofsteps/0.95-1.00.jsonl | |
| - data/code/lean_proofsteps/0.90-0.95.jsonl | |
| - data/code/lean_proofsteps/0.85-0.90.jsonl | |
| - data/code/lean_proofsteps/0.80-0.85.jsonl | |
| - data/code/lean_proofsteps/0.75-0.80.jsonl | |
| - data/code/lean_proofsteps/0.70-0.75.jsonl | |
| - data/code/lean_proofsteps/0.65-0.70.jsonl | |
| - data/code/lean_proofsteps/0.60-0.65.jsonl | |
| - data/code/lean_proofsteps/0.55-0.60.jsonl | |
| - data/code/lean_proofsteps/0.50-0.55.jsonl | |
| - data/code/maple/0.95-1.00.jsonl | |
| - data/code/maple/0.90-0.95.jsonl | |
| - data/code/maple/0.85-0.90.jsonl | |
| - data/code/maple/0.80-0.85.jsonl | |
| - data/code/maple/0.75-0.80.jsonl | |
| - data/code/maple/0.70-0.75.jsonl | |
| - data/code/maple/0.65-0.70.jsonl | |
| - data/code/maple/0.60-0.65.jsonl | |
| - data/code/maple/0.55-0.60.jsonl | |
| - data/code/maple/0.50-0.55.jsonl | |
| - data/code/python/0.95-1.00.jsonl | |
| - data/code/python/0.90-0.95.jsonl | |
| - data/code/python/0.85-0.90.jsonl | |
| - data/code/python/0.80-0.85.jsonl | |
| - data/code/python/0.75-0.80.jsonl | |
| - data/code/python/0.70-0.75.jsonl | |
| - data/code/python/0.65-0.70.jsonl | |
| - data/code/python/0.60-0.65.jsonl | |
| - data/code/python/0.55-0.60.jsonl | |
| - data/code/python/0.50-0.55.jsonl | |
| - data/code/r/0.95-1.00.jsonl | |
| - data/code/r/0.90-0.95.jsonl | |
| - data/code/r/0.85-0.90.jsonl | |
| - data/code/r/0.80-0.85.jsonl | |
| - data/code/r/0.75-0.80.jsonl | |
| - data/code/r/0.70-0.75.jsonl | |
| - data/code/r/0.65-0.70.jsonl | |
| - data/code/r/0.60-0.65.jsonl | |
| - data/code/r/0.55-0.60.jsonl | |
| - data/code/r/0.50-0.55.jsonl | |
| - data/code/tex/0.95-1.00.jsonl | |
| - data/code/tex/0.90-0.95.jsonl | |
| - data/code/tex/0.85-0.90.jsonl | |
| - data/code/tex/0.80-0.85.jsonl | |
| - data/code/tex/0.75-0.80.jsonl | |
| - data/code/tex/0.70-0.75.jsonl | |
| - data/code/tex/0.65-0.70.jsonl | |
| - data/code/tex/0.60-0.65.jsonl | |
| - data/code/tex/0.55-0.60.jsonl | |
| - data/code/tex/0.50-0.55.jsonl | |
| - config_name: code-python-0.50-to-1.00 | |
| data_files: | |
| - split: train | |
| path: | |
| - data/code/python/0.95-1.00.jsonl | |
| - data/code/python/0.90-0.95.jsonl | |
| - data/code/python/0.85-0.90.jsonl | |
| - data/code/python/0.80-0.85.jsonl | |
| - data/code/python/0.75-0.80.jsonl | |
| - data/code/python/0.70-0.75.jsonl | |
| - data/code/python/0.65-0.70.jsonl | |
| - data/code/python/0.60-0.65.jsonl | |
| - data/code/python/0.55-0.60.jsonl | |
| - data/code/python/0.50-0.55.jsonl | |
| - config_name: code-python-0.60-to-1.00 | |
| data_files: | |
| - split: train | |
| path: | |
| - data/code/python/0.95-1.00.jsonl | |
| - data/code/python/0.90-0.95.jsonl | |
| - data/code/python/0.85-0.90.jsonl | |
| - data/code/python/0.80-0.85.jsonl | |
| - data/code/python/0.75-0.80.jsonl | |
| - data/code/python/0.70-0.75.jsonl | |
| - data/code/python/0.65-0.70.jsonl | |
| - data/code/python/0.60-0.65.jsonl | |
| - config_name: code-python-0.70-to-1.00 | |
| data_files: | |
| - split: train | |
| path: | |
| - data/code/python/0.95-1.00.jsonl | |
| - data/code/python/0.90-0.95.jsonl | |
| - data/code/python/0.85-0.90.jsonl | |
| - data/code/python/0.80-0.85.jsonl | |
| - data/code/python/0.75-0.80.jsonl | |
| - data/code/python/0.70-0.75.jsonl | |
| - config_name: code-python-0.80-to-1.00 | |
| data_files: | |
| - split: train | |
| path: | |
| - data/code/python/0.95-1.00.jsonl | |
| - data/code/python/0.90-0.95.jsonl | |
| - data/code/python/0.85-0.90.jsonl | |
| - data/code/python/0.80-0.85.jsonl | |
| - config_name: code-jupyter-notebook-0.50-to-1.00 | |
| data_files: | |
| - split: train | |
| path: | |
| - data/code/jupyter-notebook/0.95-1.00.jsonl | |
| - data/code/jupyter-notebook/0.90-0.95.jsonl | |
| - data/code/jupyter-notebook/0.85-0.90.jsonl | |
| - data/code/jupyter-notebook/0.80-0.85.jsonl | |
| - data/code/jupyter-notebook/0.75-0.80.jsonl | |
| - data/code/jupyter-notebook/0.70-0.75.jsonl | |
| - data/code/jupyter-notebook/0.65-0.70.jsonl | |
| - data/code/jupyter-notebook/0.60-0.65.jsonl | |
| - data/code/jupyter-notebook/0.55-0.60.jsonl | |
| - data/code/jupyter-notebook/0.50-0.55.jsonl | |
| - config_name: code-jupyter-notebook-0.60-to-1.00 | |
| data_files: | |
| - split: train | |
| path: | |
| - data/code/jupyter-notebook/0.95-1.00.jsonl | |
| - data/code/jupyter-notebook/0.90-0.95.jsonl | |
| - data/code/jupyter-notebook/0.85-0.90.jsonl | |
| - data/code/jupyter-notebook/0.80-0.85.jsonl | |
| - data/code/jupyter-notebook/0.75-0.80.jsonl | |
| - data/code/jupyter-notebook/0.70-0.75.jsonl | |
| - data/code/jupyter-notebook/0.65-0.70.jsonl | |
| - data/code/jupyter-notebook/0.60-0.65.jsonl | |
| - config_name: code-jupyter-notebook-0.70-to-1.00 | |
| data_files: | |
| - split: train | |
| path: | |
| - data/code/jupyter-notebook/0.95-1.00.jsonl | |
| - data/code/jupyter-notebook/0.90-0.95.jsonl | |
| - data/code/jupyter-notebook/0.85-0.90.jsonl | |
| - data/code/jupyter-notebook/0.80-0.85.jsonl | |
| - data/code/jupyter-notebook/0.75-0.80.jsonl | |
| - data/code/jupyter-notebook/0.70-0.75.jsonl | |
| - config_name: code-jupyter-notebook-0.80-to-1.00 | |
| data_files: | |
| - split: train | |
| path: | |
| - data/code/jupyter-notebook/0.95-1.00.jsonl | |
| - data/code/jupyter-notebook/0.90-0.95.jsonl | |
| - data/code/jupyter-notebook/0.85-0.90.jsonl | |
| - data/code/jupyter-notebook/0.80-0.85.jsonl | |
| - config_name: code-full | |
| data_files: | |
| - split: train | |
| path: | |
| - data/code/*/*.jsonl | |
| tags: | |
| - mathematical-reasoning | |
| - reasoning | |
| - finetuning | |
| - pretraining | |
| - llm | |
| 🎉 **This work, introducing the AutoMathText dataset and the AutoDS method, has been accepted to The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025 Findings)!** 🎉 | |
| # AutoMathText | |
| **AutoMathText** is an extensive and carefully curated dataset encompassing around **200 GB** of mathematical texts. It's a compilation sourced from a diverse range of platforms including various websites, arXiv, and GitHub (OpenWebMath, RedPajama, Algebraic Stack). This rich repository has been **autonomously selected (labeled) by the state-of-the-art open-source language model**, Qwen-72B. Each piece of content in the dataset is assigned **a score `lm_q1q2_score` within the range of [0, 1]**, reflecting its relevance, quality and educational value in the context of mathematical intelligence. | |
| GitHub homepage: https://github.com/yifanzhang-pro/AutoMathText | |
| ArXiv paper: https://huggingface.co/papers/2402.07625 (https://arxiv.org/abs/2402.07625) | |
| ## Objective | |
| The primary aim of the **AutoMathText** dataset is to provide a comprehensive and reliable resource for a wide array of users - from academic researchers and educators to AI practitioners and mathematics enthusiasts. This dataset is particularly geared towards: | |
| - Facilitating advanced research in **the intersection of mathematics and artificial intelligence**. | |
| - Serving as an educational tool for **learning and teaching complex mathematical concepts**. | |
| - Providing **a foundation for developing and training AI models** specialized in processing and understanding **mathematical content**. | |
| ## Configs | |
| ```YAML | |
| configs: | |
| - config_name: web-0.50-to-1.00 | |
| data_files: | |
| - split: train | |
| path: | |
| - data/web/0.95-1.00.jsonl | |
| - data/web/0.90-0.95.jsonl | |
| - ... | |
| - data/web/0.50-0.55.jsonl | |
| default: true | |
| - config_name: web-0.60-to-1.00 | |
| - config_name: web-0.70-to-1.00 | |
| - config_name: web-0.80-to-1.00 | |
| - config_name: web-full | |
| data_files: data/web/*.jsonl | |
| - config_name: arxiv-0.50-to-1.00 | |
| data_files: | |
| - split: train | |
| path: | |
| - data/arxiv/0.90-1.00/*.jsonl | |
| - ... | |
| - data/arxiv/0.50-0.60/*.jsonl | |
| - config_name: arxiv-0.60-to-1.00 | |
| - config_name: arxiv-0.70-to-1.00 | |
| - config_name: arxiv-0.80-to-1.00 | |
| - config_name: arxiv-full | |
| data_files: data/arxiv/*/*.jsonl | |
| - config_name: code-0.50-to-1.00 | |
| data_files: | |
| - split: train | |
| path: | |
| - data/code/*/0.95-1.00.jsonl | |
| - ... | |
| - data/code/*/0.50-0.55.jsonl | |
| - config_name: code-python-0.50-to-1.00 | |
| - split: train | |
| path: | |
| - data/code/python/0.95-1.00.jsonl | |
| - ... | |
| - data/code/python/0.50-0.55.jsonl | |
| - config_name: code-python-0.60-to-1.00 | |
| - config_name: code-python-0.70-to-1.00 | |
| - config_name: code-python-0.80-to-1.00 | |
| - config_name: code-jupyter-notebook-0.50-to-1.00 | |
| - split: train | |
| path: | |
| - data/code/jupyter-notebook/0.95-1.00.jsonl | |
| - ... | |
| - data/code/jupyter-notebook/0.50-0.55.jsonl | |
| - config_name: code-jupyter-notebook-0.60-to-1.00 | |
| - config_name: code-jupyter-notebook-0.70-to-1.00 | |
| - config_name: code-jupyter-notebook-0.80-to-1.00 | |
| - config_name: code-full | |
| data_files: data/code/*/*.jsonl | |
| ``` | |
| How to load data: | |
| ```python | |
| from datasets import load_dataset | |
| ds = load_dataset("math-ai/AutoMathText", "web-0.50-to-1.00") # or any valid config_name | |
| ``` | |
| ## Features | |
| - **Volume**: Approximately 200 GB of text data (in natural language and programming language). | |
| - **Content**: A diverse collection of mathematical texts, including but not limited to research papers, educational articles, and code documentation. | |
| - **Labeling**: Every text is **scored** by Qwen-72B, a sophisticated language model, ensuring a high standard of relevance and accuracy. | |
| - **Scope**: Covers a wide spectrum of mathematical topics, making it suitable for various applications in advanced research and education. | |
| ## References | |
| - OpenWebMath [[link]](https://huggingface.co/datasets/open-web-math/open-web-math) | |
| - RedPajama [[link]](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T) | |
| - Algebraick Stack [[link]](https://huggingface.co/datasets/EleutherAI/proof-pile-2) (a subset of Proof-Pile-2) | |
| ## Citation | |
| We appreciate your use of **AutoMathText** in your work. If you find this repository helpful, please consider citing it and star this repo. Feel free to contact yifanzhangresearch@gmail.com or open an issue if you have any questions (GitHub homepage: https://github.com/yifanzhang-pro/AutoMathText). | |
| ```bibtex | |
| @article{zhang2025autonomous, | |
| title={Autonomous Data Selection with Zero-shot Generative Classifiers for Mathematical Texts}, | |
| author={Zhang, Yifan and Luo, Yifan and Yuan, Yang and Yao, Andrew Chi-Chih}, | |
| journal={The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025 Findings)}, | |
| year={2025} | |
| } | |
| ``` |