Datasets:
The dataset viewer is not available for this dataset.
Need help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.
SQL Schema Retrieval
The evaluation benchmark from "Finding the Right Tables and Columns: A Benchmark and Corpus-Adaptive Embeddings for SQL Schema Retrieval" (Zeng, Yu, Mehta, Zhao, Samdani).
Task — schema linking as retrieval. Given a natural-language question over a database,
retrieve the schema element(s) needed to answer it. Documents are schema elements: at
table granularity, each document is a table schema (columns + types rendered as markdown,
with sample rows); at column granularity, each document is a single column with its table
context. Relevance R(q) = the schema elements referenced by the question's gold SQL
(obtained by parsing table/column references from the ground-truth query). Retrieval is
performed per database group (rank the schema at hand). Metrics: nDCG@10 and
recall@10.
This recasts five text-to-SQL datasets as retrieval, spanning academic → enterprise → large live schemas — a step that pure text-to-SQL accuracy never exercises because it assumes the whole schema fits in context.
Subsets
Counts below are measured from this release (corpus docs = retrieval candidates at the subset's granularity; rel. = relevant query–document judgments).
| Subset (this release) | Source | Granularity | Corpus docs | Queries | Rel. judgments |
|---|---|---|---|---|---|
spider |
Spider (academic, cross-domain) | table | 180 | 2,147 | 3,366 |
bird |
BIRD (realistic, value semantics) | table | 75 | 1,534 | 2,956 |
beaver |
BEAVER (private enterprise warehouses) | table | 463 | 209 | 928 |
livesqlbench_table |
LiveSQLBench (base) | table | 244 | 410 | 1,075 |
livesqlbench_large_table |
LiveSQLBench-Large | table | 971 | 332 | 901 |
livesqlbench_large_column |
LiveSQLBench-Large | column | 17,709 | 332 | 2,157 |
Underlying database sizes (from the paper's Table 1): Spider 40 DB / 785 col; BIRD 11 DB / 798 col; BEAVER 6 DB / 4,238 col; LiveSQLBench 22 DB / 1,942 col; LiveSQLBench-Large 18 DB / 17,708 col. Gold sets average 1.6–4.4 tables (table level) and up to 6.5 columns (column level) per query. The paper additionally studies two document representations (schema-metadata vs. value-only) for Spider/BIRD/BEAVER; this release provides the table/column collections used for evaluation.
Format (BEIR / MTEB retrieval layout)
<subset>/
corpus.jsonl # {"_id": "<schema-element id>", "title": "", "text": "<schema as markdown>"}
queries.jsonl # {"_id": "<qid>", "text": "<natural-language question>"}
qrels/test.tsv # query-id \t corpus-id \t score (relevant judgments, score>0)
_id encodes the schema element as provided by the source: db__table for table-level
subsets, and a column identifier (e.g. db__table__column) for the column-level subset.
Sources & citation
This benchmark repackages five public text-to-SQL datasets as retrieval; please cite this work and the original datasets, and follow each source's license/terms:
- Spider — Yu et al., 2018 (EMNLP). https://yale-lily.github.io/spider
- BIRD — Li et al., 2023 (NeurIPS D&B). https://bird-bench.github.io/
- BEAVER — Chen et al., 2024, arXiv:2409.02038.
- LiveSQLBench (base + large) — BIRD Team, 2024. https://github.com/bird-bench/livesqlbench
@inproceedings{zeng2025sqlschemaretrieval,
title = {Finding the Right Tables and Columns: A Benchmark and Corpus-Adaptive Embeddings for SQL Schema Retrieval},
author = {Zeng, Qingcheng and Yu, Puxuan and Mehta, Aman and Zhao, Fuheng and Samdani, Rajhans},
year = {2025},
note = {Update venue/URL on publication}
}
- Downloads last month
- 27