zh cleaned speech corpus

This release contains refreshed full-data and token-coverage 5000-hour zh-CN and zh-TW corpora, each with a frozen eval split. All configurations reference one deduplicated, previously validated FLAC-in-Parquet audio pool.

All corpus locators are portable: audio/<subset>/<group>/part-N.parquet#row=N. The audio pool was not transcoded or repacked in this refresh.

Corpus overview

Config	Split	Rows	Hours
zh-CN	train	28,562,287	46096.335
zh-CN	5000h train	2,884,446	4999.999
zh-CN	eval	1,354	2.000
zh-TW	train	26,887,243	44914.518
zh-TW	5000h train	2,797,472	4999.999
zh-TW	eval	1,108	1.999

Cleaning

The train manifests use the converged full-data quality filters. zh-TW is normalized to Taiwan Traditional Chinese with guarded artifact repairs; zh-CN is normalized to Simplified Chinese with its own guarded artifact repairs. The 5000-hour manifests are token-coverage subsets of those cleaned train sets and reuse the exact same audio Parquet files; no audio is copied or repacked. Audited Cantonese/Hokkien contamination, foreign-script contamination, corrupt Unicode, implausible duration, and known machine-corruption cases are excluded. Three all-zero Common Voice recordings already absent from the audio pool are also excluded from each train manifest.

Train/eval portable audio-identity overlap is zero for both configurations. Exact counts, input hashes, normalization counts, and corpus checksums are in release.json and cleaning_audit.json.

Audio pool

• Unique audio rows: 34,959,126
• Audio hours: 62,343.041
• Parquet size: 5.481783 TB

from datasets import load_dataset
train = load_dataset("WTForbes/zh", "zh-TW", split="train")
eval_set = load_dataset("WTForbes/zh", "zh-TW", split="eval")