agentlans/multilingual-document-classification

Multilingual Document Classification Dataset

This dataset contains 100,000 text passages across 100 non-English language-script pairs sourced from the agentlans/HuggingFaceFW-finetranslations-100-languages-sample collection.

Each original text passage is paired with its English translation and has been programmatically annotated with domain, writing genre, and educational classifications to facilitate cross-lingual classification and domain adaptation tasks.

Dataset Overview

• Size: 100,000 original text passages + 100,000 English translations.
• Languages: 100 non-English languages (original text) paired with English translations.
• Primary Use Case: Multilingual document classification, cross-lingual domain adaptation, and translation-based text evaluation.
• Splits: All subsets are split into 80% train, 10% validation, and 10% test sets. The splits are stratified by the target labels to ensure identical class distributions across splits.

Subset Config Structure

The dataset contains subset configurations tailored for specific training objectives.

• In the main config, each original text is stored alongside its English translation within a single row.
• In subset configs (for example, configurations filtered or categorized by specific schema labels), the original texts and their English translations are stored as distinct, individual rows to allow direct training on the target language or translation.

Annotation & Classification Details

To generate granular metadata for domain, genre, and cognitive level, two primary text classifiers were applied to the English translations:

1. nvidia/domain-classifier – Extracts high-level topical domains.
2. EssentialAI/eai-distill-0.5b – Extracts genre, cognitive depth, and educational level. See the EAI Taxonomy Schema for detailed definitions.

Key Classification Fields

Column Name	Source Model	Description / Purpose
nvidia_domain	NVIDIA Domain Classifier	General topical categorization (for example, News, Food & Drink).
doc_type_v1_primary	EAI Distill 0.5B	High-level document genre classification (V1).
doc_type_v2_primary	EAI Distill 0.5B	Refined, granular document type classification (V2).

The columns nvidia_domain, doc_type_v1_primary, and doc_type_v2_primary are used as the target labels for creating subset configs.

Dataset Schema & Examples

1. main Configuration Example

The main configuration contains both the original and translated text, as well as the complete suite of metadata extracted by the classifiers.

{
  "id": "<urn:uuid:8f0799fb-7964-44e1-af9d-6565a1f85937>",
  "translated": "In Gujarat too, there was opposition to the atrocities against farmers in Madhya Pradesh. Anger has spread in Gujarat over the incident in Madhya Pradesh. A protest demonstration was held by the Pradesh Congress in Ahmedabad...",
  "original": "મધ્યપ્રદેશમાં ખેડૂતો પર અત્યાચારનો ગુજરાતમાં પણ વિરોધ થયો છે. ગુજરાતમાં પણ મધ્યપ્રદેશની ઘટનાને લઈ નારાજગી પ્રસરી છે...",
  "language": "guj_Gujr",
  "nvidia_domain": "News",
  "bloom_cognitive_primary": "Understand",
  "bloom_cognitive_secondary": "Evaluate",
  "bloom_knowledge_primary": "Factual",
  "bloom_knowledge_secondary": "Conceptual",
  "doc_type_v1_primary": "News/Editorial",
  "doc_type_v2_primary": "News Article",
  "doc_type_v2_secondary": "Knowledge Article",
  "educational_level_primary": "General",
  "educational_level_secondary": "High School",
  "extraction_artifacts_primary": "No Artifacts",
  "fdc_primary": "320.954",
  "fdc_secondary": "338.954",
  "missing_content_primary": "No Missing Content",
  "reasoning_depth_primary": "No Reasoning",
  "reasoning_depth_secondary": "Basic",
  "technical_correctness_primary": "N/A",
  "technical_correctness_secondary": "Highly Correct"
}

2. Subset Configuration Example

The subset configurations are stripped down to the target text, its language identifier, and the specific classification label for the subset.

{
  "text": "Oh sweet potato; kuinka ihana oletkaan!\nJa vielä kaunis väriltäsi...",
  "language": "fin_Latn",
  "label": "Food_and_Drink"
}

Limitations

Users should keep the following limitations in mind when utilizing this dataset:

• Source Translation Quality: Since the source texts are derived from HuggingFaceFW-finetranslations, any artifacts, vocabulary choices, or grammatical inaccuracies in the underlying translations will carry over.
• Language Distribution: The dataset contains an uniform number of samples across languages. As a result, high-resource languages (for example, Mandarin Chinese) have the same number of rows as lower-resource languages (for example, Assamese).
• Class Imbalance: Certain topical domains and document types are heavily over-represented compared to others. For instance, there are far more promotional news articles than niche categories like culinary recipes.

Licence

This dataset is released under the Open Data Commons Attribution License (ODC-BY), matching the terms of the source datasets.