flexitok
/

parity-aware-bpe-devset

Model card Files Files and versions

Parity-Aware BPE Tokenizer (dev-set balanced)

Languages

Trained on 21 languages, using FineWeb (train) and FLORES-200 (dev) per language:

Language	Training source	FLORES-200 code
English	`fw_edu`	`eng_Latn`
German	`deu_Latn`	`deu_Latn`
French	`fra_Latn`	`fra_Latn`
Arabic	`arb_Arab`	`arb_Arab`
Farsi	`fas_Arab`	`fas_Arab`
Czech	`ces_Latn`	`ces_Latn`
Chinese	`cmn_Hani`	`cmn_Hani`
Danish	`dan_Latn`	`dan_Latn`
Dutch	`nld_Latn`	`nld_Latn`
Greek	`ell_Grek`	`ell_Grek`
Hungarian	`hun_Latn`	`hun_Latn`
Indonesian	`ind_Latn`	`ind_Latn`
Italian	`ita_Latn`	`ita_Latn`
Japanese	`jpn_Jpan`	`jpn_Jpan`
Polish	`pol_Latn`	`pol_Latn`
Portuguese	`por_Latn`	`por_Latn`
Russian	`rus_Cyrl`	`rus_Cyrl`
Spanish	`spa_Latn`	`spa_Latn`
Swedish	`swe_Latn`	`swe_Latn`
Turkish	`tur_Latn`	`tur_Latn`
Vietnamese	`vie_Latn`	`vie_Latn`

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support