--- license: apache-2.0 language: - en - ja - zh - multilingual datasets: - amphion/Emilia-Dataset - nvidia/hifitts-2 - reazon-research/reazonspeech - MLCommons/peoples_speech - facebook/voxpopuli - facebook/multilingual_librispeech - facebook/omnilingual-asr-corpus - ai4bharat/IndicVoices pipeline_tag: audio-to-audio tags: - audio - speech - neural-audio-codec - speech-tokenizer - variable-frame-rate - streaming - text-to-speech --- # RubatoCodec [![Code: GitHub](https://img.shields.io/badge/Code-GitHub-black?logo=github)](https://github.com/third-intelligence/RubatoCodec) [![Paper: arXiv](https://img.shields.io/badge/arXiv-2610.29434-b31b1b)](https://arxiv.org/abs/2610.29434) [![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue)](https://www.apache.org/licenses/LICENSE-2.0) RubatoCodec is a neural speech codec that divides speech into a fixed number of variable-duration events. Each event carries discrete code indices and an explicit duration. Event placement adapts to speech content, while the number of events is determined exactly by the utterance length, so the token sequence length a speech generator must produce is predictable. A causal decoder renders completed events incrementally, which makes the codec suitable for streaming text-to-speech. The name comes from the musical term *rubato*: local timing flexes while the overall tempo is preserved. - **Adaptive Firing with an exact event budget.** Multi-scale latent dissimilarity and waveform spectral flux guide cumulative allocation on a 50-Hz latent grid. For `T` frames the event count is exactly `max(1, min(T, round(0.25 · T)))`, giving 12.5 events/s. No learned boundary predictor, boundary labels, or rate loss is used. - **Streaming decoding.** Events can be decoded one at a time or in chunks. Event-by-event decoding takes about 11 ms per event on an NVIDIA H200, well below the ~80 ms of audio each event represents on average. - **Hybrid residual quantizer with depth control.** A semantically supervised FSQ first layer is followed by 15 DAC-style residual VQ layers, each with 4,096 entries. Any prefix depth from 1 to 16 can be decoded, giving code-index bitrates of 0.15–2.4 kbit/s plus duration information. - **Multilingual.** Trained on approximately 375,000 hours of speech, mostly English, Japanese, and Chinese, with 16-kHz input and 44.1-kHz output. ![RubatoCodec architecture](assets/architecture.png) ## Model details | | | |---|---| | Input / output sample rate | 16 kHz / 44.1 kHz | | Latent grid | 50 Hz | | Event rate | 12.5 events/s (retention ratio 0.25) | | Quantizer | 16 layers × 4,096 entries; layer 0 FSQ with semantic distillation, layers 1–15 residual VQ | | Bitrate | 1.23 kbit/s at depth 8, 2.43 kbit/s at depth 16 (LibriSpeech, including duration information) | | Decoder | Fully causal Transformer + ISTFT head, streaming capable | | Parameters | 246M codec (72M encoder, 173M decoder, 1M quantizer) + frozen [w2v-BERT 2.0](https://huggingface.co/facebook/w2v-bert-2.0) (580M, encoding only) | | License | Apache 2.0 | ## Usage Install the package from the [GitHub repository](https://github.com/third-intelligence/RubatoCodec): ```bash git clone https://github.com/third-intelligence/RubatoCodec cd RubatoCodec uv sync ``` Encode and decode: ```python import soundfile as sf import torch from rubatocodec import RubatoCodec codec = RubatoCodec.from_pretrained("third-intelligence/RubatoCodec", device="cuda") data, sr = sf.read("speech.wav", dtype="float32") # mono waveform wav = torch.from_numpy(data) enc = codec.encode(wav, sr=sr) # {"codes": (16, K) int64, "durs": (K,) int64} audio = codec.decode(enc) # (1, 1, T_audio) at 44.1 kHz, full depth audio = codec.decode(enc, depth=8) # lower bitrate with the first 8 layers sf.write("recon.wav", audio[0, 0].cpu().numpy(), codec.output_sr) ``` `durs` gives each event's duration in 20-ms grid frames. Loading also downloads `facebook/w2v-bert-2.0`, which is used only for encoding. Streaming decoding, inference settings, the code format, and training are documented in the repository: [inference guide](https://github.com/third-intelligence/RubatoCodec/blob/main/docs/inference.md), [architecture](https://github.com/third-intelligence/RubatoCodec/blob/main/docs/architecture.md), [training](https://github.com/third-intelligence/RubatoCodec/blob/main/docs/training.md). ## Evaluation RubatoCodec is evaluated on LibriSpeech test-clean, VCTK, JVS nonpara30, JVNV, and seed-tts-eval test-zh against 13 other codecs. At comparable bitrates, it shows the best intelligibility preservation among the compared codecs on the English and Japanese benchmarks and high predicted quality, while the strongest comparators achieve higher PESQ-WB and STOI. dWER/dCER compare ASR transcripts of the reconstructed and original audio (Whisper-large-v3 for English and Japanese, Paraformer-zh for Mandarin). SIM is ECAPA-TDNN speaker similarity. Bitrates include duration information. The tables below show a subset of the compared codecs. **Bold** and underline mark the best and second-best scores within each bitrate band. **LibriSpeech test-clean (English)** | Codec | Event rate (Hz) | Bitrate (kbit/s) | UTMOSv2 ↑ | DNSMOS ↑ | PESQ-WB ↑ | STOI ↑ | dWER (%) ↓ | SIM ↑ | |---|---:|---:|---:|---:|---:|---:|---:|---:| | *~1.2 kbit/s* | | | | | | | | | | FlexiCodec, τ=1, depth 8 | 12.5 | 1.24 | 3.194 | 3.284 | 2.813 | 0.940 | 0.83 | 0.805 | | MiMo-Audio-Tokenizer, depth 6 | 25.0 | 1.20 | 3.183 | 3.291 | 2.677 | 0.927 | 0.86 | 0.879 | | MOSS-Audio-Tokenizer, depth 10 | 12.5 | 1.25 | 3.230 | 3.217 | **3.052** | **0.949** | 0.89 | **0.885** | | DTM-Codec, 80-Hz target | 81.5 | 1.28 | 3.421 | **3.334** | 2.971 | 0.947 | 1.41 | 0.845 | | **RubatoCodec, depth 8** | 12.5 | 1.23 | **3.478** | 3.327 | 2.684 | 0.931 | **0.75** | 0.862 | | *~2.4 kbit/s* | | | | | | | | | | Qwen3-TTS-Tokenizer-12Hz | 12.5 | 2.20 | 3.230 | 3.273 | 3.177 | 0.958 | 0.53 | 0.931 | | MiMo-Audio-Tokenizer, depth 13 | 25.0 | 2.42 | 3.210 | 3.288 | 3.094 | 0.947 | 0.67 | 0.933 | | MOSS-Audio-Tokenizer, depth 20 | 12.5 | 2.50 | 3.239 | 3.203 | **3.536** | **0.968** | 0.62 | **0.943** | | **RubatoCodec, depth 16** | 12.5 | 2.43 | **3.486** | **3.314** | 3.029 | 0.950 | **0.50** | 0.920 | **JVNV (Japanese emotional speech with nonverbal vocalizations)** On expressive speech, RubatoCodec keeps dCER below 2% at both bitrates, while every other compared codec exceeds 3%. | Codec | Event rate (Hz) | Bitrate (kbit/s) | UTMOSv2 ↑ | DNSMOS ↑ | PESQ-WB ↑ | STOI ↑ | dCER (%) ↓ | SIM ↑ | |---|---:|---:|---:|---:|---:|---:|---:|---:| | *~1.2 kbit/s* | | | | | | | | | | FlexiCodec, τ=1, depth 8 | 12.5 | 1.24 | 1.811 | 3.128 | 2.062 | 0.862 | 5.52 | 0.771 | | MiMo-Audio-Tokenizer, depth 6 | 25.0 | 1.20 | **2.681** | **3.173** | 2.295 | 0.872 | 7.88 | 0.860 | | MOSS-Audio-Tokenizer, depth 10 | 12.5 | 1.25 | 2.396 | 3.089 | **3.032** | **0.917** | 5.44 | **0.909** | | DTM-Codec, 80-Hz target | 46.1 | 0.77 | 2.523 | 3.154 | 2.081 | 0.862 | 14.82 | 0.783 | | **RubatoCodec, depth 8** | 12.5 | 1.23 | 2.628 | 3.164 | 2.650 | 0.907 | **1.92** | 0.903 | | *~2.4 kbit/s* | | | | | | | | | | Qwen3-TTS-Tokenizer-12Hz | 12.5 | 2.20 | 2.638 | 3.097 | 3.041 | 0.931 | 3.01 | 0.945 | | MiMo-Audio-Tokenizer, depth 13 | 25.0 | 2.42 | **2.708** | **3.176** | 2.949 | 0.911 | 4.21 | 0.927 | | MOSS-Audio-Tokenizer, depth 20 | 12.5 | 2.50 | 2.383 | 3.057 | **3.496** | **0.941** | 3.96 | **0.953** | | **RubatoCodec, depth 16** | 12.5 | 2.43 | 2.651 | 3.138 | 3.032 | 0.929 | **1.54** | 0.941 | **Quality versus bitrate** (RubatoCodec is marked with a red star) ![English reconstruction versus bitrate](assets/english_reconstruction.png) ![Japanese and Mandarin reconstruction versus bitrate](assets/multilingual_reconstruction.png) **Adaptive event placement.** Each rectangle in the lower strip is one event. Longer events cover pauses and low-energy regions, while speech receives shorter events, with the same 12.5 events/s budget in every utterance. ![Adaptive event placement on English, Japanese, and Mandarin speech](assets/codec_firing_examples.png) The full protocol, all compared codecs and operating points, and throughput measurements are in the [evaluation document](https://github.com/third-intelligence/RubatoCodec/blob/main/docs/evaluation.md) and the [paper](https://arxiv.org/abs/2610.29434). ## Training data Approximately 375,000 hours of public and internal speech. Public sources include [Emilia-YODAS](https://huggingface.co/datasets/amphion/Emilia-Dataset), [HiFiTTS-2](https://huggingface.co/datasets/nvidia/hifitts-2) (44-kHz subset), [ReazonSpeech](https://huggingface.co/datasets/reazon-research/reazonspeech), [The People's Speech](https://huggingface.co/datasets/MLCommons/peoples_speech) (excluding ShareAlike-licensed data), [VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli), [Multilingual LibriSpeech](https://huggingface.co/datasets/facebook/multilingual_librispeech) (excluding English), [Omnilingual ASR Corpus](https://huggingface.co/datasets/facebook/omnilingual-asr-corpus), and [IndicVoices](https://huggingface.co/datasets/ai4bharat/IndicVoices). English, Japanese, and Chinese account for approximately 44%, 27%, and 13% of the data, and other languages for the remaining 16%. ## Limitations - RubatoCodec is trained on speech only and is intended for speech. It is not designed for music or general audio. - Encoding requires the whole utterance, because the event budget is defined at the utterance level. Only decoding is streaming. - Signal-fidelity metrics (PESQ-WB, STOI) are below those of the strongest compared codecs at similar bitrates. - Evaluation covers English, Japanese, and Mandarin, all of which appear in the training data. Performance on other languages has not been evaluated. - Like other speech codecs, RubatoCodec can be used in speech generation systems capable of voice cloning. Do not use it to impersonate people or to create deceptive content without consent. ## Citation ```bibtex @misc{arata2026rubatocodecbudgetedadaptivespeech, title={RubatoCodec: Budgeted Adaptive Speech Tokenization with Streaming Decoding}, author={Chihiro Arata and Shohei Taniguchi and Gouki Minegishi and Kan Hatakeyama-Sato}, year={2026}, eprint={2610.29434}, archivePrefix={arXiv}, primaryClass={cs.SD}, url={https://arxiv.org/abs/2610.29434}, } ``` ## Acknowledgements RubatoCodec builds on [X-Codec-2.0](https://github.com/zhenye234/X-Codec-2.0) and uses [w2v-BERT 2.0](https://huggingface.co/facebook/w2v-bert-2.0) as its frozen semantic encoder. It also adapts components from [BigCodec](https://github.com/Aria-K-Alethia/BigCodec), [DAC](https://github.com/descriptinc/descript-audio-codec), [Inworld TTS](https://github.com/inworld-ai/tts), and [BigVGAN](https://github.com/NVIDIA/BigVGAN). See the repository for the full list and their licenses. ## License The model weights and code are released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0). Third-party components retain their respective licenses.