pipecat-ai/voice-reader

Pipecat Voice Reader (rc2)

⚠️ Experimental. This is a release candidate, published for trying out and feedback. Its outputs, names and calibration may change between releases, and its readings are cues for an LLM to weigh, not verdicts to act on alone. Pin a revision, and read the limitations below before relying on it.

A small CPU model that hears how a user sounds: tone of voice, sarcasm in the delivery, energy, laughter and speech rate. It is the voice half of a pair: it is made to run beside Pipecat's Tone Reader (PipecatToneReader), which reads the words. It reads the last 8 s of each user turn, once, when the user stops speaking, and Pipecat's VoiceReaderProcessor writes the reading as RCS perception at the head of the user's message, after the Tone Reader's:

([emotion happy=0.62, neutral=0.30],[sarcasm value=0.08],[tone frustrated=0.71, neutral=0.22],[arousal value=0.34],[laughter value=0.02],[speech_rate value=3.1],[sarcasm_voice value=0.63]) Oh, great.

What changed since rc1

• Licence-clean targets. Every training target in rc2 is a judge's verdict or vote share, a cue from LAION's CC BY 4.0 model, or a measurement of the audio. rc1 trained its tone head, where the two judges disagreed, on the mean of two third-party audio emotion models; rc2 uses their vote share instead and has no such model in its targets, so its lineage is Whisper (MIT), LAION's affect model (CC BY 4.0), the corpora under Training and the judges. One thread remains: the pseudo-labelled windows below carry the rc1 base model's readings, so rc1's lineage reaches rc2 second-hand through those labels.
• More judged conversation. A second round of about 24,000 conversational turns rated by both judges went into training, and its held-out turns are a new test set (below).
• Pseudo-labelled speech. 159,000 windows read by the rc1 base model (Emolia clips no judge has rated, and conversation turns) joined the training set. The sarcasm and conversation-tone gains come from them.
• tiny now has the sad output (int8 14.1 MB, from 13.3).
• Sarcasm in conversation AUROC 0.917 → 0.921 (base) and 0.904 → 0.908 (tiny); conversation tone macro-F1 0.706 → 0.754 and 0.712 → 0.739; acted sadness read 23 → 30 of 34 CREMA-D clips (base). Tone on EmoNet-Voice's synthetic clips slips for base (0.563 → 0.510).
• The tone head leans neutral a little more on borderline turns than rc1 did: see tone_bias under Outputs.

Made to pair with the Tone Reader

The two readers split the work, and each declares its own signals so the LLM sees both side by side:

	Tone Reader (the words)	Voice Reader (the voice)
Reads	the final transcript, with the bot's previous line	the last 8 s of the user's audio
Writes	emotion (nine labels), sarcasm	tone (four labels), sarcasm_voice, arousal, laughter, speech_rate
Strong on	what the words say: emotion, and sarcasm the words give away	what the words don't: a flat or frustrated delivery, sadness behind neutral words, sarcasm said with sincere words

The voice reader never writes emotion or sarcasm. Its sarcasm_voice is a weak cue on its own, and its declared description tells the LLM to weigh it with the Tone Reader's sarcasm: the case it earns its place in is words that read sincere, said in a voice that isn't. For example, from our test clips, "He's the greatest, I'd say." said sarcastically reads as happy and 2 % sarcastic from the words, and 10 % (above the 6.1 % threshold) from the voice.

It does run without the Tone Reader: the LLM then gets tone instead of emotion, and no sarcasm from the words to weigh sarcasm_voice against, so we recommend running both.

Variants

This release carries two sizes of the same model, one folder each, read by the same runtime:

Folder	variant	Params	int8 ONNX	Read time, p50 (M1 Max, 4 / 2 / 1 threads)	1 thread, x86 server	Choose it for
voice-reader-base/	"base" (default)	26M	28.6 MB	28 / 49 / 92 ms	~90–160 ms	the best tone and sadness reading
voice-reader-tiny/	"tiny"	12.5M	14.1 MB	13 / 23 / 42 ms	~45–70 ms	one-vCPU deployments: sarcasm about as well, half the CPU

The read starts when the VAD says the user stopped and runs while the speech-to-text finalises, so it rarely delays the transcription; the cost to watch is CPU at the end of each turn, beside the VAD, the turn detector and the reply. Give base two threads where the machine allows.

Each folder has model.int8.onnx (the default), model.onnx (fp32), config.json (calibration, threshold, output names) and MANIFEST.json (SHA-256 of every file, checked by the runtime on download). The config.json at the root is the release index: the variants, their folders and files. The runtime reads it first on every load.

Use

With the Tone Reader (pipecat-ai[tone]) before it, as intended:

from pipecat.processors.tone.reader import PipecatToneReader
from pipecat_voice.reader import VoiceReaderParams, VoiceReaderProcessor

pipeline = Pipeline([
    transport.input(),
    stt,
    PipecatToneReader(),  # the words: emotion, sarcasm
    VoiceReaderProcessor(params=VoiceReaderParams(revision="rc2", variant="base", threads=2)),  # the voice
    user_aggregator,
    llm,
    tts,
    transport.output(),
])

It goes after the STT (it needs the transcriptions, and the STT passes the audio on) and after the Tone Reader, whose signal declarations it merges with its own, so an LLM that keeps only the latest declaration still sees both. The runtime is numpy and onnxruntime; the model downloads from this repository at the pinned revision the first time.

Outputs

Signal	Published to the LLM	Meaning
tone	yes, top two	how the voice sounds: positive, neutral, sad, frustrated (temperature-calibrated)
sarcasm_voice	yes	the chance the delivery alone sounds sarcastic, Platt-calibrated on real conversation; below the release threshold (0.061 for base, 0.063 for tiny) it is published under 0.5, as a guess; dropped under laughter
arousal	yes	vocal energy, 0–1
laughter	yes	0–1
speech_rate	yes	syllables per second
upset	no, raw reading only	how clearly the speaker is upset, 0–1
sad	no, raw reading only	how likely listeners would hear sadness, 0–1

Nothing is written for a turn with under 0.8 s of speech. The raw reading, including upset and sad, reaches the application through the processor's on_reading event.

tone_bias. tone is calibrated on real conversation, where most turns are neutral, so on a borderline turn it leans neutral: a sad or frustrated delivery a listener would notice can come out as neutral 0.45, sad 0.40. VoiceReaderParams(tone_bias={"sad": 0.5, "frustrated": 0.5}) adds to those tones' logits before the softmax (+0.5 multiplies a tone's odds by about 1.65), so they are published more readily; our test harness runs with exactly that. The graded sad and upset readings are not affected by it. Leave it unset to read the calibrated probabilities.

Evaluation

Labels on real speech come from two audio LLM judges (Gemini 3.1 Pro preview and OpenAI gpt-audio-1.5) hearing each clip; a turn counts as sarcastic only when both say so (judge agreement on sarcasm: kappa 0.42). Test speakers never appear in training.

	base	tiny
Sarcasm, everyday conversation (3,945 turns both judges rated, 203 sarcastic; Open Yap, otoSpeech, HarperValley), AUROC	0.921	0.908
— at the release threshold: F1 / recall / false alarms on sincere turns	0.571 / 0.47 / 1.0 %	0.548 / 0.45 / 1.1 %
— second conversational test, held out from the rc2 judging round (5,037 turns, 272 sarcastic), AUROC	0.862	0.853
— mean AUROC over four test sets (two conversational, Emolia, MUStARD++)	0.798	0.797
— MUStARD++ (TV, audio only, evaluation only), AUROC	0.669	0.662
Tone macro-F1: conversation / Emolia / CREMA-D (acted) / EmoNet-Voice (synthetic)	0.754 / 0.667 / 0.902 / 0.510	0.739 / 0.647 / 0.873 / 0.477
Sadness: tone recall for sad, mean of four sets	0.54	0.45
— sad output, AUROC: conversation (any judge heard sadness) / Emolia	0.755 / 0.928	0.740 / 0.923
Upset, mean AUROC (three sets)	0.869	0.857
Arousal, CCC against the training target	0.89	0.88
Laughter, AUROC / F1	0.993 / 0.64	0.991 / 0.64
int8 vs fp32: tone agreement / sarcasm p95 |Δp|	0.983 / 0.008	0.983 / 0.009

Limitations

• Sarcasm from the voice alone is a weak cue. At the threshold it catches about half of the sarcasm both judges heard, at about 1 false alarm in 100 sincere turns. Weigh it with the words, never alone. On synthetic (TTS) sarcasm it is near chance (AUROC 0.57 on our 20 demo pairs).
• Sadness in conversation is rare (2–6 % of turns), so even a good detector's per-turn flags are mostly wrong (12–24 % precision at a 5 % false-alarm rate). Treat sad as a graded cue or average it over a conversation. It is not a measure of depression or of anyone's mental health, and a single turn cannot show either.
• Borderline tones lean neutral. Calibrated on conversation, the tone head publishes neutral on turns a listener might call sad or frustrated; tone_bias above is the knob. Acted sadness is read well (30 of 34 CREMA-D sad clips by base, 27 by tiny; the misses read as frustrated).
• Laughter ranks well but its published value is conservative: F1 0.64 at the calibrated 0.5 (rc1: 0.72), at the same AUROC.
• English only; trained on conversational and podcast speech, 8 kHz telephone audio included.
• Tone labels are LLM judgments of the audio, not human ratings.

Training

• Encoder: Whisper encoder initialised from LAION's affect-tuned laion/whisper-tiny-base-emotion-voice-burst (CC BY 4.0), base and tiny encoders respectively, cut to an 8 s window, with masked statistics pooling and one small head per output.
• Data (about 290k training windows): about 131k judged windows from Open Yap 1K and otoSpeech conversations (two judging rounds), HarperValleyBank calls (CC BY 4.0), the English YODAS part of Emilia from laion/Emolia (CC BY 4.0) and CREMA-D (ODbL / DbCL); plus 159k windows pseudo-labelled by the rc1 base model, 112,996 Emolia clips no judge has rated and 46,021 conversation turns. Augmented with gain, noise, telephone band-limiting and codecs; never speed or pitch, since rate and pitch are the signal.
• Labels: the two judges' verdicts on sarcasm, tone and upset; where they disagreed on tone, their vote share over the four tones; the sad head on the share of judges who heard sadness. No audio emotion model's output is a direct target in rc2. The pseudo-labelled windows carry rc1's readings, and rc1 did use two such models (emotion2vec+ large and the LAION model) for its disagreed turns.
• Tiny: distilled from the rc2 base model's predictions at half weight, on the same data.
• Calibration on real conversational dev turns: temperature for tone, Platt for sarcasm and laughter; the sarcasm threshold keeps false alarms at 2 % of sincere dev turns.

Evaluation-only data, never trained on: MUStARD++ (CC BY-NC) and EmoNet-Voice Bench's clearly rated clips.