Zero-Shot Voice Cloning

Dec 30, 2025 • 18 min read

We have gone from "train a custom TTS model for 10 hours with 100+ hours of recorded speech" to "clone any voice from a 3-second WhatsApp voice note." Zero-shot voice cloning doesn't need training data for the target speaker — it extracts the speaker's vocal characteristics from a short reference clip and applies them to any text in real-time. The applications range from audiobook narration and video dubbing to accessibility features — and unfortunately, voice fraud. This guide covers the technical stack, how it works, and production deployment patterns.

1. How Zero-Shot TTS Works

Traditional TTS systems (Festival, eSpeak) learned a single speaker's voice from hundreds of hours of recorded speech — a per-speaker model requiring significant data collection. Modern zero-shot TTS separates the problem into two independent components:

• Content Encoder: Encodes "what is said" — the text content and linguistic structure
• Speaker Encoder: Encodes "how it sounds" — extracts a compact speaker embedding (typically 256-512 floats) representing timbre, pitch range, accent, and speaking style from the reference audio
• Vocoder: Converts the conditioned mel spectrogram back to waveform audio

The breakthrough is that speaker embeddings generalize: the model learns a continuous space of "how voices sound" during training on thousands of speakers, so at inference it can map any new speaker into this space from just seconds of audio.

2. XTTS v2: Best Open-Source Zero-Shot TTS

pip install TTS

from TTS.api import TTS
import torch

# Load XTTS v2 (downloads ~2GB on first run, cached in ~/.local/share/tts)
device = "cuda" if torch.cuda.is_available() else "cpu"
tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2").to(device)

# Basic voice cloning: reference clip + text → speech in cloned voice
tts.tts_to_file(
    text="This sentence was generated using your voice sample.",
    speaker_wav="reference.wav",  # 3-30 seconds of clean reference audio
    language="en",                # 16 languages supported
    file_path="output.wav",
)

# Cross-language cloning: clone English speaker's voice speaking Spanish!
tts.tts_to_file(
    text="Hola, esto es una prueba de síntesis de voz entre idiomas.",
    speaker_wav="english_speaker.wav",  # English reference audio
    language="es",                       # Output in Spanish!
    file_path="spanish_output.wav",
)
# Result: The Spanish audio has the English speaker's voice characteristics —
# same timbre, similar pitch, but with Spanish phonetics

# Streaming (for real-time applications — first audio chunk in ~200ms)
def on_audio_chunk(chunk):
    play_audio(chunk)  # Play each chunk as it arrives

tts.tts_with_vc_to_file(
    text="Streaming voice synthesis for real-time applications.",
    language="en",
    speaker_wav="reference.wav",
    file_path="streaming_output.wav",
    # For true streaming, use the low-level TTS synthesizer API
)

3. Low-Latency Streaming TTS

# For real-time voice agents: stream audio while still generating text
import asyncio
import numpy as np
from TTS.tts.configs.xtts_config import XttsConfig
from TTS.tts.models.xtts import Xtts

# Load model directly for low-level streaming control
config = XttsConfig()
config.load_json("/path/to/xtts_v2/config.json")
model = Xtts.init_from_config(config)
model.load_checkpoint(config, checkpoint_dir="/path/to/xtts_v2/")
model.cuda()

# Extract speaker embedding once (reuse across requests for same speaker)
gpt_cond_latent, speaker_embedding = model.get_conditioning_latents(
    audio_path=["reference.wav"]
)

# Stream audio chunks as they generate
async def stream_tts(text: str):
    chunks = model.inference_stream(
        text,
        "en",
        gpt_cond_latent,
        speaker_embedding,
        stream_chunk_size=20,     # Generate 20 tokens, then yield audio chunk
        enable_text_splitting=True,  # Auto-split long texts
    )
    
    for i, chunk in enumerate(chunks):
        if i == 0:
            print(f"First audio chunk ready in ~{time.time() - start:.2f}s")
            # First chunk typically arrives in 150-300ms
        yield chunk.cpu().numpy()

# Pipe to WebSocket for real-time voice chat applications
# First audio chunk arrives in ~200ms on modern GPU

4. RVC: Speech-to-Speech Voice Conversion

# RVC (Retrieval-based Voice Conversion) — NOT text-to-speech
# You speak → your voice converted to sound like target speaker in real-time
# This is how "AI Cover Songs" work (Drake singing in Taylor Swift's voice)

# Install: git clone https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI

# Two modes:
# 1. Offline (file to file):
python tools/infer_batch_rvc.py \
    --input ./my_voice.wav \
    --output ./converted.wav \
    --pth_path ./models/target_speaker.pth \
    --index_path ./models/target_speaker.index \
    --f0up_key 0              # Pitch shift in semitones (0 = no shift)
    # Use -12 to +12 for male→female or female→male conversion

# 2. Real-time microphone mode:
# python tools/realtime_gui.py
# → Uses microphone input, converts voice in <50ms latency

# Training custom RVC models (requires ~5-30 min of target audio):
# python trainset_preprocess_pipeline_print.py ./target_audio 40000 2 ./logs/my_model True
# python train.py  # Fine-tune on ~30 epochs for clean audio

# RVC model quality tiers:
# < 5 min audio:   Noticeable artifacts
# 5-30 min audio:  Good quality, some accent leakage  
# 30-120 min audio: Excellent, near-indistinguishable cloning

5. Style-Bert-VITS2: Emotionally Expressive TTS

# Style-Bert-VITS2 — optimized for expressive, emotional speech
# Particularly popular for anime character voices and storytelling
# Runs efficiently on CPU (unlike XTTS which benefits from GPU)

pip install style-bert-vits2

from style_bert_vits2.tts_model import TTSModel
from style_bert_vits2.constants import Languages

model = TTSModel(
    model_path="./models/your_model/model.safetensors",
    config_path="./models/your_model/config.json",
    style_vec_path="./models/your_model/style_vectors.npy",
    device="cpu",
)

# Generate with emotion control
audio, sample_rate, _ = model.infer(
    text="I can't believe this happened!",
    language=Languages.EN,
    style="Angry",           # Pre-defined style vector
    style_weight=0.8,        # How strongly to apply the style (0=neutral, 1=full)
    sdp_ratio=0.3,           # Stochastic duration predictor ratio
    noise_scale=0.6,
    speed=1.0,
)

import soundfile as sf
sf.write("angry_output.wav", audio, sample_rate)

# Available styles (train your own or use prebuilt):
# ["Neutral", "Happy", "Sad", "Angry", "Surprised", "Whispering", "Crying"]

6. Audio Watermarking: SynthID for Generated Speech

# SynthID (from Google DeepMind) — imperceptible watermark in generated audio
# The watermark survives compression, re-encoding, and even re-recording
# Use it for all voice cloning outputs to maintain accountability

from audioseal import AudioSeal

# Encoder: embed watermark
model_encoder = AudioSeal.load_generator("audioseal_wm_16bits")
watermarked_audio, _ = model_encoder(audio_tensor, sample_rate=24000, alpha=1.0)

# Detector: verify if audio is AI-generated
model_detector = AudioSeal.load_detector("audioseal_detector_16bits")
result, message = model_detector.detect_watermark(watermarked_audio, 24000)

if result > 0.5:
    print(f"AI-generated audio detected (confidence: {result:.2%})")
else:
    print("No watermark detected — likely authentic")

Frequently Asked Questions

How long does the reference audio need to be for good cloning quality?

XTTS v2 works with as little as 3 seconds, but quality peaks around 10-30 seconds of clean speech. Key requirements for the reference clip: no background music, minimal reverb, consistent microphone distance, and natural speech pace. A single clear sentence from a phone call often works better than 5 minutes of noisy audio. The model is most sensitive to the first few seconds, which determines speaker identity.

Is there a way to prevent my voice from being cloned?

University of Chicago's "AntiFake" technique adds near-imperceptible perturbations to voice recordings that cause TTS models to fail to clone them correctly. The tool "Voice Guard" applies similar adversarial noise. However, robust defenses against all voice cloning models simultaneously don't yet exist — the arms race between cloning capabilities and defenses is ongoing. The most reliable protection is restricting public audio recordings of sensitive individuals.

Conclusion

XTTS v2 for multilingual zero-shot cloning, RVC for real-time voice conversion, and Style-Bert-VITS2 for expressive character voices cover the three main use cases in voice cloning. For production voice agents, combining XTTS streaming with a WebSocket-based audio pipeline achieves sub-300ms time-to-first-audio — perceptually real-time for conversational applications. Always implement audio watermarking (AudioSeal/SynthID) in production deployments, both for ethical compliance and as a foundation for future regulatory requirements around synthetic media disclosure.