Pixel-TTS: Image based Text Rendering for Robust Text-to-Speech

2026-06-05

AI Summary

This work proposes a vision-aware text-to-speech (TTS) approach that addresses the limitations of conventional character-level embedding-based systems, which struggle to generalize to unseen or cross-lingual characters and require expanding the embedding matrix—thereby hindering zero-shot transfer. Instead of relying on discrete character embeddings, the method renders input text as images and employs 2D convolutional neural networks to extract pixel-level visual features as character representations. This enables unified handling of structurally similar characters despite differing encodings. The proposed framework eliminates the need to extend the embedding matrix, achieves synthesis quality on par with strong baselines across multiple benchmarks, converges more rapidly during training, and demonstrates significantly enhanced robustness and generalization in zero-shot cross-lingual scenarios.

Abstract

Recent advances in pixel-based text modeling show that representing text as images enables models to exploit visual cues for language understanding. Grounding text in its visual form allows structurally similar characters with different Unicode encodings to produce similar embeddings, benefiting cross-lingual and zero-shot scenarios. Conventional text-based approaches treat each character independently, limiting generalization to unseen characters and requiring embedding expansion during cross-lingual adaptation. We propose Pixel-TTS, the first framework for visually grounded speech synthesis. It renders text as images and projects them through a 2D convolutional layer to generate embeddings. This design eliminates embedding matrix expansion during fine-tuning while improving robustness to unseen characters and orthographic variations. Extensive experiments show Pixel-TTS achieves competitive performance with strong baselines, faster convergence and robust zero-shot generalization.

Problem

• text-to-speech
• unseen characters
• cross-lingual adaptation
• embedding expansion
• orthographic variations

Innovation

• Pixel-TTS
• image-based text rendering
• visually grounded speech synthesis
• zero-shot generalization
• 2D convolutional embedding