Should We Type or Talk to LLM Agents? A Comprehensive Study of Voice and Keyboard Input Perturbations

2026-08-04

AI Summary

This study systematically investigates the mechanisms and differences in how speech transcription errors and QWERTY keyboard input perturbations affect the robustness of instruction-tuned large language models. To this end, we introduce the Human Input Variation Engine (HIVE), enabling the first direct comparison between these two perturbation types. Our experiments reveal that performance degradation primarily stems from corruption of critical question tokens rather than the introduction of extraneous noise. Speech perturbations induce significantly greater accuracy drops compared to keyboard perturbations, which models tolerate more robustly. Notably, this channel disparity vanishes in multiple-choice tasks. Furthermore, lightweight fine-tuning proves ineffective at mitigating these issues, and increased reasoning budgets only partially recover performance under keyboard input errors. These findings highlight the asymmetric impact of input modality characteristics on model comprehension and reasoning.

Abstract

Human input reaches language models by typing or speaking, and each channel leaves a distinct signature: orthographic noise for keyboards; for voice, disfluency from conventional transcription and restructuring from AI-backed dictation tools. How do they impact an LLM's performance? In this paper we present HIVE (Human Input-Variation Engine), a suite of voice transcription perturbations and QWERTY keyboard perturbations. We use HIVE to evaluate how robust models are to these perturbations. We present seven findings. (i) Voice transcription perturbations lower accuracy across every instruction-tuned model we test, and it is the structure of the transcription rather than its fillers that carries the cost. (ii) QWERTY keyboard perturbations cost less, and a model absorbs a lot of them before accuracy falls away. (iii) Both trace back to one cause, how many of the question's tokens survive the perturbation: destroying a token is what hurts, while adding new ones alongside it costs little. (iv) The gap between the two channels appears only where the answer must be constructed or deduced; on multiple choice there is none. (v) The harm does not solely come from test-set contamination. (vi) It cannot be trained away with lightweight adaptation. (vii) A thinking budget recovers the keyboard channel almost entirely but leaves the spoken registers untouched, and compressed speech is worse with it.

Problem

• voice input
• keyboard input
• input perturbations
• LLM robustness
• transcription errors

Innovation

• HIVE
• input perturbations
• voice transcription
• keyboard noise
• LLM robustness