Can LLMs Test Terminal User Interfaces?

2026-08-04

AI Summary

This work addresses the lack of effective automated testing methods for terminal user interfaces (TUIs), which suffer from low coverage and difficulty in uncovering interaction-related bugs. The authors present the first cross-language headless TUI testing benchmark, encompassing ecosystems in Rust (ratatui), Go (bubbletea), Python (Textual), and TypeScript (ink). By leveraging Docker containerization, terminal state recording, and line- and component-level coverage tracking (tuicov), they systematically evaluate random exploration against large language model (LLM)-driven testing (tuibot). Results show that random exploration excels at crash discovery due to high throughput, whereas LLM-based testing achieves higher per-interaction efficiency and can reach input-gated faults. Automatically inferred launch parameters substantially broaden the scope of testable applications. The study further reveals a weak correlation between code coverage and crash detection and releases the complete toolchain as open source.

Abstract

Terminal User Interfaces (TUIs) combine the stateful, screen-oriented behaviour of GUIs with terminal deployment and are now common in developer tools. Yet they lack a dedicated testing methodology. We survey 197 real-world TUI applications: only 12% of test code exercises the interface, and 45% of those tests never send input, checking a static frame instead. We turn these applications into a headless benchmark spanning ratatui/Rust, bubbletea/Go, textual/Python, and ink/TypeScript, packaging each as an instrumented Docker image. We record line and widget coverage where reliable, rendered terminal states, and crashes. Under equal wall-clock budgets, we compare four frontier LLMs with random exploration. No model dominates. Random is a strong time-budgeted baseline, but its crash advantage comes from higher throughput: per interaction, LLM guidance is more efficient and uniquely reaches input-gated faults. Automatically deriving launch inputs yields the largest practical gain, enabling applications that otherwise never start. Line coverage poorly predicts crash discovery, weakening it as a proxy for test effectiveness. Automated TUI testing is feasible but far from solved, and honest baselines matter more than model choice. We release the coverage tool tuicov at https://github.com/tui-testing/tuicov and the testing framework tuibot at https://github.com/tui-testing/tuibot.

Problem

• Terminal User Interfaces
• TUI testing
• test coverage
• input-gated faults
• automated testing

Innovation

• Terminal User Interface (TUI)
• LLM-based testing
• headless benchmark
• coverage measurement
• automated test input generation