Preventing Another Tessa: Modular Safety Middleware For Health-Adjacent AI Assistants

2025-09-07

AI Summary

Health AI assistants (e.g., Tessa by NEDA) pose safety risks when generating harmful recommendations. Method: This paper proposes a lightweight, modular safety middleware that integrates deterministic lexical gating with inline LLM policy filtering—enabling fail-safe detection and safety escalation within a single model invocation, thereby eliminating latency and auditability limitations of traditional multi-stage pipelines. The middleware is explicitly aligned with the OWASP LLM Top 10 and NIST SP 800-53 security governance frameworks, ensuring verifiable and auditable safety controls. Contribution/Results: In synthetic evaluations, the middleware achieves 100% interception of unsafe requests while significantly reducing inference cost and latency. Results demonstrate that explicit, last-mile detection suffices to efficiently mitigate critical safety risks in health-domain AI systems.

Technology Category

• Natural Language Processing: Safety and Robustness
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Security and Privacy: Security and privacy of machine learning and AI applications
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

In 2023, the National Eating Disorders Association's (NEDA) chatbot Tessa was suspended after providing harmful weight-loss advice to vulnerable users-an avoidable failure that underscores the risks of unsafe AI in healthcare contexts. This paper examines Tessa as a case study in absent safety engineering and demonstrates how a lightweight, modular safeguard could have prevented the incident. We propose a hybrid safety middleware that combines deterministic lexical gates with an in-line large language model (LLM) policy filter, enforcing fail-closed verdicts and escalation pathways within a single model call. Using synthetic evaluations, we show that this design achieves perfect interception of unsafe prompts at baseline cost and latency, outperforming traditional multi-stage pipelines. Beyond technical remedies, we map Tessa's failure patterns to established frameworks (OWASP LLM Top10, NIST SP 800-53), connecting practical safeguards to actionable governance controls. The results highlight that robust, auditable safety in health-adjacent AI does not require heavyweight infrastructure: explicit, testable checks at the last mile are sufficient to prevent "another Tessa", while governance and escalation ensure sustainability in real-world deployment.

Problem

• Preventing harmful AI advice in healthcare contexts
• Addressing absent safety engineering in health AI systems
• Intercepting unsafe prompts at baseline cost and latency

Innovation

• Modular safety middleware for AI assistants
• Hybrid design with lexical gates and LLM filter
• Enforces fail-closed verdicts and escalation pathways