TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructure

2026-08-06

AI Summary

This work addresses the tight coupling between tensor management—encompassing weight loading, KV caching, and checkpoint synchronization—and execution engines in existing large language model systems, which hinders cross-component reuse. To overcome this limitation, the authors propose Tensor-as-a-Service (TaaS), a novel abstraction realized through TensorCast, a distributed tensor management layer that decouples tensor lifecycle management into an independent infrastructure layer for the first time. TensorCast provides programmable tensor primitives and runtime support, enabling flexible composition and reuse of tensor management policies across diverse tasks. Integrated with vLLM and SGLang, TensorCast achieves performance on par with specialized systems across multiple tensor management scenarios and reduces median time-to-first-token latency by up to 93.2% under high-concurrency, multi-turn agent workloads.

Abstract

Modern LLM infrastructure increasingly manages tensors not only as computation data, but also as persistent states shared across distributed components. Existing systems optimize individual tensor management tasks, such as model weight loading, KV cache management, and checkpoint synchronization, by deeply integrating task-specific mechanisms with execution engines, networks, or storage backends. However, this specialization creates isolated silos that hinder the reuse and composition of tensor management strategies across evolving LLM workloads. In this paper, we identify tensor lifecycle management as a missing abstraction layer in LLM infrastructure and propose Tensor-as-a-Service (TaaS), which decouples tensor state management from computation logic. We design and build TensorCast, a distributed tensor management layer that provides first-class tensor abstractions, programmable lifecycle primitives, and a runtime that separates tensor management policies from execution mechanisms. This enables developers to write tensor management programs using TensorCast APIs while transparently leveraging distributed execution and data movement. We integrate TensorCast with vLLM and SGLang and evaluate it across diverse tensor lifecycle workloads, including model weight materialization, weight synchronization, KV cache management, and programmable request routing. Our results show that TensorCast achieves competitive performance with specialized tensor management systems while enabling new cross-component optimization policies. A programmable policy implemented with TensorCast improves median TTFT by up to 93.2% under highly concurrent multi-turn agent workloads.

Problem

• tensor management
• large language models
• lifecycle management
• distributed systems
• abstraction layer

Innovation

• Tensor-as-a-Service
• tensor lifecycle management
• distributed tensor abstraction
• decoupled execution
• programmable tensor policies