Democratizing Scalable Cloud Applications: Transactional Stateful Functions on Streaming Dataflows

2025-12-19

AI Summary

Cloud application development has long suffered from high expertise barriers due to the need to integrate distributed systems, database, and software engineering knowledge. This paper proposes Stateflow—the first cloud-native, streaming, stateful function-computing framework—designed to jointly address programmability, strongly consistent fault-tolerant transactions, and serverless semantics. Its three key contributions are: (1) an object-oriented declarative programming model that eliminates explicit error handling; (2) the Styx engine, which guarantees deterministic multi-partition serializability, snapshot consistency, and zero-loss state migration; and (3) a streaming dataflow execution model with transaction-aware state migration, enabling dynamic elastic scaling. Experimental evaluation demonstrates that Stateflow significantly outperforms existing systems in both throughput and recovery performance, substantially reducing development complexity for high-concurrency, strongly consistent cloud applications.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Data Stream Mining

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Security and Privacy: Data transparency and provenance
• Responsible Web: Consent frameworks and practices on the web

Abstract

Web applications underpin much of modern digital life, yet building scalable and consistent cloud applications remains difficult, requiring expertise across cloud computing, distributed systems, databases, and software engineering. These demands restrict development to a small number of highly specialized engineers. This thesis aims to democratize cloud application development by addressing three challenges: programmability, high-performance fault-tolerant serializable transactions, and serverless semantics. The thesis identifies strong parallels between cloud applications and the streaming dataflow execution model. It first explores this connection through T-Statefun, a transactional extension of Apache Flink Statefun, demonstrating that dataflow systems can support transactional cloud applications via a stateful functions-as-a-service API. However, this approach revealed significant limitations in programmability and performance. To overcome these issues, the thesis introduces Stateflow, a high-level object-oriented programming model that compiles applications into stateful dataflow graphs with minimal boilerplate. Building on this model, the thesis presents Styx, a distributed streaming dataflow engine that provides deterministic, multi-partition, serializable transactions with strong fault tolerance guarantees. Styx eliminates explicit transaction failure handling and significantly outperforms state-of-the-art systems. Finally, the thesis extends Styx with transactional state migration to support elasticity under dynamic workloads.

Problem

• Democratizing scalable cloud application development
• Providing high-performance fault-tolerant serializable transactions
• Enabling serverless semantics for streaming dataflows

Innovation

• Transactional extension of Apache Flink Statefun for cloud applications
• Object-oriented programming model compiling to stateful dataflow graphs
• Distributed engine with deterministic serializable transactions and fault tolerance