Investigating the Impact of Isolation on Synchronized Benchmarks

2025-11-05

AI Summary

Cloud benchmarking suffers from performance variability due to multi-tenant resource contention. While the existing Duet approach mitigates cross-VM interference via co-locating two benchmark versions on the same VM, it overlooks implicit intra-VM workload competition. This work systematically evaluates the noise robustness of multiple isolation mechanisms—cgroups/CPU pinning, Docker containers, and Firecracker microVMs—under synchronous benchmarking, using controlled noise injection to emulate resource contention. Results show that Docker containers, owing to shared kernel resources and inherent cgroups scheduler limitations, significantly amplify performance variance in synchronous settings, rendering them unsuitable for high-precision benchmarking. In contrast, process-level isolation and microVMs effectively suppress interference, reduce false positives, and improve measurement consistency. The study uncovers latent performance risks of containerized environments in synchronous benchmarking and provides empirical evidence and practical guidance for selecting isolation strategies in cloud-native benchmarking.

Technology Category

• Machine Learning: Hardware-aware ML
• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Security and Privacy: Large-scale security measurements
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Benchmarking in cloud environments suffers from performance variability from multi-tenant resource contention. Duet benchmarking mitigates this by running two workload versions concurrently on the same VM, exposing them to identical external interference. However, intra-VM contention between synchronized workloads necessitates additional isolation mechanisms. This work evaluates three such strategies: cgroups and CPU pinning, Docker containers, and Firecracker MicroVMs. We compare all strategies with an unisolated baseline experiment, by running benchmarks with a duet setup alongside a noise generator. This noise generator"steals"compute resources to degrade performance measurements. All experiments showed different latency distributions while under the effects of noise generation, but results show that process isolation generally lowered false positives, except for our experiments with Docker containers. Even though Docker containers rely internally on cgroups and CPU pinning, they were more susceptible to performance degradation due to noise influence. Therefore, we recommend to use process isolation for synchronized workloads, with the exception of Docker containers.

Problem

• Evaluating isolation mechanisms for synchronized cloud benchmarking workloads
• Mitigating performance variability from multi-tenant resource contention
• Comparing cgroups, Docker containers and MicroVMs against noise interference

Innovation

• Uses cgroups and CPU pinning for isolation
• Implements Docker containers for workload separation
• Employs Firecracker MicroVMs as isolation mechanism