prefetching strategies

This work addresses the challenge in disk-based graph indexing for approximate nearest neighbor search, where achieving both low query latency and high update throughput is difficult due to I/O-induced CPU idle cycles. The authors propose LIOS, a novel framework that leverages CPU cycles idle during I/O stalls of search threads to execute recoverable subtasks of index updates. By controlling the probability of subtask deadline violations and dynamically allocating idle time, LIOS enables cooperative scheduling between queries and updates. This approach overcomes the traditional limitation of thread schedulers being oblivious to I/O idle periods, thereby meeting strict user-specified latency targets while improving insertion and deletion throughput by up to 2.68× and 2.18×, respectively.

This work addresses the I/O bottleneck in disk-resident graph-based approximate nearest neighbor (ANN) search under memory-constrained settings, where existing approaches lack a systematic understanding of storage, layout, and execution strategies. The authors propose a unified taxonomy that decomposes such systems into five core components: storage strategy, disk layout, cache management, query execution, and update mechanism. Through fine-grained evaluation and end-to-end experiments, they uncover several non-intuitive insights: vector dimensionality significantly impacts component effectiveness; current disk layouts exhibit I/O utilization consistently below 15%; smaller page sizes outperform larger ones under optimized layouts; and update strategies must be tailored to specific workloads. These findings provide empirical foundations and practical design guidelines for building efficient disk-based ANN systems.

This work addresses the challenges of high-throughput, memory-constrained inference on general-purpose NPUs, where scheduling order, memory layout, and compute-DMA pipelining are tightly coupled, often causing excessive on-chip memory occupancy, redundant off-chip accesses, and suboptimal pipeline utilization under conventional phased compilation. To overcome these limitations, we propose DAN-Scheduler, the first deterministic offline co-optimization framework that jointly optimizes execution scheduling, memory placement, and pipeline overlapping through three integrated stages: Memory-Pressure-Aware Topological Sorting (MPAS), Deterministic Linear Repacking (DLR), and Critical Path Enhancement (CPE). Evaluated across six real-world Davinci NPU workloads, DAN-Scheduler reduces peak memory usage by 18.3%, DDR traffic by 20.4%, spill counts by 14.2%, and makespan by 16.3% on average compared to the best baseline, substantially improving resource efficiency.

This work addresses the low CPU utilization and read amplification in disk-based vector approximate nearest neighbor (ANN) search caused by poor access locality in SSD-resident graph indexes. To overcome these limitations, the authors propose a locality-aware co-optimization approach that integrates hierarchical compression, affinity-aware data layout, record-level buffer pooling, and a coroutine-driven asynchronous prefetching mechanism to significantly reduce storage stalls and memory swapping. Coupled with a beam-aware search strategy, the design enhances both I/O efficiency and computational throughput. Experimental results demonstrate that the proposed system achieves up to 5.8× higher throughput and 3.25× lower latency compared to state-of-the-art disk-based ANN systems, while attaining 92% of the throughput of in-memory systems using only 10% of the memory footprint.

In Function-as-a-Service (FaaS) environments, microVM-based serverless runtimes suffer from high latency and substantial bandwidth overhead during virtual memory hot-plug operations, compounded by the Linux kernel’s lack of explicit awareness and management of hot-plugged memory. Method: This paper proposes HotMem—a novel mechanism enabling explicit isolation and fine-grained lifecycle management of hot-plugged memory regions. It customizes the Linux v6.6 kernel to integrate memory region isolation, lightweight lifecycle tracking, and optimized, targeted memory reclamation paths. Contribution/Results: HotMem overcomes fundamental bottlenecks in conventional memory management: it accelerates memory reclamation by 10× over the state-of-the-art; achieves P99 function latency comparable to static over-provisioning models; and significantly improves microVM memory elasticity and overall resource utilization in FaaS deployments.

Existing systems struggle to efficiently evaluate scheduling, KV cache management, and routing strategies in multi-turn LLM agent services. To address this gap, this work proposes the first hardware-aware simulator capable of program-level context modeling, accurately capturing multi-turn interactions, tool invocation gaps, and cross-turn KV cache locality—thereby enabling high-fidelity simulation of stateful LLM serving. The system features a composable modular architecture integrating a program orchestrator, a tool simulator, a session-aware router, and a KV residency model, supporting multi-tier memory simulation across HBM, DRAM, and CXL. Experimental results demonstrate that, under diverse real-world deployment scenarios and hardware configurations, the simulator achieves prediction errors below 6% for key performance metrics while running on commodity CPUs, enabling cost-effective and accurate exploration of system design strategies.

This study presents the first systematic evaluation of the deployment feasibility of spiking neural networks (SNNs) in containerized edge environments. Focusing on resource- and energy-constrained virtual edge scenarios, we construct a testbed leveraging Docker Desktop, WSL2, and Windows 11 atop a single-node Kubernetes cluster orchestrated via K3d. We investigate end-to-end latency, throughput, classification accuracy, and concurrent behavior of SNN workloads under resource constraints and autoscaling conditions. Our findings reveal that SNNs are highly sensitive to CPU and memory availability: resource limitations substantially increase latency and reduce throughput, while classification accuracy remains stable. Furthermore, the default round-robin load-balancing strategy proves mismatched with SNNs’ long-duration inference tasks, leading to elevated tail latency. This work highlights the limitations of current stateless orchestration mechanisms in supporting neuromorphic computing paradigms.

This work addresses the high memory overhead and scalability limitations of large-scale approximate nearest neighbor search (ANNS) systems that rely on in-memory graph structures such as HNSW. The authors propose Helmsman, a clustering-based ANNS system that integrates a user-space I/O stack, a learning-driven hierarchical pruning mechanism, and a GPU-accelerated indexing pipeline to enable efficient billion-scale vector retrieval on all-flash storage architectures. Helmsman substantially overcomes the latency, construction speed, and resource efficiency bottlenecks of conventional clustering approaches: it reduces hardware costs by over 90%, enables billion-scale index reconstruction within hours, and in production deployment replaces a cluster requiring 35,000 CPU cores and 0.35 PB of memory with only 40 servers, maintaining stable operation for several months.

This work addresses the trade-off between high memory consumption of in-memory methods and high latency of disk-based approaches for approximate nearest neighbor (ANN) search in high-dimensional spaces by proposing SkipDisk, a hybrid memory-disk collaborative framework. Its key innovations include the introduction of point-specific pivots to derive tighter triangle inequality lower bounds, a three-tier in-memory pruning strategy to retain critical index information, and an asynchronous I/O mechanism that decouples disk access from in-memory search. Experimental results demonstrate that SkipDisk achieves 63% to 85% of HNSW’s search latency while using only 10%–20% of its memory footprint, thereby significantly reducing memory overhead without substantially compromising search efficiency.

This work addresses the challenge of concurrently serving multiple deep neural network (DNN) models on shared GPUs at the edge while meeting tail-latency and service-level objective (SLO) requirements. The authors propose a协同 scheduling framework that integrates time-sliced GPU sharing with early-exit inference, jointly optimizing model selection, exit points, and batch sizes at runtime to minimize system-wide SLO violations. A key innovation is the introduction of a stability score that quantifies the impact of scheduling decisions on future queue states, thereby expanding the feasible action space under tight constraints and enhancing latency predictability. Experimental results across diverse hardware platforms demonstrate significant improvements over existing baselines, with notable reductions in both SLO violation rates and P95 latency.