mobile deployment profiling

Existing large language model (LLM) evaluation benchmarks lack coverage of mobile application development scenarios, failing to capture platform-specific constraints, framework lifecycles, and complex API interactions inherent to such environments. To address this gap, this work introduces the first multilingual repair benchmark targeting three major mobile platforms—Android Native, React Native, and Flutter—comprising 384 real-world production issues. Each task is accompanied by an executable test patch enabling automated validation of cross-file and cross-artifact modifications. Experimental results reveal that state-of-the-art code LLMs achieve end-to-end repair success rates of only 3.39%–5.21% on this benchmark, substantially lower than their performance on existing datasets, thereby exposing critical limitations in multi-file fault localization and coordinated repair capabilities.

To address computational constraints, poor energy efficiency, and insufficient real-time performance in deploying vision-language models (VLMs) on mobile devices, this work conducts a systematic end-to-end performance evaluation of three leading inference frameworks—llama.cpp, MLC-LLM, and mllm—on state-of-the-art VLMs (LLaVA, MobileVLM, and Imp) using the OnePlus 13R as the hardware platform. We develop a full-stack benchmarking toolkit measuring CPU/GPU/NPU utilization, power consumption, thermal behavior, and end-to-end latency. Our analysis uncovers a previously unreported cross-stage hardware-resource mismatch: GPU saturation during image encoding, severe CPU bottlenecks during text generation, and low, highly volatile NPU utilization. The core contribution is a hardware-aware, VLM-specific analytical methodology for mobile deployment, complemented by an open-source lightweight monitoring tool. This work provides empirically grounded insights and concrete optimization directions for efficient on-device VLM inference.

To address the need for lightweight and efficient inference of large language models (LLMs) on mobile devices, this paper introduces the first automated benchmarking framework that jointly evaluates resource efficiency and safety risks. Methodologically, it integrates weight and activation quantization configurations, supports cross-platform performance profiling across ARM CPU/GPU/NPU backends, and incorporates real-time power consumption monitoring alongside automated hallucination and toxicity detection. Contributions include: (1) a mobile-specific, multi-dimensional evaluation paradigm that unifies assessment of generation quality, latency, throughput, memory footprint, power draw, and harmful output; (2) empirical insights into nonlinear trade-offs among mobile chip energy efficiency, quantization strategies, and their impacts on latency and memory; and (3) quantitative evidence of systematic accuracy and safety degradation under model compression—establishing a reproducible, scalable evaluation standard for on-device LLM deployment.

Neuromorphic computing lacks a unified evaluation standard, hindering objective technology assessment, cross-method comparison, and research directionality. To address this, we propose the first open, collaborative neuromorphic computing benchmarking framework, introducing a novel dual-track evaluation paradigm—“inclusive, iterative, and community-driven”—that concurrently supports algorithm-level (hardware-agnostic) and system-level (hardware-dependent) evaluation. The framework comprises a modular toolchain, a multi-domain task suite (covering speech, vision, and time-series prediction), standardized performance measurement protocols, and an open-source evaluation interface. We release the first version of multi-task baseline results, benchmarking state-of-the-art neuromorphic models alongside conventional AI approaches across accuracy, energy efficiency, and latency. All evaluations are reproducible, cross-platform, and cross-architecture, establishing a foundational infrastructure for standardized neuromorphic computing evaluation.

Existing mobile device-controlling agents lack standardized benchmarks, hindering rigorous performance evaluation and cross-method comparison. This paper introduces B-MoCA, the first benchmark for mobile agents operating on real Android systems, comprising 131 everyday tasks. It employs multi-dimensional randomization—including UI layouts and natural-language instructions—to systematically assess generalization capability. Its key contributions are: (1) an interactive benchmarking framework supporting cross-device configuration generalization; (2) the first systematic robustness evaluation of LLMs, multimodal LLMs, and imitation-learning agents within a realistic system-level simulation environment; and (3) full open-sourcing of code, execution environments, and human expert demonstration data. Experiments reveal that state-of-the-art methods achieve only ~78% success on simple tasks but drop to under 22% on complex ones, exposing critical bottlenecks in reasoning, grounding, and long-horizon planning. This work establishes a reproducible, scalable evaluation paradigm for mobile agents.

This study addresses a critical gap in current AI evaluation methodologies, which often overlook the impact of low-resource deployment conditions—such as noisy inputs, limited hardware capabilities, and unstable network connectivity—on system usability. The work proposes a novel evaluation framework that treats the deployed system as the unit of assessment, integrating task performance with real-world deployment contexts across multiple dimensions. Departing from conventional leaderboard-based approaches, the framework tailors evaluation criteria to specific application categories and introduces a standardized reporting system comprising benchmark cards, deployment profiles, and failure-handling mechanisms. By balancing comparability with contextual sensitivity, this approach provides policymakers and practitioners with clear, actionable insights for informed AI deployment decisions.

This work addresses the challenge of evaluating mobile GUI agents in real-world, closed-source applications, where existing benchmarks relying on simulated or open-source environments fall short in enabling automatic and verifiable assessment. To bridge this gap, the authors introduce AndroidDaily, a large-scale benchmark comprising 94 high-frequency Android applications and 350 everyday tasks, along with GRADE, a novel evaluation framework. GRADE enables process-aware, automatic diagnosis of long-horizon interaction trajectories in closed-source apps without requiring access to internal states, by leveraging three external, observable criteria: action fidelity, output quality, and negative constraints. Experiments demonstrate that GRADE achieves 87.37% agreement with human evaluators, while the strongest existing agent attains only a 62.0% task success rate on AndroidDaily, revealing substantial limitations of current approaches in realistic settings.

This work addresses the challenge that state-dependent behaviors in modern computing environments—such as those introduced by adaptive system mechanisms—induce time-dependent biases in traditional software benchmarking, undermining reliable performance comparisons. The paper reframes benchmarking as a decision problem aimed at identifying the fastest program and introduces an experimental paradigm centered on pairwise performance comparisons, thereby avoiding strong assumptions about modeling system dynamics. By leveraging contrastive estimators, consistent statistical inference, and test strategies under finite evaluation budgets, the approach eliminates program-specific biases without relying on absolute performance metrics. The method provides asymptotic guarantees for correct decisions in stateful environments, offering a robust and reliable benchmarking framework for performance-sensitive software development.

This study addresses the multi-objective trade-offs among generation quality, energy consumption, latency, and memory when deploying large language models on edge devices. The authors construct a reproducible empirical evaluation framework to systematically analyze the energy efficiency, performance, and privacy characteristics of models ranging from 0.5B to 9B parameters on a real-world Android device (Samsung Galaxy S25 Ultra). Leveraging non-intrusive, fine-grained power monitoring and mixed-precision inference, they uncover a “quantization-energy paradox”: model architecture—not quantization strategy—dominates energy consumption. Notably, Mixture-of-Experts architectures disrupt conventional scaling–energy relationships, and medium-scale models such as Qwen2.5-3B emerge as the optimal choice, balancing high output quality with energy efficiency, thereby offering practical deployment guidelines for on-device intelligence.

This work addresses the challenges of deploying large language models (LLMs) on edge devices, where the high-dimensional configuration space and the absence of comprehensive, real-world evaluation methodologies hinder effective trade-offs among performance, power consumption, and physical footprint. To bridge this gap, the authors propose a multidimensional benchmarking framework tailored for edge scenarios and present the first systematic evaluation of LLM inference across four representative single-board computers equipped with state-of-the-art NPU/GPU accelerators, measuring key metrics including throughput, energy efficiency, and device size. By integrating model distillation and quantization techniques, the study enables efficient on-device inference suitable for privacy-sensitive and low-connectivity environments, while quantitatively demonstrating the substantial advantages of dedicated hardware accelerators in edge-based generative AI deployments.