Opt.Gear Technical Report

2026-08-02

AI Summary

This work addresses the challenges of deploying large language models on edge devices—namely high memory consumption, slow inference speed, and limited context length—by introducing a novel hybrid architecture that integrates convolutional key-value gated mixers with a local-global attention mechanism. This design substantially reduces KV cache overhead and supports context lengths up to 64K tokens. Building upon this architecture, we present Opt.Gear-1M, the first generative model capable of running on microcontroller units (MCUs), trained efficiently on 0.5 trillion tokens and deployed on NPUs/ANEs via W4A32 quantization. Experiments demonstrate that the model achieves 20 tokens per second on an STM32H747I-DISCO board and yields up to a 4.9× speedup in prefill and decode stages on NPU hardware. All models and deployment binaries are publicly released.

Abstract

We introduce Opt.Gear, a foundation model designed for efficient on-device deployment, real-tim inference, and strong task capability. It includes a dense model (1M, 270M, and 1B) with a context length of 64K. We designed a new hybrid architecture that combines a convolutional key-value gated mixer with local-global attention to reduce the KV-cache memory that tends to increase exponentially with long context. This architecture delivers up to X4.9 faster prefill and decoding speeds on the NPUs compared to models of a similar scale models. From a 2T tokens candidate corpus, Opt.Gear is trained on a curated 0.5T tokens subset without knowledge distillation. This is the most data-efficient of the existing foundation models. All models are released with open weights and deployment binaries for ONNX, Qualcomm NPU, and Apple ANE making Opt.Gear a practical base for edge applications that need fast, memory-efficient inference and strong task capabilities. Furthermore, to expand the ecosystem of on-device generative language models, we are introducing the Opt.Gear-1M that can be deployed on Micro-Controller Units (MCUs), a Tiny Language Model (TLM). Opt.Gear-1M is the first generative language model to achieve 20 TPS with W4A32 quantization on the ARM Cortex-M7 CPU of the STM32H747I-DISCO.

Problem

• on-device deployment
• real-time inference
• memory-efficient
• edge AI
• generative language model

Innovation

• hybrid architecture
• KV-cache reduction
• on-device deployment
• data-efficient training
• Tiny Language Model