Opto-ViT-v2: Noise-Resilient On-Chip Fine-Tuning for Photonic Near-Sensor Vision Transformer Accelerators

2026-07-19

AI Summary

This work addresses the challenges of on-chip fine-tuning for silicon-photonic Vision Transformers (ViTs), which are hindered by substantial activation storage overhead, frequent weight write-backs, and sensitivity to photonic device noise. The authors propose the first parameter-efficient fine-tuning framework tailored for near-sensor photonic ViTs, decoupling pretrained optical weights from a small set of trainable electronic factors via tensorized low-rank decomposition. Combined with gradient accumulation and a sparse classifier, this approach enables efficient training while significantly reducing computational cost. Furthermore, they introduce the first system-level photonic training noise model, incorporating crosstalk, thermal drift, and laser amplitude noise. Experimental results on VTAB-1K and FGVC benchmarks demonstrate that under realistic hardware noise, the method achieves accuracy within only 0.3–0.8% of ideal software baselines, delivers over 100 KFPS/W energy efficiency, and reduces classifier training cost by approximately 40%.

Abstract

Silicon-photonic (SiPh) accelerators have emerged as a promising platform for Vision Transformer (ViT) inference by performing matrix multiplications on microring-resonator (MRR) banks with high throughput and energy efficiency. Extending these platforms to support on-chip fine-tuning remains challenging because backpropagation requires large activation storage, frequent weight write-back to MRRs, and tolerance to device-level noise. We present Opto-ViT-v2, the first framework for parameter-efficient fine-tuning (PEFT) on a near-sensor SiPh ViT accelerator. Our tensorized low-rank decomposition separates pretrained optical weights from a small set of trainable electronic factors (as few as 8K parameters for ViT-Base), greatly reducing activation storage and weight updates while enabling practical on-chip training. We further introduce a gradient-accumulated sparse classifier that freezes low-importance weights through one-shot top-k gradient masking, reducing classifier training cost by about 40 percent. We also develop the first system-level noise model for photonic on-chip training, capturing the effects of MRR crosstalk, thermal drift, and laser amplitude noise during both forward and backward propagation. Calibrated using measurements from more than 200 fabricated MRR devices, the model shows that low-rank factor updates are more robust than full fine-tuning and conventional layer-wise low-rank adaptation under identical noise conditions. Experiments on VTAB-1K (19 tasks) and FGVC few-shot benchmarks demonstrate that Opto-ViT-v2 recovers within 0.3 to 0.8 percent of clean software accuracy under measured photonic noise while achieving more than 100 KFPS/W, enabling practical on-chip domain adaptation for photonic edge vision systems.

Problem

• on-chip fine-tuning
• photonic accelerators
• noise resilience
• Vision Transformer
• microring resonators

Innovation

• parameter-efficient fine-tuning
• silicon-photonic accelerator
• low-rank decomposition
• noise-resilient training
• near-sensor computing