SceneMiner: Identity-Preserving Multi-Task Fine-Tuning for Unified BEV Scene Mining

2026-06-09

AI Summary

This work addresses the challenge that existing driving logs lack difficulty annotations and rely on single metrics that inadequately capture safety-critical scenarios. To overcome this, the authors propose a unified camera-only bird’s-eye-view (BEV) multi-task model that, in a single forward pass, simultaneously outputs text-retrieval embeddings, multi-label scene distributions, and a physical risk score. By employing an identity-preserving multi-task fine-tuning strategy—freezing the shared backbone and zero-initializing newly added submodules—the approach effectively mitigates inter-task interference, ensuring that new tasks do not degrade pre-existing performance. Built upon a frozen vision-language backbone, the model trains only approximately 102,000 parameters and achieves a mean average precision (mAP) of 0.4614 (micro-F1: 0.5557) across 20 scene categories, while enabling efficient text-prompt-driven scene retrieval.

Abstract

Mining hard, safety-critical scenes from driving logs is bottlenecked by the absence of difficulty labels, and no single proxy, collision risk, trajectory ambiguity, or semantic rarity suffices to find such scenes on its own. We present SceneMiner, a unified, camera-only bird's-eye-view pipeline that emits complementary mining signals from a frozen vision-language backbone in a single forward pass, with no LiDAR or radar: a retrieval embedding for text-prompted scenario search, a multi-label scene-tag distribution, and a continuous physics-based risk score (a motion forecast is a byproduct, not a contribution). Building such a multi-head model exposes our central finding, a failure mode we term cross-task interference: adding or upgrading one head shifts a shared activation stream and degrades weight-frozen sibling heads, so freezing parameters alone is insufficient. Our contribution, identity-preserving multi-task fine-tuning, removes this interference by zero-initializing every new sub-module and freezing every parameter that feeds the shared stream. The mining heads are thereby preserved bit-identically while training only ~102k parameters. The tagging head reaches mAP 0.4614 (micro-F1 0.5557) on 20 scene tags by pooling each scene into 32 visual tokens, and the embedding head supports text-prompted retrieval, validated qualitatively. Code is available at: https://anonymous.4open.science/r/sceneminer_anonymous-64E5

Problem

• scene mining
• safety-critical scenes
• difficulty labels
• multi-task learning
• cross-task interference

Innovation

• identity-preserving fine-tuning
• cross-task interference
• multi-task learning
• BEV scene mining
• camera-only perception