Problem
Research questions and friction points this paper is trying to address.
Image Bitstream Understanding
Privacy-friendly AIoT
Fine-grained Semantic Description
Bitstream Corruption
Image Bitstream Fine-grained Understanding for Privacy-Friendly AIoT
🤖 AI Summary
This study addresses privacy leakage concerns arising from image decoding in AIoT systems by pioneering a fine-grained understanding paradigm directly within the bitstream domain, proposing the BFG model. Specifically, BFG incorporates a bitstream semantic encoder that directly processes compressed bitstreams to autoregressively generate semantic descriptions, thereby completely circumventing explicit pixel reconstruction. Furthermore, this work constructs the CFU-D dataset to evaluate corruption robustness and introduces a large-scale corruption data augmentation training strategy. Experimental results demonstrate that BFG achieves a CIDEr score of 0.6077 under bitstream corruption, significantly outperforming mainstream multimodal large models such as Qwen-VL. Ultimately, this approach enables efficient, privacy-friendly visual analysis without requiring conventional image decoding.
📝 Abstract
Image Bitstream Fine-grained Understanding (IBFU) aims to directly perform fine-grained classification and semantic description generation from encoded image byte sequences. In contrast to conventional pixel-domain visual understanding, IBFU conducts semantic analysis without fully decoding images into the pixel domain. Since pixel-level visual content is not explicitly reconstructed during inference, this paradigm reduces visual exposure within the processing pipeline and suits privacy-friendly Artificial Intelligence of Things (AIoT) applications. In this paper, we propose Bitstream Fine-grained Generator (BFG), a novel foundation model tailored for IBFU. BFG consists of two main components: a Bitstream Semantic Encoder (BSeE) and a Fine-grained Semantic Generator (FSeG). BSeE directly models semantic representations from encoded image bitstreams without explicit pixel reconstruction, while FSeG transforms the extracted bitstream semantics into detailed natural-language descriptions through autoregressive generation. To train BFG and comprehensively evaluate IBFU in practical AIoT scenarios, where image bitstreams may suffer corruption during transmission and storage, we construct a large-scale Corrupted-bitstream Fine-grained Understanding dataset (CFU-D), containing both intact bitstreams and corrupted variants across multiple corruption types and severity levels. Experiments show that BFG maintains stable fine-grained caption generation under bitstream corruption. For example, the performance only has slight change from 0.6339 to 0.6077 in terms of average CIDEr score on Stanford Dogs Caption dataset, while vision-language models, such as Qwen-VL-Chat, BLIP-2, GLM, Gemini, and GPT suffer severe performance decrease. This paper provides a practical paradigm for privacy-friendly fine-grained understanding in AIoT.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Image Bitstream Fine-grained Understanding
Privacy-friendly AIoT
Bitstream Semantic Encoder
Autoregressive Generation
Corrupted-bitstream Robustness
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
No related jobs found.
