SIDInspector: A Mapping-First Diagnostic Resource for Semantic-ID Tokenizers

2026-06-08

AI Summary

Existing Semantic-ID (SID) tokenizers lack a unified diagnostic interface, causing mapping flaws—such as coverage gaps, full-code aliasing, and weak semantic prefixes—to remain undetected until downstream training. This work proposes the first systematic diagnostic framework tailored for SID mappings, which enables pre-training analysis by defining an adapter contract that integrates item mappings, metadata, and generation trajectories. The approach decouples addressability from behavioral semantic prefix evaluation and introduces mapping-level probes—including utilization rate, aliasing rate, neighborhood alignment, popularity distribution, and structural cost—as well as dynamic trajectory hooks. Experiments reveal that GRID-style mappings exhibit an aliasing rate as high as 0.977, whereas ReSID and GAOQ show no aliasing; deterministic category prefixes achieve the strongest co-occurrence alignment (0.447), confirming prefix alignment as a viable signal for candidate exposure.

Abstract

Semantic-ID (\sid) tokenizers are increasingly reused as standalone artifacts in generative recommendation: an exported item-to-code mapping becomes the address space that a later sequence generator must use. These mappings rarely come with a common inspection interface, so coverage gaps, full-code aliasing, behaviorally weak prefixes, tail compression, and prefix fan-out are often found only after downstream training. We present \tool, a mapping-first diagnostic resource for \sid tokenizer artifacts. \tool defines a small adapter contract over item mappings, metadata, interactions, and optional generator traces; validates the contract; and reports mapping-level probes for utilization, aliasing, neighborhood alignment, popularity allocation, and structural cost, with hooks for temporal churn and generator traces. \tool reports inspectable artifact profiles before downstream leaderboard scores. The released resource covers four tokenizer artifact lines: a same-item GRID/RQ-KMeans-style and ReSID/GAOQ contrast on 23,742 Musical items, plus released LETTER and LC-Rec item-index artifacts. In the Musical contrast, the GRID-style feature-text export has 3,749 unique full codes and a 0.977 full-code aliasing rate, while ReSID/GAOQ is aliasing-free in its exported mapping. Yet the strongest prefix--co-occurrence alignment comes from a deterministic category-prefix control, not from either learned export row (0.447 versus 0.154 and 0.055--0.080), showing that addressability and behaviorally meaningful prefixes should be inspected separately. Cross-domain, fixed-reranker, and mechanism-probe checks support the same diagnostic direction: prefix alignment is a candidate-exposure signal, while final ranking quality remains a downstream model question.

Problem

• Semantic-ID tokenizers
• mapping inspection
• code aliasing
• prefix alignment
• generative recommendation

Innovation

• Semantic-ID tokenization
• mapping-first diagnostics
• aliasing analysis
• prefix alignment
• generative recommendation