Prompting Particle Physics: Tokenized Multi-modal Foundation Models for Combinatorially Many Tasks

2026-09-25

AI Summary

This study addresses the reliance of collider experiment reconstruction and simulation on multi-step, task-specific algorithms and the absence of a unified model. To this end, it proposes a multimodal foundation model based on a shared vocabulary. The model uniformly tokenizes all physical objects within a jet and employs a multimodal Transformer architecture supporting both autoregressive and parallel decoding. By flexibly selecting input and output modalities, a single model can handle compositional multitask scenarios such as particle flow analysis while providing interpretable intermediate representations. Experimental results demonstrate that the proposed approach achieves stable training and generates realistic physical objects, with its reconstruction performance surpassing existing state-of-the-art particle flow algorithms.

Abstract

Reconstruction and simulation at a collider experiment are long chains of specialised algorithms, each tuned to a single step. We explore how one model can serve many of those steps at once, while still producing the intermediate objects (tracks, calorimeter cells, clusters, particles and jets) that make the chain interpretable. To do so, we represent every object in a jet in a single shared token vocabulary and train one model to map any subset of these modalities to any other. A task is then only a choice of which modalities to provide and which to request: particle flow, detector simulation and charged energy subtraction are all directions through the same set of weights. We train over all modality combinations, with a decoder emitting tokens either autoregressively or in parallel. With tokenisation, both architectures train stably with little tuning. In evaluations, both models produce realistic reconstruction and simulation objects, with the autoregressive model particularly faithful to output from Geant4. The autoregressive model also outperforms a state-of-the-art particle flow algorithm on many typical jet reconstruction metrics.

Problem

• particle physics
• multi-modal foundation model
• reconstruction and simulation
• tokenization
• jet reconstruction

Innovation

• multi-modal foundation model
• tokenization
• autoregressive decoding
• particle physics
• unified architecture