Bayesian genome-wide clustering and variable selection of transcriptomic data via rank-based mixtures

2026-06-04

AI Summary

This study addresses the challenges of scalability and uncertainty quantification in clustering and variable selection for ultra-high-dimensional transcriptomic ranking data by proposing the low-dimensional Bayesian Mallows mixture model (lowBM3). This work extends the Bayesian Mallows model to jointly perform clustering and variable selection for the first time, integrating Bayesian nonparametric modeling, a mixture structure over rankings, and sparsity-inducing priors within a unified framework. The approach simultaneously models sample heterogeneity, enables unsupervised parameter inference, and facilitates feature selection, complemented by a tailored post-processing strategy to effectively summarize the discrete posterior distribution. Comprehensive simulations and whole-genome RNA-seq analyses of breast cancer data demonstrate that lowBM3 achieves superior performance in both clustering accuracy and identification of biologically relevant genes.

Abstract

With the increasing availability of ranking data, there has been a growing demand for appropriate unsupervised rank-based inferential frameworks capable of handling high-dimensional datasets and providing uncertainty quantification for all estimates. Rank-based methods have also seen a growing popularity in -omics pipelines, as ranking continuous measurements provides a robust means of handling non-normally distributed data. The Bayesian Mallows model (BMM) has emerged as a promising choice because of its adaptability to various types of ranking data and its flexible framework, integrating cluster-wise rank aggregation with inference at the individual level. However, the scalability of BMM to ultra-high-dimensional settings, such as -omics analyses, has remained limited. The present paper addresses this issue by introducing the first rank-based model generalizing BMM to jointly handle clustering and variable selection, namely the lower-dimensional Bayesian Mallows Model Mixture (lowBM3). The proposed method provides a novel Bayesian framework that simultaneously handles heterogeneity in the sample, unsupervised parameter estimation, and model selection in a scalable manner for ultra-high-dimensional data. Additionally, a companion postprocessing framework is introduced to provide posterior summaries of the discrete posterior distributions of both the consensus ranking and the variable selector. Simulation studies are performed to assess the performance of the method. The usefulness of the method is also shown in an application to signature discovery for cancer genomics, where RNA-seq bulk gene expression data obtained from breast cancer patients are clustered genome-wide.

Problem

• Bayesian Mallows model
• rank-based clustering
• variable selection
• high-dimensional omics data
• unsupervised inference

Innovation

• Bayesian Mallows Model
• rank-based clustering
• variable selection
• ultra-high-dimensional data
• transcriptomics