Unity Forests: Improving Interaction Modelling and Interpretability in Random Forests

2026-01-11

AI Summary

This work proposes Unity Forests (UFOs) to address the limitation of traditional random forests in effectively modeling interaction effects among covariates with no marginal effects, which hampers both variable importance assessment and predictive performance. UFOs explicitly capture pure interaction effects by jointly optimizing splits over a randomly selected subset of covariates at the root node of each tree, while growing the remainder of the tree in the standard manner. The method introduces a Unity variable importance measure and Covariate-Representative Tree Roots (CRTRs), leveraging an Out-of-Bag criterion to select effective root splits. Experiments demonstrate that UFOs accurately identify interaction variables lacking marginal effects in simulated data and significantly outperform standard random forests on large-scale real-world datasets in terms of prediction accuracy, discriminative ability, calibration, and interpretability.

Technology Category

• Machine Learning: Ensemble Methods
• Reasoning under Uncertainty: Uncertainty Representations
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Random forests (RFs) are widely used for prediction and variable importance analysis and are often believed to capture any types of interactions via recursive splitting. However, since the splits are chosen locally, interactions are only reliably captured when at least one involved covariate has a marginal effect. We introduce unity forests (UFOs), an RF variant designed to better exploit interactions involving covariates without marginal effects. In UFOs, the first few splits of each tree are optimized jointly across a random covariate subset to form a"tree root"capturing such interactions; the remainder is grown conventionally. We further propose the unity variable importance measure (VIM), which is based on out-of-bag split criterion values from the tree roots. Here, only a small fraction of tree root splits with the highest in-bag criterion values are considered per covariate, reflecting that covariates with purely interaction-based effects are discriminative only if a split in an interacting covariate occurred earlier in the tree. Finally, we introduce covariate-representative tree roots (CRTRs), which select representative tree roots per covariate and provide interpretable insight into the conditions - marginal or interactive - under which each covariate has its strongest effects. In a simulation study, the unity VIM reliably identified interacting covariates without marginal effects, unlike conventional RF-based VIMs. In a large-scale real-data comparison, UFOs achieved higher discrimination and predictive accuracy than standard RFs, with comparable calibration. The CRTRs reproduced the covariates'true effect types reliably in simulated data and provided interesting insights in a real data analysis.

Problem

• random forests
• interaction modelling
• variable importance
• marginal effects
• interpretability

Innovation

• Unity Forests
• Interaction Modeling
• Variable Importance
• Interpretability
• Tree Root Optimization