MANILA: A Low-Code Application to Benchmark Machine Learning Models and Fairness-Enhancing Methods

2025-04-29

AI Summary

Systematic trade-off evaluation between fairness and effectiveness in machine learning models remains challenging. Method: This paper proposes a fairness benchmarking framework based on Software Product Line (SPL) engineering, implemented as a web-based low-code platform. Contribution/Results: It introduces the first SPL-based fairness benchmark, featuring an extended feature model and formal constraint mechanisms to ensure semantic correctness and experimental reproducibility. The framework automatically enumerates valid configurations, generates executable experiment pipelines, and computes Pareto-optimal trade-offs between effectiveness (e.g., accuracy) and fairness (e.g., demographic parity). Empirical evaluation demonstrates that the framework achieves high expressiveness and zero runtime configuration errors, significantly enhancing the reliability, customizability, and engineering rigor of fairness assessment.

Technology Category

• Machine Learning: Ethics, Bias, and Fairness
• Philosophy and Ethics of AI: Bias, Fairness & Equity
• Computer Vision: Bias, Fairness & Privacy

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems

Abstract

This paper presents MANILA, a web-based low-code application to benchmark machine learning models and fairness-enhancing methods and select the one achieving the best fairness and effectiveness trade-off. It is grounded on an Extended Feature Model that models a general fairness benchmarking workflow as a Software Product Line. The constraints defined among the features guide users in creating experiments that do not lead to execution errors. We describe the architecture and implementation of MANILA and evaluate it in terms of expressiveness and correctness.

Problem

• Benchmark ML models and fairness methods
• Select best fairness-effectiveness trade-off
• Prevent execution errors in experiments

Innovation

• Web-based low-code application for ML benchmarking
• Extended Feature Model for workflow modeling
• Constraint-guided experiment creation to prevent errors