Altar: Structuring Sharable Experimental Data from Early Exploration to Publication

2026-02-20

AI Summary

This work addresses the challenge of reproducibility in actively developed experimental projects, which often suffer from unstructured data management and are overlooked by conventional data management plans. We propose a lightweight, domain-agnostic framework built upon the Sacred experiment tracking model that, from the project’s inception, systematically organizes parameters, metadata, metric trajectories, and associated files. Small-scale data are stored in a NoSQL database, while large files are linked via unique identifiers to dedicated storage systems. The framework seamlessly integrates into existing research workflows, supports both local deployment and public release, and uniquely targets the dynamic exploration phase of research. By doing so, it establishes a practical bridge from early-stage experimentation to FAIR-compliant data sharing, significantly enhancing collaborative efficiency and scientific reproducibility without compromising flexibility or scalability.

Technology Category

• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust
• Search and Optimization: Sampling/Simulation-based Search
• Application Domains: Humanities & Computational Social Science

Application Category

• Security and Privacy: Data transparency and provenance
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications

Abstract

Managing the data and metadata during the active development phase of an experimental project presents a significant challenge, particularly in collaborative research. This phase is frequently overlooked in Data Management Plans included in project proposals, despite its important role in ensuring reproducibility and preventing the need for retroactive reconstruction at the time of publication. Here we present Altar, a lightweight, domain-agnostic framework for structuring experimental data from the onset of a project without imposing rigid data models. Altar is built around the Sacred experiment-tracking model and captures experimental (meta)data and structures them. Parameters, metadata, curves and small files are stored in a flexible NoSQL database, while large raw data are maintained in dedicated storage and linked through unique identifiers, ensuring efficiency and traceability. This integration is composable with exiting workflows, allowing integration with minimial disruption of work habits. We document different pathways to use Altar based on users skillset (PhD students, Post-docs, Principal Investigators, Laboratory administrators, System administrators). While getting started with Altar does not require a specialized infrastructure, the framework can be easily deployed on a server and made publicly accessible when scaling up or preparing data for publication. By addressing the dynamic phase of research, Altar provides a practical bridge between exploratory experimentation and FAIR-aligned data sharing.

Problem

• experimental data management
• metadata organization
• reproducibility
• collaborative research
• FAIR data

Innovation

• Altar
• experiment tracking
• FAIR data
• NoSQL database
• data management framework