Ichnos: A Carbon Footprint Estimator for Scientific Workflows

2024-11-19

arXiv.org

AI Summary

To address the high manual monitoring overhead, coarse-grained modeling, and insufficient integration of carbon intensity (CI) data in scientific workflow carbon footprint assessment, this paper proposes the first automated carbon estimation framework for Nextflow. The framework leverages native execution traces to eliminate manual power instrumentation, dynamically converts energy consumption to carbon emissions by fusing high- and low-resolution temporal CI data, and supports user-defined, CPU-frequency-aware fine-grained power models. Through resource-aware modeling and comparative evaluation against RAPL and GA methods, it achieves precise task-level decomposition of carbon emissions and energy consumption on two real-world Nextflow workflows. Experimental results demonstrate controlled estimation error and superior accuracy over state-of-the-art RAPL and GA approaches. Additionally, the framework enables sensitivity analysis with respect to CI granularity and CPU frequency parameters. The implementation is publicly available.

Technology Category

• Planning, Routing, and Scheduling: Plan Execution and Monitoring
• Machine Learning: Hardware-aware ML
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Sustainability and carbon-aware systems for Web, mobile, and WoT
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

We propose Ichnos, a novel and flexible tool to estimate the carbon footprint of Nextflow workflows based on detailed workflow traces, CI time series, and power models. First, Ichnos takes as input the automatically-generated workflow trace produced by Nextflow. Use of these traces is an original contribution, ensuring that users do not need to manually monitor power consumption and enabling analysis of previously executed workflows. Next, Ichnos allows users to provide their own resource power model for utilised compute resources to accurately reflect processor settings, such as the processor frequency, instead of solely relying on a linear function. Finally, Ichnos converts estimated energy consumption to overall carbon emissions using fine-grained time-series CI data for each workflow task and only resorts to coarse-grained yearly averages where high-resolution location-based CI data are not available. Additionally, Ichnos reports estimated energy consumption and carbon emissions per task, providing greater granularity than existing methodologies and allowing users to identify which of their tasks have the largest footprint to address. We provide the implementation of Ichnos as open-source. We demonstrate our tool on traces of two real-world Nextflow workflows, compare the estimated energy consumption against RAPL and the GA methodology, and show the tool's functionality by varying the granularity of provided CI data and varying the processor frequency settings of assigned compute resources.

Problem

• Estimating carbon footprint of resource-intensive scientific workflows
• Reducing user effort in carbon footprint quantification
• Providing post-hoc estimation using workflow traces and power models

Innovation

• Post-hoc carbon estimation using workflow traces
• Automated power modeling for computational resources
• Carbon intensity data aligned with execution time