From Execution to Capability: Scientific Experience Consolidation via Procedural Knowledge Synthesis

2026-07-27

AI Summary

This work addresses the challenge in scientific computing where models struggle to transfer knowledge from individual tasks to broader, reusable capabilities. The authors propose SciConsolidate, a novel framework that, for the first time, extracts procedural knowledge from runtime success and failure trajectories and employs a development-validation gating mechanism to filter effective knowledge. To bridge the gap between abstract knowledge and executable code, the approach integrates failure-driven synthesis of unanswerable queries with strong-model-guided concretization supervision. Additionally, a matched teacher-branch architecture is introduced to significantly enhance the performance of smaller models. Experimental results demonstrate that Qwen3.5-9B achieves gains of 6.25 and 3.89 points over program-free supervised fine-tuning—and improvements of 11.25 and 5.62 points over the base model—on main tasks and sub-steps, respectively, validating the efficacy of the proposed methodology.

Abstract

Large language models increasingly solve scientific-computing tasks, but executable feedback from one problem rarely becomes durable capability on subsequent problems. We study scientific-computing experience consolidation: converting verified runtime experience into transferable procedural knowledge and persistent model improvement. This setting presents two challenges: trajectory-derived artifacts may encode source-specific repairs rather than cross-task computational mechanisms; and a weaker target model may be unable to operationalize an otherwise valid abstract procedure - an abstraction-execution gap. We introduce SciConsolidate, which contrasts verified successes and failures to induce cross-task procedures, selects them through a development-validation gate, and uses failure-informed, answer-free query synthesis to expand the consolidation data without requiring pre-existing reference answers. Because the target model may not directly execute these abstractions, a stronger model concretizes them into executable code supervision for standard, procedure-free SFT; a matched no-procedure teacher branch isolates the value of procedural guidance. On SciCode, runtime procedure injection improves Qwen3.6-27B by +3.85/+6.26 sub-step/main-problem points, but yields almost no aggregate main-problem gain for Qwen3.5-9B, providing operational evidence of the abstraction-execution gap. After procedure-guided concretization, the 9B student improves under procedure-free deployment by +3.89/+6.25 points over the no-procedure SFT control and by +5.62/+11.25 over the original 9B model. These results establish an experience-to-capability pathway for scientific computing and provide a practical starting point for scaling self-improving scientific assistance.

Problem

• scientific-computing
• experience consolidation
• procedural knowledge
• abstraction-execution gap
• transferable capability

Innovation

• procedural knowledge synthesis
• experience consolidation
• abstraction-execution gap
• failure-informed query synthesis
• scientific computing