🤖 AI Summary
This work addresses the lack of reproducible and calibratable prompt engineering pipelines for evidence synthesis tasks in current large language models (LLMs). It proposes an innovative workflow that decouples scientific task specifications from prompting frameworks for the first time, leveraging annotated data and explicit metrics to drive prompt optimization. The approach operationalizes the entire pipeline into artifacts using the DSPy and GEPA toolchains, employing a small student model to execute tasks while a larger reflection model guides iterative refinement. Supporting structured task definitions, metric-driven search, and cross-framework portability, the method demonstrates strong compilability and artifact consistency in title and abstract screening tasks. Empirical validation further reveals the critical impact of optimization budgets on small-model performance, significantly enhancing the reliability and transparency of LLM-based applications.
📝 Abstract
This methods article presents a reproducible calibration workflow for prompt-based large language models (LLMs) in structured evidence-synthesis tasks. The method separates the rules that define the scientific task from the mutable prompt harness that frames and applies them. It optimises that harness against labelled or reference examples and an explicit task metric, then preserves the calibrated workflow as an inspectable artefact with its specification, metric, settings, and evaluation traces. The example code instantiates the protocol with DSPy and GEPA tools, but the underlying logic can transfer to other prompt-optimisation frameworks that support structured task definitions, metric-guided search, and artefact reuse. Title and abstract screening is the worked validation case because it provides labelled benchmark data and clear evaluation metrics. The demonstrated workflow uses a smaller student LLM for performing the scientific task execution and a larger reflection LLM to steer the prompt optimisation process during calibration. This work shows compilation, artefact round-tripping, and how optimisation budget affects a smaller student model.