🤖 AI Summary
This study investigates whether large language models (LLMs) genuinely benefit from targeted cross-domain method transfer in research idea generation or merely improve due to exposure to diverse technical seeds. To address this, we propose PaperGym, a three-stage framework comprising tool-augmented seed extraction, paraphrase-based retrieval across seven machine learning domains, and rule-based synthetic evaluation. Through controlled experiments, we present the first systematic assessment of cross-domain retrieval’s effectiveness for LLM-driven ideation. Results show that cross-domain retrieval yields ideas significantly more novel than those from no-retrieval or within-domain baselines, yet performs comparably to random diverse seeds. This suggests that LLMs do not effectively leverage the semantic structure of retrieved content, highlighting fundamental limitations in current approaches to cross-domain knowledge transfer.
📝 Abstract
The discovery of novel methodologies for emerging problems is a continuing cycle in ML, often driven by the migration of techniques across domains. Building on this observation, we ask whether current LLM ideation systems benefit from targeted cross-domain retrieval or simply from exposure to diverse mechanisms. We study this question through PaperGym, a three-stage pipeline: (1) tool-augmented seed extraction via read, grep, and bash over an isolated paper environment, (2) cross-domain seed retrieval via paraphrasing across seven ML domains, and (3) method synthesis from retrieved seeds, each scored by rubric-based judges. Tool-augmented extraction improves specificity, and paraphrase-based retrieval broadens domain coverage. In synthesis, cross-domain retrieval receives more pairwise novelty wins than no-retrieval and same-domain baselines, but shows no significant difference from a random diverse-seed control. These findings suggest LLM ideation systems benefit from diverse seed exposure, but do not yet reliably exploit the semantic reason particular seeds were retrieved. We release the seed library, rubric prompts, and run scripts at https://github.com/yunjoochoi/PaperGym