The Prompt is Mightier than the Example

2025-05-24

AI Summary

Synthetic tabular data generation using large language models (LLMs) suffers from poor scalability due to heavy reliance on in-context learning (ICL) examples. Method: This paper proposes Knowledge-Guided Prompting (KGP), a paradigm that explicitly injects structured domain knowledge into prompts to replace a subset of ICL examples. KGP integrates reasoning-aware prompt design with optimized knowledge encoding. Contribution/Results: Through systematic ablation studies and a multi-dimensional evaluation framework, we establish, for the first time, a quantifiable substitution scaling law between knowledge volume and ICL example count. KGP significantly reduces ICL dependency—by up to 80%—while preserving statistical fidelity and improving downstream task performance. This work provides both a novel paradigm and theoretical foundation for high-fidelity, low-overhead LLM-driven tabular data synthesis.

Technology Category

Natural Language Processing: Code Generation / Program Synthesis from Natural LanguageMachine Learning: Large Multimodal Models (LMMs)Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion

Application Category

Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsEconomics, Online Markets and Human Computation: Cost models of using LLMs in production systemsUser Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Numerous recent prompt optimization approaches like chain-of-thought, have been demonstrated to significantly improve the quality of content generated by large language models (LLMs). In-context learning (ICL), a recent paradigm where a few representative examples guide content generation has also led to strong improvements in generation quality of LLM generated content. This idea has been applied to great effect in synthetic tabular data generation, where LLMs, through effective use of ICL and prompt optimization, can generate data that approximate samples from complex, heterogeneous distributions based on representative examples. However, ensuring high-fidelity synthetic data often requires a very large number of ICL examples which may be unavailable or costly to obtain. At the same time, as LLMs get larger and larger, their in-built prior knowledge becomes vast and can potentially substitute for specific data examples. In this paper, we introduce Knowledge-Guided Prompting (KGP) as a new knob in prompt optimization and explore the ability of KGP-based prompt optimization to offset the cost of ICL. Specifically, we explore the question `how many examples can a prompt substitute for?' and explore knowledge-guided prompting (KGP) where domain knowledge, either inferred or available, is explicitly injected into the prompt, reducing dependence on ICL examples. Our experiments systematically explore the trade-off between ICL and KGP, revealing an empirical scaling law that quantifies how quality of generated synthetic data varies with increasing domain knowledge and decreasing example count. Our results demonstrate that knowledge-guided prompting can be a scalable alternative, or addition, to in-context examples, unlocking new approaches to synthetic data generation.

Problem

• Reducing reliance on in-context learning examples for LLMs
• Exploring prompt optimization with domain knowledge injection
• Balancing synthetic data quality and example count trade-offs

Innovation

• Knowledge-Guided Prompting reduces ICL example dependency
• Inject domain knowledge to enhance prompt optimization
• Scaling law quantifies knowledge vs example trade-off