HSS-Synth: Humanities and Social Sciences Data Synthesis for LLMs

2026-07-29

AI Summary

This work addresses the scarcity of high-quality, diverse training data in the humanities and social sciences (HSS), a domain whose open-ended nature renders existing synthetic data methods ineffective. The authors propose HSS-Synth, the first systematic data synthesis framework tailored for HSS, which employs multi-stage web corpus filtering and discriminator-driven text refinement to generate seed documents. Innovatively, it introduces a “requirement + role” inverse generation strategy to align instructions and incorporates a teacher-forced response mechanism to anchor semantics and suppress hallucinations. This approach overcomes the limitations of capability-centric and fragmented prior methods, producing 237,000 high-fidelity instruction-response pairs. Evaluated across 16 benchmarks, it outperforms 14 baselines; fine-tuning Qwen3-8B-Base with this data achieves a new state of the art, matching official model performance while significantly enhancing human preference alignment and knowledge capabilities without notable trade-offs.

Abstract

High-quality, diverse data are vital for large language models (LLMs) but remain scarce and costly. Data synthesis is a viable alternative and succeeds on closed tasks, yet the humanities and social sciences (HSS) are overlooked, and their open-ended nature makes synthesis challenging. Moving beyond prior capability-centric, fragmented attempts, we adopt a subject-centric paradigm, define the first HSS domain system covering 14 mainstream fields, and introduce HSS-Synth, the first data synthesis pipeline for HSS. HSS-Synth comprises: (1) constructing seed documents from web corpora via multi-step filtering and text refinement evaluated by a judge; (2) specifying "requirements + persona" to backtranslate seed documents into diverse yet faithful instructions with a strict Q&A alignment check; and (3) breaking LLM response limits via teacher-forced Answering that feeds seed documents during response generation to anchor semantics, reduce hallucinations, and preserve tone and integrity. HSS-Synth yields 237k high-quality, diverse instruction-tuning samples that outperform 14 leading baselines on 16 benchmarks. The fine-tuned Qwen3-8B-Base sets a new SOTA and approaches the official Qwen3-8B, improving both human preference and knowledge capabilities without performance seesaws. Extensive experiments demonstrate HSS-Synth's robustness and transferability. Our code is publicly available at https://github.com/pengr/HSS-Synth.

Problem

• Humanities and Social Sciences
• Data Synthesis
• Large Language Models
• Instruction Tuning
• Open-ended Tasks

Innovation

• HSS-Synth
• data synthesis
• instruction tuning
• teacher-forced answering
• humanities and social sciences