TreeAdapter: Hierarchical Taxonomy-Guided Adapter Composition for Fine-Grained Species Image Generation

2026-07-27

AI Summary

This work addresses the challenge that general-purpose text-to-image diffusion models struggle to generate fine-grained images of rare species, while existing fine-tuning approaches overlook shared visual features among closely related taxa. To overcome this limitation, we propose TreeAdapter, a novel framework that explicitly incorporates biological taxonomic tree structure into diffusion models for the first time. Lightweight adapters are deployed at each node of the taxonomy tree: leaf nodes model species-specific characteristics, while internal nodes capture semantic features shared by their descendants. A two-stage residual training strategy ensures that ancestral nodes learn only the residual features not already explained by their descendants. Evaluated on three biodiversity benchmarks, TreeAdapter significantly outperforms both general and domain-specific baselines, achieving state-of-the-art performance in fine-grained image generation.

Abstract

Although general text-to-image models excel in open-domain generation, their performance degrades significantly in specialized downstream domains, particularly when generating images of rare biological species. Hindered by long-tailed distributions, general models struggle to capture subtle fine-grained details, while per-species fine-tuning methods over-isolate individual species and consequently ignore the shared visual features among closely related taxa. To address this, we propose TreeAdapter, a novel framework that explicitly leverages hierarchical taxonomic data. Rather than using a monolithic model or independent per-species modules, TreeAdapter attaches lightweight adapters to every node of the taxonomic tree. Specifically, leaf-node adapters capture species-specific visual traits, while internal-node adapters encapsulate shared semantics among descendant taxa. We introduce a two-stage training paradigm where ancestor adapters are optimized to model only the residual visual features unexplained by their descendants. This model architecture and training paradigm enable the model to fully leverage hierarchical information, ensuring the accurate generation of visual features for each species. Extensive experiments across three large-scale biodiversity benchmarks demonstrate that TreeAdapter achieves state-of-the-art fine-grained generation quality, outperforming both general-purpose and domain-specific baselines.

Problem

• fine-grained species image generation
• hierarchical taxonomy
• long-tailed distribution
• shared visual features
• text-to-image generation

Innovation

• hierarchical taxonomy
• adapter composition
• fine-grained image generation
• residual feature learning
• species-specific generation