QDAG: Declarative Composition of Reusable Analytics Methodologies at LinkedIn

2026-06-03

AI Summary

This work addresses the challenges of implementing analytical logic in production-grade systems using imperative code, which often leads to code duplication, definition drift, and poor testability. To overcome these issues, the authors propose modeling analytical workflows as typed, declarative directed acyclic graphs (QDAGs), enabling, for the first time, a unified representation of complex analytical pipelines in large-scale production environments. The system efficiently orchestrates multi-step operations through on-demand execution, memoization, pruning, and parallel scheduling, supporting diverse primitives such as Apache Pinot queries, service invocations, SQLite joins, jq transformations, conditional logic, differentially private aggregations, and nested QDAG calls. Deployed across over 100 production use cases at LinkedIn on more than 500 hosts, the system achieves median orchestration overhead of approximately 10 milliseconds and 99th-percentile latency under 50 milliseconds, significantly enhancing reusability, consistency, and testability while meeting interactive latency requirements.

Abstract

Production analytics products often depend on reusable methodologies: multi-step definitions such as headcount growth, top-skill growth, or differentially-private impression distributions. Although these methodologies define business-critical numbers, they are commonly implemented as imperative glue around OLAP queries, service calls, joins, transformations, and conditional logic. As a result, teams duplicate orchestration code, definitions drift across products, and methodologies are difficult to test or analyze. We present QDAG, a production system at LinkedIn that represents an analytics methodology as a declarative directed acyclic graph of typed steps. Nodes may execute Apache Pinot queries, downstream service calls, in-memory SQLite joins, jq transformations, conditionals, differentially-private aggregations, or calls to other QDAGs. The engine evaluates graphs demand-driven, memoized, pruned, and parallelized in the per-request analytics mid-tier. QDAG is deployed across more than 500 hosts and over 100 production use cases, adding roughly 10 ms median orchestration overhead and under 50 ms at the 99th percentile. Our experience shows that making methodologies declarative improves reuse, testability, and cross-product consistency while preserving interactive latency.

Problem

• reusable analytics methodologies
• imperative orchestration
• definition drift
• testability
• cross-product consistency

Innovation

• declarative analytics
• directed acyclic graph
• reusable methodologies
• differentially-private aggregation
• demand-driven evaluation