⌘K
Change language Switch ThemeSign In
Narrow Mode
How Datadog Cut the Size of Its Agent Go Binaries by 77% ========================================================
!Image 2: InfoQ InfoQ @Sergio De Simone
One Sentence Summary
Datadog reduced its Go agent binary size by 77% through systematic dependency auditing, strategic use of build tags, and eliminating reflection and plugin-related linker limitations.
Summary
This article details a major engineering initiative at Datadog to combat 'binary bloat' in their Go-based Agent, which had grown from 428 MiB to 1.22 GiB. The investigation revealed that the growth was driven by transitive dependencies, large third-party SDKs (like Kubernetes), and specific Go compiler/linker behaviors. Engineers employed a multi-pronged strategy: auditing imports using tools like 'goda', isolating optional features via build tags, and refactoring code to prevent unnecessary package inclusion. Crucially, they identified that heavy use of reflection and the 'plugin' package disables the linker's dead-code elimination, leading to massive overhead. By optimizing these areas and contributing fixes back to upstream libraries, they achieved a 77% size reduction without removing any features.
Main Points
* 1. Transitive dependencies in Go can lead to massive binary bloat if not strictly audited.Go's dependency model pulls in all transitive imports, meaning a minor change can inadvertently include hundreds of packages. Systematic auditing is required to maintain a lean binary. * 2. Build tags and package isolation are effective tools for excluding unnecessary code.Using '//go:build' tags and moving functions into dedicated packages allows developers to exclude optional features from the final build, significantly reducing the generated code footprint. * 3. Reflection can silently disable critical linker optimizations like dead-code elimination.When non-constant method names are used via reflection, the linker cannot determine which methods are needed at runtime, forcing it to keep every exported method and its dependencies. * 4. The 'plugin' package forces dynamic linking behavior that increases binary size.Simply importing the 'plugin' package causes the Go linker to treat the binary as dynamically linked, which disables method-level dead-code elimination and preserves unexported methods. * 5. Upstream contributions are often necessary to solve systemic bloat in large ecosystems.Datadog engineers had to submit PRs to major projects like Kubernetes and Uber-go/dig to eliminate dynamic reflection, benefiting the entire Go community's binary sizes.
Metadata
AI Score
86
Website infoq.com
Published At Yesterday
Length 515 words (about 3 min)
Sign in to use highlight and note-taking features for a better reading experience. Sign in now
After the Datadog Agent grew from 428 MiB to 1.22 GiB over a period of 5 years, Datadog engineers set out to reduce its binary size. They discovered that most Go binary bloat comes from hidden dependencies, disabled linker optimizations, and subtle behaviors in the Go compiler and linker.
> This growth impacted both us and our users: network costs and resource usage increased, perception of the Agent worsened, and it became harder to use the Agent on resource-constrained platforms.
To address this, Datadog software engineer Pierre Gimalac wrote, their approach consisted in auditing imports, isolating optional code, and eliminating reflection/plugin pitfalls to shrink binaries as much as possible.
Indeed, after analyzing the Agent's growth, Datadog engineers found out it was driven by new features, additional integrations, and large third-party dependencies (e.g., Kubernetes SDKs). In particular, Go’s dependency model includes transitive imports, making it so that even a small change can pull in hundreds of packages.
Datadog engineers devised two practical ways to remove unnecessary dependencies: using build tags (//go:build feature_x) to exclude optional code, and moving code into separate packages so that non-optional packages remain as small as possible. Both techniques required systematically auditing imports to identify which files or packages could be excluded from a given build. For example, simply moving one function into its own package removed ~570 packages and ~36 MB of generated code from a binary not using it.
Auditing dependencies is not an easy task, but the Go ecosystem provides three useful tools to help: go list, which lists all packages used in a build; goda, which visualizes dependency graphs and import chains to help understand why a given dependency is required; and go-size-analyzer, which shows how much space each dependency contributes to a binary.
Besides dependency optimization, Datadog engineers got an additional 20% size reduction by minimizing the use of reflection, which can silently disable some linker optimizations, including dead-code elimination:
> if you use a non-constant method name, the linker can no longer know at build time which methods will be used at runtime. So it needs to keep every exported method of every reachable type, and all the symbols they depend on, which can drastically increase the size of the final binary.
To address this issue, they eliminated dynamic reflection where possible, both in their codebase and in dependencies. The latter step required submitting several PRs to projects such as kubernetes/kubernetes, uber-go/dig, google/go-cmp and others.
Another feature that disables dead-code elimination is Go plugins, a mechanism that enables a Go program to dynamically load Go code at runtime. In fact, simply importing the plugin package causes the linker to treat the binary as dynamically linked, "which disables method dead code elimination and even forces the linker to keep all unexported methods". This change yielded an additional ~20% reduction in some builds.
As a final note, Gimalac emphasizes that these improvement were achieved over a six month period and, most importantly, did not require removing any feature. His account includes much more detail than can be covered here, so be sure to read it to get the full story.
!Image 3: InfoQ InfoQ @Sergio De Simone
One Sentence Summary
Datadog reduced its Go agent binary size by 77% through systematic dependency auditing, strategic use of build tags, and eliminating reflection and plugin-related linker limitations.
Summary
This article details a major engineering initiative at Datadog to combat 'binary bloat' in their Go-based Agent, which had grown from 428 MiB to 1.22 GiB. The investigation revealed that the growth was driven by transitive dependencies, large third-party SDKs (like Kubernetes), and specific Go compiler/linker behaviors. Engineers employed a multi-pronged strategy: auditing imports using tools like 'goda', isolating optional features via build tags, and refactoring code to prevent unnecessary package inclusion. Crucially, they identified that heavy use of reflection and the 'plugin' package disables the linker's dead-code elimination, leading to massive overhead. By optimizing these areas and contributing fixes back to upstream libraries, they achieved a 77% size reduction without removing any features.
Main Points
* 1. Transitive dependencies in Go can lead to massive binary bloat if not strictly audited.
Go's dependency model pulls in all transitive imports, meaning a minor change can inadvertently include hundreds of packages. Systematic auditing is required to maintain a lean binary.
* 2. Build tags and package isolation are effective tools for excluding unnecessary code.
Using '//go:build' tags and moving functions into dedicated packages allows developers to exclude optional features from the final build, significantly reducing the generated code footprint.
* 3. Reflection can silently disable critical linker optimizations like dead-code elimination.
When non-constant method names are used via reflection, the linker cannot determine which methods are needed at runtime, forcing it to keep every exported method and its dependencies.
* 4. The 'plugin' package forces dynamic linking behavior that increases binary size.
Simply importing the 'plugin' package causes the Go linker to treat the binary as dynamically linked, which disables method-level dead-code elimination and preserves unexported methods.
* 5. Upstream contributions are often necessary to solve systemic bloat in large ecosystems.
Datadog engineers had to submit PRs to major projects like Kubernetes and Uber-go/dig to eliminate dynamic reflection, benefiting the entire Go community's binary sizes.
Key Quotes
* Most Go binary bloat comes from hidden dependencies, disabled linker optimizations, and subtle behaviors in the Go compiler and linker. * Simply moving one function into its own package removed ~570 packages and ~36 MB of generated code from a binary not using it. * If you use a non-constant method name, the linker can no longer know at build time which methods will be used at runtime. * Simply importing the plugin package causes the linker to treat the binary as dynamically linked, which disables method dead code elimination. * These improvements were achieved over a six month period and, most importantly, did not require removing any feature.
AI Score
86
Website infoq.com
Published At Yesterday
Length 515 words (about 3 min)
Tags
Go
Binary Optimization
Dependency Management
Linker
Reflection
Related Articles
* Architecture in a Flow of AI-Augmented Change * Where Architects Sit in the Era of AI to describe human-AI collaboration levels, and highlighting the extende...") * Engineering Speed at Scale — Architectural Lessons from Sub-100-ms APIs * 4 Patterns of AI Native Development * DeepWiki and the Malleability of Software: Ripping Out Dependencies with AI Agents * InfoQ Java Trends Report 2025 HomeArticlesPodcastsVideosTweets
How Datadog Cut the Size of Its Agent Go Binaries by 77% ... ===============