Extract trace context from Kafka producer record headers by PlugaruT · Pull Request #10020 · DataDog/dd-trace-java

PlugaruT · 2025-11-23T15:34:08Z

What Does This Do

Enables Kafka producers to extract and continue existing traces from record headers.

Motivation

Fix for #5092

Currently, when a Kafka producer sends a record that already has trace context in its headers (e.g., when forwarding messages between services), the tracer creates a new root span instead of continuing the existing trace. This breaks distributed tracing continuity

This change extracts the trace context from record headers using extractContextAndGetSpanContext() and uses it as the parent for the produce span, maintaining trace continuity across service boundaries. It's also making dd-trace behave the same way opentelemetry does, see here.

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any useful labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

Allow Kafka producers to continue existing traces by extracting trace context from record headers and using it as parent for the produce span. This enables distributed tracing when messages are forwarded between services with pre-existing context.

PerfectSlayer

Hi @PlugaruT 👋 Thanks for taking time to contribute.

Quick feedback about the change but I think this might break the context tracking later as KafkaProducerCallback won't use the extracted context.

What I would do is:

Declare an AgentSpanContext
Try to extract from record headers
If null, set it with active span
Reuse this span context for both startSpan and KafkaProducerCallback.

I will send the current PR into CI to get more feedback (see #10022) and ping @DataDog/apm-idm-java to get a proper review

PlugaruT · 2025-11-24T14:53:08Z

Hi @PlugaruT 👋 Thanks for taking time to contribute.

Quick feedback about the change but I think this might break the context tracking later as KafkaProducerCallback won't use the extracted context.

What I would do is:
* Declare an AgentSpanContext

* Try to extract from record headers

* If null, set it with active span

* Reuse this span context for both `startSpan` and `KafkaProducerCallback`.
I will send the current PR into CI to get more feedback (see #10022) and ping @DataDog/apm-idm-java to get a proper review

Hi @PerfectSlayer, are you referring to how parent is passed down to the KafkaProducerCallback? Seems that indeed it's a flaw in my logic, for some reason I was under the impression that it's automatically passed with the span. My miss. Will get it fixed.

PerfectSlayer · 2025-11-25T07:45:56Z

are you referring to how parent is passed down to the KafkaProducerCallback?

Yes

Seems that indeed it's a flaw in my logic, for some reason I was under the impression that it's automatically passed with the span. My miss. Will get it fixed.

The active span content is implicitly used for startSpan() if you don’t set one, but the KakfaProducerCallback need an explicit one. So better declare one explicit parent context and use it for both.

PlugaruT · 2025-12-02T19:27:01Z

Hey @DataDog/apm-idm-java, can I get a review on this one 🙏

PerfectSlayer · 2025-12-03T09:40:33Z

I pinged the team in their internal Slack channel

ygree · 2025-12-03T18:40:20Z

.../src/main/java/datadog/trace/instrumentation/kafka_clients/KafkaProducerInstrumentation.java

+      final AgentSpan span;
+      final AgentSpan callbackParentSpan;
+
+      if (extractedContext != null) {


This could be a breaking change if users expect activeSpan to take precedence. Adding a configuration option, would allow users to configure this behavior.

I actually was expecting the other way around, especially coming from opentelemetry, I expected to have the context reuse the one from headers and attach new spans to the same trace and not break it.
I can do it, no problem there, but I'd say this is a bug fix rather than a feature.

It's done. Let me know if the name works

Makes sense. I'll exclude the commit that adds the setting, then. Let's see if all the CI checks pass in the mirroring PR: #10022

I'm looking into the failing test. Just found how to run the system tests locally.

@ygree I force pushed without the commit that added the config and also fixed the test, at least it passes on my computer 😅

Thanks! I'll pull your changes to the mirroring PR and run the CI checks.

ygree

The same change should be made in datadog.trace.instrumentation.kafka_clients38.ProducerAdvice to cover Kafka 3.8+ clients.

ygree · 2025-12-05T19:36:18Z

@PlugaruT Thank you for your contribution! Mirror PR #10022 has been merged.

PlugaruT marked this pull request as ready for review November 23, 2025 15:44

PlugaruT requested a review from a team as a code owner November 23, 2025 15:44

PerfectSlayer reviewed Nov 24, 2025

View reviewed changes

PerfectSlayer added type: bug Bug report and fix tag: community Community contribution inst: kafka Kafka instrumentation labels Nov 24, 2025

PerfectSlayer mentioned this pull request Nov 24, 2025

🪞 10020 - Extract trace context from Kafka producer record headers #10022

Merged

Link the parent span to the extracted context for KafkaProducerCallback

5e9b05d

PlugaruT requested a review from PerfectSlayer November 28, 2025 08:50

ygree reviewed Dec 3, 2025

View reviewed changes

ygree requested changes Dec 3, 2025

View reviewed changes

Port the change to Kafka Clients 3.8+

9cf2fd6

PlugaruT requested a review from a team as a code owner December 4, 2025 14:00

PlugaruT requested a review from ygree December 4, 2025 17:39

Add TextMapExtractAdapter to Kafka producer instrumentation

973db4e

PlugaruT force-pushed the fix/propagate-kafka-producer-trace-context-if-exists branch from 79f8032 to 973db4e Compare December 5, 2025 15:18

ygree closed this in #10022 Dec 5, 2025

Conversation

PlugaruT commented Nov 23, 2025

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

PerfectSlayer left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PlugaruT commented Nov 24, 2025

Uh oh!

PerfectSlayer commented Nov 25, 2025

Uh oh!

PlugaruT commented Dec 2, 2025

Uh oh!

PerfectSlayer commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ygree Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PlugaruT Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

PlugaruT Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

ygree Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

PlugaruT Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

PlugaruT Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

ygree Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

ygree left a comment

Choose a reason for hiding this comment

Uh oh!

ygree commented Dec 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

PerfectSlayer left a comment •

edited

Loading

PerfectSlayer commented Dec 3, 2025 •

edited

Loading

ygree Dec 3, 2025 •

edited

Loading