[Discuss][Observability] OpenTelemetry GenAI export for Agent Traces (follow-up to #900)

[Zhuoxi2000]

Motivation

#900 defines how Agent Traces are recorded in the Event Log, and deliberately lists OTel export as a non-goal of the recording scope. In that thread, @joeyutong agreed that OpenTelemetry export should be a supported follow-up once the recording contract settles, with the exact GenAI attribute mapping to be worked out as part of the exporter design. This discussion is that follow-up: a config-gated exporter that maps recorded Execution Events to the OpenTelemetry GenAI semantic conventions, so any OTLP-compatible backend (Jaeger, Grafana Tempo, Langfuse, Phoenix, vendor APMs) can visualize and analyze agent runs without flink-agents shipping its own UI.

Proposed mapping

Trace topology, aligned with the #900 recording model:

flink-agents	OTel GenAI
Input run (input_run_id)	Trace (one run = one trace)
Execution (execution_id / parent_execution_id)	Span / parent Span
Agent execution	span invoke_agent {agent_name}; gen_ai.operation.name = invoke_agent, gen_ai.agent.name
Tool execution	span execute_tool {tool_name}; gen_ai.operation.name = execute_tool, gen_ai.tool.name
Model call	span chat {model}; gen_ai.operation.name = chat, gen_ai.request.model, gen_ai.provider.name
business_key	gen_ai.conversation.id
Lifecycle events (started/finished)	span start/end timestamps
status / problem_category	span status + error.type
Token usage	gen_ai.usage.input_tokens / gen_ai.usage.output_tokens

ID derivation. Framework IDs stay opaque, as preferred in #900; the exporter derives fixed-width OTel IDs deterministically, e.g. trace_id = truncate(SHA-256(input_run_id), 128 bits) and span_id = truncate(SHA-256(execution_id), 64 bits). Deterministic derivation keeps re-exports idempotent.

Design points to agree on

1. Where the exporter runs. Two candidate shapes:

   • (a) In-process: an Execution Event listener in the runtime feeds an async OTLP exporter (batching, bounded queue, drop-on-overflow so it can never backpressure the job).
   • (b) Out-of-band: a separate consumer reads the Event Log and exports — zero impact on the running job, works retroactively, but requires event-log trace recording ([api][runtime][python] Add Agent Trace recording to Event Log #924) to be enabled.

   I lean toward (b) first, since it derives everything from the recording contract and adds no runtime risk, keeping (a) as a later option. Opinions welcome.

2. Config surface. event-log.trace.otel.enabled (default false, mirroring the recording flag), OTLP endpoint/protocol/headers, service.name. Content capture (prompt/response bodies) stays off by default and is opted into separately — gen_ai content capture has privacy and volume implications.

3. Semconv version. The gen_ai.* conventions are still development stability; we should pin the exported schema_url and document the target version. I'm engaging in the OTel GenAI SIG, so we can track upstream changes as they stabilize.

4. Scope for 0.4. Phase 1 = Java, spans-only, out-of-band exporter + config; Phase 2 = usage-metrics enrichment; Phase 3 = Python parity. Is Phase 1 realistic for the 0.4 feature freeze (Sept 15)?

What I'd deliver

I've already implemented the inverse mapping (OTel GenAI → storage) in a side project, so the attribute table above is grounded in working code. If maintainers agree with the general shape, I'll break this into concrete issues and start with Phase 1.

---

Updates (2026-08-07) — incorporating the review feedback above:

Rejected alternative: Flink's OpenTelemetryTraceReporterFactory. Flink 2.3 ships a native
OTel trace reporter, but it cannot serve as the export path for agent traces: it does not
preserve caller-derived trace/span ids or span status, it requires related spans to be reported
together, and it is not consistently available across the 1.20 / 2.0 / 2.1 distributions this
project ships for. A standalone converter avoids all three constraints.

Scope clarifications:

• Phase 1 is a batch, offline converter over already-collected Event Log files — not a
  continuous tailer. Durability, partition discovery, checkpoints, and retention are properties
  of a durable Event Log sink, which deserves its own design track; a continuous OTel consumer
  would build on that. Delivery is at-least-once; deterministic ids make duplicates identifiable,
  and idempotent coalescing is a backend property.
• Since the converter is an external process, its settings are plain converter arguments
  (endpoint, protocol, service.name) — the earlier event-log.trace.otel.* naming is dropped.
• The contract is the serialized Event Log record, which both runtimes produce through the same
  writer, so one converter serves both Java and Python agents. "Java" refers only to the
  converter's implementation language; there is no separate Python-parity phase.
• The converter lives in an optional standalone module that is not bundled into dist, with
  its OTel dependencies aligned through the opentelemetry-bom, keeping them off the
  distribution classpath entirely.
• Attribute-quality gaps (resolved model vs resource alias, provider identity, per-call token
  usage) are best addressed by recording them in entityMetadata at recording time, as raised
  above — tracked as part of the recording-contract conversation on [api][runtime][python] Add Agent Trace recording to Event Log #924.

[wenjin272]

Thanks for driving this discussion. OpenTelemetry is becoming a widely adopted standard for agent observability, so I believe supporting the conversion of Flink Agents Event Logs into OpenTelemetry GenAI traces would be very valuable. It would also allow users to integrate with existing observability backends without Flink Agents having to provide its own visualization stack. The proposed mapping and overall design look reasonable to me, and I have no major concerns with this direction.

[Zhuoxi2000]

Thanks for the review! will open a tracking issue for the exporter and start with Phase 1: Java, spans only, reading from the Event Log, and gated by config. Usage-metrics enrichment and Python parity can follow as separate issues.
#970

[weiqingy]

Thanks for making the mapping explicit rather than leaving it implicit in #900. OTLP export is worth pursuing, and a standalone exporter looks like the right direction.

Could the proposal include a short rejected-alternatives note for Flink's OpenTelemetryTraceReporterFactory? Although it is available with the pinned Flink 2.3.0, it cannot preserve caller-derived IDs or span status, requires related spans to be reported together, and lacks consistent support across the 1.20, 2.0, and 2.1 distributions shipped here. That seems enough to explain why the native trace pipeline cannot represent an agent trace.

Before implementation, could we clarify the architecture and recording contract on which the exporter would depend?

1. What durable source would the standalone exporter consume?

Today, Event Log output is either SLF4J or TaskManager-local files under java.io.tmpdir. Both are local, and the logger type is a closed enum that does not support adding a durable sink alongside the existing one. A separate consumer would therefore amount to tailing files that may disappear during failover.

Could Phase 1 define durability, partition discovery, checkpoints, replay, retention, failover, and handling of incomplete or out-of-order lifecycle events? Would event-log.trace.otel.* also be clearer as consumer configuration, since an external process cannot access AgentPlan?

2. Could #924 capture the telemetry-neutral metadata while its contract is still open?

Several proposed OTel attributes currently have no reliable source:

• The LLM entity name is a resource alias such as _default_chat_model, not necessarily the exact requested model.
• Provider identity exists only indirectly in the resource descriptor.
• Token counts are cumulative Flink counters keyed by model, rather than per-execution values.

The existing entityMetadata channel is already serialized and used for tool executions, while both Java and Python LLM paths currently leave it empty. Would recording the resolved model, provider, and per-call token usage there keep the exporter focused on conversion rather than inference and metric joins?

Could the contract also define how agent, action, LLM, parser, and tool executions become spans? It currently records no agent entity, while action roots have no parent and parser executions have no proposed mapping. Without explicit run, root, and parent semantics, could chained actions appear as unrelated roots or recorded executions be omitted?

3. Could the contract remain language-neutral from the start?

Python actions already report through Java, and Java's EventLogWriter serializes their events. Could "Java first, Python parity later" therefore refer only to implementation work, with one serialized contract and one exporter serving both runtimes?

4. Would an optional module provide a safer dependency boundary?

The Elasticsearch integration already brings an unrelocated OTel API 1.29.0 into the uber JAR, and the default OTLP sender would add a second okhttp package alongside the one already there. Introducing the 1.54.1 SDK/exporter can produce incompatible dependency mediation, including the observed missing ComponentLoader failure. Could a standalone optional artifact align its OTel dependencies through the BOM and avoid exposing that conflict to every distribution?

5. Could delivery semantics be defined separately from ID derivation?

Stable derived IDs would help with reconstruction, but they would not make OTLP delivery idempotent. Could the proposal describe consumer checkpoints, retries, and replay behavior for complete and partial spans?

Would it make sense to frame Phase 1 as:

1. Finalize a language-neutral execution and metadata contract.
2. Define a durable, replayable Event Log source and consumer-checkpoint model.
3. Provide an optional standalone OTLP converter on top of that contract.

In-process export, content capture, and further enrichment could then build on that foundation in later phases.

[Zhuoxi2000]

Thanks for the thorough review — these are exactly the questions worth settling before implementation.

Rejected alternative. Agreed. I’ll document why OpenTelemetryTraceReporterFactory isn’t a good fit: no caller-defined IDs or status, related spans need to be exported together, and support differs across Flink distributions.

1. Source and scope. Phase 1 is a batch converter over already-collected Event Log files, not a continuous tailer. Durability, checkpoints, retention, and failover therefore belong to a separate durable Event Log sink design, which a future continuous OTel consumer could build on.

Since the converter is standalone, its settings will be regular arguments such as endpoint, protocol, and service.name, rather than event-log.* config.

2. Recording metadata. Strong +1 on recording the resolved model, provider, and per-call usage in entityMetadata. That keeps the exporter a pure converter without inference or metric joins. The run/root/parent relationships should also be explicit in the contract rather than synthesized by the exporter. I’m happy to help with the LLM metadata follow-up once #924 settles.

3. Language neutrality. Agreed. Both runtimes produce the same serialized Event Log records, so one converter serves both. “Java first” only refers to the converter implementation language.

4. Dependency boundary. The converter will remain an optional standalone module and won’t be bundled into the distribution, keeping its OTel SDK/exporter dependencies off the main classpath.

5. Delivery semantics. The batch converter is at-least-once. Deterministic IDs make duplicates identifiable, but idempotency is ultimately backend-dependent. Checkpointing and replay guarantees belong to the future continuous-consumer track.

So the revised plan is:

1. Clarify metadata and run/root/parent semantics in [api][runtime][python] Add Agent Trace recording to Event Log #924.
2. Track the durable, replayable Event Log sink separately.
3. Keep [Feature] OpenTelemetry GenAI exporter for Agent Traces (Phase 1: Java, spans-only) #970 focused on the optional batch converter, with a continuous consumer as follow-up work.

I’ll update #970 accordingly.

[weiqingy]

Thanks for working through these. I think this narrows the remaining questions considerably.

Assuming each converter invocation receives a fixed, complete set of already-collected files, the batch shape seems to resolve the ordering and timeout questions: the converter can group records by inputRunId, sort them, and use end-of-input rather than a timeout to determine that no more records are coming.

Incomplete executions may still be present, though. EventLogWriter calls append and flush for each record, but treats failures as best-effort and does not retry them. A crash can also leave a start record in the collected files without a terminal record. A restored Action execution keeps its executionId and inputRunId, since both are checkpointed with the ActionTask. By contrast, the state pairing nested LLM, Parser, and Tool starts with their terminal reports is transient, so recovery can produce new child executions under the same run.

How would we like those incomplete executions to appear in the exported trace? Would omitting them hide useful evidence of failed work, while synthesizing an end timestamp risk implying a duration we did not observe? Could emitting them as errors with an explicit "incomplete/synthetic close" marker preserve that distinction, or would a diagnostic-only record be clearer?

Could trace_tree.py provide a useful precedent for input validation and diagnostics? It reports conditions such as MISSING_PARENT for business-event lineage and MALFORMED_RECORD for invalid input. Although it intentionally ignores execution lifecycle records and therefore does not answer the incomplete-span policy itself, might its warning model be reusable?

It also discovers only events-*.log, which is the FileEventLogger naming contract rather than the default SLF4J output. Would it help for Phase 1 to state whether the converter consumes specifically FileEventLogger JSONL files, including how a complete multi-subtask file set is identified, or normalized records collected from any Event Log sink?

I also wonder whether the Parser mapping could be explicit. The run/root/parent fields establish its topology, but the converter still needs to decide its span name, gen_ai.operation.name, and OTel span kind. Since #924 records PARSER as a first-class execution type, would defining one stable mapping here make traces more consistent across converter versions?

The OTel GenAI convention permits a custom gen_ai.operation.name when none of its well-known values applies, and OpenLLMetry uses that mechanism for names such as execute_task and llm_request. OpenInference's CHAIN, GUARDRAIL, and EVALUATOR values belong to the separate openinference.span.kind classification. Since Phoenix and Langfuse recognize that convention, would it be worth deciding whether Phase 1 emits only OTel GenAI attributes or also an OpenInference classification for AI-aware rendering?

Separately, for the OTel span kind, would SpanKind.INTERNAL fit Parser, and perhaps Action, as in-process work, while preserving CLIENT for spans that directly represent remote calls?

[Zhuoxi2000]

Thanks for working through these. I think this narrows the remaining questions considerably.

Assuming each converter invocation receives a fixed, complete set of already-collected files, the batch shape seems to resolve the ordering and timeout questions: the converter can group records by inputRunId, sort them, and use end-of-input rather than a timeout to determine that no more records are coming.

Incomplete executions may still be present, though. EventLogWriter calls append and flush for each record, but treats failures as best-effort and does not retry them. A crash can also leave a start record in the collected files without a terminal record. A restored Action execution keeps its executionId and inputRunId, since both are checkpointed with the ActionTask. By contrast, the state pairing nested LLM, Parser, and Tool starts with their terminal reports is transient, so recovery can produce new child executions under the same run.

How would we like those incomplete executions to appear in the exported trace? Would omitting them hide useful evidence of failed work, while synthesizing an end timestamp risk implying a duration we did not observe? Could emitting them as errors with an explicit "incomplete/synthetic close" marker preserve that distinction, or would a diagnostic-only record be clearer?

Could trace_tree.py provide a useful precedent for input validation and diagnostics? It reports conditions such as MISSING_PARENT for business-event lineage and MALFORMED_RECORD for invalid input. Although it intentionally ignores execution lifecycle records and therefore does not answer the incomplete-span policy itself, might its warning model be reusable?

It also discovers only events-*.log, which is the FileEventLogger naming contract rather than the default SLF4J output. Would it help for Phase 1 to state whether the converter consumes specifically FileEventLogger JSONL files, including how a complete multi-subtask file set is identified, or normalized records collected from any Event Log sink?

I also wonder whether the Parser mapping could be explicit. The run/root/parent fields establish its topology, but the converter still needs to decide its span name, gen_ai.operation.name, and OTel span kind. Since #924 records PARSER as a first-class execution type, would defining one stable mapping here make traces more consistent across converter versions?

The OTel GenAI convention permits a custom gen_ai.operation.name when none of its well-known values applies, and OpenLLMetry uses that mechanism for names such as execute_task and llm_request. OpenInference's CHAIN, GUARDRAIL, and EVALUATOR values belong to the separate openinference.span.kind classification. Since Phoenix and Langfuse recognize that convention, would it be worth deciding whether Phase 1 emits only OTel GenAI attributes or also an OpenInference classification for AI-aware rendering?

Separately, for the OTel span kind, would SpanKind.INTERNAL fit Parser, and perhaps Action, as in-process work, while preserving CLIENT for spans that directly represent remote calls?

Thanks! totally agreed that batch input gives us a clear ordering/completion model. The converter can group by inputRunId, sort by timestamp, and treat end-of-input as completion. I’ll make that explicit in #970.

Incomplete executions. I’d combine the two approaches: still emit the span, close it at the only timestamp we observed (so zero duration rather than inventing an end time), keep the status UNSET, add something like flink_agents.execution.incomplete = true, and surface a diagnostic as well.

I’d avoid ERROR because a missing terminal record doesn’t necessarily mean the execution failed — it could also be a dropped best-effort append or a recovery/replay artifact. Marking it incomplete keeps that evidence queryable without claiming an outcome we didn’t observe. A terminal record with no matching start would get the same treatment, with a MISSING_START diagnostic.

Diagnostics. I agree we should follow trace_tree.py here and reuse the same machine-readable shape (code, id, message, filePath, lineNumber). We can keep MALFORMED_RECORD and add INCOMPLETE_EXECUTION / MISSING_START rather than inventing a separate model.

Input contract. Phase 1 should operate on the JSON records themselves, not depend on a file naming convention. It can take explicit files, ignore unknown fields, and handle concatenated JSON objects. Directory discovery for events-*.log can just be a convenience matching trace_tree.py. For multi-subtask jobs, completeness remains the caller’s responsibility, but per-file counts and diagnostics should make missing input easier to spot.

Parser mapping. Agreed: parse {entityName}, gen_ai.operation.name = "parse", SpanKind.INTERNAL.

OpenInference. I’d keep Phase 1 OTel GenAI-only. openinference.span.kind can be an additive opt-in follow-up for backends like Phoenix/Langfuse without changing topology or IDs.

Span kind. Agreed on INTERNAL for Parser and Action. I’d use INTERNAL for Tool as well for now, since the Event Log doesn’t tell us whether a tool is local or remote. LLM is the only clearly remote operation, so that can be CLIENT. If tool transport is recorded later in entityMetadata, remote/MCP tools can be classified as CLIENT then.

I’ll fold these into #970’s scope/design notes.