Fullsend

Appearance

108. Tool-call span topology ​

Date: 2026-09-03

Status ​

Accepted

Context ​

ADR 0050 chose OpenTelemetry and a three-level opt-in but never named the spans: the run → sandbox_create → agent tree lives only in the guides, and granularity was left to #294. Level 3 content (#6429, #6603) put tool calls and results on the agent span as gen_ai.output.messages parts — at semconv v1.37.0 the execute_tool span is metadata-only; its content attributes (gen_ai.tool.call.arguments, .result) arrive in v1.38.0 as Opt-In and Development. Review of #6603 asked why tool calls are not spans.

fullsend observes the runtime's stream rather than executing tools. The normalized ToolUseEvent/ToolResultEvent pairs carry a call id, a tool name and an is_error flag; Claude Code's assistant (tool_use) and user (tool_result) stream lines each carry a sandbox-clock timestamp the parser does not decode; pi's tool_execution lines — the only pi lines the parser reads for calls — carry none (its session header and message lines do) and codex's items carry none; several calls are open at once (parallel sub-agent dispatch); parent_tool_use_id is dropped at decode; the pi and codex parsers pass no call ids through.

Options ​

  1. Message record only (status quo): tool calls stay parts of gen_ai.output.messages; no per-call timing or status in the span tree.
  2. execute_tool child spans from the normalized events: one span per id-bearing call under the iteration's agent span, metadata only; content stays on the message record.
  3. The runtime's native OpenTelemetry: Claude Code emits claude_code.tool spans (beta) and honours inbound TRACEPARENT — true timing, but redaction would leave fullsend's pipeline, the threat model keeps runtime telemetry out of scope, and the sandbox would need egress.
  4. Per-tool content on the spans: needs v1.38.0's Opt-In attributes, and the scorers read the message record today.

Decision ​

Option 2. toolSpanTracker (internal/cli/tool_spans.go) opens an execute_tool <tool name> span (kind Internal) when the runtime reports a call and ends it when the result arrives. Both timestamps are runner-side receipt instants: the parent agent span's clock, so a child never falls outside its parent through cross-host skew, and the one source every runtime provides. The cost is timing that is loose in both directions — receipt trails the sandbox's own timestamps by the pipe latency and the parser's decode of the line, and the start also by the console renderer's output for the call, which the runner prints before the tracker stamps the span (it prints nothing for a result). The Level 3 collector runs after the tracker, so its redaction pass over an event never delays that event's own stamp; events are handled one at a time, though, so with several calls open a stamp can still trail the collector's pass over an event queued ahead of it (another call's large result) — a start delayed that way makes the span shorter than the execution, and it can begin after a quick call has already finished. A call still open when the stream ends is closed then, before content assembly. The start is arguments-complete. Decoding Claude Code's timestamps into trace.WithTimestamp would tighten that runtime alone and is left open; upstream documents the assistant-line field as optional, host-clock and display-only, and the user-line field as optional with a receive-time fallback, so a decode must fall back to receipt time. Attributes follow semconv v1.37.0: gen_ai.operation.name=execute_tool, gen_ai.tool.name, gen_ai.tool.call.id; a result flagged is_error sets error.type=tool_error and status Error. Calls that never get a result close as error.type=unanswered, a result whose stream line was too long to decode ends its span marked fullsend.tool.result_oversized with no status (the tool answered; its outcome is unknown), results for calls never reported are marked fullsend.tool.unmatched, events without a call id (pi, codex) get no span, and neither do Claude Code's server-side tools, whose result never arrives as a tool_result — the edge cases are specified in the dev guide. Names and call ids pass through the same sanitizer as span content — names bounded, ids dropped on any finding; at most 1,024 spans are recorded per iteration, so an agent- controlled burst cannot fill the OTLP batch queue and evict the agent span, with the overflow counted in fullsend.tool_spans.dropped; the cap assumes the default OTEL_BSP_MAX_QUEUE_SIZE (2048), and a smaller queue lowers the protection. The spans are Level 1 metadata, emitted regardless of the content gate. Tool content — results now, full arguments next — stays on the agent span's gen_ai.output.messages record, which is the scorer contract (ADR 0087).

Option 3 is the candidate to revisit once the runtime's tracing is stable and a redaction stage outside fullsend is designed; option 4 waits for the convention bump.

Consequences ​

  • The span tree gains one level; readers that select spans by name (evalmeasure) are unaffected, and the execute_tool count can be below fullsend.tool_calls, which counts every reported call, id or not.
  • Tool-heavy iterations add hundreds of spans, each one synchronous write to run-telemetry.jsonl and one OTLP batch entry.
  • Full per-call volume is intended: the spans are the unit of tool-level observability, the cap is a pathological-case backstop rather than a knob, and no switch or sampler is added — sampling is per trace and would drop whole runs, not thin these spans; the levers stay the standard ones (endpoint unset keeps the file only; OTEL_SDK_DISABLED).
  • Sub-agent calls are flat children of the agent span; nesting them under the dispatching Agent call is a small follow-up now that the parent span exists while its children run (ADR 0050's deferred item 1 stays deferred).
  • execute_tool spans are Claude-only until the pi and codex parsers emit call ids and results (#7414).
  • This settles the span-granularity question in #294; retention and access remain open there.