|
| 1 | +--- |
| 2 | +title: pole-instrument Technical Design |
| 3 | +tags: [architecture, python, instrumentation, sidecar] |
| 4 | +links: [pole-python-monorepo] |
| 5 | +updated: 2026-08-20 |
| 6 | +sources: 4 |
| 7 | +--- |
| 8 | + |
| 9 | +# pole-instrument Technical Design |
| 10 | + |
| 11 | +## Status and decisions |
| 12 | + |
| 13 | +Proposed for the first implementation increment. This document defines the runtime and packaging |
| 14 | +contract; it does not add executable instrumentation. |
| 15 | + |
| 16 | +Decisions made now: |
| 17 | + |
| 18 | +- Publish a separate `pole-instrument` distribution with import package `pole_instrument`, console |
| 19 | + entry point `pole-instrument`, and explicit API `pole_instrument.activate()`. |
| 20 | +- Support HTTPX 0.27–0.28 (sync and async request APIs) and gRPC Python 1.71–1.x (sync unary calls) |
| 21 | + initially. Defer gRPC async/streaming, Dubbo, and Thrift automatic adapters. |
| 22 | +- Activation is explicit. Installing either distribution never starts a session or patches a |
| 23 | + framework. `sitecustomize.py`, `usercustomize.py`, `.pth` activation, and implicit import hooks are |
| 24 | + prohibited. |
| 25 | +- Targeted calls fail closed when a valid Sidecar endpoint cannot be obtained. Untargeted calls are |
| 26 | + left unchanged. Activation itself is fail open by default after a bounded diagnostic, with a |
| 27 | + strict option for deployments that require readiness. |
| 28 | +- A live `SidecarSession` belongs to one worker process and must never cross `fork()`. |
| 29 | + |
| 30 | +Open questions for a later ADR are the target declaration syntax for framework integrations beyond |
| 31 | +the initial APIs, Dubbo/Thrift client selection, and whether gRPC async/streaming can preserve all |
| 32 | +call semantics without private framework hooks. |
| 33 | + |
| 34 | +## Goals and non-goals |
| 35 | + |
| 36 | +Goals are explicit opt-in instrumentation, reuse of the stable Thin SDK seam, deterministic routing, |
| 37 | +safe recovery, reversible narrowly scoped adapters, and diagnostics that do not disclose request |
| 38 | +content. The initial release supports Python 3.9–3.13, matching `pole-client-python`. |
| 39 | + |
| 40 | +Non-goals are server instrumentation, target discovery, remote Sidecar configuration, protocol |
| 41 | +contract changes, tracing correlation through `traceparent`, automatic application target guessing, |
| 42 | +and blanket support for every client that speaks HTTP, gRPC, Dubbo, or Thrift. |
| 43 | + |
| 44 | +## Distribution and dependency boundaries |
| 45 | + |
| 46 | +`pole-instrument` depends on a compatible `pole-client-python` minor series and exposes framework |
| 47 | +extras: `pole-instrument[httpx]` and `pole-instrument[grpc]`. Its base install contains the activation, |
| 48 | +registry, lifecycle, configuration, and diagnostic surfaces but no framework client dependency. |
| 49 | +The monorepo releases both distributions independently; compatibility metadata records supported |
| 50 | +Python, framework, Thin SDK, and Sidecar contract versions. Incompatible versions are diagnosed and |
| 51 | +their adapter is not installed. |
| 52 | + |
| 53 | +The existing `pole-client-python` distribution, `pole_client` import path, exports, dependency set, |
| 54 | +and explicit-call behavior remain unchanged. Its wheel must continue to exclude launchers, |
| 55 | +instrumentation modules, startup files, and framework monkey patches. |
| 56 | + |
| 57 | +## Startup and activation semantics |
| 58 | + |
| 59 | +The launcher form is: |
| 60 | + |
| 61 | +```text |
| 62 | +pole-instrument [instrument options] -- python -m application [application arguments] |
| 63 | +``` |
| 64 | + |
| 65 | +The launcher parses only arguments before `--`; it passes the remaining argument vector and |
| 66 | +environment to the child unchanged and never logs them. The API form calls `activate(config)` before |
| 67 | +the application creates instrumented clients. |
| 68 | + |
| 69 | +Configuration precedence is explicit API values, `POLE_INSTRUMENT_*` environment variables, then |
| 70 | +documented defaults. Unknown keys and invalid values fail activation before patching. Secrets are |
| 71 | +not accepted as instrumentation configuration. |
| 72 | + |
| 73 | +Activation performs these steps in order: |
| 74 | + |
| 75 | +1. Validate configuration, process identity, Python/framework compatibility, and requested adapters. |
| 76 | +2. Start one worker-owned `SidecarSession` with bounded initialization. |
| 77 | +3. Install requested adapters in the registry. |
| 78 | +4. Import or execute the application only after successful launcher activation. |
| 79 | +5. Register shutdown callbacks after all owned resources exist. |
| 80 | + |
| 81 | +`activate()` is idempotent for an identical effective configuration and returns the existing runtime. |
| 82 | +A conflicting second activation raises `AlreadyActivatedError` without modifying the active runtime. |
| 83 | +Each adapter installation uses an ownership marker and cannot double-wrap a callable. |
| 84 | + |
| 85 | +Framework modules imported before API activation are supported only where the adapter can replace a |
| 86 | +documented public method safely. Otherwise activation skips that adapter with |
| 87 | +`framework_imported_too_early`; strict mode raises before any partial installation. The launcher is |
| 88 | +the recommended path because it activates before application import. |
| 89 | + |
| 90 | +## Existing public seam and request flow |
| 91 | + |
| 92 | +The runtime owns, but does not reimplement, `pole_client.SidecarSession`. An outbound adapter reads a |
| 93 | +fresh `listener_snapshot()` for each new pooled transport and records its generation. The protocol |
| 94 | +address comes only from `SidecarSession.endpoint(ListenerProtocol...)`; there is no fallback business |
| 95 | +listener, remote override, or cached direct destination. |
| 96 | + |
| 97 | +Target identity is an explicit immutable adapter argument or request extension containing |
| 98 | +`namespace` and `service`. Missing or ambiguous identity leaves an untargeted call untouched; APIs |
| 99 | +declared as targeted raise `MissingTargetError`. Adapters never infer a target from a URL hostname, |
| 100 | +authority, method name, tracing fields, or application package. |
| 101 | + |
| 102 | +At the final outbound request assembly point, adapters construct `TargetService(namespace, service)` |
| 103 | +and call `to_metadata()` for HTTP or `to_grpc_metadata()` for gRPC. The optional explicit |
| 104 | +`TrafficContext` is passed at that same point; otherwise the Thin SDK reads the current scope. |
| 105 | +Consequently caller metadata is preserved except for canonical replacement of reserved target keys |
| 106 | +and cleanup/replacement of reserved TrafficContext baggage. `traceparent` remains unrelated data. |
| 107 | +No adapter imports generated protobuf modules or depends on private `pole_client` state. |
| 108 | + |
| 109 | +## Adapter contract and support matrix |
| 110 | + |
| 111 | +| Protocol / adapter | Initial status and versions | Interception and target source | Semantics and exclusions | |
| 112 | +|---|---|---|---| |
| 113 | +| HTTP / HTTPX | Supported: HTTPX 0.27–0.28, sync and async | Public `Client.send` / `AsyncClient.send`; explicit request extension or bound client target | Rewrites scheme/authority to the Sidecar HTTP endpoint at send time and uses `TargetService.to_metadata`. Redirect hops require an explicit retained target. Existing client pools are partitioned by snapshot generation and closed on invalidation. Custom transports are unsupported unless they opt into the adapter wrapper. | |
| 114 | +| gRPC Python | Supported: grpcio 1.71–1.x, sync unary-unary only | Public client interceptor; explicit target bound when constructing the instrumented channel | Creates the channel against the Sidecar gRPC endpoint and injects `to_grpc_metadata`. User retry policy is retained. Async, client/server/bidi streaming, and user-supplied existing channels are rejected as unsupported, not partially patched. | |
| 115 | +| Dubbo | Deferred | No initial client API selected | `ListenerProtocol.DUBBO` availability does not imply framework support. No patching or routing claim. | |
| 116 | +| Thrift | Deferred | No initial client API selected | `ListenerProtocol.THRIFT` availability does not imply framework support. No patching or routing claim. | |
| 117 | + |
| 118 | +HTTPX uninstrumentation restores only methods owned by this runtime when no later patch has replaced |
| 119 | +them. gRPC interceptors are attached to newly created instrumented channels, so shutdown closes only |
| 120 | +owned channels. Retries reuse canonical metadata but perform endpoint availability/generation checks |
| 121 | +before opening a new transport. Unsupported streaming and custom-transport cases fail at adapter |
| 122 | +construction, before a request can be rerouted incorrectly. |
| 123 | + |
| 124 | +## Failure and recovery semantics |
| 125 | + |
| 126 | +Activation defaults to fail open: if the socket is absent, initialization times out, or the first |
| 127 | +snapshot is invalid, no adapters remain installed and the application starts with an error diagnostic. |
| 128 | +Strict activation fails closed and does not start the application. Rollback of partial activation is |
| 129 | +reverse ordered and closes the session. |
| 130 | + |
| 131 | +Routing is stricter. Once a request is explicitly targeted, `SidecarUnavailableError`, an invalid |
| 132 | +snapshot, or adapter failure raises a targeted-routing error. It must never fall back to the original |
| 133 | +destination. An in-flight request already accepted by the local Sidecar follows the underlying client |
| 134 | +result; instrumentation does not replay it automatically. |
| 135 | + |
| 136 | +On control-stream disconnect, the Thin SDK invalidates the snapshot. The next targeted call fails |
| 137 | +before acquiring a stale pooled connection. On snapshot-generation change, adapters atomically mark |
| 138 | +all older-generation pools/channels draining, prevent new acquisition from them, close them when |
| 139 | +their current users release them, and create transports using the new endpoint. Reconnection is |
| 140 | +exclusively the existing `SidecarSession` lifecycle. |
| 141 | + |
| 142 | +An adapter error affects only calls using that adapter. It is reported with a stable reason code and |
| 143 | +does not disable unrelated adapters or mutate untargeted requests. |
| 144 | + |
| 145 | +## Process, concurrency, and fork lifecycle |
| 146 | + |
| 147 | +Activation records the owning PID. The launcher activates inside the final application worker. A |
| 148 | +Gunicorn-style master must use a worker post-fork hook or application factory; activation in the |
| 149 | +master is rejected with `prefork_parent_unsupported`. If a fork occurs after activation, the child |
| 150 | +detects the PID mismatch before the next adapter operation, discards inherited locks/channels without |
| 151 | +using them, and requires child reactivation. The parent retains ownership of its resources. |
| 152 | + |
| 153 | +Registry and generation transitions are protected by locks, while request-specific target and |
| 154 | +`TrafficContext` data remains in arguments and the Thin SDK's `contextvars`/OTel storage. Threads and |
| 155 | +asyncio tasks therefore do not share mutable request context. Nested scopes retain the existing |
| 156 | +restore behavior. |
| 157 | + |
| 158 | +Shutdown is idempotent: stop accepting new targeted calls, remove owned patches, close owned client |
| 159 | +resources, then close `SidecarSession`. Each close is bounded; interpreter shutdown is not held open |
| 160 | +by non-daemon instrumentation threads. |
| 161 | + |
| 162 | +## Diagnostics and security |
| 163 | + |
| 164 | +`pole_instrument.status()` returns immutable activation state, PID, enabled adapter names and |
| 165 | +versions, Sidecar availability, current snapshot generation, and the last stable reason code. Logs |
| 166 | +use the same reason codes for activation, compatibility decisions, generation changes, skipped |
| 167 | +adapters, and shutdown. |
| 168 | + |
| 169 | +By default diagnostics never include target namespace/service values, baggage, credentials, request |
| 170 | +headers, URLs with query strings, application arguments, or environment contents. Debug mode may |
| 171 | +include framework method names and numeric generations but not request data. Metrics use bounded |
| 172 | +adapter and reason-code labels only. |
| 173 | + |
| 174 | +Only validated loopback endpoints returned by `SidecarSession` are trusted. Caller-supplied endpoint |
| 175 | +overrides are not part of the API. Canonical Thin SDK metadata assembly defeats case-variant forged |
| 176 | +target fields and removes stale reserved baggage. Patches are limited to documented public seams, |
| 177 | +carry ownership markers, and are reversible when no third party has subsequently replaced them. |
| 178 | + |
| 179 | +## Compatibility, rollout, and rollback |
| 180 | + |
| 181 | +Users who install only `pole-client-python`, or install `pole-instrument` without invoking its CLI/API, |
| 182 | +observe no behavior change. Rollout is opt-in per adapter and supports `POLE_INSTRUMENT_ENABLED=0` as |
| 183 | +an emergency disable switch below explicit API configuration. Operators can use one worker group as |
| 184 | +a canary, inspect status/reason codes, then expand adoption. |
| 185 | + |
| 186 | +Rollback disables activation or removes the separate distribution; no Thin SDK downgrade is required. |
| 187 | +CI builds both distributions in isolation and checks their contents, entry points, optional dependency |
| 188 | +resolution, Python matrix, framework matrix, and declared version compatibility. Release notes state |
| 189 | +supported combinations and reject incompatible major contract versions. |
| 190 | + |
| 191 | +Alternatives considered: |
| 192 | + |
| 193 | +- Implicit `sitecustomize`/`.pth` activation was rejected because import order, process ownership, |
| 194 | + debugging, and rollback become nondeterministic. |
| 195 | +- An explicit launcher/API was selected because it provides a testable lifecycle and a clean no-op |
| 196 | + path for existing users. |
| 197 | +- Direct per-application framework integration remains valid for unsupported clients and is safer |
| 198 | + than pretending a generic adapter can preserve unknown transport semantics. |
| 199 | + |
| 200 | +## Verification strategy |
| 201 | + |
| 202 | +Every implementation behavior begins with a focused test at its public seam: |
| 203 | + |
| 204 | +| Behavior | Smallest test | Broader evidence | |
| 205 | +|---|---|---| |
| 206 | +| Configuration and idempotent activation | Unit test `activate()` with precedence, duplicate, conflict, and unsupported-version cases | Launcher subprocess integration | |
| 207 | +| HTTPX target routing | One sync and one async request through a recording public transport | Real HTTP listener, redirects/retries, pool-generation change | |
| 208 | +| gRPC target routing | Unary public interceptor call against a recording server | Real UDS Sidecar fixture plus loopback gRPC listener | |
| 209 | +| Metadata/context integrity | Assert adapter output through `TargetService` public methods | Existing conformance vectors and concurrent context isolation | |
| 210 | +| Missing target | Public instrumented-client call without identity | Framework integration negative suite | |
| 211 | +| Sidecar unavailable/disconnect | Public adapter call with a `SidecarSession` fixture whose stream closes | E2E proves no stale endpoint use and later-generation recovery | |
| 212 | +| Fork ownership | PID-change unit test before adapter acquisition | Gunicorn-style master/worker subprocess test | |
| 213 | +| Diagnostics/redaction | Public `status()` and captured log test with secret-shaped inputs | Packaging/integration log audit | |
| 214 | +| Shutdown/uninstrumentation | Repeated public `close()` and method identity assertion | Worker termination timeout test | |
| 215 | +| Packaging compatibility | Install each wheel in an isolated environment | Python 3.9–3.13 and supported framework CI matrix | |
| 216 | + |
| 217 | +The public-seam E2E starts a real UDS gRPC `SidecarSessionService`, activates through the launcher or |
| 218 | +public API, makes a supported client call, and observes the validated loopback endpoint plus canonical |
| 219 | +TargetService and TrafficContext metadata. It then terminates the control stream, proves the next |
| 220 | +targeted call cannot use the stale endpoint, publishes a new snapshot, and proves recovery uses the |
| 221 | +new generation. |
| 222 | + |
| 223 | +Test doubles are appropriate for configuration, registry, PID, diagnostic, and adapter assembly unit |
| 224 | +tests. Availability, disconnect, reconnection, snapshot generation, and claimed framework routing |
| 225 | +require real public-seam fixtures. Unsupported gRPC streaming, custom transports/channels, Dubbo, |
| 226 | +Thrift, early-import incompatibility, double activation, and ambiguous identity remain mandatory |
| 227 | +negative cases rather than untested implied support. |
| 228 | + |
| 229 | +## Related pages |
| 230 | + |
| 231 | +- [[pole-python-monorepo]] |
0 commit comments