Skip to content

Commit e48caf6

Browse files
feat(otel): client-side redaction of PII and/or secrets (#584)
* feat: first shot * fix: tweaks * fix: resolve redaction and policy changes * feat: add secret regexes to default redaction patterns Fold AWS, Google, JWT, PEM and Stripe key patterns into DEFAULT_TOKEN_PATTERNS and compose DEFAULT_PII_SECRET_PATTERNS from it, with tests covering each secret and the composition invariant. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai> * refactor: type RedactingSpanExporter against SpanExporter base Use a conditional base (SpanExporter under TYPE_CHECKING, object at runtime) so linters verify the export/shutdown/force_flush overrides while keeping the OpenTelemetry SDK an optional import. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai> * feat: redact string elements of homogeneous sequences Add a shared _redact_value helper so both policies scan the string elements of list/tuple attribute values instead of passing them through verbatim, preserving the container type and leaving numeric/bool sequences untouched. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai> * test: convert redaction tests to pytest style Rewrite test_redaction.py as pytest classes with fixtures and parametrization, and convert the TestTelemetryRedaction class in test_telemetry.py to a plain pytest class (using caplog for log assertions). Optional span attributes/context are narrowed with asserts instead of file-level pyright suppressions. Older telemetry tests are left as-is. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai> * fix: tweaks * fix: additional sensitive keys * docs: add observability docs * feat: more secret patterns * feat: make RegexRedactionPolicy the default redaction policy Switch default_redaction_policy() from the key-oriented AttributeRedactionPolicy to the content-oriented RegexRedactionPolicy, which preserves keys/structure and redacts only matched secret/PII substrings. Update docstrings, README policy table, the example, and adapt the behavioural tests accordingly. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai> * chore: split in 2 files and adjust public interface * docs: more detailed redaction docs * test: reconfiguring dedicated telemetry should apply new redaction Regression test for the case where a second configure_telemetry call in dedicated mode is ignored because of the early return on an existing auto provider. Currently fails: the provider is not rebuilt so the new redaction setting never takes effect. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai> * fix: apply new settings when reconfiguring dedicated telemetry The early return on an existing auto telemetry provider fired before the replace_existing check, so an explicit configure_telemetry call (which always requests replacement) was a no-op: the provider was never rebuilt and a changed redaction setting was silently ignored. Only short-circuit when replace_existing is False, letting the rebuild path shut down the old provider and recreate it with the new settings. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai> * feat: optional metadata to attribute redaction policy * fix: typing in test --------- Co-authored-by: Mistral Vibe <vibe@mistral.ai>
1 parent de4f18d commit e48caf6

11 files changed

Lines changed: 1585 additions & 10 deletions

File tree

‎README.md‎

Lines changed: 141 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ Mistral AI API: Our Chat Completion and Embeddings APIs specification. Create yo
4646
* [Resource Management](#resource-management)
4747
* [Debugging](#debugging)
4848
* [IDE Support](#ide-support)
49+
* [Telemetry \& Observability](#telemetry--observability)
4950
* [Development](#development)
5051
* [Contributions](#contributions)
5152

@@ -836,7 +837,7 @@ print(res.choices[0].message.content)
836837
operations. These operations will expose the stream as [Generator][generator] that
837838
can be consumed using a simple `for` loop. The loop will
838839
terminate when the server no longer has any events to send and closes the
839-
underlying connection.
840+
underlying connection.
840841

841842
The stream is also a [Context Manager][context-manager] and can be used with the `with` statement and will close the
842843
underlying connection when the context is exited.
@@ -1277,9 +1278,146 @@ Generally, the SDK will work well with most IDEs out of the box. However, when u
12771278

12781279
<!-- Placeholder for Future Speakeasy SDK Sections -->
12791280

1281+
1282+
## Telemetry & Observability
1283+
1284+
The SDK can emit [OpenTelemetry](https://opentelemetry.io/) traces for the API calls it makes (chat, agents, embeddings, OCR, …), following the
1285+
[GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/).
1286+
Spans capture the operation, model, token usage, and — unless redacted — the input/output messages and tool calls. Telemetry is **opt-in** and lives in the `mistralai.extra.observability` module.
1287+
1288+
### Installation
1289+
1290+
Install the `telemetry` extra:
1291+
1292+
```bash
1293+
pip install "mistralai[telemetry]"
1294+
# or: uv add "mistralai[telemetry]"
1295+
```
1296+
1297+
### Enabling telemetry
1298+
1299+
Either set an environment variable before creating the client:
1300+
1301+
```bash
1302+
export MISTRAL_SDK_TELEMETRY=dedicated # dedicated | global | false
1303+
```
1304+
1305+
or configure it in code:
1306+
1307+
```python
1308+
import os
1309+
from mistralai.client import Mistral
1310+
from mistralai.extra.observability import configure_telemetry
1311+
1312+
with Mistral(api_key=os.environ["MISTRAL_API_KEY"]) as client:
1313+
# Dedicated mode (default): the SDK creates and owns an OTLP exporter that
1314+
# ships spans to the Mistral telemetry endpoint. Spans are redacted before
1315+
# export.
1316+
configure_telemetry(client)
1317+
1318+
client.chat.complete(
1319+
model="mistral-small-latest",
1320+
messages=[{"role": "user", "content": "Hello!"}],
1321+
)
1322+
```
1323+
1324+
### Provider modes
1325+
1326+
`configure_telemetry(client, provider=...)` selects where spans go and who owns the export pipeline:
1327+
1328+
| `provider` | Who owns the exporter | Where spans go | Redaction |
1329+
| ---------- | --------------------- | -------------- | --------- |
1330+
| `"dedicated"` (default) | The SDK | Mistral telemetry endpoint | Applied automatically |
1331+
| `"global"` | Your application | Your global OpenTelemetry provider | **Not** applied — you need to wrap your own exporter |
1332+
| a `TracerProvider` | Your application | The provider you pass | **Not** applied — you need to wrap your own exporter |
1333+
1334+
In `global`/custom modes your application owns the pipeline, so the `redaction` argument is ignored (a warning is logged). Wrap your own exporter with `RedactingSpanExporter` to redact spans there:
1335+
1336+
```python
1337+
from opentelemetry import trace
1338+
from opentelemetry.sdk.trace import TracerProvider
1339+
from opentelemetry.sdk.trace.export import BatchSpanProcessor
1340+
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
1341+
from mistralai.extra.observability import RedactingSpanExporter, configure_telemetry
1342+
1343+
provider = TracerProvider()
1344+
provider.add_span_processor(
1345+
BatchSpanProcessor(RedactingSpanExporter(OTLPSpanExporter()))
1346+
)
1347+
trace.set_tracer_provider(provider)
1348+
1349+
# SDK spans now flow through your global provider (already redacted above).
1350+
configure_telemetry(client, provider="global")
1351+
```
1352+
1353+
### Redaction
1354+
1355+
In dedicated mode, redaction is on by default. Control it with the `redaction` argument, which also accepts any of the reusable policies from `mistralai.extra.observability`:
1356+
1357+
```python
1358+
from mistralai.extra.observability import AttributeRedactionPolicy
1359+
1360+
configure_telemetry(client) # default policy (regex)
1361+
configure_telemetry(client, redaction=AttributeRedactionPolicy()) # very conservative key-oriented policy
1362+
configure_telemetry(client, redaction=False) # disabled - no redaction
1363+
configure_telemetry( # custom callback to control how attributes are redacted
1364+
client,
1365+
redaction=lambda key, value: None if "email" in key else value,
1366+
)
1367+
```
1368+
1369+
| Policy | Strategy | Trade-off |
1370+
| ------ | -------- | --------- |
1371+
| `RegexRedactionPolicy` (default, `redaction=True`) | Content-oriented: keeps keys and structure, redacts matched substrings (secret tokens plus PII — emails, card-like sequences, IPv4). | Redacts most sensitive data while preserving observability value; may miss free-form PII or secrets not in the pattern set. |
1372+
| `AttributeRedactionPolicy` | Key-oriented: redacts whole values for sensitive keys (explicit set, fragment match, or non-primitive value), then scans kept values for secret token patterns. | Very conservative, but erases most prompt/response content. |
1373+
| `CallbackRedactionPolicy` (`redaction=<callable>`) | Your `(key, value) -> value \| None` masker per attribute; return `None` to drop the attribute. | Full control; you own the logic. |
1374+
1375+
The built-in defaults are exported as constants, so you can extend them instead of replacing them wholesale:
1376+
1377+
```python
1378+
import re
1379+
1380+
from mistralai.extra.observability import (
1381+
DEFAULT_PII_SECRET_PATTERNS,
1382+
DEFAULT_SENSITIVE_ATTRIBUTE_KEYS,
1383+
AttributeRedactionPolicy,
1384+
RegexRedactionPolicy,
1385+
)
1386+
1387+
# Content-oriented: add a custom secret pattern to the default set.
1388+
configure_telemetry(
1389+
client,
1390+
redaction=RegexRedactionPolicy(
1391+
patterns=(*DEFAULT_PII_SECRET_PATTERNS, re.compile(r"\bacme-[a-z0-9]{16}\b")),
1392+
),
1393+
)
1394+
1395+
# Key-oriented: mask an extra application attribute on top of the defaults.
1396+
configure_telemetry(
1397+
client,
1398+
redaction=AttributeRedactionPolicy(
1399+
sensitive_keys=DEFAULT_SENSITIVE_ATTRIBUTE_KEYS | {"app.customer.email"},
1400+
),
1401+
)
1402+
```
1403+
1404+
*Note: the `RedactingSpanExporter` primitive is reusable by any OpenTelemetry application, independent of the Mistral client.*
1405+
1406+
### Environment variables
1407+
1408+
| Variable | Description | Default |
1409+
| -------- | ----------- | ------- |
1410+
| `MISTRAL_SDK_TELEMETRY` | Auto-enable telemetry: `dedicated`, `global`, or `false`. | unset (disabled) |
1411+
| `MISTRAL_OTLP_TRACES_ENDPOINT` | Override the OTLP traces endpoint used in dedicated mode. | `https://api.mistral.ai/telemetry/v1/traces` |
1412+
| `MISTRAL_SDK_DEBUG_TRACING` | Set to `true` for verbose tracing logs. | `false` |
1413+
| `MISTRAL_API_KEY` | Used as the bearer token for the dedicated-mode exporter. | — |
1414+
1415+
Runnable examples live in [`examples/mistral/observability`](/examples/mistral/observability).
1416+
1417+
12801418
# Development
12811419

12821420
## Contributions
12831421

1284-
While we value open-source contributions to this SDK, this library is generated programmatically. Any manual changes added to internal files will be overwritten on the next generation.
1285-
We look forward to hearing your feedback. Feel free to open a PR or an issue with a proof of concept and we'll do our best to include it in a future release.
1422+
While we value open-source contributions to this SDK, this library is generated programmatically. Any manual changes added to internal files will be overwritten on the next generation.
1423+
We look forward to hearing your feedback. Feel free to open a PR or an issue with a proof of concept and we'll do our best to include it in a future release.
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
#!/usr/bin/env python
2+
"""Dedicated telemetry mode.
3+
4+
The SDK creates and owns an OTLP exporter that ships spans to the Mistral
5+
telemetry endpoint. Spans are redacted before export.
6+
7+
Requires the telemetry extra: pip install "mistralai[telemetry]"
8+
"""
9+
10+
import os
11+
12+
from mistralai.client import Mistral
13+
from mistralai.extra.observability import configure_telemetry
14+
15+
16+
def main() -> None:
17+
api_key = os.environ["MISTRAL_API_KEY"]
18+
19+
with Mistral(api_key=api_key) as client:
20+
# Dedicated mode is the default; redaction is on by default.
21+
configure_telemetry(client)
22+
23+
response = client.chat.complete(
24+
model="mistral-small-latest",
25+
messages=[{"role": "user", "content": "What is the best French cheese?"}],
26+
)
27+
print(response.choices[0].message.content)
28+
29+
30+
if __name__ == "__main__":
31+
main()
Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
#!/usr/bin/env python
2+
"""Global provider mode with application-owned redaction.
3+
4+
In global (or custom TracerProvider) mode your application owns the OTEL export
5+
pipeline, so `configure_telemetry`'s redaction argument is ignored. To redact
6+
spans you wrap your own exporter with RedactingSpanExporter.
7+
8+
Requires the telemetry extra: pip install "mistralai[telemetry]"
9+
"""
10+
11+
import os
12+
13+
from opentelemetry import trace
14+
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
15+
from opentelemetry.sdk.trace import TracerProvider
16+
from opentelemetry.sdk.trace.export import BatchSpanProcessor
17+
18+
from mistralai.client import Mistral
19+
from mistralai.extra.observability import RedactingSpanExporter, configure_telemetry
20+
21+
22+
def main() -> None:
23+
api_key = os.environ["MISTRAL_API_KEY"]
24+
25+
# Build your own provider and wrap the exporter with redaction.
26+
provider = TracerProvider()
27+
provider.add_span_processor(
28+
BatchSpanProcessor(RedactingSpanExporter(OTLPSpanExporter()))
29+
)
30+
trace.set_tracer_provider(provider)
31+
32+
with Mistral(api_key=api_key) as client:
33+
# SDK spans flow through the global provider configured above.
34+
configure_telemetry(client, provider="global")
35+
36+
response = client.chat.complete(
37+
model="mistral-small-latest",
38+
messages=[{"role": "user", "content": "Say hello."}],
39+
)
40+
print(response.choices[0].message.content)
41+
42+
43+
if __name__ == "__main__":
44+
main()
Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
#!/usr/bin/env python
2+
"""Choosing and extending a redaction policy in dedicated telemetry mode.
3+
4+
The `redaction` argument of `configure_telemetry` accepts:
5+
- True (default): the regex (content-oriented) policy
6+
- False: redaction disabled
7+
- a RedactionPolicy instance (e.g. AttributeRedactionPolicy)
8+
- a (key, value) -> value | None callback
9+
10+
Built-in policies expose their defaults as public constants so you can
11+
extend them instead of replacing them wholesale:
12+
- AttributeRedactionPolicy: DEFAULT_SENSITIVE_ATTRIBUTE_KEYS,
13+
DEFAULT_SENSITIVE_ATTRIBUTE_FRAGMENTS, DEFAULT_SAFE_ATTRIBUTE_KEYS,
14+
DEFAULT_TOKEN_PATTERNS
15+
- RegexRedactionPolicy: DEFAULT_PII_SECRET_PATTERNS
16+
17+
CallbackRedactionPolicy is different in kind: instead of extending constants you
18+
supply a `(key, value) -> value | None` function and decide per attribute. It is
19+
the only policy that can return None to drop an attribute entirely; the built-in
20+
policies replace values in place and never remove keys.
21+
22+
`demonstrate()` applies the policies directly to a sample attribute mapping and
23+
prints the before/after; it needs no API key and no network. `main()` wires an
24+
extended policy into `configure_telemetry` and issues a live request (requires
25+
the telemetry extra: pip install "mistralai[telemetry]").
26+
"""
27+
28+
import os
29+
import re
30+
31+
from opentelemetry.util.types import AttributeValue
32+
33+
from mistralai.client import Mistral
34+
from mistralai.extra.observability import (
35+
DEFAULT_PII_SECRET_PATTERNS,
36+
DEFAULT_SENSITIVE_ATTRIBUTE_KEYS,
37+
AttributeRedactionPolicy,
38+
CallbackRedactionPolicy,
39+
RegexRedactionPolicy,
40+
configure_telemetry,
41+
)
42+
43+
# A custom attribute key your application sets and wants masked, on top of the
44+
# built-in sensitive keys.
45+
CUSTOMER_EMAIL_KEY = "app.customer.email"
46+
47+
# A custom secret shape (e.g. an internal token) the default patterns don't know.
48+
ACME_TOKEN_PATTERN = re.compile(r"\bacme-[a-z0-9]{16}\b")
49+
50+
51+
def build_attribute_policy() -> AttributeRedactionPolicy:
52+
"""Key-oriented policy: defaults plus one extra sensitive key."""
53+
return AttributeRedactionPolicy(
54+
sensitive_keys=DEFAULT_SENSITIVE_ATTRIBUTE_KEYS | {CUSTOMER_EMAIL_KEY},
55+
)
56+
57+
58+
def build_regex_policy() -> RegexRedactionPolicy:
59+
"""Content-oriented policy: default patterns plus one extra secret shape."""
60+
return RegexRedactionPolicy(
61+
patterns=(*DEFAULT_PII_SECRET_PATTERNS, ACME_TOKEN_PATTERN),
62+
)
63+
64+
65+
def custom_mask(key: str, value: AttributeValue) -> AttributeValue | None:
66+
"""Callback: full control over each attribute.
67+
68+
Return the value to keep it, a transformed value to mask it, or None to drop
69+
the attribute entirely (something the built-in policies cannot do).
70+
"""
71+
# Drop your application's private namespace outright.
72+
if key.startswith("app."):
73+
return None
74+
# Leave model/usage metadata untouched.
75+
if key.startswith(("gen_ai.request", "gen_ai.usage")):
76+
return value
77+
# Mask anything else with your own marker.
78+
return "***"
79+
80+
81+
def build_callback_policy() -> CallbackRedactionPolicy:
82+
"""Callback-based policy wrapping the custom_mask function above."""
83+
return CallbackRedactionPolicy(custom_mask)
84+
85+
86+
def demonstrate() -> None:
87+
"""Show, without any network call, what each extended policy redacts."""
88+
attributes = {
89+
# Safe key, survives both policies.
90+
"gen_ai.request.model": "mistral-small-latest",
91+
# Built-in sensitive key: whole value dropped by the attribute policy.
92+
"gen_ai.input.messages": "user: my card is 4111 1111 1111 1111",
93+
# Custom key we added to the attribute policy's sensitive set.
94+
CUSTOMER_EMAIL_KEY: "jane@example.com",
95+
# Free-form value carrying secrets: caught by the regex policy's patterns.
96+
"http.request.body": "Authorization: Bearer sk-abcdefghijklmnopqrst; "
97+
"internal=acme-0123456789abcdef",
98+
}
99+
100+
for label, policy in (
101+
("AttributeRedactionPolicy (extended keys)", build_attribute_policy()),
102+
("RegexRedactionPolicy (extended patterns)", build_regex_policy()),
103+
("CallbackRedactionPolicy (custom function)", build_callback_policy()),
104+
):
105+
print(f"\n{label}")
106+
redacted = policy.redact_attributes(attributes)
107+
for key in attributes:
108+
# A callback may drop a key entirely (returns None), so it can be
109+
# absent from the redacted mapping.
110+
after = redacted[key] if key in redacted else "<dropped>"
111+
print(f" {key}")
112+
print(f" before: {attributes[key]}")
113+
print(f" after : {after}")
114+
115+
116+
def main() -> None:
117+
demonstrate()
118+
119+
api_key = os.environ.get("MISTRAL_API_KEY")
120+
if not api_key:
121+
print("\nSet MISTRAL_API_KEY to run the live configure_telemetry example.")
122+
return
123+
124+
with Mistral(api_key=api_key) as client:
125+
# Wire the extended attribute policy into dedicated telemetry mode.
126+
configure_telemetry(client, redaction=build_attribute_policy())
127+
128+
# Alternatives:
129+
# configure_telemetry(client, redaction=build_regex_policy())
130+
# configure_telemetry(client, redaction=custom_mask) # bare callback
131+
# configure_telemetry(client, redaction=build_callback_policy())
132+
# configure_telemetry(client, redaction=False) # disable entirely
133+
134+
response = client.chat.complete(
135+
model="mistral-small-latest",
136+
messages=[{"role": "user", "content": "Say hello."}],
137+
)
138+
print(response.choices[0].message.content)
139+
140+
141+
if __name__ == "__main__":
142+
main()

0 commit comments

Comments
 (0)