Redact or omit sensitive data before it crosses your telemetry export boundary. Keep approved operational fields and enough consistent structure to investigate failures. Do not assume that removing an email address from the user message also removes it from tool arguments, results, errors, or copied metadata.
This guide describes a technical implementation pattern. Your organization’s data-handling requirements determine which content is permitted to leave the application.
Inventory every capture path
An agent integration may capture prompts, completions, retrieved documents, tool payloads, exception messages, model attributes, and user traits. Automatic instrumentation can introduce new fields without a developer explicitly adding a logging statement.
Inspect a real staging export at the wire boundary. Review the serialized payload rather than only the dashboard display, because a UI may hide fields that were still transmitted.
Agnost’s data-governance documentation states that it does not currently provide automatic PII redaction or DLP before ingestion. Apply your policy in the application or export pipeline before sending data.
Prefer an allowlist for structured metadata
Begin with fields whose purpose is clear: operation type, duration, application version, result state, and pseudonymous relationship identifiers. Avoid forwarding arbitrary dictionaries such as request headers or complete user objects.
SAFE_FIELDS = {"tool_name", "status", "latency_ms", "agent_version"}
def operational_metadata(record):
return {key: record[key] for key in SAFE_FIELDS if key in record}
sample = {"tool_name": "lookup_policy", "status": "ok", "token": "secret"}
assert operational_metadata(sample) == {"tool_name": "lookup_policy", "status": "ok"}This example only filters known structured fields. It is not a general-purpose PII detector and does not make arbitrary text safe. Validate permitted field values too; an error string can contain sensitive content even when its key is allowlisted.
Preserve relationships without exposing identities
Use stable internal pseudonymous IDs where the investigation requires joining turns or actions. Keep the mapping to a real person in your own system. Treat pseudonymous identifiers as controlled data, not anonymous public content.
For a wrong-order failure, the reviewer needs to know that the requested order and the result order differ. They usually do not need the customer’s email or shipping address. Replace identifiers consistently so redaction does not erase the distinction you are trying to test.
Do not hash low-entropy values and assume the result is anonymous. If tokenization is appropriate, design its scope, key management, rotation, and access controls with your data team.
Decide what text evidence is necessary
| Investigation | Potentially sufficient evidence |
|---|---|
| Duplicate policy reads | Tool name, scoped normalized inputs, version, internal result fingerprint |
| False completion claim | Approved claim excerpt, action identity, verified result state |
| Wrong policy answer | Relevant policy excerpt and answer claim, with personal context removed |
| Excessive latency | Timestamps, dependency structure, operation labels |
Some semantic investigations genuinely need text. Capture the minimum approved excerpt and explain the resulting coverage limits. Metadata-only telemetry cannot establish every conversation-level failure.
OpenTelemetry’s sensitive-data guidance describes handling sensitive information in telemetry. Review both application instrumentation and collector processing so data is not retained or exported earlier than intended.
Test redaction as an export contract
Create fixtures with secrets in nested tool arguments, arrays, URLs, exception text, and copied user messages. Assert that disallowed values do not appear anywhere in the serialized export. Also verify that allowed fields remain useful and that identifiers remain consistent where required.
If the redactor fails, define the fallback explicitly. Dropping sensitive content and emitting a safe diagnostic counter is often preferable to exporting the original payload silently. Monitor that counter so redaction failures do not become invisible telemetry loss.
Reinspect exports after SDK or instrumentation upgrades. The monitoring pilot checklist includes this check alongside evidence completeness. Choose sampling separately: collecting fewer sensitive records is not the same as making each collected record appropriate to export.