Self-hosted agent observability gives your team responsibility for running the monitoring system and controlling its deployment. It does not automatically keep every data flow inside your network or eliminate operating costs. Model-based evaluators, object storage, authentication, and external tools may create additional boundaries.
Evaluate the complete data path and the team’s ability to operate it before choosing a deployment model.
Draw the data path first
List the application, collector, ingestion queue, storage, query service, dashboard, evaluator, and identity provider. For every connection, record which data crosses it and who operates the destination.
If an evaluator sends transcripts to an external model provider, a locally hosted trace database does not keep that evaluation traffic local. If the dashboard loads attachments from a separate bucket, that bucket needs the same scrutiny as the database.
Use your actual deployment architecture rather than assuming all features share one container or one network boundary.
Compare responsibilities, not only license prices
| Responsibility | Managed service question | Self-hosted question |
|---|---|---|
| Availability | What service commitments and failure visibility exist? | Who handles outages and capacity changes? |
| Upgrades | How are changes introduced and communicated? | Who tests migrations and rollback? |
| Backups | What recovery commitments apply? | Have restores been tested? |
| Access control | What controls are available on your plan? | Who configures and audits them? |
| Evaluation | Where do model calls and code execution run? | What isolation and egress controls exist? |
| Retention | What deletion and retention behavior is supported? | Does deletion cover every storage layer? |
These are evaluation questions, not claims that either model is inherently safer or cheaper.
Understand the actual product architecture
For one concrete example, Langfuse’s self-hosting documentation describes application services and multiple storage components, and distinguishes local Docker Compose use from production deployment options. That illustrates why “we can run a container” is not a complete operations plan.
Check the documentation for the exact version you intend to operate. SDK and server compatibility, migration requirements, and feature licensing can affect the rollout. Do not base a production design on an old tutorial that happens to start successfully on a laptop.
For Agnost deployment and procurement requirements, use the security contact and public guidance to confirm the current offering. A generic reference to enterprise or VPC deployment should not be treated as a complete contract for your environment.
Test failures in the monitoring system
Simulate an unavailable collector, full queue, slow storage, dropped batch, and expired credentials. Confirm how the application behaves and how the monitoring owner learns about the problem.
Instrumentation should have an explicit policy for backpressure and export failure. Losing telemetry is bad; unexpectedly blocking the primary user workflow can be worse. Make the tradeoff deliberate and observable.
Measure ingestion lag and incomplete traces. A system that accepts data but takes hours to make it searchable may not meet an incident-response requirement even if eventual storage is reliable.
Test recovery and deletion
Restore a backup into an isolated environment and verify the records you expect. Check what happens to attachments, derived evaluations, caches, and exports when the source record is removed. Retention is a property of the full data lifecycle, not one database table.
This is an engineering validation exercise. Legal or contractual retention requirements need the relevant agreement and your organization’s review; a blog checklist cannot establish compliance.
Make the deployment decision with a workload
Estimate trace volume, spans per trace, content size, retention, query concurrency, evaluation traffic, and growth. Include engineering time, on-call ownership, infrastructure, and backup costs. Compare that workload with a managed quote using the same assumptions.
The monitoring pilot checklist helps turn those requirements into observed results. Before enabling content capture in either deployment model, implement the redaction boundary. Hosting location matters, but the most useful decision considers evidence quality, operational ownership, and every place the data travels.