What do I need to send my first span?
Three things: an API key, the SDK, and an actual LLM call. The last one catches most people.
- Create a workspace and an API key at app.sentrinode.com → API Keys. The raw key is shown once.
- Install the SDK and your provider library.
sentrinode-llmdoes not pull the provider in for you. - Put the code in a file and run that file.
pip install sentrinode-llm anthropic
# test_llm.py
import os, sentrinode_llm, anthropic
sentrinode_llm.instrument(
tenant="your-workspace-slug",
api_key=os.environ["SENTRINODE_API_KEY"],
)
client = anthropic.Anthropic()
client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=64,
messages=[{"role": "user", "content": "hello"}],
)
# PowerShell
$env:SENTRINODE_API_KEY="snk_..."; $env:ANTHROPIC_API_KEY="sk-ant-..."; python test_llm.py
# bash / zsh
export SENTRINODE_API_KEY="snk_..."; export ANTHROPIC_API_KEY="sk-ant-..."; python test_llm.py
The call appears on the dashboard within a few seconds.
“The term 'import' is not recognized as the name of a cmdlet”
You pasted Python into PowerShell. PowerShell has no import statement, and f(x="y") is not its call syntax either — the next error would have been Missing argument in parameter list.
Python goes in a file. Save it as test_llm.py, then run python test_llm.py. If you want to type Python interactively, run python first to get the >>> prompt, then paste.
ModuleNotFoundError: No module named 'anthropic'
sentrinode-llm deliberately does not install provider SDKs — you might use one, both, or neither, and pulling both in would bloat every install.
pip install anthropic # and/or
pip install openai
I ran instrument() and nothing showed up
Expected. instrument() only patches the client — it does not send anything by itself. A real LLM call has to follow it; that call is what produces a span.
If you have made a call and still see nothing, in order:
- Wrong workspace slug. The tenant must match your workspace exactly. A mismatch returns
401, not a silent drop. - Looking at the wrong window. The live panels show the last 60 seconds. If your script finished a few minutes ago, the rates are back to zero — that is correct behaviour, not a fault. Month-to-date spend will still show it.
- The process exited too fast. Spans are exported in the background; a script that calls the API and exits immediately can die before the flush. Add a short sleep at the end to check.
What can SentriNode actually see?
The SDK records metadata only: model name, input and output token counts, latency, finish reason, computed cost, and any customer or feature label you attach.
Prompts and responses never leave your process. The SDK does not read message content, so there is nothing to redact, leak, or explain to your legal team. If you self-host, nothing leaves your network at all.
The optional attribution labels — sentri.customer_id and sentri.feature — are values you choose. Do not put personal data in them; they are stored and displayed as-is.
Creating, rotating and revoking API keys
Keys live at app.sentrinode.com → API Keys. Only a SHA-256 hash is stored, so the raw key is displayed once and cannot be recovered — if you lose it, revoke it and issue another.
Rotate immediately if a key has ever been pasted into a chat window, a screenshot, a support ticket, or committed to a repository. Revoking takes effect within about a minute, which is the auth cache TTL.
Keep keys in environment variables rather than in source. A key in a file is one git add away from being public.
Self-hosting without a dashboard? Mint keys in Postgres:
select sentrinode_create_tenant('my-team');
select sentrinode_create_api_key('my-team', 'laptop');
-- prints the raw key once
What 401, 413 and 429 mean
| Code | Meaning | Fix |
|---|---|---|
401 | Unknown workspace, or a key that is wrong, revoked or expired | Check the slug and issue a fresh key |
413 | Batch larger than 1 MB | Lower your exporter's batch size |
429 | Over the per-workspace rate limit | Honour Retry-After; send fewer, larger batches |
400 | Malformed JSON, or more than 2000 spans in one request | Reduce batch size |
503 | The service is misconfigured, not your request | Check status, or your own env if self-hosting |
Invalid credentials are rejected before the rate limiter, so someone guessing keys cannot exhaust your workspace's quota.
Limits
| Limit | Default | Why |
|---|---|---|
| Requests per workspace | 600 / minute | One noisy tenant cannot crowd out another |
| Request body | 1 MB | Rejected at the proxy before it reaches the app |
| Spans per request | 2000 | Keeps a single batch from monopolising parsing |
| Nodes per workspace | 500 | Oldest are evicted first |
| Node inactivity | 15 minutes | A node that stops reporting drops off the list |
Batch your spans rather than sending them one at a time — the limit is on requests, so 600 requests of 100 spans is 60,000 spans a minute.
How much traffic can it take?
A single ingest instance sustains roughly 14,500 spans/second — about 870,000 a minute — measured under load rather than estimated.
Latency depends far more on your batch shape than on volume. Same throughput, very different experience:
| Batch shape | Throughput | p50 latency |
|---|---|---|
| 4 exporters × 100 spans | 14,800/s | 25 ms |
| 32 exporters × 500 spans | 16,600/s | 888 ms |
Moderate batches from a few exporters is the sweet spot. Very large batches raise latency without buying throughput.
How is cost computed?
From token counts and a per-model price table, at the moment the call is recorded. Input and output tokens are priced separately, because for most models output costs several times more than input.
It is an independent estimate, not a copy of your provider invoice. It will not match to the cent — providers apply caching discounts, batch pricing and negotiated rates that the SDK cannot see. Use it to find which model, customer or feature is expensive, not to reconcile a bill.
If a model is missing from the price table the call is still recorded, with tokens and latency, at zero cost.
Budgets and hard caps
Set a monthly limit per workspace, an alert threshold (default 80%), and optionally a hard cap.
- Alerts fire once per level per month — at the threshold, and again at 100%. They do not repeat every minute.
- Hard cap makes the SDK refuse further LLM calls once you are over. The SDK polls the budget endpoint and fails open: if SentriNode is unreachable, your calls go through. An observability tool must never be the reason your product stops working.
Month-to-date spend is accumulated per call and is not affected by the 60-second display window. If the live rate looks lower than you expect, month-to-date is the number to trust.
Where does my data live, and for how long?
| What | Where | Retention |
|---|---|---|
| Live rates | Memory | 60 seconds |
| Cost chart buckets | Memory | 24 hours, lost on restart |
| Month-to-date spend | Redis | Calendar month, survives restart |
| 1-minute rollups | Postgres | Long-term |
| Per-call detail | Local files or InfluxDB | Configurable, 30 days by default |
Per-call history is optional and off unless configured. Two ways to turn it on:
- Local files — set
TRACE_LOG_DIR. One JSON object per call, per workspace, per UTC day. No database, no account, readable withgrepandjq. - InfluxDB — set all four
INFLUX_*variables, for querying history over time.
# today's spend for one workspace, from the files
cat /data/traces/my-team/$(date -u +%F).jsonl | jq -s 'map(.cost_usd) | add'
Files are pruned by age and total size — TRACE_RETENTION_DAYS (30) and TRACE_MAX_TOTAL_MB (2048). The size cap is the one that matters: a busy month fills a disk long before anything is 30 days old, and a full volume stops ingestion.
Can I monitor a machine, not just LLM calls?
Yes. Point an OpenTelemetry Collector with the hostmetrics receiver at the ingest endpoint — CPU, memory, disk, network. Config is on the Install page under “This machine”.
Host metrics arrive as node state, so they populate the nodes view rather than the LLM cost panels. Both can run at once.
Can I self-host?
Yes. The self-hosting guide walks through it step by step; in short there are three options:
- Standalone — Postgres holds workspaces and keys. Ingest, budgets, alerts and the
/v1API, with no Supabase. No web dashboard. - Full stack — runs Supabase's own components locally, so you get the dashboard and accounts too. Around 1.3 GB of RAM.
- With hosted Supabase — the stack we run in production.
Neo4j is not required for LLM observability — the ingest path never touches it. It exists for the service-graph features and is the single largest component, so leaving it out saves about 456 MB.
The repository is currently private; email support@sentrinode.com for access.
A node vanished from the dashboard
Nodes drop off after 15 minutes without reporting. That is the design — the list shows what is live, not everything ever seen. If a node disappeared, it stopped sending; check the process and its network path.
Also worth knowing: a workspace is capped at 500 nodes, and the oldest are evicted first. If you are cycling through short-lived hostnames, use a stable sentri.node_name instead of the machine's hostname.
A metric shows blank when it should read zero
This was a real bug, fixed in August 2026. Values of exactly 0 were discarded on the way in — so 0% CPU, zero errors, or an empty queue vanished instead of showing zero.
If you are running an older self-hosted build and see blanks where zeros belong, update. Hosted workspaces already have the fix.