Telemetry & Savings

MemStrata ships four metrics on the dashboard. Every number is computed locally from your own session data. This page explains exactly what each metric measures, how the baseline is established, and how the money-back reconciliation works.

---

The four metrics

1. Tokens saved vs. RAG

What it is: The percentage reduction in input tokens MemStrata sends to your LLM provider compared to a naive retrieval-augmented generation baseline.

How it’s computed:

1. For each AI request, MemStrata records the token count of the context it actually sent.
2. It also records what a naive RAG pipeline would have sent for the same query — the top-K chunk retrieval from your codebase using a standard embedding search.
3. tokens_saved_pct = 1 - (actual_tokens / rag_tokens)

What “naive RAG” means: Top-5 chunks from a BM25+embedding hybrid retrieval, no deduplication, no dependency graph pruning. This is the baseline used by most AI coding tools that claim “codebase awareness.”

Median across active users: 42%. Individual results vary by codebase size, coding style, and which integrations are active.

---

2. Retrieval recall

What it is: Of the entities (symbols, files, functions) the AI actually referenced in its response, what fraction did MemStrata include in the context it sent?

How it’s computed:

1. After each AI turn, MemStrata parses the response for entity references (function names, file paths, class names, etc.).
2. It checks which of those entities were in the context it sent.
3. recall = (entities_referenced_that_were_in_context) / (total_entities_referenced)

Why it matters: High recall means MemStrata is reliably surfacing the relevant code. Low recall means the AI is hallucinating or referencing things MemStrata missed — either way, useful signal.

Median across active users: 87% per turn.

---

3. Retrieval precision

What it is: Of the entities MemStrata included in context, what fraction did the AI actually use in its response?

How it’s computed:

1. After each AI turn, MemStrata compares what it sent against what the response referenced.
2. precision = (entities_in_context_that_AI_used) / (total_entities_in_context)

Why it matters: High precision means MemStrata is sending focused, relevant context — not padding. Low precision means the context graph is over-fetching, which wastes tokens (and shows up as a lower “tokens saved” score).

Median across active users: 64% per turn.

---

4. 24-hour edit survival

What it is: The fraction of AI-suggested edits that are still present in the repo 24 hours after being applied.

How it’s computed:

1. When MemStrata detects an AI-suggested edit was applied (via the harness or extension), it records the diff.
2. 24 hours later, it checks whether those lines are still present in the file (accounting for git moves and renames).
3. survival = (edits_still_present_after_24h) / (total_edits_applied)

Why it matters: It’s a proxy for edit quality. Suggestions that get immediately reverted indicate the AI was off-base. High survival rate = the AI understood your codebase well enough to produce durable changes.

Median across active users: 73% per session.

---

The cohort baseline

The “vs. RAG” comparison uses a cohort baseline, not a global average. Your baseline is computed from:

• Codebases of similar size (±1 order of magnitude in file count)
• Similar primary language (Python, TypeScript, etc.)
• Similar AI tool category (inline completion vs. chat vs. agentic)

The baseline is updated weekly. You can inspect the current baseline values in Settings → Telemetry → Cohort baseline.

---

Dollar savings calculation

dollar_savings = tokens_saved × provider_rate

Where:

• tokens_saved = actual tokens sent subtracted from the RAG-baseline tokens, summed over all turns in the billing cycle
• provider_rate = your configured per-token rate for your LLM provider (set in Settings → Providers, or auto-detected from your API key’s usage logs)

The dashboard Money tab shows this in real time. At the end of the cycle, the total is what’s compared against your subscription cost for the money-back guarantee.

---

Money-back reconciliation

At the end of every billing cycle:

1. MemStrata reads ~/memstrata/telemetry.db and computes total_dollar_savings for the cycle.
2. If total_dollar_savings < subscription_cost, MemStrata sends the difference to the license server.
3. The license server creates a Stripe customer.credit_note for the shortfall.
4. The credit appears on your next Stripe invoice automatically — no action required.
5. You receive a receipt email from Stripe showing the math.

Floor: If you were not active for ≥7 days in the cycle, the guarantee does not apply (you likely didn’t incur the full subscription charge either, depending on your trial/pause state).

Cap: Credits are capped at your subscription cost for the cycle. Surplus savings don’t roll over as cash — they roll over as continued good performance.

---

Reading your dashboard

Tab	What’s shown
Money	Dollar savings this cycle, rate per day, projected end-of-cycle total
Tokens	Raw token counts — sent, saved, RAG baseline
Quality	Recall and precision per turn, 7-day trend
Edits	24h survival rate, number of AI suggestions applied

All data is local. The dashboard reads directly from ~/memstrata/telemetry.db. Nothing is sent to a server to generate this view.