Closed-Loop Knowledge Mining: From Resolved Conversations to Draft KB Articles

Quick answer: A knowledge base decays at the same rate customers invent new questions, and the decay is invisible until containment numbers slip. Devotel Orbit closes the loop on that decay with shipped machinery: every question the agent misses is persisted with its original wording, the misses cluster into quantified topics with a count of customers and conversations behind each, each cluster carries a suggestion chip that says how to close it, and a one-click flow drafts a full knowledge-base answer card from the transcripts that produced the gap — staged as a draft that only grounds answers after an operator reviews and publishes it. This guide walks the loop end to end: where the misses come from, how to read the two report surfaces, and the operating rhythm that keeps a knowledge base current against live traffic.

Why knowledge bases decay in production

The pre-launch picture for an AI agent is tidy: compile the answers customers ask, load them into a knowledge base, point the agent at it. The production picture is different. Product names change, pricing tiers move, a carrier policy shifts the phrasing of every porting question — and none of that arrives as a notification. It arrives as a slow drift between what contacts ask and what the knowledge base answers, visible only in aggregate symptom: more escalations to humans, more low-confidence answers, more "I didn't find that" outcomes reported after the fact.

The standard remedy is a quarterly content audit: someone exports chat transcripts, skims a sample, and writes the articles they can spot. Two problems. The audit runs months after the questions started recurring, and the person sampling transcripts re-checks material that is already covered — the misses drown in the hits. A workable loop has to do three things a spreadsheet-side audit does not: capture every miss with its original wording, quantify it so the biggest gap is obvious, and produce a draft answer from the same evidence before asking a human to spend time on it.

The loop, stage by stage

Four stages, each with one home surface. Every stage is live today.

1. Capture. When an inbound conversation deflects to a human — and when a deflected conversation later reopens, confirming the answer failed — the original customer question is persisted as a knowledge-gap record with the exact phrasing the customer used. No export, no sampling, no tag-and-hope ticket discipline. The capture runs in the background alongside the inbox, so the report reflects everything the agent missed in the selected window, not the misses someone logged.

1. Cluster and quantify. The Knowledge Gap Report on each knowledge base, and the Knowledge gap miner on each agent's Knowledge tab, group the recorded misses into clusters of phrasing variants and rank them. The agent-scoped view adds a coverage ratio — the share of that agent's conversations that ended in a knowledge miss — so you can rank agents, not just topics. Each cluster carries the numbers that size the fix: how many contacts asked, how many conversations it touched, and the representative question the draft will target.

1. Suggest the remedy. Each cluster reads its own evidence and tags the fix it needs. A cluster whose conversations escalated to a human is a missing topic — the chip says "add article," because no published answer covers it. A cluster that only triggered low-confidence phrasing is a response-quality gap — the chip says "add exemplar," meaning a QA-approved example answer that tunes how the agent responds. A cluster carrying both kinds of evidence gets both tags. Hover the chip for the one-line reason, so the suggestion is auditable rather than a black box.

1. Draft and approve. The "Create draft article" button on a cluster generates a full answer-card draft from the transcripts behind that cluster and uploads it into the knowledge base it will serve — agent-scoped views draft into the first knowledge base attached to that agent; on the knowledge-base view the target is the knowledge base itself. The document carries a draft badge and never grounds an agent answer on its own — an operator reviews it in the editor and publishes. If a knowledge base has publish approval required, the document queues for a reviewer before anything becomes retrievable; review stays the gate, the draft just does the blank-page work.

Three surfaces, one report

The same knowledge-gap data is reachable from three places, and none of them needs an export:

• Dashboard — per knowledge base. The Knowledge Gap Report lives on the knowledge base itself and answers "what is this knowledge base missing?" with the ranked cluster list, the suggestion chips, and the draft-article button. Agents → Knowledge base.
• Dashboard — per agent. The Knowledge gap miner panel mounts on the agent detail page's Knowledge tab, under the knowledge-base attach section, and answers "how much of this agent's traffic is ending in a miss?" — coverage ratio first, clusters second. Agents → <agent> → Knowledge. This is the view to open when deciding which agent gets attention this week.
• API. The clustered report is a first-class endpoint — GET /agents/:id/kb-unanswered returns the coverage ratio and ranked clusters for a chosen window (7, 14, 30, 60, or 90 days), so the report can feed an internal weekly review or a BI export without a dashboard session. The draft flow is likewise API-first: POST /agents/knowledge-bases/:id/gaps/draft takes a cluster id from the report and returns the staged draft document — the dashboard button is a caller of the same endpoint, not a separate implementation.

The operating rhythm

Two checks of attention a week is the whole maintenance plan, and the second one is optional.

• Daily, when volume is high: open the largest cluster on the primary agent's Knowledge gap miner and follow the chip. Add the article when the topic is missing; add the exemplar when the phrasing is weak. Closing the biggest chip each day shrinks the report on its own.
• Weekly: review the coverage ratio per agent. A ratio that trends down week over week means the loop is working; a ratio that climbs on a specific agent means that agent's topic inventory is drifting and its clusters deserve a closer pass.
• Before a launch or pricing change: generate new FAQ articles from the change before the first contact has to miss on the answer. The loop catches misses after the fact; the launch checklist is where coverage still belongs to the operator.

Notice what none of this asks for: a transcript export, a sample, or a copy-paste into a blank editor. The loop instruments the misses, sizes them, and hands the reviewer a draft — the reviewer's job is judgement, not archaeology.

Frequently asked questions

Does a draft article start answering customers immediately? No. Drafts carry a draft badge and stay out of retrieval until an operator publishes them. If the knowledge base requires publish approval, the document additionally queues for a reviewer before it can ground anything.

Where does the draft land? On the knowledge-base view, into that knowledge base. On the per-agent view, into the first knowledge base attached to the agent — the draft button is disabled until a knowledge base is attached, so there is never a half-written artifact with nowhere to live.

Which questions get captured? Deflection misses from inbound conversations, plus the original question again when a deflected conversation reopens — that second signal marks the answer that demonstrably failed, which is the highest-value evidence in the report.

Can the report feed our own metrics pipeline? Yes. The per-agent report is an API endpoint (GET /agents/:id/kb-unanswered) with a selectable window, so the coverage ratio and cluster list can land wherever the rest of your reporting lives.

Try it on your biggest agent: Agents → <agent> → Knowledge → Knowledge gap miner, open the largest cluster, and let the chip pick the remedy.