Why we listen to the kernel

A refusal is information

The consultant on this site answers from our published pages, and cites the section he took the answer from. When he cannot find one, he says he is not confident and stops. That behaviour is deliberate and it is measured: on the last off-corpus check, zero of thirteen questions outside the published material were answered anyway. A system that answers everything is not more helpful than one that refuses; it is less trustworthy, because you cannot tell its guesses from its knowledge.

The number that moved, and what moved it

We keep a fixed set of twenty-six questions a visitor to this site might genuinely ask, with the page each one should be answered from. In September the consultant got 13 of 26 onto the right page first time. After one round of work it got 24 of 26. These are not the twenty-six on our home page: those ask about our own systems and score the answer and its source, while these ask about this site and score whether the right page came first.

We changed nothing about the engine. What we changed was our own pages: the answers to half those questions were sitting in text the index was quietly dropping. The engine had been telling us, in every refusal, that the material was not there — and for a while we read that as the engine being weak.

That is the whole argument for listening to it. The refusals were a map of the holes in our own documentation, published weekly, for free, and we ignored it.

What you can observe

• A refusal on a question you know the answer to. Almost always your documentation, not the reader. Somewhere the thing everybody knows was never written down.
• A confident answer from the wrong place. Rarer and more serious: one of our own twenty-six still does this. It means two pages of ours make the same claim in different words, and nothing says which is current.
• The cited section itself. Every answer names where it came from, so you can disagree with the source rather than with the answer — which is a much more productive argument to have.

What you can decide with it

How you act on a refusal is yours to choose; we will not prescribe a workflow for reading your own organisation. What we would say is that the three obvious responses are genuinely different:

• Accept it when the question is outside what you have published. The refusal is correct and the fix is a document, not a setting.
• Re-ask it in the vocabulary your documents actually use. We have watched a question fail and then succeed on one word, because the policy it was about never used the word the asker did — and that gap is worth knowing about on its own.
• Escalate it when two sources disagree. That is not a question for a reader of any kind; it is a decision somebody in your organisation has to make and record.

The honest limit: these numbers are about our fixture, on our box, against our own published pages. They say what this behaviour looks like when it is measured. They are not a forecast of what it would score on your material, and we will not quote you one before we have seen it.

Where this starts

Ask our consultant something — including something he should refuse — and watch what he does with it. If what you want is that behaviour pointed at your own material, how engagements work describes the diagnostic that starts it.