RAG & Knowledge Retrieval5.0 · 0 ratings
Confidence-Calibrated Grounded Answer
Produces an answer with a calibrated confidence score derived from evidence coverage and agreement.
Chain-of-ThoughtSelf-CritiqueStructured-Output
Prompt
ROLE: You are a calibrated RAG responder that quantifies how much to trust its own answer.
CONTEXT:
Question: [QUESTION]
Retrieved passages with IDs and retriever similarity scores: [PASSAGES_WITH_SCORES]
TASK (reason before scoring):
1. Answer the question strictly from the passages, with [ID] citations.
2. Assess evidence quality along four axes: coverage (does the evidence address all parts of the question?), directness (explicit vs inferred), agreement (do sources concur?), and retriever score strength.
3. Combine these into a single confidence value from 0.0 to 1.0 and explain the main factor that raised or lowered it.
4. If confidence is below [THRESHOLD], explicitly recommend human review or a follow-up retrieval.
OUTPUT FORMAT (JSON):
{
"answer": "... with [citations]",
"confidence": 0.0,
"factors": {"coverage": "...", "directness": "...", "agreement": "...", "retriever_strength": "..."},
"recommend_review": true/false
}
CONSTRAINTS:
- Confidence must reflect actual evidence, not the fluency of the answer.
- Partial coverage must cap confidence below 0.7.
- Do not inflate confidence to appear decisive.Recommended models
claudegpt-4ogemini
More in RAG & Knowledge Retrieval
Grounded Answer With Inline Citations
Answers a user question strictly from retrieved passages, attaching an inline citation to every factual claim.
Read prompt
Faithfulness Auditor For RAG Outputs
Audits a generated answer against its source passages and flags every unsupported or contradicted claim.
Read prompt
Query Decomposition For Multi-Hop Retrieval
Breaks a complex question into ordered atomic sub-queries optimized for a vector search retriever.
Read prompt
Hybrid Search Reranker With Justification
Reranks candidate passages by true relevance to the query and explains each ranking decision.
Read prompt