Free resources · For businesses

Does asking Claude to quote your document first make its answers more accurate?

A prompt that makes Claude quote your document before it answers. It did not change accuracy in a test of 144 replies, but every answer came with a findable sentence.

Can you trust an answer about your own document?

You give Claude a document and ask a question about it. Anthropic's advice is to "explicitly give Claude permission to admit uncertainty". For long documents, its advice is to "ask Claude to extract word-for-word quotes first before performing its task".

What was tested?

We used one made-up bakery information sheet of 145 words, and 24 questions about it. Twelve questions the sheet answers. Twelve it does not, and six of those assume a fact the sheet lacks, such as the flavors of the gluten-free cake. We asked each question 3 times with each of two prompts, so Claude Haiku wrote 144 replies. The plain prompt said "Answer the question." The other asked for the exact sentence first, then a one-sentence answer. It said to write "I don't know" if the sheet did not say.

What happened?

Both prompts answered all 36 answerable questions correctly.
The plain prompt said the sheet did not know for 35 of 36 unanswerable questions. The one exception inferred "almond" from the sheet.
The quote-first prompt said so for 36 of 36.
Its replies held 43 quotes, and all 43 were in the sheet.

The cost was $0.0009 a reply for the plain prompt and $0.0010 for quote-first.

What does it mean?

Here the plain prompt almost never guessed, so quote-first changed little in accuracy. What it added was a way to check: each answer came with a sentence you can find in your own document.

How far can this be trusted?

One model and one short document. Anthropic's advice on quotes is for long documents.
We wrote the questions for this test. None asked for a fact outside any document, where Claude must rely on memory.

What we would do

Paste your document and your question into Claude, then add this prompt:

Quote the exact sentence from the document that answers the question, then answer in one sentence. If the document does not contain the answer, say "I don't know" and do not guess.

Search your document for the quote Claude gave.
If you cannot find the quote, treat the answer as unconfirmed and read the nearby text.

Check

Each answer you rely on has a quote you found in your own document, and the quote supports the answer. We checked only that quotes existed.
This result covers one model and one short document.