QA-001
Scenario
This case tests basic factual retrieval from a single, clearly stated field. The Acme Corporation facility agreement contains the facility amount in one clause (Clause 2.1) with unambiguous wording. The system must return the exact amount, cite the correct clause, and include the verbatim quote. This is the minimum viable Q&A behavior — a system that cannot answer this question is not production-ready. The correct answer requires no reasoning or synthesis; it is a direct lookup.
Input
Document: acme-corp-facility-agreement.pdf
Question: What is the total facility amount under this agreement?
Expected Output
Answer: The total facility amount is USD 350,000,000 (United States Dollars three hundred and fifty million).
Citation:
"The Total Commitments under this Agreement are USD 350,000,000 (United States Dollars three hundred and fifty million), being a term loan facility made available to the Borrower on the terms and conditions set out herein."
Source: acme-corp-facility-agreement.pdf, Page 8, Clause 2.1 (The Facility)
Ground Truth Citation
"The Total Commitments under this Agreement are USD 350,000,000 (United States Dollars three hundred and fifty million), being a term loan facility made available to the Borrower on the terms and conditions set out herein."
Source: acme-corp-facility-agreement.pdf, Page 8, Clause 2.1 (The Facility)
Scoring Criteria
| Condition | Score |
|---|---|
| Correct amount (USD 350,000,000); correct clause citation (2.1); verbatim quote included | 1.0 |
| Correct amount; citation to wrong clause (e.g., cover page rather than Clause 2.1) | 0.75 |
| Correct amount; no citation at all | 0.50 |
| Amount stated as "USD 350 million" (semantic match, no verbatim quote) with citation | 0.75 |
| Amount stated as "USD 350 million" with no citation | 0.50 |
| Any incorrect amount (e.g., USD 300M, USD 450M) | 0.0 |
| Amount not present in source document (fabricated) | 0.0 (case-level) |
Known Failure Modes
- Answering with a different amount from another document in the corpus (cross-document contamination if multi-document context is passed).
- Providing the answer without a citation — factually correct but fails the provenance requirement.
- Stating the currency as "dollars" without specifying USD — ambiguous and not verbatim.
- Rounding to "approximately USD 350 million" rather than stating the exact contractual amount.
Regression Note
N/A — initial case