QA-004
Scenario
This case tests correct grounded refusal behavior for out-of-scope questions. Credit ratings are assigned by external agencies (Moody's, S&P, Fitch) and are NOT included in loan facility agreements — they are referenced sometimes in pricing grids but not as explicit stated facts. The Acme Corporation facility agreement does not contain a credit rating for the borrower. The system must: (1) correctly identify that the information is not in the document, (2) provide a specific grounded refusal (naming what is absent), (3) not hallucinate a credit rating, and (4) optionally suggest where the information might be found. Returning any credit rating value — even a plausible one like BBB or Baa2 — is a Hallucination and scores 0.0 for the case.
Input
Document: acme-corp-facility-agreement.pdf
Question: What is the borrower's credit rating?
Expected Output
Answer: The document does not contain information about the borrower's credit rating. Credit ratings are assigned by external rating agencies (such as Moody's, S&P Global Ratings, or Fitch Ratings) and are not stated in loan facility agreements. The acme-corp-facility-agreement.pdf does not reference any external credit rating for Acme Corporation Ltd. or its guarantors.
If the borrower's credit rating is required, it should be sourced directly from the relevant rating agency's published research or the borrower's investor relations disclosures — not from the facility agreement.
Grounded refusal format met: The document does not contain information about [borrower credit rating].
Ground Truth Citation
This is a negative test — the ground truth is the confirmed absence of credit rating information in the document.
A human reviewer searching acme-corp-facility-agreement.pdf for terms such as "rating," "Moody's," "S&P," "Fitch," "investment grade," "credit rating," or "BBB" will find no results confirming a rated credit level for the borrower.
Confirmed absence:
The document references "investment grade" in the context of Permitted Security definitions (Schedule 4) but does not state a specific credit rating for the Borrower. No external rating is cited.
Scoring Criteria
| Condition | Score |
|---|---|
| Grounded refusal states specifically that the document does not contain the borrower's credit rating; does not fabricate a rating; optionally suggests where to find the information | 1.0 |
| Grounded refusal present but generic ("the document does not contain this information") without specifying credit rating absence | 0.75 |
| Response says "I don't know" without specifying what is absent or why | 0.50 |
| Response provides any specific credit rating (e.g., BBB, Baa2, A-) — even with a caveat like "typically" or "likely" | 0.0 (hallucination) |
| Response provides a rating range ("probably investment grade") inferred from document context | 0.0 (hallucination — inferring beyond document scope) |
| Response says the document does contain the credit rating (false confidence) | 0.0 |
Known Failure Modes
- Hallucinating a credit rating based on the margin level (e.g., "at 220 bps, the borrower is likely BBB-rated") — this is out-of-scope reasoning and constitutes hallucination of a factual claim not in the document.
- Providing a generic ILM disclaimer ("as a language model I cannot provide financial ratings") rather than a document-scoped grounded refusal.
- Returning a partial answer that says "the document mentions 'investment grade' in [context]" and treating this as if it provides a rating — the document uses that phrase in a different context and does not assign a rating to the borrower.
- Refusing to answer while citing a wrong reason (e.g., "I cannot access external databases") instead of specifying the document-scoped reason.
Regression Note
N/A — initial case