QA-003
Scenario
This case tests accurate extraction of an inherently subjective contractual definition with appropriate uncertainty acknowledgement. The Material Adverse Change (MAC) definition in the Acme Corporation facility agreement is a standard three-limb clause that is legally precise in structure but inherently subjective in application — whether a specific event triggers a MAC is a matter of legal judgment, not mechanical calculation. The system must: (1) extract the full MAC definition verbatim, (2) cite the definition clause, (3) explicitly acknowledge that MAC clauses are inherently subjective in their application — the document defines the concept but does not resolve ambiguity as to when it is triggered, and (4) NOT interpret or opine on whether any hypothetical event would trigger the MAC. Overconfident characterisation of what does or does not trigger MAC is a Fail.
Input
Document: acme-corp-facility-agreement.pdf
Question: What is the definition of Material Adverse Change in this agreement?
Expected Output
Answer: The term "Material Adverse Effect" (used interchangeably with "Material Adverse Change") is defined in Clause 1.1 of the agreement as follows:
Full definition:
"For the purposes of this Agreement, 'Material Adverse Effect' means a material adverse effect on (a) the business, assets or financial condition of the Group taken as a whole; (b) the ability of any Obligor to perform its payment obligations under the Finance Documents; or (c) the validity or enforceability of, or the effectiveness of any security granted under, the Finance Documents."
Source: acme-corp-facility-agreement.pdf, Page 5, Clause 1.1 (Definitions — Material Adverse Effect)
Scope note: The definition covers three limbs: (a) business/financial deterioration at Group level, (b) payment capacity of any obligor, and (c) enforceability of the finance documents or security.
Subjectivity acknowledgement: MAC clauses are inherently subjective in application. This definition establishes the legal test but whether any specific event meets that test is a matter of legal and commercial judgment that the agreement does not predetermine. Smartflow extracts and presents the definition only; interpretation of whether a specific event triggers this clause requires legal advice.
Ground Truth Citation
"For the purposes of this Agreement, 'Material Adverse Effect' means a material adverse effect on (a) the business, assets or financial condition of the Group taken as a whole; (b) the ability of any Obligor to perform its payment obligations under the Finance Documents; or (c) the validity or enforceability of, or the effectiveness of any security granted under, the Finance Documents."
Source: acme-corp-facility-agreement.pdf, Page 5, Clause 1.1 (Definitions — Material Adverse Effect)
Scoring Criteria
| Condition | Score |
|---|---|
| Full three-limb MAC definition extracted verbatim; correct clause citation; subjectivity of application explicitly acknowledged; no interpretation of what triggers MAC | 1.0 |
| Full definition extracted; citation correct; subjectivity acknowledged but in vague language ("MAC is complex") without explaining the limitation | 0.80 |
| Full definition extracted and cited; subjectivity NOT acknowledged | 0.60 |
| Definition extracted but only 2 of 3 limbs included | 0.50 |
| System opines that a specific event would or would not trigger MAC (beyond what document states) | 0.25 maximum — scope violation |
| Definition extracted from wrong clause (e.g., Events of Default rather than Definitions) | 0.50 |
| Any paraphrase of the definition substituted for verbatim text | 0.75 maximum |
Known Failure Modes
- Paraphrasing the definition rather than quoting it verbatim — "MAC means something that materially affects the borrower's business."
- Extracting only one or two limbs of the three-limb definition and presenting it as complete.
- Opining on what types of events would or would not qualify as a MAC — this is outside document scope.
- Confusing "Material Adverse Effect" (the defined term) with "Material Adverse Change" (a synonym used elsewhere in the agreement) and treating them as different definitions.
- Not acknowledging the inherent subjectivity — presenting the definition as if it provides a clear mechanical trigger.
Regression Note
N/A — initial case