DoubleAgent — Generator-Evaluator Dual Agent Pattern
This skill should be used when designing, implementing, or improving any AI system that requires quality assurance through separation of generation and evaluation roles. It implements the Generator-Evaluator dual-agent architecture (inspired by Anthropic's engineering blog and GAN design), where a Generator produces outputs and a dedicated Evaluator agent independently validates them through real interaction (e.g., Playwright browser operations), eliminating AI self-evaluation bias. Use when: building AI-generated UIs/code that needs quality checks, designing multi-agent pipelines with feedback loops, or upgrading existing coder+tester workflows to real interaction-based evaluation.
DoubleAgent Skill
Purpose
The DoubleAgent pattern solves a fundamental problem in AI-generated software: AI self-evaluation bias.
When a single AI agent both generates and evaluates its own output, it systematically overestimates quality — the same cognitive conflict that occurs when a student grades their own exam. The solution is to forcibly separate the two cognitive roles into independent agents with different prompts, goals, and evaluation criteria.
This skill provides:
- Architecture templates for Generator-Evaluator agent pairs
- Evaluator prompt templates calibrated with few-shot scoring examples
- Iteration loop design for 5-15 round refinement cycles
- Playwright integration patterns for real browser-based evaluation
- Scoring rubric design to prevent score drift and grade inflation
Core Architecture
User Goal / Spec
↓
┌─────────────┐
│ Generator │ ← Produces output (code, UI, content, data)
└──────┬──────┘
│ output artifact
↓
┌────────────────────────────────────┐
│ Evaluator │
│ • Reads spec (NOT generator output)│
│ • Operates artifact via Playwright │
│ (click, fill form, navigate) │
│ • Scores on rubric (0-100) │
│ • Writes structured feedback │
└────────────────┬───────────────────┘
│ score + feedback
↓
┌────────────────┐
│ Score ≥ target? │
│ YES → Done │
│ NO → Loop │
└────────┬────────┘
│
└──→ Generator (next iteration)
Key principle: The Evaluator reads the original spec, not the Generator's output. It evaluates independently, as if it were a real user encountering the product for the first time.
When to Apply
| Scenario | Apply DoubleAgent? |
|---|---|
| AI-generated frontend UI with interactions | ✅ Yes |
| Multi-step workflow code (forms, flows) | ✅ Yes |
| API endpoint implementation + validation | ✅ Yes |
| Content generation (reports, copy, docs) | ✅ Yes (text-based evaluator) |
| Single-function refactoring | ⚠️ Optional |
| Simple config changes | ❌ Not needed |
Implementation Steps
Step 1: Define the Spec Contract
Write a clear spec that both agents will reference independently. The spec must be:
- Concrete (measurable outcomes, not vague goals)
- Observable (evaluable through interaction or inspection)
- Versioned (so both agents work from the same contract)
See references/architecture.md for spec template.
Step 2: Configure the Generator Agent
Assign the Generator a single role: produce output that satisfies the spec.
- Do NOT ask the Generator to self-evaluate
- Do NOT include evaluation criteria in the Generator's prompt
- Provide: spec + iteration history + previous evaluator feedback
Step 3: Configure the Evaluator Agent
Assign the Evaluator a single role: independently verify the spec is satisfied.
- Load
references/evaluator-prompts.mdfor calibrated prompt templates - Use Playwright MCP for UI/web artifacts (real browser interaction)
- Use structured JSON output for scores to enable automated loop control
- Calibrate with few-shot examples BEFORE running (prevents grade inflation)
Step 4: Design the Iteration Loop
MAX_ROUNDS = 15
PASS_THRESHOLD = 80 # out of 100
for round in range(MAX_ROUNDS):
output = generator.run(spec, history)
evaluation = evaluator.run(spec, output) # Playwright-based
history.append({"round": round, "score": evaluation.score, "feedback": evaluation.feedback})
if evaluation.score >= PASS_THRESHOLD:
break
if evaluation.score_trend == "plateauing":
generator.switch_approach() # Complete strategy reset
See scripts/iteration_loop.py for a complete implementation template.
Step 5: Calibrate the Evaluator
To prevent score drift, run the Evaluator on 3-5 known examples FIRST:
- 1 example at ~30/100 (clearly bad)
- 1 example at ~60/100 (mediocre)
- 1 example at ~85/100 (good)
- 1 example at ~95/100 (excellent)
If scores deviate >15 points from expected, adjust the Evaluator's prompt or rubric weights before the real run.
Scoring Rubric Design
Effective rubrics for software systems:
| Dimension | Weight | What to Measure |
|---|---|---|
| Functional completeness | 30% | Does each spec requirement work end-to-end? |
| Interaction quality | 25% | Click/form/navigation behavior as a real user |
| Edge case handling | 20% | Error states, empty data, boundary inputs |
| Code/design quality | 15% | Consistency, readability, no obvious anti-patterns |
| Originality / craft | 10% | Avoids generic/template outputs when spec requires uniqueness |
Adjust weights based on the domain. For content systems, increase "originality". For data pipelines, increase "edge case handling".
Playwright Integration (for UI artifacts)
When evaluating web/H5/mini-program outputs, the Evaluator should:
- Navigate to the deployed artifact URL
- Execute each spec requirement as a user action sequence
- Observe actual behavior (DOM state, network requests, visual output)
- Record pass/fail per requirement with screenshots
- Report structured JSON with score breakdown
Playwright MCP tool calls to use:
playwright_navigate→ open URLplaywright_click→ interact with elementsplaywright_fill→ fill form inputsplaywright_screenshot→ capture evidenceplaywright_get_visible_text→ verify content
Reference Files
references/architecture.md— Detailed architecture patterns, spec templates, and design rationalereferences/evaluator-prompts.md— Ready-to-use Evaluator prompt templates for different artifact types
Scripts
scripts/iteration_loop.py— Complete iteration loop implementation templatescripts/calibrate_evaluator.py— Evaluator calibration utility
Related skills
- Downloads
- 894All time
- Security audit
- Pass
- Last updated
- 6mo ago
- Current version
- v1.0.0
- License
- MIT-0
