Multi-Model Evaluator

Run identical prompts across multiple LLM models. Compare cost, quality, and latency to find the optimal model for each task type.

Quick Start

# Run a benchmark across default models
python3 scripts/model-evaluator.py benchmark --prompt "Summarize this text in 3 bullet points" --text "Your content here"

# Compare specific models
python3 scripts/model-evaluator.py benchmark --prompt "Write a function to sort a list" --models "haiku,sonnet,opus"

# View historical results
python3 scripts/model-evaluator.py history

# Get model recommendations by task type
python3 scripts/model-evaluator.py recommend --task coding

# Show model pricing reference
python3 scripts/model-evaluator.py pricing

# Export results for analysis
python3 scripts/model-evaluator.py export --format csv --output results.csv

Commands

| Command | Purpose |

|---------|---------|

| benchmark | Run prompt across models, compare results |

| history | View past benchmark results |

| recommend | Get model recommendation by task type |

| pricing | Show current model pricing table |

| export | Export benchmark data as CSV/JSON |

| rank | Rank models by cost-efficiency for a task category |

Task Categories

The evaluator classifies tasks and maintains per-category model rankings:

• coding — Code generation, debugging, refactoring
• writing — Content, emails, documentation
• analysis — Data analysis, reasoning, research
• summary — Summarization, extraction, compression
• creative — Creative writing, brainstorming
• simple — Quick lookups, formatting, translations

How It Works

1. Same prompt sent to each model

2. Response captured with timing and token counts

3. Cost calculated from pricing data

4. Quality scored (coherence, completeness, accuracy)

5. Results stored in memory/model-eval-history.json

6. Rankings updated per task category