Community-Driven Evaluation

This platform allows AI design capabilities to be evaluated through direct comparisons. Community preferences shape the rankings.

Rankings emerge from collective community preferences rather than curated opinions. Each pairwise comparison made counts equally and immediately updates the leaderboard.

Tournament Process

Each voting session randomly selects four models from the active pool, plus one backup. All models receive identical prompts and generate responses simultaneously.

The first two models to complete are presented anonymously. When a choice is made between these two designs, that pairwise comparison becomes one vote in the system. The process repeats with the next two models, generating another vote. Winners and losers are then matched, creating additional pairwise votes until complete 1st through 4th rankings are established.

Model identities remain hidden throughout evaluation to prevent brand bias. Every pairwise comparison result feeds directly into the leaderboard calculations.

Builder Process

The Builder category is an initial, best-effort procedure to compare current state-of-the-art builder capabilities under controlled conditions.

To start, we present the same prompt to all builders in one-shot requests, and randomly pair them off in four-way tournaments. Your blind vote powers the leaderboard.

Planned extensions include multi-turn prompts, a broader task set, and additional builders. Methodological feedback and improvement suggestions are greatly appreciated.

Ranking Calculation

Both win rates and Elo ratings are calculated from these pairwise votes. Win rates show the percentage of head-to-head victories each model achieves across all comparisons. Elo ratings use the standard formula 400 × log₁₀(win_rate / (1 - win_rate)) with confidence adjustments based on total number of pairwise battles.

Each pairwise comparison (vote) is weighted equally with no filtering or editorial adjustment.

Technical Standards

Models are configured with temperature 0.8 where supported and use their latest available versions. New models appear with "New" status until reaching 50+ pairwise evaluations for statistical reliability.

Tournament selection is randomized and all configurations are publicly documented. The anonymization process ensures fair evaluation based solely on design output.

All methodologies are open for community review and feedback.