Methodology / v1

A score is only as good as its evidence.

Four scoring axes. Explicit evidence states. A public rubric that shows what was checked, whether a review claims a hands-on record, and what is still missing.

Explore the rubric ↓Read evidence states

Sources → Hands-on evidence → Editorial judgment

aicoolies Score v1

The four axes

Scored reviews may carry four values on a 0–100 scale. The first three are scored dimensions; the fourth is an editorial verdict that deliberately is nottheir average. A score's presence alone is not proof that its evidence metadata is complete.

01 / SCORED DIMENSION

Speed

score_speed · 0–100

Raw responsiveness in day-to-day use: completion and generation latency, indexing time on a real mid-size repository, cold-start time, and how the tool behaves under a slow connection. On Verified reviews, speed is measured hands-on during real work sessions, and the run is published in the review's test record. On Documented reviews, the speed score is an editorial estimate from vendor documentation and public reports, and the review page says so.

02 / SCORED DIMENSION

Privacy

score_privacy · 0–100

What leaves your machine and under which terms: training on your code, prompt retention windows, telemetry defaults and opt-outs, self-hosting options, and whether a zero-retention or enterprise mode exists. Claims are checked against the vendor’s own privacy policy and documentation, not marketing pages.

03 / SCORED DIMENSION

Developer Experience

score_dev_experience · 0–100

How it feels to work with: setup friction, editor and CLI integration quality, configurability, documentation depth, failure modes, and how well it fits into an existing workflow rather than demanding a new one.

04 / EDITORIAL VERDICT

Overall

score_overall · 0–100

The editor’s holistic verdict on a 0–100 scale. It weighs the three dimensions above together with pricing fairness and project health (maintenance activity, license, longevity risk). It is a considered editorial judgment, not an automated average — two tools with identical sub-scores can earn different overall scores when their pricing or trajectory differs.

practice

How a score is earned

Hands-on before verified

A review is marked Verified only when it records the tested date and version, environment, rubric version, evidence artifact, and source-check dates. Older scores remain visible but are labeled separately. The artifact must be a record we produced; vendor pricing, docs, changelog, or release pages do not qualify.

Claims are verified at the source

Pricing, licensing, and feature claims should cite official documentation or repository evidence. A source-check date records when those links were actually revisited.

Scores decay honestly

A visible last-verified date is shown only when an editor independently rechecks the full record. Routine CMS saves and updatedAt timestamps are not treated as verification.

Sponsorships never touch rankings

Sponsored placements are sold separately, always carry a visible Sponsored label, and never influence scores, verdicts, recommendations, or the order of any list. There is no affiliate weighting and no way to buy a score.

Reviews marked Verified link to a public test record we produced ourselves: commands, versions, and outputs. Documented reviews are built from sources and do not claim hands-on testing. Meet the editor. Imports and automation may prepare records, but they do not grant Verified status.

evidence

Evidence states

Evidence status describes the record, not the product. It lets readers distinguish a reproducible review from a legacy score without deleting useful historical content.

Verified: Rubric version, test date and version, environment, public evidence artifact, source-check date, and full-record verification date are complete. The artifact must be a record we produced; vendor pricing, docs, changelog, or release pages do not qualify.
Partial: Some source or hands-on evidence exists, but one or more required verification fields are missing. Incomplete hands-on records are labeled Editorial & Architecture Review. Documentation-based records (an editor sets the documented badge, or the review still uses the stock VS Code AST template environment) are labeled Documented evidence. Both labels stay on the partial evidence state, and the recorded scores stay as they are.
Not yet evidenced: The editorial record has no reproducible test evidence yet and must not be described as hand-tested.
Legacy / unclassified: The review predates the evidence contract. Its score can remain visible, but readers should not infer Verified status.

changelog

Versioning

This page documents v1 of the rubric. If an axis definition, the scale, or the weighting philosophy ever changes, the rubric gets a new version number and this page keeps a changelog. New or re-verified reviews store the rubric version that produced their score; a missing version marks a legacy/unclassified record rather than silently asserting v1. Scores may still be revisited under the same version when a re-verification changes the underlying facts.

reuse

Use the scores

Current score values are published as machine-readable open data in reviews.json under CC-BY 4.0. When you cite a number, reference it as “aicoolies Score v1”and link this page or the tool's review. Attribution is the only thing the license asks for.