AI Pull-Request Review: Domain Explainer#

What problem this solves#

Code review is the step where someone other than the author reads a change before it merges. It catches bugs, spreads knowledge and enforces conventions. It is also slow: reviewers are busy, and a PR waiting for review is work that is done but not shipped.

AI PR reviewers put a language model at that step. They read the diff, sometimes with the files around it, and post comments the way a human reviewer would. The promise is a first pass that is always available. The risk is a reviewer that is confidently wrong, noisy, or silent when it matters.

The question has become more pressing as coding agents write more of the code. A team whose agents open dozens of PRs a day cannot give each one a careful human read, and a reviewer from the same model as the author may share the author’s blind spots.

Key concepts#

Diff vs context. The diff is the lines that changed. Many bugs are visible only with context: the caller of a changed function, the schema a query assumes, the test that no longer covers a branch. Tools differ mostly in how much context they fetch and how. More context is not automatically better; one 2026 benchmark found that adding raw repository context lowered detection rates.

Precision and recall. Precision is the share of a reviewer’s comments that point at a real issue. Recall is the share of real issues it commented on. A reviewer that posts forty comments will have high recall and low precision; one that posts two will have the reverse. F1 combines them into one number, at the cost of hiding which way a tool leans.

Golden comments. Benchmarks need ground truth: a list of the issues a PR actually had. Usually humans write it. Any issue the humans missed counts against a tool that found it, so benchmark precision understates real precision somewhat.

LLM judge. Matching a tool’s comment to a golden issue is itself done by a language model in most benchmarks. A different judge model gives different scores.

Ambient vs invoked. An ambient reviewer runs on every PR without being asked. An invoked reviewer runs when someone, or an agent, asks. Ambient gives coverage without discipline; invoked gives control without coverage.

Advisory vs gating. An advisory reviewer posts comments. A gating reviewer can block or approve a merge. Some AI reviewers can satisfy a “requires approval” rule on their own, which changes what that rule means.

Self-preference. Language models rate their own outputs higher than others’ outputs of similar quality. When the same model family writes and reviews code, the reviewer may be more likely to accept the author’s mistakes.

Verification pass. A second stage that tries to confirm each candidate finding before posting it, by re-reading the code, running a test, or reproducing the bug. It trades latency and cost for fewer false alarms.

The shape of the market#

Three kinds of product:

1. Hosted bots you install as a code-host app (CodeRabbit, Greptile, Cursor Bugbot, Qodo and others). The vendor’s servers read your code.
2. Platform-native reviewers inside a code host or suite you already pay for (GitHub Copilot, GitLab Duo, Gemini Code Assist enterprise, Amazon Q).
3. Self-run reviewers you install in your own CI or server with your own model key (PR-Agent, Kodus, and coding agents such as Claude Code, Codex and OpenCode packaged for CI).

Pricing comes in two shapes: a flat monthly fee per developer, or a charge per review, per line or per credit. The second shape grows with the number of PRs, which agent- written code tends to increase sharply.

Common misconceptions#

“The benchmark leader is the best reviewer.” On the one open, independent benchmark, the top few tools are separable from the bottom few, but most of the middle cannot be told apart on 50 PRs. Vendors each cite a different snapshot in which they are first.

“An AI reviewer makes review faster.” The one industrial before/after study found PRs took longer to close after an AI reviewer was added, because developers spent time on its comments.

“One good reviewer is enough.” Reviewers miss most issues and miss different ones. Four reviewers combined caught about ten points more of a benchmark’s issues than the best one alone.

“It’s free because it runs in our CI.” Self-run reviewers cost model tokens and CI minutes. Since June 2026, GitHub’s own Copilot reviewer also consumes Actions minutes on private repositories.

How to choose#

1. Decide whether a new party may read your code. That splits the market in half.
2. Decide who will read the comments: humans (favor precision) or agents (can absorb recall).
3. Estimate PRs per developer per month. That picks the price shape.
4. Keep the reviewer advisory, keep its rules in the repository, and watch for it going quiet.

Related surveys#

• 1.212 AI Coding Agent Harnesses: the agents that write the code being reviewed, and several of which also review.
• 1.250 Linters & Static Analysis: deterministic checks that should run before any AI reviewer.
• 1.205 LLM Evaluation: the LLM-as-judge methods that benchmarks in this field depend on.

S1 Rapid Discovery — Approach#

What this category is#

AI pull-request review tools read a proposed change (a PR or merge request diff, usually with some surrounding repository context) and post findings back: inline comments, a summary, sometimes a suggested fix. They sit at the merge gate, between “the author thinks it is done” and “it lands on main.”

The test for inclusion: the tool’s primary output is review findings on a diff, and it can attach those findings to a PR on a code host (GitHub, GitLab, Bitbucket, Azure DevOps, Gitea/Forgejo), whether it runs on its own or when someone asks.

Category boundary#

In scope

• Hosted review bots installed as a code-host app: CodeRabbit, Greptile, Graphite, Cursor Bugbot, Qodo, Sourcery, cubic, CodeAnt, Bito, Baz, Kodus Cloud, Augment.
• Review built into the code host or a platform suite: GitHub Copilot code review, GitLab Duo Code Review, Gemini Code Assist (enterprise), Amazon Q Developer.
• Self-run reviewers you install in your own CI or on your own server: PR-Agent, Kodus (self-hosted), claude-code-action, openai/codex-action, run-gemini-cli, the OpenCode GitHub agent.
• Invoked review from a coding agent: Claude Code /code-review and its cloud ultra mode, Codex @codex review, CodeRabbit CLI, plus Anthropic’s managed Code Review service.

Out of scope

• Deterministic static analysis (linters, type checkers, SAST rule engines). These are rule-based and covered in 1.250 Linters & Static Analysis and 1.251 Type Checkers. Several AI reviewers run linters internally and post the results; that does not move linters into this category.
• Coding agent harnesses as such (1.212). Claude Code and Codex appear here only in their review role.
• Human review workflow tools (stacked-diff UIs, review queues, merge queues) except where the vendor’s AI reviewer is the product being compared.
• AI that applies fixes to human review comments (the reverse direction; e.g. Meta’s 2025 work on auto-fixing reviewer comments).

Evaluation dimensions#

Each option is profiled on the same six dimensions:

1. Trigger model — ambient (reviews every PR on open/push), summoned (a comment command), or invoked (someone runs it from a terminal or a CI job they wrote).
2. Where it runs and what it consumes — vendor cloud, your CI runners (and their minutes), your own server, or your workstation.
3. Pricing shape — flat per seat, metered per review / per line / per credit, or your own model tokens plus compute.
4. Data posture — retention, training on customer code, self-host or VPC option, bring-your-own model key.
5. Output and control — inline comments, severity, suggested fixes, whether it can approve or block, config files and learned rules.
6. Evidence of quality — who measured it, on what, with what sample size.

How volatile this is#

High. In the twelve months to October 2026: Cursor agreed to acquire Graphite (source: https://cursor.com/blog/graphite, accessed 2026-10-02); Qodo handed PR-Agent to a community org (source: https://www.qodo.ai/blog/qodo-is-handing-pr-agent-over-to-the-community/, accessed 2026-10-02); Google shut down the free consumer Gemini Code Assist GitHub app (source: https://developers.google.com/gemini-code-assist/docs/deprecations/consumer-code-review, accessed 2026-10-02); Cursor Bugbot, Greptile and GitHub Copilot all moved toward metered billing; Augment rebuilt its reviewer on a new agent platform and retired individual plans. decay_class: fast.

Sources for every figure in this pass are inline. Vendor statements are marked (vendor: ...); figures in circulation that could not be traced to a primary page are marked (unverified).

Hosted review bots#

Install an app on the code host; the vendor’s cloud receives each PR, runs its models, and posts back. None of these consume the customer’s CI minutes. All prices are list prices read on 2026-10-02.

Flat per-seat#

CodeRabbit is the volume leader of this group and the default many teams meet first. Its configuration file (.coderabbit.yaml) and per-path instructions are the most developed rule system among the hosted bots. CodeRabbit states that code is sent to OpenAI and Anthropic models and is not used for training “regardless of retention settings” (vendor: https://docs.coderabbit.ai/faq, accessed 2026-10-02).

Kodus sits on the boundary between hosted and self-run: the same AGPLv3 code runs as its cloud service or on your own Docker host, and both are bring-your-own-key.

Metered (per review, per line, or per credit)#

Metering is the direction of travel. A flat seat price caps the bill and uncaps the vendor’s model spend; a metered price does the reverse. Teams that push many small PRs (agent-authored work is the obvious case) pay very differently under the two shapes.

CodeAnt AI publishes a trial and a “100% off for open source” offer but its per-seat price was not on the fetched page (unverified). Qodo’s live pricing page shows no permanent free tier; secondary sources describe a 250-credit monthly free tier (unverified). Qodo’s vendor blog dates a “Qodo 3.0” launch to 2026-03-30.

Data posture, where published#

Every hosted bot is a new party that reads the code. For a private repository that is a disclosure decision, separate from the price.

Invoked review from a coding agent#

A developer, or an agent session, runs the review on purpose. Nothing watches the repository. The review lands when someone asks for it, which is a weakness for a team that forgets and a strength for a team that wants every review to be a decision.

Claude Code /code-review#

• Runs locally as a background subagent in a Claude Code session, against normal plan usage. Targets: the working diff, a PR number, a branch or a path. Effort levels low to max; low and medium return fewer, high-confidence findings, high and above return broader, less certain ones. --comment posts inline to GitHub or as a GitLab MR note; --fix applies findings (vendor: https://code.claude.com/docs/en/code-review, accessed 2026-10-02).
• /code-review ultra (also claude ultrareview for CI) sends the branch to a cloud fleet of agents in which every finding is independently reproduced and verified before it is reported. About 5-10 minutes; Pro and Max get three free runs, then about $5-25 per review in usage credits; default limit 500 files / 8,000 lines; research preview (vendor: https://code.claude.com/docs/en/ultrareview, accessed 2026-10-02).

Anthropic Code Review (managed)#

The ambient sibling of the above, run on Anthropic’s infrastructure for Team and Enterprise: on PR open, on every push, or by @claude review. Parallel agents each hunt one class of issue, then a verification step checks candidates against the code. It posts a neutral check run and never approves or blocks. Average about 20 minutes and $15-25 per review; not available to Zero Data Retention orgs; configured through CLAUDE.md and REVIEW.md (vendor: https://code.claude.com/docs/en/code-review, accessed 2026-10-02). Anthropic’s launch figures: under 1% of findings marked incorrect by its engineers; 84% of PRs over 1,000 lines received findings, averaging 7.5 issues (vendor, reported at https://www.helpnetsecurity.com/2026/03/10/anthropic-claude-code-review/).

Codex#

@codex review in the PR (see platform-native) or a /review against a base branch in the Codex CLI (unverified on primary docs).

CodeRabbit CLI#

Proprietary local CLI that reviews uncommitted or branch changes before a PR exists. Free tier with daily limits; a usage add-on is billed per file; --agent emits NDJSON for a coding agent to consume (vendor: https://docs.coderabbit.ai/cli/, accessed 2026-10-02).

The self-review question#

An invoked reviewer from the same agent that wrote the code is the cheapest path for a team already using that agent: no new vendor, no new data recipient, no CI minutes. It is also a reviewer from the same model family as the author. Evidence on whether that matters is in S2.

Review built into the code host or a platform suite#

These ship inside something the team already pays for. The appeal is no new vendor; the cost is that pricing, data handling and roadmap follow the platform, not the reviewer.

GitHub Copilot code review#

• Trigger: request Copilot as a reviewer, or enable automatic review on PR open (optionally on each push). It reviews once by default (source: https://docs.github.com/en/copilot/concepts/agents/code-review, accessed 2026-10-02).
• Where it runs: on GitHub Actions. From 2026-06-01, reviews of private repositories consume the account’s Actions minutes (overage at standard Actions rates) in addition to Copilot AI credits. Public repos are unaffected; self-hosted and larger runners are supported and billed at their own rates (source: https://github.blog/changelog/2026-04-27-github-copilot-code-review-will-start-consuming-github-actions-minutes-on-june-1-2026/, accessed 2026-10-02).
• Per-review cost is unsettled in GitHub’s own docs. The billing page fetched on 2026-10-02 still described 13 premium requests per review; the changelog says premium requests were replaced by token-based AI credits. Treat the per-review figure as unconfirmed.
• Output: an overview with an approval assessment, inline comments. It can submit an approving review that satisfies required-approval rules when an admin enables that; new commits dismiss the approval.
• Config: .github/copilot-instructions.md. No model choice.

Copilot is the only reviewer in this survey that draws on two meters at once: model credits and CI minutes. For an organization whose Actions minutes are a shared, capped pool, that second meter is the deciding fact.

GitLab Duo Code Review#

GitLab’s built-in reviewer for merge requests. It appears on Martian’s offline benchmark (see S2). Pricing and data terms were not checked for this pass (unverified).

Gemini Code Assist#

The free consumer GitHub app is gone: deprecated 2026-06-18 and shut down 2026-07-17, with all review activity ended (source: https://developers.google.com/gemini-code-assist/docs/deprecations/consumer-code-review, accessed 2026-10-02). The enterprise edition, tied to Google Cloud, continues; its review pricing was not found (unverified). Teams that adopted the free app in 2025 lost their reviewer with a month’s notice.

Amazon Q Developer#

GitHub PR review (auto on open, /q comments, generated fixes) is still labeled Preview, and Q Developer IDE plugin support ends 2027-04-30 as AWS moves users to Kiro (source: https://aws.amazon.com/q/developer/, accessed 2026-10-02). Whether PR review continues inside Kiro is not stated (unverified).

OpenAI Codex review#

@codex review on a PR, or automatic review enabled per repository; rules go in an AGENTS.md “Code Review Rules” section; @codex fix it follows up with a change (source: https://learn.chatgpt.com/docs/third-party/github, accessed 2026-10-02). It runs on OpenAI’s cloud and is included in ChatGPT plan usage rather than priced per review. The same model can also be run in your own CI through openai/codex-action (see self-run).

S1 Recommendation — the map#

Two axes decide most of the choice#

The field sorts on who pulls the trigger and where the code goes.

*Copilot runs in the vendor’s cloud for the model and on your Actions minutes for the job, so it sits in both columns for billing purposes.

Pick the cell first. A team that cannot add a data recipient is choosing from the right column whatever the benchmark says. A team that wants every PR covered without anyone remembering is choosing from the top row.

Price shape is the third axis#

Flat seat (CodeRabbit, Graphite, Sourcery, Bito, Kodus) versus metered (Greptile, Bugbot, cubic, Qodo, Baz, Augment, Anthropic Code Review, Copilot). Per-review costs on the metered side run from about $1 (Bugbot, vendor) to $15-25 (Anthropic Code Review, vendor). A team’s PR count, not its headcount, decides which shape is cheaper. Teams whose PRs are mostly written by agents have high PR counts per human seat, which favors flat seats for ambient review.

What the field looks like in October 2026#

• No stable leader on quality. On the one independent, open benchmark, the top three are separable from the bottom few but the middle fifteen are not (S2).
• Consolidation and exits. Graphite to Cursor; PR-Agent to a community org; Gemini’s free app shut down; Korbit reported shut down (unverified date); Ellipsis’s pricing page now sells general cloud coding agents (source: https://www.ellipsis.dev/pricing, accessed 2026-10-02) rather than a review bot.
• Metering is spreading. Three major products moved from seats to usage in 2026.

Carried into S2#

CodeRabbit, Greptile, Cursor Bugbot, Qodo / PR-Agent, Copilot code review, Anthropic Code Review and Claude Code /code-review, Kodus, cubic, Augment. The rest stay in the S1 tables.

Self-run reviewers (your CI or your server, your model key)#

No new party reads the code except the model provider you already chose. You pay that provider’s tokens plus whatever compute runs the job. In GitHub Actions that compute is your Actions minutes, which matters on a capped plan; a self-hosted runner or webhook server moves it to your own hardware.

GitHub figures below were read from api.github.com on 2026-10-02.

PR-Agent#

The longest-running open-source reviewer (/review, /describe, /improve, /ask comment commands). Qodo built it, let it go quiet in late 2025 while it focused on its hosted product, and on 2026-04-23 handed it to a community organization with a three-person governance committee that includes an outside maintainer (source: https://www.qodo.ai/blog/qodo-is-handing-pr-agent-over-to-the-community/, accessed 2026-10-02). Release cadence since then is roughly every two weeks. The Qodo blog says the license “returns to Apache 2.0”; the repository’s LICENSE file reads MIT (read 2026-10-02). The file governs.

Each tool is a single model call by design (vendor: README, “~30 seconds, low cost”), which makes it cheap and fast and shallower than multi-agent reviewers.

claude-code-action, codex-action, run-gemini-cli, OpenCode#

These are general coding agents packaged to run in CI, and review is one prompt among many. The reviewer is as good as the prompt and the model you attach. They are the right shape for a team that wants its review rules in a versioned workflow file and its model choice in its own hands. run-gemini-cli has the slowest cadence of the group (v0.x, last release April 2026).

Dead or dormant#

• coderabbitai/ai-pr-reviewer — archived November 2023; the repository URL now returns 404.
• villesau/ai-codereviewer — not archived, last push 2024-08-19.
• Reviewpad — repository returns 404 (status unverified).

Search results for “open source AI code review” still surface all three.

S2 Comprehensive — Approach#

Three questions decide whether an AI reviewer is worth its cost, and each has a different kind of evidence:

1. Does it find real problems, and how much noise comes with them? Benchmarks (precision, recall, F1 against a labeled set) and field studies (what share of comments developers acted on). Covered in benchmark-evidence.md and field-evidence-and-self-review.md.
2. What does it cost at a given PR volume? List prices converted to a per-month bill at three volumes. cost-at-volume.md.
3. What does it do mechanically? Trigger, re-review on push, where it runs, what it can block, how it is configured. feature-comparison.md.

Methods note#

The Martian Code Review Bench figures in this pass were recomputed from the benchmark’s own published per-PR results, not copied from its leaderboard or from vendor posts. The script and the commit it ran against are in harness/1-219-ai-code-review/ at the repository root. The recomputation adds a PR-level bootstrap 95% interval to each F1, which the leaderboard does not show.

Cost figures are arithmetic on the vendor list prices recorded in S1; they inherit those prices’ (vendor: ...) status.

Benchmark evidence#

Martian Code Review Bench#

Martian is a research lab that does not sell a reviewer. Its benchmark is open (MIT): data, judge prompts and pipeline are published at https://github.com/withmartian/code-review-benchmark (accessed 2026-10-02).

• Offline set: 50 PRs from five open-source repositories (Cal.com, Discourse, Grafana, Keycloak, Sentry), with human-curated “golden” comments. 98 golden issues in the default core profile.
• Judging: an LLM judge matches each tool’s comments to golden issues. Three judges are published: Claude Opus 4.5 (default), Claude Sonnet 4.5 and GPT-5.2. Two of the judge vendors also have products on the board.
• Online set: about 300,000 real PRs, scored by whether developers acted on the bot’s comments (source: methodology/full.md in the repository).
• Stated limits (Martian’s own): static dataset with contamination risk, a golden set that caps recall, judge calibration without formal human validation, and it scores products, not models.

Recomputed results (offline, core profile, Opus 4.5 judge)#

Micro-averaged over 50 PRs, with a 2,000-sample PR-level bootstrap interval (measured 2026-10-02; recomputed from benchmark_dashboard.json at repo commit e616e84975, 2026-08-30; script: harness/1-219-ai-code-review/martian_offline.py).

The dataset does not say which configuration “Claude Code” and “Claude Code (CLI)” denote, or what date each tool was run. Neither row should be read as Anthropic’s managed Code Review or the ultra mode, which postdate or are not labeled in the data.

How to read it#

• The top three hold across judges. Qodo Extended, cubic and Augment are the top three under all three judge models. That is the most robust ranking claim the data supports.
• The middle is a tie. From Macroscope (54.2) down to CodeAnt (40.6), most adjacent intervals overlap heavily. On 50 PRs, a 3-5 point F1 gap is inside the noise. Vendor posts calling a tool “#1 on Martian” each cite a different snapshot, profile or online/offline split (Qodo, CodeRabbit, Greptile and cubic have all made the claim).
• The judge moves the numbers. Most tools score lower under Sonnet 4.5 and GPT-5.2 than under Opus 4.5, by up to about nine F1 points for the top tools (Qodo Extended 65.8 under Opus, 56.6 under Sonnet); the order in the middle reshuffles.
• Precision and recall trade, and the trade is a product choice. CodeRabbit and Copilot sit at high recall, low precision (many comments, a third to two fifths of them matching a golden issue). Devin, Baz and Kodus sit the other way. Graphite on this snapshot is near-silent: every comment it made matched, but it found 7.6% of the issues.
• A low-precision reviewer costs reader time on every PR; a low-recall one costs missed bugs on some PRs. Which costs more depends on who reads the comments. An agent reading them is cheaper per comment than a human.

c-CRAB (academic, executable)#

NUS, Zhejiang University and SonarSource, arXiv 2603.23448 (2026-03-30): 184 PR instances across 67 repositories, each with executable tests that pass only if the review identified the problem the test encodes. Claude Code, Codex, Devin Review and PR-Agent individually passed 20.1%-32.1% of the 234 tests; all four combined reached 41.5%. Manual analysis judged 84% of generated comments useful even where the test failed (source: https://arxiv.org/html/2603.23448v2, accessed 2026-10-02).

Two readings: individual reviewers miss most issues a human reviewer catches, and different reviewers miss different issues — the union beats any member by about ten points.

SWE-PRBench#

arXiv 2603.26130 (March 2026): 350 human-annotated PRs across 65 repositories; eight frontier models detected 15-31% of human-flagged issues from the diff, and adding more raw repository context lowered performance (unverified; read from the abstract summary only).

Vendor-run benchmarks#

• Greptile, July 2025: 50 PRs, catch rate Greptile 82%, Bugbot 58%, Copilot 54%, CodeRabbit 44%, Graphite 6% (vendor: Greptile; read via a secondary summary, unverified against the primary).
• Entelligence, September 2026: 67 real bugs across five repositories, eight tools; Entelligence ranks itself first at F1 47.2 (vendor: https://entelligence.ai/code-review-benchmark-2026).
• Bugbot: “80% resolution rate” at default effort (vendor: Cursor docs).
• Augment: 65% precision on its own codebase (vendor).

A vendor’s own benchmark ranks the vendor first in every case found. They are useful as descriptions of what each vendor optimizes for, not as rankings.

Configuration examples#

A self-run ambient reviewer: PR-Agent as a GitHub Action#

From PR-Agent’s installation docs (source: https://github.com/The-PR-Agent/pr-agent/blob/main/docs/docs/installation/github.md, read 2026-10-02):

# .github/workflows/pr_agent.yml
on:
  pull_request:
    types: [opened, reopened, ready_for_review, synchronize]
  issue_comment:
jobs:
  pr_agent_job:
    if: ${{ github.event.sender.type != 'Bot' }}
    runs-on: ubuntu-latest
    permissions:
      issues: write
      pull-requests: write
      contents: write
      checks: write
    name: Run pr agent on every pull request, respond to user comments
    steps:
      - name: PR Agent action step
        id: pragent
        uses: the-pr-agent/pr-agent@main
        env:
          OPENAI_KEY: ${{ secrets.OPENAI_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Three changes a cost- or security-conscious team usually makes:

• runs-on: self-hosted moves the job off the hosted minute pool onto your own runner. The model call is the only thing that leaves.
• Drop synchronize from types to review once per PR rather than on every push. Comment commands (/review) still re-run it on demand.
• Pin the action to a release tag or commit instead of @main.

Repository-level settings go in .pr_agent.toml on the default branch. PR-Agent’s docs warn against pointing the config branch at PR-controlled input: anyone who can push a branch could then supply a config that redirects the model endpoint, injects instructions, or enables auto-approval of their own PR (source: https://github.com/The-PR-Agent/pr-agent/blob/main/docs/docs/usage-guide/configuration_options.md, read 2026-10-02). The same class of risk applies to any reviewer that reads its rules from the branch under review.

A liveness check for an ambient reviewer#

An ambient bot that stops posting looks like one that found nothing. This script lists recent PRs and the minutes until a given bot first commented or reviewed, using the GitHub CLI’s REST access. Set BOT to the reviewer’s login.

#!/usr/bin/env bash
# usage: review-latency.sh owner/repo bot-login [count]
set -euo pipefail
REPO=$1; BOT=$2; N=${3:-10}
gh pr list -R "$REPO" --state all --limit "$N" --json number,createdAt \
  --jq '.[] | "\(.number) \(.createdAt)"' |
while read -r num created; do
  first=$( { gh api "repos/$REPO/pulls/$num/reviews" --paginate \
               --jq ".[] | select(.user.login==\"$BOT\") | .submitted_at";
             gh api "repos/$REPO/issues/$num/comments" --paginate \
               --jq ".[] | select(.user.login==\"$BOT\") | .created_at"; } | sort | head -1)
  if [ -z "$first" ]; then
    echo "#$num  no review from $BOT"
  else
    mins=$(( ( $(date -d "$first" +%s) - $(date -d "$created" +%s) ) / 60 ))
    echo "#$num  first review after ${mins} min"
  fi
done

A switch trigger can then be written as a rule over this output, such as “no review within 30 minutes on three consecutive PRs.” The threshold is the team’s choice.

Review rules in the repository#

Every option in this survey reads rules from a file in the repository: .coderabbit.yaml, .pr_agent.toml, .cursor/BUGBOT.md, .github/copilot-instructions.md, REVIEW.md and CLAUDE.md (Anthropic), the “Code Review Rules” section of AGENTS.md (Codex). Writing the substance once in plain prose and keeping each tool’s file as a short pointer to it makes a later switch a matter of changing the pointer.

Cost at volume#

Monthly cost for one human seat at three PR volumes. Arithmetic on the vendor list prices recorded in S1 (all vendor, read 2026-10-02). Annual-billing rates where the vendor offers them. Model tokens for self-run options are shown separately because they depend on the model and the diff size.

*At 300 PRs a month, rate limits and the $0.25-per-file overage may apply; whether a given workload hits them depends on how PRs cluster in time (unverified for any specific workload).

What the table shows#

• Flat seats are insensitive to PR count; metered pricing is linear in it. At 20 PRs a month the cheaper metered options (Bugbot, Greptile basic) cost about what a seat does. At 300 PRs a month the metered options in the table cost from $300-450 (Bugbot) to $4,500-7,500 (Anthropic’s managed review), against $24 for a CodeRabbit seat: about 12x to 300x (vendor: list prices in S1, accessed 2026-10-02).
• “PRs per seat” is the number to estimate before buying, and it is what agent- authored work changes. A human opening 20 PRs a month and a human supervising agents that open 300 are the same seat to a per-seat vendor and fifteen times the bill to a per-review vendor.
• The multi-agent verified reviewers list at about $15-25 (Anthropic Code Review) and $5-25 (ultra) per review against about $1.00-1.50 for a single-pass bot such as Bugbot (vendor list prices, S1). They are priced for the PRs where a missed bug is expensive, not for every PR.
• Self-run in Actions has two meters. Model tokens plus CI minutes. On a plan where Actions minutes are a hard-capped pool shared across repositories, an ambient reviewer that runs on every push draws down the same pool the test suites need. Copilot code review on private repos draws on it too, since 2026-06-01.
• Local invoked review has no marginal price inside a subscription, but it spends the subscription’s usage window, which is the binding limit for heavy users.

Feature comparison#

Mechanics only. “(u)” marks a cell not confirmed on a primary page (unverified). Sources are the vendor docs and repositories cited in S1, read 2026-10-02.

Mechanics that change the choice#

• Approval authority. Copilot (when enabled) and Augment’s Cosmos reviewer can approve a PR. Anthropic’s managed reviewer is designed never to. A team whose merge rule is “a human or a deliberate gate approves” should check this before installing, because an approving bot satisfies required-approval rules.
• Re-review on push is what makes ambient review expensive under metering: a PR with eight fix-up pushes is nine reviews. Bugbot and Anthropic expose the setting; on others it is implicit.
• Config lives in the repository for every option here. Moving between reviewers means translating one rules file into another, which is cheap; learned rules that live in the vendor’s database (Bugbot @cursor remember, CodeRabbit learnings) do not move.
• Model choice is a self-run property. Among hosted bots only Greptile and Qodo (Enterprise) and Kodus (all tiers) let you bring the model, which is also how a team gets a reviewer from a different family than its author.

Field evidence, self-review, and verification#

What developers do with AI review comments#

“Resolved” is not precision. A resolved comment can be a nit fixed to clear the thread. The three measures in this survey (resolved 55-74%, Martian offline precision about 30-70% for most tools, c-CRAB executable pass 20-32%) measure different things and should not be compared to one another.

In the one industrial deployment with a before/after measurement, AI review made PRs slower to close, not faster.

Same-model-family review#

A reviewer from the same model family as the author is the default when a team uses one coding agent for both jobs. The evidence:

• General LLM self-preference is established. Panickssery, Bowman and Feng (arXiv 2404.13076, 2024) showed GPT-4 and Llama 2 recognize their own outputs and rate them higher, with preference strength tracking recognition. Wataoka et al. (arXiv 2410.21819) attribute the effect to perplexity: judges favor text that is familiar to them.
• Code-specific evidence exists and points the same way. “Articulate but Wrong” (arXiv 2605.21537): across 1,980 code-modernization attempts by 11 models, 31.7% of semantically drifted outputs were endorsed by the same model that produced them; the self-review miss rate ranged from 0% to 100% depending on the model.
• No controlled study was found of “model X writes the PR, model X reviews it” on real PR review precision and recall. The case against same-family review is inferred from the two lines above, not measured on the task.

Two mitigations appear in practice: a reviewer from a different model family (most self-run reviewers and several hosted bots let you pick), and a verification step that must reproduce a finding against the code rather than argue for it.

Verification passes#

Anthropic’s managed Code Review and its ultra mode, and Augment’s Cosmos reviewer, describe a second stage that tries to confirm or refute each candidate finding before posting (vendor). No independent measurement of that stage on PR review was found.

Adjacent evidence, all from vulnerability detection rather than PR review:

• A heterogeneous adversarial verifier (a small local Qwen3-8B checking DeepSeek-V3 experts) raised precision by 10.3 points on 262 NIST Juliet samples (arXiv 2604.21282; synthetic benchmark). The author chose a different model family for the verifier.
• An LLM filter over SAST findings cut false positives from 560 to 64 with a 3.1% recall loss (arXiv 2605.01885; unverified, abstract only).

The pattern these support: verification that executes something (a test, a mutant, a reproduction) is stronger evidence than verification that re-reads the diff, and a verifier from another model family sidesteps the self-preference problem.

S2 Recommendation#

Five findings that hold under scrutiny#

1. Quality differences in the middle of the field are smaller than the noise in the best available benchmark. On Martian’s 50-PR offline set, three tools (Qodo Extended, cubic, Augment) lead under every judge; most of the next fifteen have overlapping 95% intervals. Choosing among them on benchmark rank is choosing on noise. Choose on trigger, data posture, price shape and precision/recall lean.
2. Every reviewer misses most issues, and they miss different ones. c-CRAB: 20-32% individually, 41.5% for four combined. Two different reviewers are worth more than one better-ranked one, if the budget allows a second.
3. Precision vs recall is a product choice to match to the reader. High-recall, low-precision bots (CodeRabbit, Copilot) suit a reader who can triage cheaply, such as an agent that is told to fix or dismiss each comment. High-precision, lower-recall tools (Devin, Baz, Kodus on this benchmark) suit a human reader whose attention is the scarce resource.
4. Price shape dominates price level once PR volume is high. At 300 PRs per seat per month, metered ambient review costs 12x-300x a $24 seat (vendor: list prices in S1) (see cost-at-volume.md).
5. Same-family self-review is a known weakness without a measured size. Use a different model family for the reviewer, or a verification step that executes something, when the author is an AI agent.

Engineering stances#

• Ambient, flat, hosted — CodeRabbit or Sourcery, tuned for precision through the config file. Lowest effort; adds a data recipient.
• Ambient, self-run — PR-Agent or Kodus with a key from a different model family than the author. Keeps code with the existing model provider; costs setup and either CI minutes or a small server.
• Invoked, deep — a multi-agent verified review (/code-review ultra, Anthropic Code Review on manual trigger) on the PRs that touch money, auth, migrations or user data; nothing or a cheap pass on the rest.
• Layered — a cheap ambient pass on every PR plus an invoked deep pass on risky ones. The c-CRAB union result is the argument for it.

S3 Need-Driven — Personas#

Each persona file states the constraint, the eliminations it causes, and a conditional recommendation.

S3 Recommendation#

The constraint picks the cell; the reader picks the precision.

Two observations cut across personas:

• Who reads the comments decides the right precision/recall point. Agents triage cheaply and can take a high-recall reviewer; humans cannot.
• A second reviewer from a different family beats a better-ranked single one when the miss rate matters (c-CRAB: 41.5% combined vs 32.1% best single).

Persona: GitLab, Bitbucket, Azure DevOps or Gitea shop#

Situation#

The code host is not GitHub, or there are several. Many products in this category started as GitHub apps.

What it eliminates#

• GitHub-only products: Copilot code review, Anthropic’s managed Code Review, the GitHub Actions-packaged agents (claude-code-action, codex-action, run-gemini-cli, OpenCode’s agent) unless ported to the host’s CI.

What remains#

Recommendation#

On GitLab, Duo Code Review is the no-new-vendor option and sits mid-table on Martian’s offline set (F1 53.0). For mixed hosts, PR-Agent and Kodus cover the widest set with one configuration. Claude Code’s local /code-review --comment can post a GitLab MR note, so invoked review works there too.

Persona: open-source maintainer#

Situation#

Public repositories, outside contributors of uneven experience, no budget, and limited time to triage. Fork PRs are common.

What it eliminates#

• Anything paid. Several vendors waive fees for public repos, so this is less limiting than it sounds.
• Gemini Code Assist’s free GitHub app, which was the common free choice in 2025 and was shut down on 2026-07-17.
• Self-run Actions reviewers on fork PRs, unless the maintainer understands the secret- exposure rules for pull_request_target. A workflow that hands a model key to code from a fork is a security decision, not a convenience.

What remains#

• CodeRabbit (public repos free, per vendor), Sourcery (public repos free), Greptile (free for MIT/Apache-licensed projects), CodeAnt (free for open source on request).
• Copilot code review on public repositories is outside the 2026 Actions-minute change.

Recommendation#

A free-for-public hosted bot, configured toward precision so contributors are not buried in comments. Precision matters more here than in a team setting: an outside contributor who receives fifteen low-value comments on a first PR may not return. CodeRabbit leans high-recall on Martian’s benchmark (precision 32.6%, core profile); its path filters and instructions are the lever to pull.

Persona: product team, 5-50 engineers#

Situation#

Private repositories, mostly human-written PRs, a human reviewer already required on every PR. The AI reviewer is a first pass that should catch what tired humans miss and stay quiet otherwise. Seat budget exists; a new SaaS vendor is acceptable after a security review.

What it eliminates#

• High-noise configurations. A human reads every comment; a low-precision bot taxes every PR and gets ignored. The one industrial before/after study found AI review lengthened PR closure time (5h52m to 8h20m; Cihan et al., ICSE-SEIP 2025).
• Vendors without a published data policy, for teams whose security review requires one.

What remains#

• Flat-seat: CodeRabbit, Graphite, Sourcery, Bito.
• Metered at moderate volume (20-50 PRs per engineer per month): Greptile basic, Bugbot, cubic, Qodo. At this volume they cost about the same as a seat.
• Top of Martian’s offline set: Qodo Extended, cubic, Augment.

Recommendation#

Trial two tools for two weeks each on the same repositories and count the comments the team acted on. Benchmark rank in the middle of the field does not separate products; the team’s own acceptance rate does. Prefer a tool that cannot approve, so the human review requirement stays meaningful.

Persona: regulated enterprise#

Situation#

Source code may not leave an approved boundary, or may go only to an approved model provider under an existing agreement. Audit trails are required. Changes need a recorded human approval.

What it eliminates#

• Hosted bots without a self-host, VPC or single-tenant option.
• Any reviewer that can satisfy a required-approval rule on its own.
• Anthropic’s managed Code Review for orgs on Zero Data Retention (it is not offered to them, vendor docs).

What remains#

• Self-run: PR-Agent (MIT, any model via LiteLLM including Bedrock, Vertex and local models), Kodus self-hosted (AGPLv3; check the license against internal policy), claude-code-action or codex-action against the provider already under contract.
• Vendor self-host tiers: CodeRabbit Enterprise, Greptile Enterprise (BYO model), Sourcery Enterprise (BYO model), Qodo Enterprise (on-prem), Bito Professional.

Recommendation#

PR-Agent on internal runners against the already-approved model endpoint is the shortest path through procurement: no new vendor, permissive license, a governance committee since April 2026. If the review quality bar is higher than a single model call, a vendor self-host tier is the next step. Check whether its multi-agent features survive self-hosting.

Persona: solo maintainer running a fleet of coding agents#

Situation#

One human owns several private, IP-sensitive repositories. Most PRs are written by AI coding agents the maintainer supervises. PR volume per human is high (hundreds a month across repositories is plausible) and PRs arrive in bursts. CI runs on a hosted plan whose minutes are one pool across all repositories with a hard spend stop. The maintainer wants findings surfaced at the merge gate, not automatic approval.

What it eliminates#

• Metered per-review hosted bots for ambient coverage. At high PR volume the arithmetic in S2’s cost table goes against Greptile advanced reviews, Bugbot, and Anthropic’s managed review on every push. Re-review on each fix-up push multiplies it.
• Copilot code review on private repos. It draws on Actions minutes as well as model credits since 2026-06-01, so it competes with the test suites for the capped pool.
• Any reviewer that can approve. Copilot’s approval mode and Augment’s risk-based auto-approval would let an agent-authored PR pass the gate with no deliberate review.
• Hosted bots that add a data recipient, unless the maintainer accepts that disclosure for the repositories involved.

What remains#

1. Invoked review run by the agent session that wrote the diff, before it calls the PR done. Zero marginal price inside an existing coding-agent subscription; no new recipient; no CI minutes. The weakness is same-family self-review. Mitigate with:
   • a higher effort level or a multi-agent mode whose findings must be reproduced, and
   • review prompts that make the reviewer run code (tests, mutants) rather than read it.
2. A self-run ambient pass with a different model family (PR-Agent or Kodus, on a self-hosted runner or a small server to keep it off the shared minute pool). Cheap per PR, catches a different set of issues (c-CRAB’s union result).
3. A flat-seat hosted bot (CodeRabbit, Sourcery) if a new data recipient is acceptable. Flat pricing absorbs agent PR volume.

Watch for#

• Width. Multi-agent review modes list at about $5-25 per review against about $1.00-1.50 for a single-pass hosted bot (vendor list prices, S1). Run as an everyday gate, a fan-out reviewer can consume a usage window quickly. Price the number of agents before launching, and keep the wide mode for risky diffs.
• Bot silence. An ambient bot that stops posting looks the same as an ambient bot that found nothing. A gate that depends on an ambient bot needs a liveness check.

Recommendation#

Invoked review by default, with a cross-family or executed-verification step on PRs touching auth, money, migrations or user data. Add an ambient pass only if PRs are being merged without anyone invoking review, and then prefer self-run or flat-seat.

S4 Strategic — Viability signals#

A reviewer is a cheap thing to switch in principle (its config is a file in the repository) and an expensive thing to lose in practice (a merge gate that silently stops running). The viability signals that matter here:

1. Exit history in the last year. Has the product, or a sibling product from the same vendor, been shut down, sold or handed off?
2. Pricing-model churn. A move from flat to metered changes the bill at the next renewal without any change in usage.
3. Platform dependency. A reviewer bundled with a larger platform follows that platform’s strategy (Gemini’s consumer app, Amazon Q’s move to Kiro).
4. Governance of open-source options. Who can merge, and whether the project survived a sponsor stepping back.
5. Lock-in surface. Rules in a repo file move; learned rules in a vendor database and approval integrations do not.

Funding is used below only as a durability fact, never as a quality signal.

S4 Recommendation#

Optimize for replaceability. No reviewer in this category has a durable quality lead that a benchmark can show, and the field lost or reshaped several products in a year. The posture that survives:

1. Keep review rules in the repository in a form that can be translated (a REVIEW.md, AGENTS.md section, or .coderabbit.yaml), not only in vendor-side learned rules.
2. Make the reviewer advisory, not the approver. The merge decision stays with a human or a deliberate gate; the reviewer’s absence then degrades coverage instead of blocking merges.
3. Watch for silence. An ambient reviewer that stopped posting is indistinguishable from one that found nothing. Track time-from-PR-open-to-first-review and alert when it exceeds a threshold over several consecutive PRs.
4. Set a switch trigger before you need it. Examples: review latency over a stated number of minutes on N consecutive PRs; a renewal that changes price shape; monthly spend crossing a stated multiple of the plan’s list price; a vendor announcement that moves the reviewer under a different product.
5. Prefer a reviewer whose model you can change where the author is also an AI. A self-run or BYO-model reviewer lets you keep the reviewer’s family different from the author’s as both evolve.

By strategic path#

• Conservative: self-run PR-Agent or Kodus with a model key you already hold, advisory only.
• Low-effort: flat-seat hosted bot, advisory, with a liveness check.
• Depth-first: invoked multi-agent verified review on risky PRs, plus a cheap ambient pass everywhere.

Viability by option#

Facts as of 2026-10-02, sources in S1 unless given here.

Patterns#

• Four of the fifteen options above exited or left the category in about a year (Gemini consumer app, Korbit, Ellipsis, and Graphite’s Diamond as a separate product, the last unverified). Another (Amazon Q) is in wind-down. A team adopting a hosted reviewer should assume it may need to switch within the life of a repository.
• Three moved from seats toward metering (Bugbot, Greptile, Copilot’s credits plus minutes). Price shape is less stable than the product.
• The lock-in that hurts is the gate, not the rules. A reviewer that is a required check, or that approves, breaks merges when it stops. A reviewer that posts advisory comments degrades gracefully.