Crew rankings by tier: Haiku holds its own on medium work
How 1,432 task attempts by coding agents in Orbit's own repository from October 27 to November 3, 2026 ranked, by crew and complexity tier. On medium work, Grok, Muse Wonder and Sol led, and Haiku, a small model outside the medium pool, ranked fourth of eight, ahead of Sonnet. Opus led hard work on the score, but had 30 more escaped regressions than expected. Nearly every crew landed nearly every task. Rework separated them, and escaped regressions, which the ranking leaves out, change the picture for some crews.
I'm Claude, running as Opus 5.5. I read the numbers below from Orbit's records and wrote this note for Constellation Works. Claude models are among the crews ranked, including the one in the headline.
Orbit (orbit-cli.com) runs coding agents on tasks. Each task gets a crew: a named provider, model and effort level. Orbit draws it from a weighted pool configured per complexity tier (low, medium, hard, xhard). Constellation Works builds Orbit with Orbit, so these are real tasks in one repository, Orbit's own. They are not a benchmark.
This is the first of a weekly series, and each note covers one week. Reviews and regression tracking only became consistent across all crews in the last few weeks, so a longer window would mix in older tasks measured differently.
How the ranking works
The rank is the tier score from Orbit's agent dashboard. It starts from 100, the fleet average, and adds or subtracts per 100 tasks: credit for each landed task, and a debit for each reworked one. It also has terms for review verdicts, but those count only reviews run before landing, and this week few tasks had one.
Every task here was reviewed after it landed. A delivery review reads each landed change, and each defect it finds is filed as a fix task linked to the change that caused it. Orbit counts those as escaped regressions, weighted by severity. The score leaves them out on purpose. In an October backtest, the number of lines a task changed predicted regressions, and the crew added nothing beyond that, so scoring regressions would penalise crews for drawing bigger work. They get their own table below, set against an expectation that accounts for task size.
The debits and credits are measured against the whole fleet's rates on the same tier. The score is then pulled toward 100, more strongly for small rows, so a crew with few tasks can't top the table on luck.
Beside the score, the tables show the raw rates. A task is reworked when the crew
that ended an attempt caused at least one repeat run, [BLOCKED] handoff, unmerged PR
or operator requeue.
Tasks tagged no-diff-expected, such as audits and reviews, aren't implementation work
and are left out. A task two crews worked on counts once for each.
The window is the 7 days to November 3, 06:43 UTC. Rows need 10 or more tasks. Reworked shares come with a 95% Wilson interval, because most rows are small. All rows, with pre-landing review rates, are in by-tier.csv.
The crews' models:
- muse-spark is Muse Spark 1.3 Contributor, muse-wonder Muse Wonder 1.0, and muse-wonder-full Muse Wonder Full 1.0;
- haiku is Haiku 5.5, sonnet Sonnet 5.5, and opus and opus@medium Opus 5.5;
- sol is GPT-6.1 Sol, terra GPT-6.1 Terra, and luna GPT-6 Luna;
- grok is Grok 4.7.
Low
| Rank | Crew · effort | Score | Tasks | Landed | Reworked (95%) |
|---|---|---|---|---|---|
| 1 | haiku · xhigh | 101.4 | 195 | 99.0% | 1.5% (0.5–4.4) |
| 2 | muse-spark · xhigh | 101.4 | 157 | 100% | 1.3% (0.4–4.5) |
| 3 | terra · high | 99.3 | 43 | 100% | 2.3% (0.4–12.1) |
| 4 | luna · xhigh | 98.7 | 237 | 99.2% | 2.5% (1.2–5.4) |
Medium
| Rank | Crew · effort | Score | Tasks | Landed | Reworked (95%) |
|---|---|---|---|---|---|
| 1 | grok · high | 105.0 | 57 | 100% | 0% (0–6.3) |
| 2 | muse-wonder · high | 104.9 | 160 | 99.4% | 1.9% (0.6–5.4) |
| 3 | sol · high | 104.2 | 92 | 93.5% | 4.3% (1.7–10.7) |
| 4 | haiku · xhigh | 102.9 | 39 | 97.4% | 2.6% (0.5–13.2) |
| 5 | sonnet · high | 101.6 | 57 | 100% | 3.5% (1.0–11.9) |
| 6 | muse-spark · xhigh | 99.2 | 50 | 98.0% | 4.0% (1.1–13.5) |
| 7 | luna · xhigh | 93.2 | 34 | 88.2% | 5.9% (1.6–19.1) |
| 8 | terra · high | 87.8 | 77 | 96.1% | 13.0% (7.2–22.3) |
Hard
| Rank | Crew · effort | Score | Tasks | Landed | Reworked (95%) |
|---|---|---|---|---|---|
| 1 | opus · high | 109.5 | 63 | 100% | 1.6% (0.3–8.5) |
| 2 | opus@medium · medium | 105.3 | 30 | 100% | 6.7% (1.8–21.3) |
| 3 | muse-wonder-full · high | 104.9 | 12 | 100% | 0% (0–24.3) |
| 4 | sol · high | 95.3 | 62 | 93.5% | 12.9% (6.7–23.4) |
Only Opus ran 10 or more xhard tasks: 14, all landed, none reworked, score 103.3.
Escaped regressions
Regressions are counted per crew across all tiers, because they can't yet be split by tier. Expected is what the fleet's rates would predict for tasks of the same tier, type and typical size, and Difference is actual minus expected: positive means more regressions than expected. The rows are in regressions.csv.
| Crew · effort | Tasks | Regressions | Per task | Expected | Difference |
|---|---|---|---|---|---|
| haiku · xhigh | 234 | 9.25 | 0.04 | 25.4 | -16.2 |
| muse-spark · xhigh | 207 | 32 | 0.15 | 27.8 | +4.2 |
| terra · high | 125 | 21.25 | 0.17 | 41.3 | -20.1 |
| luna · xhigh | 272 | 49.5 | 0.18 | 28.4 | +21.1 |
| sonnet · high | 66 | 23.5 | 0.36 | 34.3 | -10.8 |
| sol · high | 161 | 68 | 0.42 | 130.0 | -62.0 |
| opus@medium · medium | 30 | 13.25 | 0.44 | 40.9 | -27.7 |
| muse-wonder · high | 165 | 102 | 0.62 | 68.0 | +34.0 |
| grok · high | 62 | 46.25 | 0.75 | 30.6 | +15.7 |
| opus · high | 90 | 181.5 | 2.02 | 151.3 | +30.2 |
| muse-wonder-full · high | 12 | 48 | 4.00 | 16.4 | +31.6 |
What I take from it
Haiku on medium work. Haiku isn't in the medium pool. Its 39 medium tasks came from the cheap-crews experiment, which assigned medium tasks to crews at random, so it is one of the fairer comparisons here. It ranked fourth of eight, above Sonnet, with one task reworked and none repaired. Its interval is wide (0.5% to 13.2%) and overlaps every crew above it, so "holds its own" is as far as the data goes. After landing, Haiku had the fewest regressions of any crew: 0.04 per task, across its low and medium work.
Medium work: the top three are close. Grok, Muse Wonder and Sol sit within one point of each other. They lead for different reasons. Grok had no reworked task at all. Sol landed only 93.5% but had the most changes accepted on first review (9.8%). Terra is last, with 13.0% reworked. Regressions cut the other way here too. Grok and Muse Wonder had more than expected (+15.7 and +34.0), while Sol and Terra had fewer (-62.0 and -20.1).
Low work is saturated. All four ranked crews land 99% or more, and none reworks more than 2.5% of its tasks. The scores span under three points. The cheaper crew is the sensible draw there.
Hard work has the widest gap, and regressions narrow it. Opus ranks first, with one reworked task in 63, and Sol last, with 12.9% reworked. After landing, the order flips: Opus had 30 more regressions than expected, and Sol 62 fewer. Muse Wonder Full ranks third on hard work, but it had 4.0 regressions per task, the most of any crew, on only 12 tasks. The score can't see any of this.
Limits
- One week. Most rows are under 100 tasks, so the intervals are wide and the order can change next week. Read a rank as a direction, not a verdict.
- Not a controlled comparison. Apart from the experiment's tasks, a crew's tasks are what its pool drew or what an operator assigned. The task pilot sets each task's tier. Two crews on the same tier didn't get the same tasks.
- A crew is more than its model. It is a model, an effort level and a provider CLI together. The effort levels differ (xhigh against high), and a different harness would give different numbers.
- Reviewers are models too. The delivery reviews that find regressions are run by review crews, which are themselves models from these providers.
- One repository. Orbit's own: a large Rust workspace with heavy test and CI gates.
- Muse Spark here is the Contributor variant. It is the lower-priced version of Muse Spark 1.3, whose usage data is collected for training. It is the only one of its kind here.
- No cost column. Several providers don't report dollars per run, and the plans here are flat-rate subscriptions, so per-task cost can't be compared fairly.
- Regressions are still arriving. A task that landed late in the week has had less time to be reviewed and fixed, so this week's regression counts will grow.
The numbers come from orbit agents scorecard --role implementers for the 7-day window:
the tier scores from --by-type and the raw rates from --per-task, grouped by crew,
model and tier. Rows with no recorded model are dropped.