Foundation Model Token Cost Comparison: 14 Amazon Bedrock Models Measured with a Coding-Agent Ralph Loop

Published 2026-10-13·68 min read

AI documentation guide

Reference Benchmark

This benchmark was built with reference to the Ralph loop completion by model report by Jung Do Hyun (정도현, Roboco) and its public repository. The task (a RealWorld backend implementation), the completion criteria, the structure of the Ralph loop harness and the Hurl tests used for grading follow the reference report. Running on Amazon Bedrock in us-east-1, the builder-plus-verifier two-agent setup, five runs per model and Bedrock-price cost calculation were defined for this report. The numbers here are therefore not results of the reference report and were not measured under its conditions.

Measurement Status

Measured — 14 models ran five times each on Amazon Bedrock in us-east-1, 70 runs in total, of which 69 completed. Thirteen models ran between 2026-10-12 22:28 and 2026-10-13 00:18 UTC. Claude Fable 5.1 ran separately from 2026-10-13 04:36 to 05:21 UTC, after the account's data retention setting was changed. Every number in the tables was generated by script from the run records (token usage, grading results, session transcripts).

• Completed69/7014 models
• Cheapest modelGPT-6 LunaMedian $0.027 per run
• Priciest modelGPT-6 AstraMedian $3.495 per run
• Cost ratio (max/min)130.4×
• Cache-read token share94%
• Total spend$106.02results $88.42 + discarded first attempt $17.60

Overview

This report measures, on Amazon Bedrock, how many tokens, how much cost and how many LLM requests each model needs for an AI coding agent to finish the same backend implementation task.

A Ralph loop gives the same prompt to a fresh agent session over and over. Only the repository (files and git history) carries over between iterations; the conversation does not. The agent therefore has to read the repository each time, work out what is left, and create a completion marker file when it judges the task done. The harness grades the result with the official tests after every iteration.

The models compared are frontier models callable on Amazon Bedrock (group 1) and Chinese open-weight models callable on Amazon Bedrock (group 2). Open-weight means the trained weights are published. The reference report ran each model with a different agent harness and provider API; this report ran every model with the same harness (pi coding agent) and the same API (Bedrock Converse), so the differences shown are differences between models under one set of conditions.

Summary

• Scope [fact]
  • 14 models ran the Ralph loop five times each on Amazon Bedrock in us-east-1, 70 runs in total.
  • Thirteen models ran between 2026-10-12 22:28 and 2026-10-13 00:18 UTC. Claude Fable 5.1 ran separately from 2026-10-13 04:36 to 05:21 UTC, after the account's Bedrock data retention mode was changed.
  • Each run used two agents built on the same model: a builder and a verifier.
  • Group 1 (frontier models callable on Bedrock) has 12 models; group 2 (Chinese open-weight models callable on Bedrock) has 2.
• Completion [fact]
  • 69 of 70 runs passed the official grader, which requires all 13 RealWorld Hurl test files (154 requests) to pass.
  • 13 models completed 5/5; GLM 5.3 completed 4/5.
  • Every model's median Ralph iteration count is 1. Only 3 completed runs needed two or more iterations.
• Cost [calculated: reported tokens × Bedrock US cross-Region inference prices as of 2026-10-12, median per run]
  • Lowest: GPT-6 Luna $0.027 and Claude Haiku 5.5 $0.065
  • Highest: GPT-6 Astra $3.495, Claude Fable 5.1 $2.566 and Claude Opus 4.8 $2.018
  • The most expensive model cost about 130 times as much as the cheapest.
• Group comparison [calculated]
  • The 2 Chinese open-weight models (Kimi K3 $1.422 and GLM 5.3 $1.821) have a group value (median of model medians) of $1.621, about 2.0 times the frontier group value ($0.831).
  • They used about 2.3 times the output tokens, 2.3 times the total tokens and 1.5 times the LLM requests.
  • The spread inside the frontier group (about 130×) is far larger than the gap between groups. The two models rank 9 and 10 of 14 by cost (cheapest first) and were cheaper than Claude Opus 5, Claude Opus 4.8, Claude Fable 5.1 and GPT-6 Astra.
  • Cost decisions should therefore be made per model, not per group.
• What drove cost [calculated]
  • 94% of all tokens in completed runs were cache reads: prompt prefixes stored by an earlier request and read again at a lower price than regular input.
  • Cache-read prices and volumes therefore mattered as much as output prices. For GLM 5.3, about 68% of the run cost came from cache reads.
• Claude Fable 5.1 [fact/calculated]
  • It became callable after the account's us-east-1 data retention mode was changed from inherit to aws_review, and completed 5 of 5 runs.
  • Its median cost per run was $2.566, rank 13 of 14 (cheapest first). Its input/output prices ($11/$55) equal GPT-6 Astra's and are the highest among the Claude models.
  • In aws_review mode, prompts and outputs of Fable 5 and 5.1 requests are retained inside the AWS boundary for up to 30 days and may be reviewed by AWS staff. They are not shared with the model provider.
• Comparison with the published benchmark [calculated]
  • Published runs of 6 models that the reference report ran with the same pi harness were re-priced at Bedrock prices and compared with this measurement.
  • Claude Opus 5.5, Claude Sonnet 5.5, Claude Haiku 5.5 and GPT-6 Sol matched this measurement (verifier included) within ±10%.
  • GPT-6.1 Sol came out about 1.6× higher and Kimi K3 came out about 1.9× higher.
• Total spend [calculated]
  • Runs used in the results cost $88.42 (including $12.87 for Claude Fable 5.1); the discarded first attempt cost $17.60; the total is $106.02.
  • The largest per-model spend, GPT-6 Astra at $16.82, stayed within the $500 per-model cap. The actual bill must be checked in AWS Cost Explorer.

Notation

This report labels values by their nature.

• [fact]: Observed directly by running or querying something, with the time of observation.
• [calculated]: Derived by a fixed rule, such as reported token counts multiplied by published prices. Not a billed amount.
• [estimate]: Involves an assumption, which is stated.
• [source]: The material a number or judgment rests on.

Values that cannot be confirmed from the material are not invented; they are marked "not verifiable" or "n/a".

Reference Benchmark and Task

Task and completion criteria [source: the reference report, PROMPT-en.md and harness scripts in its repository]

Every model receives the same task: implement the RealWorld backend API with TypeScript, Fastify, Prisma and SQLite. The scope covers JWT authentication, users and profiles, article CRUD, comments, favorites, tags and feeds.

A run counts as completed only if both conditions hold:

1. The agent verifies the completion conditions itself and creates a .ralph-done file at the repository root.
2. The harness grader re-runs the official RealWorld Hurl API tests (13 files, 154 requests) and all 13 files pass. Hurl is a test tool that runs HTTP requests and expected responses written in plain text files.

If .ralph-done exists but grading does not pass all 13 files, the harness deletes the file and starts the next iteration.

Safeguards

The reference harness has three safeguards, and this report used the same values.

• Stall watchdog: ends a session when none of the session transcript, logs or repository files change for 300 seconds.
• Run time cap: ends a run that exceeds 20 minutes (1,200 seconds).
• Cost guard: stops a run when the running cost estimate exceeds the configured limit.

Metrics in the reference report [fact: reference page data, checked 2026-10-12 15:40 UTC]

For each condition (model × harness), the reference report gives per-run elapsed time, commit count, output tokens, cost at each provider's public list prices, completion and iteration count. It has 28 conditions, 19 models and 117 runs, all completed. Because its costs are not Bedrock prices, this report re-priced the published token counts of the same models at Bedrock prices for comparison.

Models Compared

Classification and Bedrock availability [fact: checked 2026-10-12]

Availability was checked in three steps. First, the model cards and per-Region availability tables in the Bedrock User Guide: none of the target models is offered on an in-Region endpoint in us-east-1, so they are called through the US geographic cross-Region inference profiles (model IDs starting with us.). Second, a read-only role in the test account queried each model's agreement, authorization, entitlement and Region availability. Third, a short prompt was sent to every model.

The Bedrock model cards describe Kimi K3 and GLM 5.3 as an "open-weight model" and an "open weight text generation model" [source]. GLM 5.3 is not in the reference report: its glm-5.3-flash is not offered on Bedrock, so GLM 5.3, the model from the same family that Bedrock offers, was added.

Model in the reference report	Provider	Classification	us-east-1 Bedrock	In this report
Opus 5.5	Anthropic	Group 1 frontier	Callable	Measured
Sonnet 5.5	Anthropic	Group 1 frontier	Callable	Measured
Haiku 5.5	Anthropic	Group 1 frontier	Callable	Measured
Fable 5.1	Anthropic	Group 1 frontier	Callable (requires account data retention aws_review)	Measured
Opus 5	Anthropic	Group 1 frontier	Callable	Measured
Sonnet 5	Anthropic	Group 1 frontier	Callable	Measured
Opus 4.8	Anthropic	Group 1 frontier	Callable	Measured
gpt-6.1-sol	OpenAI	Group 1 frontier	Callable	Measured
gpt-6-astra	OpenAI	Group 1 frontier	Callable	Measured
gpt-6-sol	OpenAI	Group 1 frontier	Callable	Measured
gpt-6-luna	OpenAI	Group 1 frontier	Callable	Measured
gpt-5.6-sol	OpenAI	Group 1 frontier	Callable	Measured
kimi-k3	Moonshot AI	Group 2 Chinese open-weight	Callable	Measured
DeepSeek V4-Pro	DeepSeek	Chinese model (open-weight status not verified)	Not offered	Excluded
DeepSeek V4.1-Flash	DeepSeek	Chinese model (open-weight status not verified)	Not offered	Excluded
qwen3.8-max	Alibaba (Qwen)	Chinese model (open-weight status not verified)	Not offered	Excluded
qwen3.8-27b	Alibaba (Qwen)	Chinese model (open-weight status not verified)	Not offered	Excluded
glm-5.3-flash	Z.AI	Chinese model (open-weight status not verified)	Not offered (GLM 5.3 from the same family is offered)	Excluded; GLM 5.3 measured instead
gemini-3.8-flash	Google	Neither group	Not offered	Excluded
GLM 5.3 (not in the reference report)	Z.AI	Group 2 Chinese open-weight	Callable	Measured

DeepSeek, Qwen and glm-5.3-flash were excluded because us-east-1 Bedrock did not offer them as of 2026-10-12. Whether their weights are published was not checked, because the lack of Bedrock availability is reason enough. gemini-3.8-flash is a Google model, belongs to neither group, and is not offered on Bedrock.

Claude Fable 5.1 and data retention

When first checked, Bedrock rejected Claude Fable 5.1 with data retention mode 'default' is not available for this model [fact]. According to the Data retention page of the Bedrock User Guide (checked 2026-10-13), Claude Fable 5 and 5.1 can be called only when the account's effective data retention mode is aws_review or higher [source]. The setting is per account and per Region and cannot be changed with a request parameter.

After the account owner changed the us-east-1 data retention mode from inherit to aws_review (2026-10-13 04:25:43 UTC), the new value was confirmed with the query API and a short request succeeded; then the five runs were executed [fact]. The setting behaves as follows [source: same page].

• It is set through the API, without a console page. Example: aws bedrock put-account-data-retention --region us-east-1 --mode aws_review. Setting --mode inherit reverts it.
• It applies to every bedrock-runtime request the account sends in that Region.
• Prompts and outputs of Fable 5 and 5.1 requests are retained inside the AWS boundary for up to 30 days and may be reviewed by AWS staff. They are not shared with the model provider.
• Other models follow the modes they allow. For example, requests to a model that allows no retention (none) are not retained even with this setting on.
• With cross-Region inference, retained data is stored in the Region that actually served the request.

Because this changes a data retention policy, anyone repeating the experiment needs the account owner to review this impact and decide.

Test Configuration

Agent harness [fact]

• Every model used pi coding agent 0.87.1, the same version as the reference report's pi conditions.
• Calls went through the Bedrock Converse API with thinking set to medium. Claude models receive adaptive thinking with effort=medium; for other models pi sends no reasoning field, so the model default applies.
• One difference from the reference report: pi 0.87.1 has no settings for Claude Haiku 5.5, so Bedrock rejected its requests. haiku-5 was added to two model-name lists in the pi bundle (adaptive thinking and prompt caching). Requests to other models are unaffected.

Multi-agent setup

The reference report runs a single agent and does not define a multi-agent setup. This report used the following one: two agents on the same model, run one after the other within each iteration, exchanging information through files in the repository.

1. Builder: receives the reference PROMPT-en.md unchanged and implements the task. From the second iteration on, one sentence is appended telling it that a verification agent reviewed the previous iteration and that it should read REVIEW.md first. Only the builder may create .ralph-done.
2. Verifier: receives a verification prompt in a fresh session, starts the server, runs the official Hurl tests itself and writes a verdict (PASS/FAIL), failing items, causes and a fix list to REVIEW.md. It does not modify other files.
3. Grading: a grader with the reference report's criteria decides completion.

Isolation and credentials [fact]

• Every agent session and every grading pass ran in its own Docker container built from Amazon Linux 2023, Node 22.23.3, pi 0.87.1 and hurl 8.0.1.
• Each container was limited to 8 GB of memory, 4 CPUs and 4,096 processes, and mounted only that run's repository, session records and home directory.
• No AWS access keys entered the containers. They received only a Bedrock short-term API key, usable only for Bedrock calls and expiring within an hour, re-issued every 10 minutes.

Safeguards and scheduling [fact]

• Up to 30 iterations per run, a 1,200-second run time cap and a 300-second stall watchdog.
• Cost limits of $100 per run, $500 per model and $7,000 overall.
• The 13 models ran with up to seven models in parallel; each model's five runs were sequential. Claude Fable 5.1 ran alone after all other models had finished.
• English prompts only.

Metric definitions

• Completed: a run that passed official grading. Medians and min/max use completed runs only.
• Tokens [fact]: the usage Bedrock reported with each response, summed over all builder and verifier requests. Input means uncached input tokens; output includes reasoning tokens; cache reads and cache writes are counted separately.
• Cost [calculated]: token counts multiplied by the prices under "Prices Applied". Cache writes use the 5-minute cache price, because pi does not set a cache duration and Bedrock defaults to 5 minutes.
• Ralph iterations: the number of iterations in which the builder ran, the same definition as the reference report's iteration count.
• LLM requests: the number of responses the model returned, the model-level "turns", builder and verifier combined.
• Run time: from the start to the end of a run, including both sessions and grading. The reference report's elapsed time is the agent session time, a different definition.

Results

Cost and time

Each point is one model: the median cost per completed run (horizontal, log scale) against the median run time (vertical).

AnthropicOpenAIMoonshot AIZ.AI

FrontierChinese open-weight

Show data

Model	Vendor	Group	Cost/run (median)	Run minutes (median)
Claude Opus 5.5	Anthropic	Frontier	$0.907	6.1
Claude Sonnet 5.5	Anthropic	Frontier	$0.306	3.8
Claude Haiku 5.5	Anthropic	Frontier	$0.065	7.4
Claude Fable 5.1	Anthropic	Frontier	$2.566	8.9
Claude Opus 5	Anthropic	Frontier	$1.999	8.4
Claude Sonnet 5	Anthropic	Frontier	$1.179	9.7
Claude Opus 4.8	Anthropic	Frontier	$2.018	7.2
GPT-6.1 Sol	OpenAI	Frontier	$0.527	8.1
GPT-6 Astra	OpenAI	Frontier	$3.495	7.9
GPT-6 Sol	OpenAI	Frontier	$0.291	3.1
GPT-6 Luna	OpenAI	Frontier	$0.027	4.2
GPT-5.6 Sol	OpenAI	Frontier	$0.755	4.1
Kimi K3	Moonshot AI	Chinese open-weight	$1.422	10.5
GLM 5.3	Z.AI	Chinese open-weight	$1.821	10.1

Cost per run versus run time by model

The x-axis is median cost per run on a log scale; the y-axis is median run time in minutes. Color encodes vendor and marker shape encodes group (circle = frontier, diamond = Chinese open-weight). Thin whiskers show each axis min–max range.

Source: This report's 70 measured runs (Amazon Bedrock us-east-1, 2026-10-12 to 2026-10-13 UTC); cost computed at prices checked on 2026-10-12.

Results by model

Bars span the minimum–maximum of completed runs and the vertical tick marks the median. Switch metrics with the buttons and expand a row for per-run values. The table below gives the same values as numbers: medians over completed runs, with the minimum–maximum of the five runs in parentheses. Cost is [calculated]; the rest are [fact].

Metric

• Completed run
• Median
• ×Timed out (no completed value)

Model

Completed-run range

$0.024$0.130$0.697$3.726

Frontier

Claude Opus 5.55/5 completedMedian $0.907▸Show the five runs

Claude Opus 5.5 — Show the five runs

Run	Status	Iterations	Minutes	LLM requests	Output tokens	Total tokens	Cost	Verifier	Commits	Self-terminated
1	done	1	7.1	41	22,812	1,079,865	$1.034	PASS	2	0
2	done	1	5.7	30	20,467	689,505	$0.872	PASS	2	0
3	done	1	6.1	30	20,402	631,889	$0.861	PASS	2	0
4	done	1	8.0	45	23,773	1,088,484	$1.054	PASS	3	0
5	done	1	6.1	35	20,667	796,558	$0.907	PASS	2	0

Claude Sonnet 5.55/5 completedMedian $0.306▸Show the five runs

Claude Sonnet 5.5 — Show the five runs

Run	Status	Iterations	Minutes	LLM requests	Output tokens	Total tokens	Cost	Verifier	Commits	Self-terminated
1	done	1	3.7	25	13,929	489,440	$0.319	PASS	2	0
2	done	1	3.8	24	14,407	418,465	$0.305	FAIL	1	0
3	done	1	4.1	27	14,309	603,161	$0.328	FAIL	2	0
4	done	1	3.4	25	14,161	445,388	$0.306	PASS	3	0
5	done	2	3.9	23	12,677	226,794	$0.257	PASS / PASS	2	1

Claude Haiku 5.55/5 completedMedian $0.065▸Show the five runs

Claude Haiku 5.5 — Show the five runs

Run	Status	Iterations	Minutes	LLM requests	Output tokens	Total tokens	Cost	Verifier	Commits	Self-terminated
1	done	1	6.2	39	42,838	1,155,347	$0.046	PASS	4	0
2	done	1	8.3	62	52,713	2,727,358	$0.070	PASS	4	0
3	done	1	7.4	50	54,257	2,156,808	$0.065	PASS	4	0
4	done	2	9.9	77	67,557	2,711,858	$0.084	FAIL / PASS	7	0
5	done	1	7.0	49	47,874	2,032,399	$0.059	PASS	6	0

Claude Fable 5.15/5 completedMedian $2.566▸Show the five runs

Claude Fable 5.1 — Show the five runs

Run	Status	Iterations	Minutes	LLM requests	Output tokens	Total tokens	Cost	Verifier	Commits	Self-terminated
1	done	1	8.9	44	29,187	1,238,647	$2.796	PASS	2	0
2	done	1	10.9	58	28,567	1,564,029	$2.837	PASS	2	0
3	done	1	6.9	36	22,047	787,229	$2.118	PASS	7	0
4	done	1	9.3	46	25,586	1,183,739	$2.555	PASS	2	0
5	done	1	8.4	44	26,693	1,062,192	$2.566	PASS	6	0

Claude Opus 55/5 completedMedian $1.999▸Show the five runs

Claude Opus 5 — Show the five runs

Run	Status	Iterations	Minutes	LLM requests	Output tokens	Total tokens	Cost	Verifier	Commits	Self-terminated
1	done	1	6.8	50	24,165	1,138,625	$1.635	PASS	2	0
2	done	1	7.3	56	26,436	1,452,008	$1.876	PASS	2	0
3	done	1	14.0	91	40,478	3,851,983	$3.726	PASS	3	0
4	done	1	10.3	69	32,053	1,769,367	$2.258	PASS	2	0
5	done	1	8.4	57	28,503	1,555,721	$1.999	PASS	2	0

Claude Sonnet 55/5 completedMedian $1.179▸Show the five runs

Claude Sonnet 5 — Show the five runs

Run	Status	Iterations	Minutes	LLM requests	Output tokens	Total tokens	Cost	Verifier	Commits	Self-terminated
1	done	1	11.3	102	36,492	3,758,302	$1.412	PASS	1	0
2	done	1	8.3	80	32,142	2,787,977	$1.128	PASS	2	0
3	done	1	9.7	85	31,074	3,059,198	$1.179	PASS	2	0
4	done	1	8.1	77	31,127	2,909,277	$1.155	PASS	2	0
5	done	1	11.4	93	34,760	4,019,069	$1.455	PASS	2	0

Claude Opus 4.85/5 completedMedian $2.018▸Show the five runs

Claude Opus 4.8 — Show the five runs

Run	Status	Iterations	Minutes	LLM requests	Output tokens	Total tokens	Cost	Verifier	Commits	Self-terminated
1	done	1	6.6	60	23,680	1,749,809	$1.971	PASS	2	0
2	done	1	6.6	56	22,316	1,675,909	$1.859	PASS	2	0
3	done	1	9.6	75	29,333	2,735,633	$2.678	none	2	1
4	done	1	7.6	62	27,140	1,873,275	$2.134	PASS	2	0
5	done	1	7.2	60	23,804	1,877,268	$2.018	PASS	2	0

GPT-6.1 Sol5/5 completedMedian $0.527▸Show the five runs

GPT-6.1 Sol — Show the five runs

Run	Status	Iterations	Minutes	LLM requests	Output tokens	Total tokens	Cost	Verifier	Commits	Self-terminated
1	done	1	8.8	53	22,847	1,385,667	$0.559	PASS	6	0
2	done	1	6.7	46	18,490	1,112,514	$0.457	PASS	5	0
3	done	1	8.1	45	20,621	1,164,468	$0.536	PASS	7	0
4	done	1	8.4	44	22,189	1,096,209	$0.527	PASS	4	0
5	done	1	6.5	49	19,083	1,251,798	$0.495	PASS	5	0

GPT-6 Astra5/5 completedMedian $3.495▸Show the five runs

GPT-6 Astra — Show the five runs

Run	Status	Iterations	Minutes	LLM requests	Output tokens	Total tokens	Cost	Verifier	Commits	Self-terminated
1	done	1	7.5	46	22,221	1,325,517	$3.495	PASS	4	0
2	done	1	7.3	48	21,151	1,208,409	$3.153	PASS	4	0
3	done	1	7.9	48	21,299	1,139,606	$3.102	PASS	6	0
4	done	1	8.8	50	23,685	1,340,715	$3.511	PASS	5	0
5	done	1	8.0	55	22,250	1,479,623	$3.562	PASS	5	0

GPT-6 Sol5/5 completedMedian $0.291▸Show the five runs

GPT-6 Sol — Show the five runs

Run	Status	Iterations	Minutes	LLM requests	Output tokens	Total tokens	Cost	Verifier	Commits	Self-terminated
1	done	1	3.2	32	8,428	456,816	$0.281	PASS	2	0
2	done	1	2.9	36	9,605	592,640	$0.326	PASS	3	0
3	done	1	2.5	33	7,967	432,181	$0.258	PASS	2	0
4	done	1	3.3	51	9,441	818,653	$0.358	PASS	2	0
5	done	1	3.1	41	8,544	564,091	$0.291	PASS	2	0

GPT-6 Luna5/5 completedMedian $0.027▸Show the five runs

GPT-6 Luna — Show the five runs

Run	Status	Iterations	Minutes	LLM requests	Output tokens	Total tokens	Cost	Verifier	Commits	Self-terminated
1	done	1	5.8	59	16,847	1,227,597	$0.029	FAIL	2	0
2	done	1	4.0	43	14,265	848,865	$0.025	PASS	3	0
3	done	3	15.7	113	45,643	1,979,669	$0.066	FAIL / FAIL / PASS	5	0
4	done	1	4.2	43	13,818	1,057,982	$0.027	PASS	2	0
5	done	1	4.1	50	14,272	961,021	$0.024	PASS	2	0

GPT-5.6 Sol5/5 completedMedian $0.755▸Show the five runs

GPT-5.6 Sol — Show the five runs

Run	Status	Iterations	Minutes	LLM requests	Output tokens	Total tokens	Cost	Verifier	Commits	Self-terminated
1	done	1	4.1	34	12,032	664,801	$0.818	PASS	4	0
2	done	1	5.8	27	12,787	524,621	$0.755	FAIL	2	0
3	done	1	3.4	32	10,223	584,537	$0.708	PASS	2	0
4	done	1	5.8	31	12,480	775,728	$0.908	FAIL	2	0
5	done	1	3.7	29	10,402	505,256	$0.667	PASS	3	0

Chinese open-weight

Kimi K35/5 completedMedian $1.422▸Show the five runs

Kimi K3 — Show the five runs

Run	Status	Iterations	Minutes	LLM requests	Output tokens	Total tokens	Cost	Verifier	Commits	Self-terminated
1	done	1	9.8	61	39,505	1,658,186	$1.422	PASS	5	0
2	done	1	12.0	61	47,056	1,856,453	$1.640	PASS	4	0
3	done	1	10.6	65	35,008	1,637,495	$1.326	PASS	2	0
4	done	1	10.5	63	40,008	1,815,345	$1.479	PASS	4	0
5	done	1	8.3	57	32,640	1,276,997	$1.154	PASS	4	0

GLM 5.34/5 completed×Median $1.821▸Show the five runs

GLM 5.3 — Show the five runs

Run	Status	Iterations	Minutes	LLM requests	Output tokens	Total tokens	Cost	Verifier	Commits	Self-terminated
1	done	1	10.3	74	48,793	2,876,201	$1.402	PASS	4	0
2	done	1	9.9	89	68,386	4,364,926	$2.062	none	4	1
3	done	1	8.2	86	50,777	3,389,784	$1.595	PASS	5	0
4	timeout	2	20.1	85	52,892	2,491,280	$1.433	FAIL	3	0
5	done	1	15.0	73	77,914	4,143,819	$2.046	PASS	6	0

Group	Model	Completed	Cost per run (USD)	Output tokens	Total tokens	LLM requests	Ralph iterations	Run time (min)
Frontier	Claude Opus 5.5	5/5	$0.907 ($0.861–$1.054)	20,667 (20,402–23,773)	796,558 (631,889–1,088,484)	35 (30–45)	1	6.1 (5.7–8.0)
Frontier	Claude Sonnet 5.5	5/5	$0.306 ($0.257–$0.328)	14,161 (12,677–14,407)	445,388 (226,794–603,161)	25 (23–27)	1–2	3.8 (3.4–4.1)
Frontier	Claude Haiku 5.5	5/5	$0.065 ($0.046–$0.084)	52,713 (42,838–67,557)	2,156,808 (1,155,347–2,727,358)	50 (39–77)	1–2	7.4 (6.2–9.9)
Frontier	Claude Fable 5.1	5/5	$2.566 ($2.118–$2.837)	26,693 (22,047–29,187)	1,183,739 (787,229–1,564,029)	44 (36–58)	1	8.9 (6.9–10.9)
Frontier	Claude Opus 5	5/5	$1.999 ($1.635–$3.726)	28,503 (24,165–40,478)	1,555,721 (1,138,625–3,851,983)	57 (50–91)	1	8.4 (6.8–14.0)
Frontier	Claude Sonnet 5	5/5	$1.179 ($1.128–$1.455)	32,142 (31,074–36,492)	3,059,198 (2,787,977–4,019,069)	85 (77–102)	1	9.7 (8.1–11.4)
Frontier	Claude Opus 4.8	5/5	$2.018 ($1.859–$2.678)	23,804 (22,316–29,333)	1,873,275 (1,675,909–2,735,633)	60 (56–75)	1	7.2 (6.6–9.6)
Frontier	GPT-6.1 Sol	5/5	$0.527 ($0.457–$0.559)	20,621 (18,490–22,847)	1,164,468 (1,096,209–1,385,667)	46 (44–53)	1	8.1 (6.5–8.8)
Frontier	GPT-6 Astra	5/5	$3.495 ($3.102–$3.562)	22,221 (21,151–23,685)	1,325,517 (1,139,606–1,479,623)	48 (46–55)	1	7.9 (7.3–8.8)
Frontier	GPT-6 Sol	5/5	$0.291 ($0.258–$0.358)	8,544 (7,967–9,605)	564,091 (432,181–818,653)	36 (32–51)	1	3.1 (2.5–3.3)
Frontier	GPT-6 Luna	5/5	$0.027 ($0.024–$0.066)	14,272 (13,818–45,643)	1,057,982 (848,865–1,979,669)	50 (43–113)	1–3	4.2 (4.0–15.7)
Frontier	GPT-5.6 Sol	5/5	$0.755 ($0.667–$0.908)	12,032 (10,223–12,787)	584,537 (505,256–775,728)	31 (27–34)	1	4.1 (3.4–5.8)
Chinese open-weight	Kimi K3	5/5	$1.422 ($1.154–$1.640)	39,505 (32,640–47,056)	1,658,186 (1,276,997–1,856,453)	61 (57–65)	1	10.5 (8.2–12.0)
Chinese open-weight	GLM 5.3	4/5	$1.821 ($1.402–$2.062)	59,582 (48,793–77,914)	3,766,802 (2,876,201–4,364,926)	80 (73–89)	1	10.1 (8.2–15.0)

Token composition (medians)

Model	Input (uncached)	Output	Cache read	Cache write	Total	Cache-read share
Claude Opus 5.5	74	20,667	722,474	53,343	796,558	91%
Claude Sonnet 5.5	54	14,161	392,240	38,933	445,388	88%
Claude Haiku 5.5	104	52,713	2,008,107	93,162	2,156,808	93%
Claude Fable 5.1	92	26,693	1,096,620	61,437	1,183,739	93%
Claude Opus 5	114	28,503	1,467,821	59,283	1,555,721	94%
Claude Sonnet 5	170	32,142	2,960,551	70,704	3,059,198	97%
Claude Opus 4.8	120	23,804	1,787,192	58,440	1,873,275	95%
GPT-6.1 Sol	92	20,621	1,074,525	59,843	1,164,468	92%
GPT-6 Astra	96	22,221	1,236,901	58,048	1,325,517	93%
GPT-6 Sol	72	8,544	525,901	30,494	564,091	93%
GPT-6 Luna	100	14,272	982,784	60,391	1,057,982	93%
GPT-5.6 Sol	62	12,032	528,733	49,072	584,537	90%
Kimi K3	427	39,505	1,556,409	60,969	1,658,186	94%
GLM 5.3	160	59,582	3,612,806	87,281	3,766,802	96%

Builder and verifier split (medians)

"Self-terminated sessions" counts sessions in which the agent killed its own process and exited with code 143 (see "Limitations").

Model	Builder LLM requests	Verifier LLM requests	Builder cost	Verifier cost	Verifier share	Self-terminated sessions
Claude Opus 5.5	23	11	$0.772	$0.135	15%	0
Claude Sonnet 5.5	17	8	$0.269	$0.037	12%	1
Claude Haiku 5.5	41	10	$0.061	$0.005	8%	0
Claude Fable 5.1	33	13	$2.192	$0.375	15%	0
Claude Opus 5	44	14	$1.773	$0.203	10%	0
Claude Sonnet 5	71	10	$1.101	$0.061	5%	0
Claude Opus 4.8	52	9	$1.887	$0.133	7%	1
GPT-6.1 Sol	38	8	$0.442	$0.055	10%	0
GPT-6 Astra	41	9	$3.196	$0.278	8%	0
GPT-6 Sol	28	7	$0.259	$0.047	16%	0
GPT-6 Luna	44	10	$0.022	$0.007	25%	0
GPT-5.6 Sol	22	8	$0.552	$0.181	24%	0
Kimi K3	47	16	$1.300	$0.122	9%	0
GLM 5.3	64	16	$1.729	$0.082	4%	1

Group comparison

Each value is the median of the group's per-model medians; parentheses show the range across models in the group. Group 2 has only 2 models, so per-model values are the better basis for comparison.

Group	Models	Cost per run	Output tokens	Total tokens	LLM requests	Run time (min)
Frontier	12	$0.831 ($0.027–$3.495)	21,444 (8,544–52,713)	1,174,104 (445,388–3,059,198)	47 (25–85)	7.3 (3.1–9.7)
Chinese open-weight	2	$1.621 ($1.422–$1.821)	49,543 (39,505–59,582)	2,712,494 (1,658,186–3,766,802)	70 (61–80)	10.3 (10.1–10.5)

Comparison with published runs of the same model [calculated]

The published runs that the reference report ran with the same pi harness (single agent) were re-priced at Bedrock prices and compared with this measurement's medians (builder plus verifier). "Builder-only ratio" divides this measurement's builder cost, without the verifier, by the re-priced published cost. The other models are not compared directly because their published runs used other harnesses (Claude Code, Codex) or have no token records.

Published runThis runBuilder only

Show data

Model	Published run	This run	Builder only	Ratio (this/published)
Claude Opus 5.5	$0.896	$0.907	$0.772	1.01x
Claude Sonnet 5.5	$0.282	$0.306	$0.269	1.09x
Claude Haiku 5.5	$0.067	$0.065	$0.061	0.97x
GPT-6.1 Sol	$0.332	$0.527	$0.442	1.59x
GPT-6 Sol	$0.298	$0.291	$0.259	0.98x
Kimi K3	$0.749	$1.422	$1.300	1.90x

Same-model cost comparison (dumbbell)

For each model, compares the published single-agent pi run cost (re-priced at Bedrock prices), this measurement’s median cost, and the builder-only median on one axis. The multiplier at the end of each row is this median divided by the published cost.

Source: Published runs: token counts from the public repository by Jung Do Hyun (정도현, Roboco); this measurement: this report's measured tokens. Both priced at Amazon Bedrock us-east-1 prices checked on 2026-10-12.

Model	Reference experiment	Published run, re-priced	This measurement	Ratio	Builder-only ratio	Output tokens (published → this)
Claude Opus 5.5	EXP-030	$0.896	$0.907	1.01	0.86	18,990 → 20,667
Claude Sonnet 5.5	EXP-032	$0.282	$0.306	1.09	0.95	12,223 → 14,161
Claude Haiku 5.5	EXP-038	$0.067	$0.065	0.97	0.91	54,120 → 52,713
GPT-6.1 Sol	EXP-034	$0.332	$0.527	1.59	1.33	13,251 → 20,621
GPT-6 Sol	EXP-030	$0.298	$0.291	0.98	0.87	7,599 → 8,544
Kimi K3	EXP-029	$0.749	$1.422	1.90	1.74	18,667 → 39,505

Model overview

Card values are medians over completed runs, with the minimum–maximum in parentheses. The notes come from per-run metrics, exit codes, verifier verdicts and session transcripts.

• Claude Opus 5.5

  Frontier · Anthropic
  us.anthropic.claude-opus-5-5

  Cost

  $0.907 ($0.861–$1.054)

  Cost rank

  7 of 14 (1 = cheapest)

  Output tokens

  20,667 (20,402–23,773)

  Total tokens

  796,558 (631,889–1,088,484)

  Cache-read tokens

  722,474 (558,985–1,008,530)

  Run minutes

  6.1 (5.7–8.0)

  LLM requests

  35 (Builder 23 / Verifier 11)

  Completed

  5/5

  Iterations

  1, 1, 1, 1, 1

  Verifier cost share

  15%

  All five runs completed in the first iteration, and the verifier said PASS every time.

• Claude Sonnet 5.5

  Frontier · Anthropic
  us.anthropic.claude-sonnet-5-5

  Cost

  $0.306 ($0.257–$0.328)

  Cost rank

  4 of 14 (1 = cheapest)

  Output tokens

  14,161 (12,677–14,407)

  Total tokens

  445,388 (226,794–603,161)

  Cache-read tokens

  392,240 (178,476–548,677)

  Run minutes

  3.8 (3.4–4.1)

  LLM requests

  25 (Builder 17 / Verifier 8)

  Completed

  5/5

  Iterations

  1, 1, 1, 1, 2

  Verifier cost share

  12%

  Run 5: in iteration 1 the builder killed its own process (exit 143); it completed in iteration 2. Runs 2, 3: the server listened on another port than PORT, so the verifier said FAIL, but official grading passed.

• Claude Haiku 5.5

  Frontier · Anthropic
  us.anthropic.claude-haiku-5-5

  Cost

  $0.065 ($0.046–$0.084)

  Cost rank

  2 of 14 (1 = cheapest)

  Output tokens

  52,713 (42,838–67,557)

  Total tokens

  2,156,808 (1,155,347–2,727,358)

  Cache-read tokens

  2,008,107 (1,034,505–2,581,355)

  Run minutes

  7.4 (6.2–9.9)

  LLM requests

  50 (Builder 41 / Verifier 10)

  Completed

  5/5

  Iterations

  1, 1, 1, 2, 1

  Verifier cost share

  8%

  Run 4: iteration 1 ended without a completion claim (grader 0/13 files); it completed in iteration 2. pi 0.87.1 sends a thinking format that Bedrock rejects for Haiku 5.5, so haiku-5 was added to two model-name lists in the pi bundle.

• Claude Fable 5.1

  Frontier · Anthropic
  us.anthropic.claude-fable-5-1

  Cost

  $2.566 ($2.118–$2.837)

  Cost rank

  13 of 14 (1 = cheapest)

  Output tokens

  26,693 (22,047–29,187)

  Total tokens

  1,183,739 (787,229–1,564,029)

  Cache-read tokens

  1,096,620 (713,581–1,472,841)

  Run minutes

  8.9 (6.9–10.9)

  LLM requests

  44 (Builder 33 / Verifier 13)

  Completed

  5/5

  Iterations

  1, 1, 1, 1, 1

  Verifier cost share

  15%

  Ran alone, about 4 h 18 min after the other 13 models finished, once the account's us-east-1 data retention mode had been set to aws_review. All five runs completed in the first iteration, and the verifier said PASS every time.

• Claude Opus 5

  Frontier · Anthropic
  us.anthropic.claude-opus-5

  Cost

  $1.999 ($1.635–$3.726)

  Cost rank

  11 of 14 (1 = cheapest)

  Output tokens

  28,503 (24,165–40,478)

  Total tokens

  1,555,721 (1,138,625–3,851,983)

  Cache-read tokens

  1,467,821 (1,057,977–3,729,782)

  Run minutes

  8.4 (6.8–14.0)

  LLM requests

  57 (Builder 44 / Verifier 14)

  Completed

  5/5

  Iterations

  1, 1, 1, 1, 1

  Verifier cost share

  10%

  All five runs completed in the first iteration, and the verifier said PASS every time. Run 3 cost about 1.9× the median (91 LLM requests, 1 iteration).

• Claude Sonnet 5

  Frontier · Anthropic
  us.anthropic.claude-sonnet-5

  Cost

  $1.179 ($1.128–$1.455)

  Cost rank

  8 of 14 (1 = cheapest)

  Output tokens

  32,142 (31,074–36,492)

  Total tokens

  3,059,198 (2,787,977–4,019,069)

  Cache-read tokens

  2,960,551 (2,689,255–3,906,673)

  Run minutes

  9.7 (8.1–11.4)

  LLM requests

  85 (Builder 71 / Verifier 10)

  Completed

  5/5

  Iterations

  1, 1, 1, 1, 1

  Verifier cost share

  5%

  All five runs completed in the first iteration, and the verifier said PASS every time.

• Claude Opus 4.8

  Frontier · Anthropic
  us.anthropic.claude-opus-4-8

  Cost

  $2.018 ($1.859–$2.678)

  Cost rank

  12 of 14 (1 = cheapest)

  Output tokens

  23,804 (22,316–29,333)

  Total tokens

  1,873,275 (1,675,909–2,735,633)

  Cache-read tokens

  1,787,192 (1,600,449–2,645,739)

  Run minutes

  7.2 (6.6–9.6)

  LLM requests

  60 (Builder 52 / Verifier 9)

  Completed

  5/5

  Iterations

  1, 1, 1, 1, 1

  Verifier cost share

  7%

  Run 3: the verifier session killed its own process (exit 143) and left no verdict; the run still passed official grading.

• GPT-6.1 Sol

  Frontier · OpenAI
  us.openai.gpt-6.1-sol

  Cost

  $0.527 ($0.457–$0.559)

  Cost rank

  5 of 14 (1 = cheapest)

  Output tokens

  20,621 (18,490–22,847)

  Total tokens

  1,164,468 (1,096,209–1,385,667)

  Cache-read tokens

  1,074,525 (1,011,677–1,302,871)

  Run minutes

  8.1 (6.5–8.8)

  LLM requests

  46 (Builder 38 / Verifier 8)

  Completed

  5/5

  Iterations

  1, 1, 1, 1, 1

  Verifier cost share

  10%

  All five runs completed in the first iteration, and the verifier said PASS every time.

• GPT-6 Astra

  Frontier · OpenAI
  us.openai.gpt-6-astra

  Cost

  $3.495 ($3.102–$3.562)

  Cost rank

  14 of 14 (1 = cheapest)

  Output tokens

  22,221 (21,151–23,685)

  Total tokens

  1,325,517 (1,139,606–1,479,623)

  Cache-read tokens

  1,236,901 (1,062,948–1,399,215)

  Run minutes

  7.9 (7.3–8.8)

  LLM requests

  48 (Builder 41 / Verifier 9)

  Completed

  5/5

  Iterations

  1, 1, 1, 1, 1

  Verifier cost share

  8%

  All five runs completed in the first iteration, and the verifier said PASS every time.

• GPT-6 Sol

  Frontier · OpenAI
  us.openai.gpt-6-sol

  Cost

  $0.291 ($0.258–$0.358)

  Cost rank

  3 of 14 (1 = cheapest)

  Output tokens

  8,544 (7,967–9,605)

  Total tokens

  564,091 (432,181–818,653)

  Cache-read tokens

  525,901 (393,654–778,994)

  Run minutes

  3.1 (2.5–3.3)

  LLM requests

  36 (Builder 28 / Verifier 7)

  Completed

  5/5

  Iterations

  1, 1, 1, 1, 1

  Verifier cost share

  16%

  All five runs completed in the first iteration, and the verifier said PASS every time.

• GPT-6 Luna

  Frontier · OpenAI
  us.openai.gpt-6-luna

  Cost

  $0.027 ($0.024–$0.066)

  Cost rank

  1 of 14 (1 = cheapest)

  Output tokens

  14,272 (13,818–45,643)

  Total tokens

  1,057,982 (848,865–1,979,669)

  Cache-read tokens

  982,784 (774,123–1,778,975)

  Run minutes

  4.2 (4.0–15.7)

  LLM requests

  50 (Builder 44 / Verifier 10)

  Completed

  5/5

  Iterations

  1, 1, 3, 1, 1

  Verifier cost share

  25%

  Run 3: In iteration 1 the builder ran npm run dev in the foreground and the session hung until the 300-second stall watchdog ended it. After iteration 2 the grader passed 5 of 13 files; the run completed in iteration 3. Run 1: the server listened on another port than PORT, so the verifier said FAIL, but official grading passed. Run 3 cost about 2.5× the median (113 LLM requests, 3 iterations).

• GPT-5.6 Sol

  Frontier · OpenAI
  us.openai.gpt-5.6-sol

  Cost

  $0.755 ($0.667–$0.908)

  Cost rank

  6 of 14 (1 = cheapest)

  Output tokens

  12,032 (10,223–12,787)

  Total tokens

  584,537 (505,256–775,728)

  Cache-read tokens

  528,733 (451,226–704,329)

  Run minutes

  4.1 (3.4–5.8)

  LLM requests

  31 (Builder 22 / Verifier 8)

  Completed

  5/5

  Iterations

  1, 1, 1, 1, 1

  Verifier cost share

  24%

  Runs 2, 4: the server listened on another port than PORT, so the verifier said FAIL, but official grading passed.

• Kimi K3

  Chinese open-weight · Moonshot AI
  us.moonshotai.kimi-k3

  Cost

  $1.422 ($1.154–$1.640)

  Cost rank

  9 of 14 (1 = cheapest)

  Output tokens

  39,505 (32,640–47,056)

  Total tokens

  1,658,186 (1,276,997–1,856,453)

  Cache-read tokens

  1,556,409 (1,190,363–1,739,007)

  Run minutes

  10.5 (8.3–12.0)

  LLM requests

  61 (Builder 47 / Verifier 16)

  Completed

  5/5

  Iterations

  1, 1, 1, 1, 1

  Verifier cost share

  9%

  All five runs completed in the first iteration, and the verifier said PASS every time. Run 3 hit 2 transient Bedrock stream errors; pi retried automatically. Its model card says explicit prompt caching is not offered through Converse, yet Bedrock reported cache-read and cache-write tokens (mechanism not verified).

• GLM 5.3

  Chinese open-weight · Z.AI
  us.zai.glm-5.3

  Cost

  $1.821 ($1.402–$2.062)

  Cost rank

  10 of 14 (1 = cheapest)

  Output tokens

  59,582 (48,793–77,914)

  Total tokens

  3,766,802 (2,876,201–4,364,926)

  Cache-read tokens

  3,612,806 (2,752,084–4,200,005)

  Run minutes

  10.1 (8.2–15.0)

  LLM requests

  80 (Builder 65 / Verifier 16)

  Completed

  4/5

  Iterations

  1, 1, 1, ×, 1

  Verifier cost share

  4%

  Run 4: In iteration 1 the builder started a node test server in the foreground and hung; the 300-second stall watchdog ended the session. Iteration 2 hit the 20-minute run cap before the builder created .ralph-done. Grading right after the cap passed all 13 files, but under the completion rule (.ralph-done plus 13 passing files) the run counts as incomplete. Run 2: the verifier session killed its own process (exit 143) and left no verdict; the run still passed official grading.

Interpretation

Between groups.

The Chinese open-weight models sent more LLM requests (about 1.5× by group value), produced more output tokens (about 2.3×) and took longer (about 1.4×) to finish the same task. Their Bedrock prices were not low either.

• Kimi K3's input/output prices ($3.3/$16.5 per million tokens) are higher than those of Claude Sonnet 5.5 and GPT-6.1 Sol ($2.2/$11).
• GLM 5.3 has a low output price ($5.808), but its cache-read price ($0.3432) is about 3× Sonnet 5.5's ($0.11) and it read many cached tokens (median 3,612,806). About 68% of its run cost came from cache reads. [calculated: median cache-read tokens × price divided by median run cost]

As a result its cost per run was about 2.0× the frontier group value. Two caveats apply:

• The comparison uses Bedrock prices. The cost of self-hosting the open-weight models is not covered.
• Group 2 contains only 2 models, so its group value is weak evidence.

Between models (frontier).

• Price is the main driver of cost differences.
  • GPT-6 Luna and Claude Haiku 5.5 cost $0.11/$0.55 for input/output. They used 1,057,982 and 2,156,808 total tokens, yet cost only $0.027 and $0.065 per run.
  • GPT-6 Astra and Claude Fable 5.1 have the highest input/output prices ($11/$55). Cache reads cost $1.1 for GPT-6 Astra and $0.275 for Fable 5.1. Per run, GPT-6 Astra cost $3.495 and Fable 5.1 $2.566, so GPT-6 Astra was the more expensive of the two. Output tokens were 22,221 and 26,693; cache-read tokens were 1,236,901 and 1,096,620.
• Within one price tier, usage differences show.
  • Claude Sonnet 5 and Sonnet 5.5 share input/output prices ($2.2/$11); only Sonnet 5's cache-read price is twice as high ($0.22).
  • Sonnet 5 sent 85 LLM requests and used 3,059,198 total tokens for $1.179. Sonnet 5.5 sent 25 requests and used 445,388 tokens for $0.306, so Sonnet 5 cost about 3.9× as much.
  • The gap between Claude Opus 5.5 ($0.907) and Opus 4.8 ($2.018) combines price (input/output $4.4/$22 vs $5.5/$27.5, cache read $0.22 vs $0.55) and usage (35 vs 60 requests).
• Speed: GPT-6 Sol was both the fastest (3.1 minutes per run) and the model with the fewest output tokens (8,544).

Variation across the five runs. For 12 models the most expensive run cost at most 1.35× the median. The exceptions:

• Claude Opus 5: run 3 used 91 LLM requests, 3,851,983 total tokens and 1 iteration, costing $3.726 (about 1.9× the median).
• GPT-6 Luna: run 3 used 113 LLM requests, 1,979,669 total tokens and 3 iterations, costing $0.066 (about 2.5× the median).

Cost and effect of the multi-agent setup.

• The verifier's median cost was about 4% (GLM 5.3) to 25% (GPT-6 Luna) of the median run cost. [calculated: ratio of per-model medians]
• Because most runs finished in one Ralph iteration, the builder actually read the verifier's REVIEW.md only in the 4 runs with two or more iterations.
• Of the 69 iterations that passed the official grader, the verifier also said PASS in 62.
  • In 5, the server listened on port 8000 instead of the PORT environment variable, so the verifier reported FAIL (Claude Sonnet 5.5 2, GPT-6 Luna 1 and GPT-5.6 Sol 2).
  • In 2, the verifier session terminated itself and left no verdict.
  • No iteration had a verifier PASS while the official grader failed to pass all 13 files.
• There is no control group without a verifier, so the verifier's effect on completion or cost cannot be quantified (not verifiable).

Comparison with published runs of the same model.

• Claude Opus 5.5, Claude Sonnet 5.5, Claude Haiku 5.5 and GPT-6 Sol stayed within ±10% of the re-priced published runs even with the verifier added, because the builder's cost alone was 0.86–0.95× the published runs.
• GPT-6.1 Sol produced 20,621 output tokens against 13,251 in the published runs, and cost about 1.6× as much.
• Kimi K3 produced 39,505 output tokens against 18,667 in the published runs, and cost about 1.9× as much.
• Which endpoint the published runs used (Bedrock or the provider's API) was not verified.
• GLM 5.3's published baseline ($0.544) is an [estimate] that borrows glm-5.3-flash's tokens. The measured GLM 5.3 produced 59,582 output tokens, about 2.6× that baseline, and cost $1.821, about 3.3×.

Incomplete runs and anomalies.

• 1 run(s) did not complete.
  • GLM 5.3 run 4: In iteration 1 the builder started a node test server in the foreground and hung; the 300-second stall watchdog ended the session. Iteration 2 hit the 20-minute run cap before the builder created .ralph-done. Grading right after the cap passed all 13 files, but under the completion rule (.ralph-done plus 13 passing files) the run counts as incomplete.
• Stall watchdog: GPT-6 Luna run 3. In iteration 1 the builder ran npm run dev in the foreground and the session hung until the 300-second stall watchdog ended it. After iteration 2 the grader passed 5 of 13 files; the run completed in iteration 3.
• 3 sessions terminated themselves by killing their own process (Claude Sonnet 5.5 builder, Claude Opus 4.8 verifier and GLM 5.3 verifier). Only Claude Sonnet 5.5 run 5 needed an extra iteration as a result.
• Transient Bedrock stream errors ("Bedrock stream ended without a stop reason") occurred (Kimi K3 run 3, 2×). pi retried automatically and completion was not affected.

Prices Applied [source]

Prices are for calls to the us-east-1 endpoint through the US cross-Region inference profiles, in USD per million tokens, checked on 2026-10-12. For Anthropic and OpenAI models these prices are 10% higher than the global cross-Region inference prices.

• Anthropic: the "Geo and In-region Cross-region Inference" table on the Amazon Bedrock pricing page, read for US East (N. Virginia) from the pricing data the page uses (published 2026-10-12T08:40:08Z).
• OpenAI: the US cross-Region inference row of the standard price table (input up to 272K tokens) in the Bedrock model cards. The largest context of a single request in this measurement (input + cache read + cache write) was 94,172 tokens, so the long-context price above 272K tokens does not apply [fact].
• Kimi K3: the price table in its Bedrock model card. Cache writes use the 30-minute cache price, the only cache-write price published.
• GLM 5.3: the Z AI GLM 5.3 table on the Bedrock pricing page.
• The public AWS Price List API data for us-east-1 Bedrock (published 2026-10-10) has no entries for these models, so the pricing page and model cards were used.

Model	Bedrock model ID	Input	Output	Cache read	Cache write
Claude Opus 5.5	us.anthropic.claude-opus-5-5	$4.4	$22	$0.22	$5.5
Claude Sonnet 5.5	us.anthropic.claude-sonnet-5-5	$2.2	$11	$0.11	$2.75
Claude Haiku 5.5	us.anthropic.claude-haiku-5-5	$0.11	$0.55	$0.011	$0.1375
Claude Fable 5.1	us.anthropic.claude-fable-5-1	$11	$55	$0.275	$13.75
Claude Opus 5	us.anthropic.claude-opus-5	$5.5	$27.5	$0.55	$6.875
Claude Sonnet 5	us.anthropic.claude-sonnet-5	$2.2	$11	$0.22	$2.75
Claude Opus 4.8	us.anthropic.claude-opus-4-8	$5.5	$27.5	$0.55	$6.875
GPT-6.1 Sol	us.openai.gpt-6.1-sol	$2.2	$11	$0.11	$2.75
GPT-6 Astra	us.openai.gpt-6-astra	$11	$55	$1.1	$13.75
GPT-6 Sol	us.openai.gpt-6-sol	$2.2	$11	$0.22	$2.75
GPT-6 Luna	us.openai.gpt-6-luna	$0.11	$0.55	$0.011	$0.1375
GPT-5.6 Sol	us.openai.gpt-5.6-sol	$4.4	$22	$0.44	$5.5
Kimi K3	us.moonshotai.kimi-k3	$3.3	$16.5	$0.33	$4.125
GLM 5.3	us.zai.glm-5.3	$1.848	$5.808	$0.3432	$2.31

Spend [calculated]

Item	Cost (reported tokens × price)	Notes
Runs used in the results	$88.42	All 70 runs in isolated containers, complete or not; includes $12.87 for the five Claude Fable 5.1 runs
Discarded first attempt	$17.60	Agents ran directly on the host and could interfere with each other, so the results were dropped
Total	$106.02	Excludes the small pre-flight probe calls (one or two per model)

These amounts are reported token counts multiplied by prices. The actual bill must be checked in AWS Cost Explorer; models billed through AWS Marketplace appear on the bill under each provider.

Limitations

1. One task, five runs per model. Rankings are observations for this task and this setup. Even within one model, the most expensive run cost up to about 2.7× the cheapest (GPT-6 Luna).
2. The setup differs from the reference report. The reference report ran a single agent per model with different harnesses (Claude Code, Codex CLI, pi and others). Here every model ran on pi and Bedrock Converse with a builder plus a verifier, so tokens and cost include the verifier's share. This compares models within one harness, not harnesses.
3. The first attempt was discarded. The first attempt ran up to seven agents directly on one host without containers. Session records showed agents running host-wide commands such as pkill -f and fuser -k, which could stop other runs' servers or the grader's server. Because contamination could not be ruled out, those results were dropped and the runs repeated in isolated containers. The reference report runs one run at a time and does not have this problem.
4. The verifier checked the server port more strictly than the original criteria. The verifier prompt asks the server to listen on the PORT environment variable, which the original completion criteria do not require. The official grader finds the port the server actually opened, so this changes only the content of REVIEW.md, not completion.
5. Some sessions killed their own process (exit 143). pi receives the whole prompt as a command-line argument, as in the reference report. When an agent runs pkill -f with a phrase from its prompt or kills all node processes, it kills itself as well. The verifier's exposure comes from phrases such as npm run dev in this report's verifier prompt. The prompts were not changed mid-experiment so that finished runs stayed comparable.
6. The verifier's effect cannot be judged. There is no control group without a verifier, and most runs completed in the first iteration, so few builders actually read REVIEW.md.
7. Group 2 has only 2 models. GLM 5.3 is not in the reference report, so its published baseline is an [estimate] borrowing glm-5.3-flash's tokens.
8. Kimi K3's caching differed from its model card. The card says explicit prompt caching is offered only through APIs other than Converse, yet Bedrock reported cache-read and cache-write tokens for Converse calls. Costs use the reported values; how caching was applied could not be verified.
9. Timing and concurrency differ. Claude Fable 5.1 ran alone, about 4 h 18 min after the other 13 models had finished. Bedrock load and the number of concurrent runs on the host may have differed; the effect on run time was not measured. Tokens and cost are assumed to depend less on timing, but this was not verified [estimate].
10. Costs are calculated. They use 2026-10-12 prices and change when prices change.

Reproduction

• Tool versions: pi coding agent 0.87.1 (one change for Haiku 5.5), hurl 8.0.1, Node 22.23.3, container base Amazon Linux 2023
• Settings: thinking medium, English prompts, up to 30 iterations, 1,200-second run cap, 300-second stall watchdog, five runs per model
• Prompts: the builder used PROMPT-en.md from the reference repository unchanged; the verifier prompt was written for this report.
• Per-run values: expand a model row in the "Results by model" chart above to see each of the five runs. The same values are in src/data/ralph-loop-bedrock.json (runs) in this repository.
• Harness scripts and raw data (session transcripts, per-run metric CSVs, aggregate files) are not included in this repository. The basic harness structure is in the reference repository.

References

• Jung Do Hyun (정도현, Roboco), Ralph loop completion by model (reference report, checked 2026-10-12 15:40 UTC)
• roboco-io/coding-agent-benchmark (the reference harness, prompts and grading Hurl tests)
• Amazon Bedrock pricing
• Amazon Bedrock User Guide: Data retention
• RealWorld backend API specification
• pi coding agent