Is AGI here?

Not in public: 0 of 8 parts met (4 partial, 4 far).

AGI anywhere, public or hidden: 1% today · 36% by end-2030 · 59% by end-2035 How we get it →

AGI: a system with the full scope of human cognitive autonomy: it can learn, reason, plan and generalize knowledge across any intellectual domain, without task-specific retraining.

Definitions v2.0. Physical and embodied work is out of scope. One system must meet all eight parts below at the same time, and the v1.0 remote-work test stays as one of its tests. A part counts as met only when a public system passes its published test. A few tests ask for more than the definition strictly needs (each part's "The full test" says which); our probabilities judge the definition itself. How this relates to v1.0 · How others define AGI

The eight parts

Best public evidence for each part, against its bar. A part's status is the highest of its test's published marks that the evidence reaches. Select a part for the test, the measurements, the hard parts, who is working on it and what we watch.

0 of 8 met · 0 close · 4 partial · 4 far

The eight parts: status, best public result and bar
PartStatusBest public resultBar
Breadth of workPartial41.25% (Claude Fable 5.2)80% of real remote projects (Remote Labor Index), and 80% wins or ties in at least 36 of GDPval's 44 occupations.
Professional qualityPartial41.25% (Claude Fable 5.2)Parity with professionals: wins or ties 50%+ on expert-graded tasks in 80% of occupations, and 50%+ of real projects judged as good.
ReliabilityFar35.1% (Qwen 3.8 Max)Tasks right on all four tries (pass^4) 80% on tau-bench's hardest domain and the tau2 average, and an 80% time horizon of a 167-hour work-month.
ReasoningPartial4 solves (Public models (Claude Fable 5, GPT-5.6 Sol, GPT-6 Astra))Expert-exam scores (met), plus 10 AI-alone solves of open math problems, one a major advance, and 3 credited results in two other fields.
Planning and long projectsFar3.1 h (Claude Mythos Preview (early snapshot))An 80% time horizon of a work-month (167 h) on a suite able to measure it, plus a documented multi-week project. Close at a work-week (40 h).
Autonomy without scaffoldingFar44.3% (Claude Opus 5)80% strict on OSWorld 2.x with the official general harness, and 80% of multi-day tasks done with no human stepping in. Close at 65%.
Learning on the jobFar25.4% (In-context learning with Claude Sonnet 4.6)Provisional: capture 50% of the gain from experience (CL-Bench), match a human learner from documents, and keep 90% of it.
Generalization to the newPartial1.5% (Claude Opus 4.8)80% on a new ARC generation within 3 months of its launch, at no more than 10 times human cost, with the lead holding on a held-out suite.

Breadth of work , status Partial, step 2 of 4

On real freelance projects the best system, Claude Fable 5.2, delivers 41.25%, against 80%; results by occupation for the other half are unpublished.

Partial: Claude Fable 5.2's RLI 41.25% reaches the 40% Partial mark; Close needs 60%, and GDPval results by occupation are unpublished.

Now: 41.25% (Remote Labor Index: real freelance projects delivered at client-acceptable quality; Claude Fable 5.2, Nov 9, 2026) verified fact source. Bar: 80% of real remote projects (Remote Labor Index), and 80% wins or ties in at least 36 of GDPval's 44 occupations.

Hardest: Whole projects aren't single deliverables. Who: Scale AI and Center for AI Safety (CAIS), OpenAI, Artificial Analysis and 4 more. Next to watch: Top Remote Labor Index automation rate.

The test, the evidence and who is working on it

Question. Can one AI system do at least four-fifths of the paid work people do at a computer (design, analysis, writing, law, finance, media, engineering), rather than a few kinds of task?

Where it stands. Claude Fable 5.2 delivers 41.25% of real freelance projects at client-acceptable quality on the Remote Labor Index (Nov 9), nearly double the previous best (Claude Opus 5.5, 21.25%) and past the 40% Partial mark, against our 80%. On one-shot GDPval tasks, lab-reported overall scores have been above 80% since March 2026, with no results by occupation.

0 of 2 parts of the test met.

  • Not yet: Remote Labor Index at 80% or more (41.25% (Claude Fable 5.2))
  • Not measured: GDPval wins or ties 80%+ in at least 36 of 44 occupations (84.9% overall (GPT-5.5); no per-occupation results published)

Measure. Primary: the Remote Labor Index (RLI; Scale AI and CAIS) automation rate: the share of 240 real paid freelance projects (23 of Upwork's 64 work categories; median 11.5 hours of human work, mean 28.9) where human evaluators judge the AI's deliverable at least as good as the professional's, so that a reasonable client would accept it. Cross-check: OpenAI's GDPval 'wins or ties' rate against professionals' own deliverables (44 occupations; tasks average about 7 hours of expert work), broken down by occupation as in the GDPval paper. Context only, not tests: LLM-judged reruns of GDPval (Artificial Analysis, Mercor), which don't compare against the professionals; the Anthropic Economic Index, which measures use; and O*NET exposure studies, which measure potential exposure.

Marks. Partial: Remote Labor Index (RLI) 40% or more, or the GDPval per-occupation half passed. Close: RLI 60% or more. Met: RLI 80% or more, and on GDPval wins or ties in 80% or more of comparisons in at least 36 of the 44 occupations.

Details. Met when the best public system reaches an RLI automation rate of 80% or more and, on GDPval, wins or ties in 80% or more of comparisons in at least 36 of the 44 occupations (80% of them), not only on the overall average. No per-occupation GDPval results are published for current models, so that half is unmeasured today. Reliability (repeat success) and multi-week length are scored by their own components.

Why this test. The breadth clause asks for a share of real economic work, so the test should be a share of real work. RLI is the closest public match: whole paid projects, judged against what a client would accept. GDPval checks quality against experienced professionals (its task writers averaged 14 years' experience) but uses one-shot, well-specified tasks, so it overstates breadth. Neither samples the whole remote economy: GDPval covers 44 occupations in 9 industries, and RLI covers 23 freelance categories and leaves out work that needs client interaction or long-term evaluation. We'd switch to a benchmark that samples remote O*NET tasks by wage share if one is published.

Also. GDPval, well-specified tasks: 84.9% wins or ties overall (GPT-5.5), but no per-occupation results are published, so that half is unmeasured. verified fact source

The Remote Labor Index also measures Quality, against a lower 50% mark.

Part of the v1.0 remote-work test, kept in v2.0. How v2.0 relates to v1.0

Latest measurements (8)

On whole real projects the best public system now does about half of what the bar needs: Claude Fable 5.2, released Nov 9, has a Remote Labor Index automation rate of 41.25% (Scale AI; Anthropic's system card counts 99 of 240 projects), against 80%, nearly double Claude Opus 5.5's 21.25% (GPT-6 Astra: 20.83%). That reaches the test's 40% Partial mark; Close needs 60%. The jump has not shown on the LLM-judged rerun of GDPval: on Artificial Analysis's GDPval-AA, Fable 5.2 (1831 Elo) trails Opus 5.5 (1866). On one-shot, well-specified GDPval tasks, lab-reported overall scores have been above 80% since March 2026 (GPT-5.4 83.0%, GPT-5.5 84.9%; OpenAI's figures), but no per-occupation results are published for current models.

MeasureResultSystemAs ofRating and source
Remote Labor Index automation rate (best public system)41.25%Claude Fable 5.2 (Claude Opus 5.5 second at 21.25%)Nov 9, 2026 (entry date from the leaderboard's page data; read Nov 13)verified fact source
Scale's leaderboard, read Nov 13; our Nov 10 reading recorded 41.30 for the same entry. Anthropic's Fable 5.2 system card gives 41.3% (99 of 240 projects) and says Scale ran Claude models at high effort with a 200k-token context window and a 32k-token response limit. No confidence interval is published.
Remote Labor Index automation rate (best OpenAI system, third overall)20.83%GPT-6 AstraSep 17, 2026 (entry date from the leaderboard's page data; re-read Nov 13)verified fact source
Remote Labor Index automation rate (best Google system, fifth overall)17.08%Gemini 4 Argon (listed as 'Gemini Argon 4')Oct 22, 2026 (entry date from the leaderboard's page data; re-read Nov 13)verified fact source
GDPval wins or ties against professionals' deliverables, as reported by OpenAI (the last GDPval win rate OpenAI has published; its GPT-5.6 post cites Artificial Analysis's GDPval-AA instead, and its GPT-6 Astra and GPT-6.1 Sol posts give no GDPval figure)84.9%GPT-5.5Apr 23, 2026verified fact source
GDPval score self-reported by another lab (ByteDance doesn't say how it was graded, so it isn't comparable with OpenAI's figures)87.9ByteDance Seed2.1 ProJun 23, 2026verified fact source
GDPval-AA v2.1 (Artificial Analysis rerun of the 220 public GDPval tasks; a panel of three LLM judges compares models with each other; Elo anchored to DeepSeek V4.1 Flash (max) at 1600; no human baseline)1866 Elo (±26), first; Kimi K3.1 Pro 1861, Muse Wonder Full (Max) 1857, Claude Sonnet 5.5 (Max) 1838, Claude Fable 5.2 (Max) 1831; Gemini 4 Carbon (High) 1733; GPT-6.1 Sol (Max) 1608; GPT-6 Astra (Max) 1586Claude Opus 5.5 (Max, Default Fallback)Nov 13, 2026 (read; the page gives no entry dates)verified fact source
Anthropic Economic Index: share of jobs that have seen Claude used for at least a quarter of their tasks, pooled across Anthropic's samples (measures use, not ability)about 49%Claude (all models); Claude.ai conversations pooled across reports since the first one (Feb 2025)Jan 15, 2026 (first reported); the Mar 24, 2026 report, with data to Feb 12, says it barely changedverified fact source
GDPx v1.0 (SemiAnalysis): pass@1 averaged across 11 expert-built domains weighted equally, graded by deterministic checks or LLM judges against expert rubrics; no human baseline16.4%Claude Fable 5.2 (Claude Opus 5.5 12.0% and Gemini 4 Carbon 10.9%, the same as at launch on Nov 1; GPT-6 Astra 9.6%)Nov 13, 2026 (read; the page gives no entry dates)verified fact (builder-run, not peer-reviewed) source
SemiAnalysis's own benchmark, not peer-reviewed; the leaderboard (read Nov 13) gives no entry dates or changelog, so we date the reading, not the run.

Timeline (13 milestones) and what lies ahead

  • Mar 17, 2023new test OpenAI and University of Pennsylvania researchers map GPT models onto O*NET tasks: about 80% of US workers could have at least 10% of their tasks affected, and about 19% at least half. This measures exposure, not ability. verified fact source
  • Feb 10, 2025new test Anthropic launches the Economic Index. In Claude.ai usage, about 36% of jobs show AI use for at least a quarter of their tasks and about 4% for at least three-quarters. verified fact source
  • Sep 25, 2025new test OpenAI publishes GDPval: 1,320 tasks in 44 occupations, written by professionals averaging 14 years' experience and graded blind by same-occupation experts. The best model, Claude Opus 4.1, ties or beats the professionals on 47.6% of 220 gold-set tasks. verified fact source
  • Oct 30, 2025new test Scale AI and CAIS publish the Remote Labor Index. The best agent, Manus, completes 2.5% of real freelance projects to a standard a client would accept. verified fact source
  • Dec 11, 2025GPT-5.2 Thinking ties or beats industry professionals on 70.9% of GDPval comparisons, and GPT-5.2 Pro on 74.1% (OpenAI). verified fact source
  • Mar 5, 2026GPT-5.4 matches or beats industry professionals in 83.0% of GDPval comparisons (OpenAI), the first overall GDPval score above 80% we found. verified fact (part our inference) source
  • Apr 23, 2026GPT-5.5 reaches 84.9% on GDPval (OpenAI). It is the last GDPval win rate OpenAI has published: its GPT-5.6 post (July 9) cites Artificial Analysis's GDPval-AA instead, and its GPT-6 Astra and GPT-6.1 Sol posts give none. verified fact source
  • Jul 1, 2026CAIS reports Claude Fable 5 at 15.8% on the RLI, nearly double Claude Opus 4.8's 8.3%, on a stronger setup than at launch (labs' own agent tools, longer budgets); none of the Fable 5 work it shows would be accepted. verified fact source
  • Sep 17, 2026GPT-6 Astra leads the RLI at 20.83%, the highest score so far (entry date from the leaderboard's page data; Astra was released on Sept 3, and we first recorded the score on Sept 29). verified fact source
  • Oct 7, 2026Claude Opus 5.5 leads the RLI at 21.25%, 0.42 points above GPT-6 Astra (entry date from the leaderboard's page data; no confidence interval is published). verified fact source
  • Oct 26, 2026new test Agents' Last Exam v1.1 is released: 122 public tasks in 55 sub-industries referenced to O*NET, scored by deterministic graders. Gemini 4 Carbon fully completes 45.9%; Kimi 3.1 Pro and Claude Opus 5.5 43.4% each. verified fact source
  • Nov 1, 2026new test SemiAnalysis publishes GDPx v1.0: 206 tasks in 11 domains from 14 data partners, with a default 8-hour limit, graded by deterministic checks or LLM judges against expert rubrics. Claude Opus 5.5 leads at 12.0%, Gemini 4 Carbon 10.9%. verified fact (builder-run, not peer-reviewed) source
  • Nov 9, 2026Claude Fable 5.2, released the same day, leads the RLI at 41.25%, nearly double Claude Opus 5.5's 21.25% and the first score above 40% (entry date from the leaderboard's page data). verified fact source

Ahead. No published forecast or fitted trend gives a date for 80% breadth. The Remote Labor Index (RLI) record (2.5% in Oct 2025; 15.8% in Jul 2026, on a stronger agent setup; 20.83% in Sept 2026; 21.25% in Oct 2026) is short, has no confidence intervals and rose partly on scaffolding, so, as on our trends page, we give no date. CAIS found that RLI success doesn't fall as projects take humans longer, so METR's time-horizon trend can't date this part either. our inference source

Hard parts (7)

  • Whole projects aren't single deliverables. On one-shot, well-specified GDPval tasks, lab-reported scores are above 80%, but the best system, Claude Fable 5.2, still fails about three in five real freelance projects on the RLI (41.25% delivered). In the RLI paper's analysis of the launch agents' deliverables, 45.6% were poor quality, 35.7% were incomplete, 17.6% had corrupt or unusable files and 14.8% had inconsistencies (one deliverable can count in several), for example videos of 8 seconds where 8 minutes were requested, and floor plans that didn't match the supplied sketches. verified fact source
  • Much remote work produces files other than text: 3D and CAD models, architecture drawings, video, audio and design files. The RLI includes them. Scale reports that the most common failure is amateur-quality work, while successes cluster in audio work, image generation (logos, ads), report writing and data retrieval. The RLI still leaves out work that needs direct client interaction, physical labor or long-term evaluation, and covers 23 of Upwork's 64 categories. verified fact source
  • GDPval is one-shot and well specified. OpenAI says it doesn't capture building context, improving over multiple drafts or navigating ambiguity, such as talking to a client before deciding what to produce. OpenAI said in September 2025 that future versions would be more interactive; we have found none published. verified fact (part our inference) source
  • LLM judges overrate newer models, and the most often updated GDPval reruns use them. CAIS calibrated an automated judge on earlier models and found it overestimated GPT-5.5's RLI rate about 2.9 times (17.9% against 6.25% from human evaluators) and Claude Opus 4.8's about 2.3 times (18.8% against 8.33%), although it ranked the models correctly. Artificial Analysis's GDPval-AA and Mercor's GDPval leaderboards are LLM-judged and don't compare with the professionals' deliverables. The rankings only partly agree: Claude Fable 5.2 leads the RLI at 41.25%, nearly double Claude Opus 5.5's 21.25%, yet trails Opus 5.5 on GDPval-AA (1831 against 1866 Elo); GPT-6 Astra, third on the RLI, is at 1586. verified fact source
  • New tasks score lower than the public ones. Mercor's GDPval Extended adds 132 tasks it wrote to the 213 public GDPval tasks: Gemini 4 Argon (High), first on the extended benchmark, scores 85.9% on the public tasks and 76.4% on the extended one, Claude Opus 5.5 (Max) 88.3% and 75.8%, and GPT-6 Astra (Max) 86.7% and 64.1%. These are LLM-judged mean rubric scores, not wins or ties against professionals. verified fact source
  • Deployment evidence leans on coding. Tasks tied to Computer and Mathematical occupations were 35% of Claude.ai conversations in February 2026, and more coding is moving to Anthropic's API through Claude Code, although Anthropic reports that Claude.ai use cases diversified (the top 10 tasks fell from 24% to 19% of traffic). The index measures what people bring to Claude, not what it can do. verified fact source
  • Benchmarks see little of the work. Mercor estimates that about 90% of US working hours go to tasks with little or no coverage in the 29 benchmarks of its initial index: an LLM judge mapped their prompts onto O*NET and found 924 of 18,797 tasks covered. Mercor calls the results provisional. expert opinion source

Who is working on it (7)

  • Scale AI and Center for AI Safety (CAIS): Run the Remote Labor Index: 240 real paid Upwork projects in 23 categories, judged by trained human evaluators (majority of three) against the professional's own deliverable. Scale hosts the leaderboard and runs the evaluations; CAIS publishes results and says it will keep adding major model releases. verified fact source
  • OpenAI: Built GDPval (44 occupations, 1,320 tasks, a 220-task open gold set and an experimental automated grader) and reported GDPval win rates in launch posts through GPT-5.5 (April 2026). In September 2025 it said future versions would be more interactive and context-rich. verified fact source
  • Artificial Analysis: Independently reruns the 220 public GDPval tasks as GDPval-AA v2.1 in its open-source Stirrup agent harness (code execution, web search and fetch, image viewing). A panel of three frontier LLM judges compares models' deliverables in pairs, and Elo is anchored to DeepSeek V4.1 Flash (max) at 1600. The leaderboard lists 309 model entries (Nov 6). verified fact source
  • Anthropic Economics team: Publishes the Anthropic Economic Index, which maps Claude usage onto O*NET tasks and occupations (latest report 'Cadences', June 26, 2026), and the Anthropic Economic Index Survey (announced Apr 22, 2026). verified fact source
  • Mercor: Runs APEX-Agents (long, cross-application tasks written by investment-banking analysts, management consultants and corporate lawyers, graded against expert rubrics by an LLM judge, pass@1) and GDPval leaderboards, including GDPval Extended with 132 tasks of its own. Its GDPval scores are LLM-judged mean rubric scores. verified fact source
  • UC Berkeley RDI and the RDI Foundation (Agents' Last Exam): Co-lead Agents' Last Exam (ALE), a benchmark of agent tasks in non-physical industries referenced to O*NET/SOC 2018: 55 sub-industries in 13 industry groups, deterministic graders against hidden references, and a new public subset about every six months. Snorkel AI contributes funding (Open Benchmarks Grants) and hosts the leaderboard. verified fact source
  • SemiAnalysis (GDPx): Built GDPx v1.0 (Nov 1, 2026) with 14 specialized data partners: 206 tasks in 11 domains (finance, legal, quant trading, chip design, hardware design, biology, materials science, cybersecurity, software engineering, alignment and recursive self-improvement). It runs each model in its native harness where one exists and reports pass@1 across three attempts, with all domains weighted equally. verified fact source

What we watch (6)

  • Top Remote Labor Index automation rate. Counts toward the test's marks: Partial at 40%, Close at 60%, Met at 80% together with the GDPval per-occupation check. Source: Scale AI leaderboard link Why: It is the primary measure and the breadth half of alarm condition Y1. Claude Fable 5.2 rose from Fable 5.1's 17.92% to 41.25% (Nov 9), 23.3 points in one model generation and 20 points above the previous best, past the 20-point line we set as off-trend; the largest jump before it was GPT-5.5's 6.25% to GPT-6 Astra's 20.83% (14.6 points).
  • Per-occupation GDPval results, and any interactive GDPval successor with clarifying requests and revisions. Cross-check only; it can't move the status by itself. We'd log it if a lab publishes per-occupation results with at least 36 of the 44 occupations at 80% or more wins or ties, or if a system scores 70% or more on an interactive version. Source: OpenAI link Why: It tests whether quality holds across occupations and once tasks stop being well specified. OpenAI's last GDPval win rate is GPT-5.5's (April 2026); its GPT-6 Astra and GPT-6.1 Sol posts report none.
  • RLI entries for Kimi K3.1 Pro (1861 on GDPval-AA, second), Claude Sonnet 5.5 (1838) and Gemini 4 Carbon (1733; generally available since Nov 10); Gemini 4 Argon entered at 17.08% on Oct 22. Cross-check only: a GDPval-AA top-five model scoring under 25% on the RLI would further confirm the gap between single deliverables and whole projects; a score at the test's Partial mark (40%) or above would narrow it. Source: Scale AI leaderboard link Why: Claude Fable 5.2, fifth model on GDPval-AA (1831), scored 41.25% on the RLI (Nov 9), at the test's Partial mark, so the gap between single deliverables and whole projects has narrowed; Claude Opus 5.5, first on GDPval-AA, is still at 21.25%. These entries show whether LLM-judged Elo gains carry over to whole projects a client would accept.
  • Anthropic Economic Index: share of jobs where Claude is used for at least three-quarters of tasks, its 'effective AI coverage' measure, and the share of use that is automated rather than augmented. We'd count it as corroboration from real deployment when the share of jobs at three-quarters or more of tasks passes 25% (about 4% in the first report, Feb 2025; the 2026 reports we read don't update this cut). Source: Anthropic link Why: Benchmarks can be gamed or saturated, and broad use in real work is a separate check. It covers Claude only.
  • A new benchmark that samples remote professional tasks by O*NET occupation and wage share, or an RLI v2 with more categories and longer projects. If one is published with human grading and a public leaderboard, we'd adopt it as the primary measure and re-score this component. Source: arXiv and evaluator announcements (Scale AI, CAIS, OpenAI, Epoch AI, METR) Why: Neither current test covers the whole remote economy, so 80% on either one overstates 80% of all remote work.
  • Agents' Last Exam (ALE): share of tasks fully completed, and any results by industry group. Cross-check only; it can't move the status. ALE is referenced to O*NET but scored by deterministic graders against hidden references, not by comparison with professionals' deliverables. We'd log it when it publishes pass rates by industry group. Source: UC Berkeley RDI and the RDI Foundation (leaderboard hosted by Snorkel AI) link Why: It covers 55 sub-industries in 13 industry groups, wider than the RLI's 23 freelance categories, so it can show whether whole-task success holds across fields. In v1.1 (Oct 26) Gemini 4 Carbon, generally available since Nov 10, fully completes 45.9% of the 122 public tasks, and Kimi 3.1 Pro and Claude Opus 5.5 43.4% each; Claude Fable 5.2 has no entry and no results by industry group are published (read Nov 13).

Could it be hidden?

Breadth is hard to observe for an undisclosed system. RLI scores exist only for systems Scale and CAIS can run (public releases, or models a lab submits), and GDPval win rates only for systems a lab chooses to report, so a hidden system passing this component would leave no trace on the leaderboards (A). It would more likely show up indirectly. The first route is economic footprint, which is hypothesis C's sanctioned path: the work gets sold. Signs would be an unexplained surge of high-quality remote deliverables on freelance platforms, or a sharp drop in hiring for outsourced design, analysis and media work. The second is a lab's own productivity claims: Anthropic's prototype R&D Automation Index says Claude 'leads' 26% of its AI R&D work as of August 2026, a measure that tracks A and B better than breadth. METR's Nov 5 assessment adds a view from inside one lab: working from Anthropic's internal benchmark scores, without query access, METR puts the internal-only Model 4 (finished training September to October 2026) at 178.6 on AECI, Anthropic's version of the Epoch Capabilities Index, and above 50% on all 15 benchmarks it was tested on. AECI is a general capability index, not a test of paid work, so it can't speak to breadth directly (our inference). The third is a jump in a public model's RLI score after long internal use, which would show how far behind disclosure runs. Claude Fable 5.2 nearly doubled the best RLI rate on its release day (41.25%, Nov 9), and its system card shows outside testing a month earlier: its AutomationBench scores 'come from Zapier's October 8, 2026 run' (Anthropic). The same card reports capability-index scores for 'a separate, unreleased internal research model that is somewhat more capable than Claude Mythos 5.2', evaluated nowhere else in the card, so no RLI figure is published for it (Anthropic). Breadth on its own isn't evidence for B. For D, broad advisory work in government would show up through disclosed contracts and use (D-open), not through these benchmarks.

Partial since Nov 13, 2026 · reviewed Nov 13, 2026.

Professional quality , status Partial, step 2 of 4

On real projects the best system delivers 41.25%, against 50%; on well-specified tasks, wins or ties reach 84.9% overall, with no occupation results.

Partial: RLI 41.25% (Claude Fable 5.2) passes the 25% Partial mark and the 35% Close line, but Close also needs the well-specified half passed by occupation.

Now: 41.25% (Remote Labor Index: real projects judged at least as good as the professional's; Claude Fable 5.2, Nov 9, 2026) verified fact source. Bar: Parity with professionals: wins or ties 50%+ on expert-graded tasks in 80% of occupations, and 50%+ of real projects judged as good.

Hardest: Clean tasks overstate real quality. Who: OpenAI (GDPval), Scale AI and Center for AI Safety (Remote Labor Index), Mercor (APEX benchmarks) and 3 more. Next to watch: Expert-graded GDPval, or a successor with ambiguous and multi-draft tasks, reported per occupation.

The test, the evidence and who is working on it

Question. When the system hands in a piece of work (a memo, a spreadsheet model, a legal draft, a design), experienced people in that job judge it at least as good as what a typical skilled professional would have delivered.

Where it stands. The real-project half has moved: Claude Fable 5.2 delivers 41.25% of Remote Labor Index projects at least as well as the professional (Nov 9), against our 50%, past the 25% Partial mark. Close also needs the well-specified half passed by occupation: GPT-5.5 wins or ties in 84.9% of GDPval comparisons (OpenAI, April 2026), aggregate only.

0 of 2 parts of the test met (1 on the overall figure only, which doesn't count).

  • Overall figure only: Expert-graded wins or ties 50%+ on well-specified tasks, in 80% of occupations (84.9% overall (GPT-5.5); per-occupation figures unpublished)
  • Not yet: Remote Labor Index at 50% or more (real projects) (41.25% (Claude Fable 5.2))

Measure. Human grading of the system's deliverable against a professional's deliverable for the same task. Two parts: (1) well-specified tasks: blind head-to-head grading by experts from the same occupation, as the rate of wins or ties against professionals (GDPval or a successor), reported per occupation; (2) real end-to-end projects: the share of real commissioned projects where human evaluators, taking a reasonable client's view, judge the system's deliverable at least as good as the professional's (Remote Labor Index automation rate). All-pass legal rubrics (Harvey LAB, applied by an LLM judge to attorney-written criteria) are a domain check only.

Marks. Partial: either half passed, or RLI 25% or more. Close: RLI 35% or more with the well-specified half passed by occupation. Met: (a) expert-graded wins or ties 50%+, with at least as many wins as losses where wins are reported, on well-specified tasks in at least 80% of occupations, and (b) RLI 50% or more, held across two leaderboard updates.

Details. Met when, under a general harness, the best public system (a) wins or ties the professional at least 50% of the time, with at least as many wins as losses where wins are reported, on expert-graded well-specified tasks in at least 80% of occupations tested, and (b) has its deliverable judged at least as good as the professional's on at least 50% of real end-to-end projects (RLI automation rate 50% or more). 50% is parity: the system's work matches or beats the professional's on half the tasks; with comparison professionals who are typical or better, that is at least a median professional's level (our inference).

Why this test. The definition asks for the level of a median skilled professional, so the natural measure is a comparison with professionals' own deliverables, judged by people. Part (a) alone is not enough: GDPval's own authors say its tasks are precisely specified and one-shot, while real work involves ambiguity, iteration and long deliverables; part (b) tests quality on real projects. GDPval's comparison workers average 14 years of experience, above the median, so parity with them is a conservative test (our inference). Wins-or-ties alone would let a model that loses more often than it wins pass when ties are common, hence the wins-versus-losses check. LLM-judged scores do not count toward the threshold: CAIS found an automated judge overstated RLI rates two to three times for the newest models. Exam scores (bar, medical licensing) are excluded as the main test because they measure knowledge, not work product, and have overstated professional competence before. How many occupations and how reliably are left to the breadth and reliability components.

Also. Well-specified half cleared in aggregate: GDPval 84.9% wins or ties against a 50% bar; the per-occupation split is unpublished. verified fact source

Same Remote Labor Index figure as Breadth, against a lower 50% mark.

Part of the v1.0 remote-work test, kept in v2.0. How v2.0 relates to v1.0

Latest measurements (5)

On real projects Claude Fable 5.2, released Nov 9, has its deliverable judged at least as good as the professional's on 41.25% of Remote Labor Index projects (Scale AI; Anthropic's system card counts 99 of 240), against our 50%, nearly double Claude Opus 5.5's 21.25%. That passes the 25% Partial mark and the 35% RLI line of the Close mark, but Close also needs the well-specified half passed by occupation, and no per-occupation GDPval breakdown has been published since the original paper: GPT-5.5's 84.9% wins or ties (OpenAI, Apr 23, 2026) is aggregate only. The jump has not shown on LLM-judged tests: Fable 5.2 trails Opus 5.5 on Artificial Analysis's GDPval-AA (1831 against 1866 Elo) and AA-Briefcase (1785 against 1801), and on Harvey's legal benchmark it passes 10.0% of tasks all-pass (±3.3), against 27.4% for the best, Gemini 4 Argon.

MeasureResultSystemAs ofRating and source
GDPval: wins or ties against industry professionals on well-specified tasks in 44 occupations (OpenAI's run; GDPval's method is blind grading by experts from the same occupation)84.9%GPT-5.5 (Claude Opus 4.7 80.3% and Gemini 3.1 Pro 67.3% in the same OpenAI table)Apr 23, 2026verified fact (part our inference) source
OpenAI's own evaluation. The post does not restate who graded; OpenAI said "expert human judges" for GPT-5.2's figure, so "expert-graded" here is our inference. Aggregate only: no per-occupation figures. It is the latest GDPval figure; OpenAI's later launch posts give the LLM-judged GDPval-AA or no GDPval.
Remote Labor Index automation rate: share of 240 real freelance projects where human evaluators judge the AI's deliverable at least as good as the professional's, that is, acceptable to a reasonable client41.25%Claude Fable 5.2 (Claude Opus 5.5 21.25%, GPT-6 Astra 20.83%, Fable 5.1 17.92%, Fable 5 15.83%; GPT-5.5 in Codex 6.25%)Nov 9, 2026verified fact source
Evaluator leaderboard, read Nov 13. The Fable 5.2 entry was created on Nov 9 according to the page data (Opus 5.5 on Oct 7; GPT-6 Astra and Fable 5.1 on Sept 17). Anthropic's system card gives 41.3% (99 of 240 projects), with Scale running Claude models at high effort. Three evaluators per project; no confidence intervals are published.
Harvey LAB, attorney-built legal tasks scored all-pass (a task counts only if every rubric criterion passes), Pass@1 as run by Mercor27.4% (±5.4)Gemini 4 Argon (High); Muse Wonder 1.0 (Max) 23.0% (±4.9); Gemini 4 Carbon (High) 23.0% (±5.0); Grok 4.7 (xHigh) 19.5% (±4.5); Claude Fable 5.1 (Max) 17.4% (±4.2); Claude Opus 5.5 (Max) 10.8% (±3.4); Claude Fable 5.2 (Max) 10.0% (±3.3); GPT-6 Astra (Max) 9.2% (±3.3) (read Nov 13)Nov 13, 2026 (read; the board gives no entry dates)verified fact (LLM-judged) source
Evaluator leaderboard, re-read Nov 13 from the page's own data, which lists every entry; it gives no entry dates. An LLM judge (Gemini 3 Flash Preview, per Mercor's methodology page) checks each attorney-written criterion, so this is not human grading. Gemini 4 Carbon has been generally available since Nov 10 (Google). Entries whose release status we have not checked are left out. No lawyer baseline is published.
GDPval-AA v2.1, Artificial Analysis's model-judged version of GDPval (Elo from blind pairwise comparisons by a panel of three LLM judges)1866 Elo (±26)Claude Opus 5.5 (Max); Kimi K3.1 Pro 1861, Muse Wonder Full (Max) 1857, Claude Sonnet 5.5 (Max) 1838, Claude Fable 5.2 (Max) 1831, Claude Fable 5.1 (Max) 1763, Gemini 4 Carbon (High) 1733, GPT-6.1 Sol (Max) 1608, GPT-6 Astra (Max) 1586Nov 13, 2026 (read; the page gives no entry dates)verified fact (LLM-judged) source
Evaluator leaderboard, values read Nov 13 (the judge panel as read Oct 5); Anthropic's Opus 5.5 page (Sept 22) gives 1846. The judges are LLMs (GPT-5.6 Sol, Gemini 3.8 Flash and Claude Opus 5). Since September (v2.1) the scale is pinned to DeepSeek V4.1 Flash at 1600, so it has no human reference and can't say whether a model reaches professional level.
AA-Briefcase v1.1, Artificial Analysis's private test of long-horizon knowledge work (multi-week projects with many linked tasks), graded by rubric and pairwise judging by a panel of frontier LLMs1816 EloClaude Sonnet 5.5 (Max); Claude Opus 5.5 (Max) 1801, Claude Fable 5.2 (Max) 1785, Muse Wonder Full (Max) 1729, Gemini 4 Carbon (High) 1611, GPT-6 Astra (Max) 1569Nov 13, 2026 (read; the page gives no entry dates)verified fact (LLM-judged) source
Evaluator leaderboard, read Nov 13. Anthropic's Fable 5.2 system card reports a rubric score of 66.1% for Fable 5.2, against 61.5% for Fable 5.1 and 61.2% for Opus 5.5 (Artificial Analysis's runs, via Anthropic). LLM judges and no human reference, so it can't show parity with professionals.

Timeline (12 milestones) and what lies ahead

  • Sep 7, 2020new test MMLU benchmark published: the largest GPT-3 beats random guessing by almost 20 points on average across 57 subjects, with near-random accuracy on law and morality. verified fact source
  • Mar 30, 2024A peer-reviewed re-analysis (Martínez, Artificial Intelligence and Law) of GPT-4's simulated bar exam, which OpenAI put around the top 10%, places it near the 48th percentile among those who passed, and about the 15th on the essays. verified fact source
  • Sep 25, 2025new test OpenAI launches GDPval: blind grading by experts from the same occupation on 220 public tasks across 44 occupations, each about 7 hours of expert work. The best model, Claude Opus 4.1, wins or ties on 47.6%, just under half. verified fact source
  • Oct 30, 2025new test Scale AI and the Center for AI Safety publish the Remote Labor Index: 240 real freelance projects averaging about 29 hours of professional work. The best agent (Manus) delivers client-acceptable work on 2.5%. verified fact source
  • Dec 11, 2025GPT-5.2 Thinking beats or ties industry professionals in 70.9% of GDPval comparisons, according to expert human judges; OpenAI calls it its first model at or above human expert level. GPT-5.2 Pro wins outright in 60.0%. verified fact (lab's own evaluation) source
  • Apr 23, 2026GPT-5.5 reaches 84.9% wins or ties on GDPval (GPT-5.4 in March: 83.0%, with 70.8% outright wins). It is the last GDPval figure OpenAI has published; its later launch posts give the LLM-judged GDPval-AA or none. verified fact (lab's own evaluation) source
  • Jul 1, 2026Remote Labor Index: Fable 5 reaches 15.8%, up from 2.5% in under eight months, run with stronger scaffolding. The Center for AI Safety says today's AIs still fall short of professional quality on most projects. verified fact source
  • Oct 1, 2026Mercor reports frontier models outscoring all 12 licensed junior accountants (about 5.5 years' experience) on simplified month-end close tasks: models scored perfectly, the accountants averaged about 37%. Mercor says the tasks played to models' strengths and left out client work. verified fact (not peer-reviewed) source
  • Oct 8, 2026Epoch's first Automation Report: six public models on 11 of its work tasks, graded by one person. Fable 5.1 and GPT-6 Astra lead but can't yet automate Epoch's work; one design task was dropped after Fable 5.1 produced Epoch-quality work. verified fact (single grader, one run) source
  • Oct 22, 2026Gary King and Steven Worthington report that Claude Opus 4.8's reviews of 124 manuscripts beat 134 high-quality human reviews: a GPT-5.5 judge preferred the AI review in all 132 decided pairs, and human journal editors in 87%. expert opinion source
  • Oct 22, 2026unreleased model Microsoft AI says its MAI-DxO diagnostic system, in work published in Science, reached 83% accuracy on 304 complex NEJM cases, against 22% for physicians without reference resources and 36% with them. verified fact (lab's own evaluation) source
  • Nov 9, 2026Remote Labor Index: Claude Fable 5.2, released the same day, delivers work judged at least as good as the professional's on 41.25% of projects, nearly double the previous best (Claude Opus 5.5, 21.25%). verified fact source

Ahead. We have no date to give. In aggregate the well-specified half was crossed in December 2025 (GPT-5.2 Thinking, 70.9%). For real projects we found no published forecast of when the Remote Labor Index reaches 50%. Its best rate rose from 2.5% (Oct 2025) to 21.25% (Oct 7, 2026), but it has no confidence intervals and the July jump came with stronger scaffolding and bigger budgets, so we fit no trend, as on our trends page. our inference source

Hard parts (8)

  • Clean tasks overstate real quality. GDPval's authors say its tasks are precisely specified and one-shot: no ambiguity, no client conversation, no revision rounds. GPT-5.5 wins or ties professionals in 84.9% of GDPval comparisons, yet in Codex it delivered client-acceptable work on 6.25% of Remote Labor Index projects, and it passes 4.1% of Harvey's legal tasks. verified fact (part our inference) source
  • Professional work is judged by its worst error. Harvey's LAB passes a legal task only if every rubric criterion passes, as a partner would check, and the best model passes 27.4%. On GDPval, models lost mostly on instruction-following and formatting, all sometimes made up data or miscalculated, and experts rated about 3% of GPT-5's losing deliverables catastrophic. verified fact (part expert opinion) source
  • OpenAI has stopped publishing expert-graded scores, and labs now report model-judged ones. Since GPT-5.5 in April, OpenAI's launch posts give the LLM-judged GDPval-AA (GPT-5.6, July) or no GDPval at all (GPT-6 Astra, GPT-6 Sol and Luna, GPT-6.1 Sol), and Anthropic's launch pages give GDPval-AA. verified fact source
  • Expert grading is slow: in the GDPval paper, expert graders took about 109 minutes on average to review each model deliverable. verified fact source
  • Model judges overstate quality. CAIS calibrated an automated judge on older models and found it rated GPT-5.5 at 17.9% on the Remote Labor Index where human judges gave 6.25% (Opus 4.8: 18.8% against 8.33%). It ranked models correctly but overstated how good they are, so LLM-judged scores can track progress but cannot show parity with professionals. verified fact (part expert opinion) source
  • Human baselines are thin and uneven. Benchmark tasks are made harder over time to catch models out, so people do badly on them; Mercor says its accounting tasks tested what models do best and left out client work. Exams have overstated professional skill before (GPT-4's bar result). expert opinion (part verified fact) source
  • Expert-built work with open decisions is still mostly failed. Surge AI's DAYJOB: Tax has 50 assignments in eight areas of tax practice, built and reviewed by tax experts, with decisions the instructions don't fully specify; an attempt passes only if it meets at least 95% of the task's criteria. The best agent, Claude Sonnet 5.5, passes 10.8%, and GPT-6 Astra (Max) 4.8%. Neither page gives a human baseline. verified fact (vendor-run, not peer-reviewed) source
  • Expert-built finance and legal work still fails all-or-nothing checks. On SemiAnalysis's GDPx the finance tasks are 'so difficult that all the models score 0%' under binary grading, and the legal tasks 'too difficult for all models', so its leaderboard reports partial scores; on one public Series B redline task, no redline was usable in 20 runs across four models. verified fact (builder-run, not peer-reviewed) source

Who is working on it (6)

  • OpenAI (GDPval): Built GDPval: 1,320 tasks across 44 occupations (220 public), written by professionals averaging 14 years' experience and blind-graded by experts from the same occupation. Says future versions will add interactivity and ambiguity. Its last published GDPval figure is GPT-5.5's (April 2026). verified fact source
  • Scale AI and Center for AI Safety (Remote Labor Index): 240 real paid freelance projects across many formats (3D and CAD, architecture, design, video, audio, data, web, writing). Three trained evaluators per project judge whether the AI's deliverable is at least as good as the professional's, from a reasonable client's view (94.4% agreement). Scale Labs runs the evaluations; CAIS reports the results. verified fact source
  • Mercor (APEX benchmarks): Professional-services benchmarks with rubrics written by practitioners in investment banking, consulting, law, medicine and accounting (APEX-1, APEX-Agents, APEX-Accounting), scored by an LLM judge against those rubrics. Runs Harvey LAB on new models and published the junior-CPA baseline study (Oct 1, 2026). verified fact source
  • Harvey (Legal Agent Benchmark, LAB): Open benchmark launched May 6, 2026 with more than 1,200 long legal tasks in 24 practice areas and over 75,000 expert-written rubric criteria (1,749 tasks and over 100,000 criteria on Mercor's leaderboard page in October). All-pass scoring: a task counts only if every criterion passes; an LLM judge applies the criteria. verified fact source
  • Artificial Analysis (GDPval-AA): Runs the 220 public GDPval tasks on new models with a rotating panel of three LLM judges and publishes Elo ratings. It is the most current and widest comparison, but no human grades it, and since September its scale has no human reference. verified fact source
  • Epoch AI (Epoch Automation Reports): Launched Oct 8, 2026: 11 tasks from Epoch's own work in five categories (graphic design, data insights, data explorers, data-center research, research design), attempted without further intervention in each model's own harness and graded by one person against Epoch's staff standards. Epoch plans to add results as new models are released. verified fact source

What we watch (5)

  • Expert-graded GDPval, or a successor with ambiguous and multi-draft tasks, reported per occupation. Part (a) is confirmed when a lab or evaluator publishes expert-graded wins or ties of 50% or more, with at least as many wins as losses, in at least 80% of occupations. The same result on a version with interactive or ambiguous tasks would also bear on part (b). Source: openai.com/index/gdpval, evals.openai.com, lab launch posts and system cards Why: Since the original paper, the only expert-graded figures are aggregates on clean one-shot tasks, last published in April 2026. With the RLI at 41.25% (Claude Fable 5.2, Nov 9), this half is all that stands between the part and its Close mark (our inference).
  • Remote Labor Index best automation rate. Counts toward the test's marks: Partial at 25%, Close at 35% (with the well-specified half passed by occupation), Met at 50% held across two leaderboard updates. Source: Scale AI leaderboard link Why: It is the clearest public, human-judged test of professional-quality work on real projects. Claude Fable 5.2 reached 41.25% on Nov 9, past the RLI lines of the Partial and Close marks, so Close now waits on the per-occupation half.
  • Harvey LAB all-pass score, and any lawyer baseline on it. A cross-check: we'd log a step when the best model passes 35% of tasks or matches a published baseline of practising lawyers. Status moves only on the test's marks. Source: Mercor link Why: Every criterion must pass, which is how a partner checks a deliverable; it tests whether models avoid the one costly miss. It is LLM-judged, so a lawyer baseline would add a human anchor.
  • Blinded human-baseline studies on less structured work (client clarification, revision after feedback). A cross-check: we'd log a step when a peer-reviewed or evaluator study rates AI deliverables at or above median experienced professionals on tasks with ambiguity and iteration. Source: mercor.com/blog, arXiv, NBER working papers Why: The accountant study shows parity on structured tasks. Epoch's first Automation Report (Oct 8) found the leading public models below its staff's standard on the open-ended parts of its own work, with one grader per task. The open question is the rest of the job.
  • Deployment outcomes: firms reporting AI-produced client deliverables shipped without professional review. A cross-check: we'd log a step when two or more large firms in one occupation disclose this in SEC filings, earnings calls or regulator filings. Source: SEC EDGAR, earnings call transcripts, bar and accounting regulators Why: Real acceptance by clients is the end test that benchmarks stand in for.

Could it be hidden?

Professional quality is among the easiest components to test in private (our inference). Labs grade professional work before release: OpenAI published GDPval results with its launches through April 2026 and reports internal investment-banking, consulting, design and data-science task suites, and it has published no expert-graded GDPval figure since GPT-5.5, so a jump would not show at once. Under A (Hidden AGI), the first traces would be internal-only professional evals in system cards running well ahead of the public model, or a lab's own back-office and client work done without staff rework. Under C (Covert AGI actor), on its sanctioned but undisclosed path, an operator could sell deliverables through agencies or freelance platforms, the kind of work RLI samples; that would appear only as unexplained output or price shifts in particular occupations, which we cannot tell apart from public models plus human review. Under D-open, it would appear as government contracts for AI-produced analyses or drafts. B links only through the quality of AI-led research work (Claude leads 26% of Anthropic's AI R&D on Anthropic's own index, August). Detection is weak: quality shows only through human grading someone chooses to publish, so a benchmark that stops moving does not rule out a hidden system (our inference).

Partial since Nov 13, 2026 · reviewed Nov 13, 2026.

Reliability , status Far, step 1 of 4

On a hard agent test the best model gets 35.1% of tasks right on all four tries, against 80%.

Far: pass^4 35.1% is below the 50% Partial mark, and the 80% horizon (about 3.1 h) is short of a workday.

Now: 35.1% (tau3-Banking pass^4: share of tasks done right on all four tries; Qwen 3.8 Max, Aug 3, 2026) verified fact source. Bar: Tasks right on all four tries (pass^4) 80% on tau-bench's hardest domain and the tau2 average, and an 80% time horizon of a 167-hour work-month.

Hardest: Success falls as tasks get longer. Who: METR, Sierra (tau-bench), Princeton (Rabanser, Kapoor, Kirgis, Liu, Utpala, Narayanan) and 3 more. Next to watch: METR's 80% time horizon for a newer frontier model (Claude Fable 5.2, Claude Opus 5.5, GPT-6 Astra, Gemini 4 Argon, Gemini 4 Carbon) (Nov 30, 2026).

The test, the evidence and who is working on it

Question. Whether the system does tasks right at least four times out of five, consistently from one try to the next, rather than succeeding only on lucky runs.

Where it stands. On tau3-Banking the best model does 35.1% of tasks right on all four tries (Qwen 3.8 Max; 55% on one try), against 80%. METR's best published 80% time horizon is about 3.1 hours (Claude Mythos Preview, early snapshot), about 54 times short of the 167-hour work-month the test also needs.

0 of 2 parts of the test met.

  • Not yet: 80% time horizon of a work-month (167 h) (about 3.1 h (Claude Mythos Preview, early))
  • Not yet: pass^4 of 80% on tau3-Banking and on the tau2 average (35.1% banking; 73.2% tau2 average)

Measure. Primary: METR's 80% time horizon, the length of task (in skilled-human working time) that the system completes successfully four times in five, with a general harness. Check: repeat-run consistency on a current agent benchmark, read as pass^k (the share of tasks done right on every one of k independent tries; tau-bench reports k up to 4), alongside wrong-answer rates (how often a model answers wrongly instead of saying it doesn't know).

Marks. Partial: pass^4 50% or more on tau3-Banking, or an 80% time horizon of a workday (8 h). Close: pass^4 65% on tau3-Banking and an 80% time horizon of a work-week (40 h). Met: pass^4 80% on tau3-Banking and on the tau2 average, and an 80% time horizon of a work-month (167 h) on a suite able to measure it.

Details. An 80% time horizon of at least one work-month (167 working hours, 10,020 minutes, the bar our trend fit on Trend watch uses), with a general harness, on a task suite able to measure that length; and pass^4 of at least 80% on the hardest current tau-bench domain (today tau3-Banking) and on the average of the three tau2 domains (retail, airline, telecom). Wrong-answer rates are tracked but carry no pass mark yet.

Why this test. The definition asks for 80%-or-better success on at least 80% of tasks, including multi-week projects. The 80% horizon is the task length at which a system succeeds four times in five, so a work-month reading is the closest published measure of that clause; it folds the two 80% figures into one number, which is why we add a repeat-run check. Pass^4 is the share of tasks done right on all four tries. With four tries it is the natural estimate of the share of tasks done at least 80% of the time (three of four is only 75%), and an average over lucky and unlucky runs cannot pass it. It is strict: a task done right exactly 80% of the time comes out four for four only 41% of the time, so we also show the share done right on at least three of four tries (our calculation from Sierra's pass^3 and pass^4) as a lenient read. What it misses: METR's suite is mostly software, machine-learning and cybersecurity tasks and cannot measure beyond about 16 hours today, and tau-bench is customer-service work, so neither covers all professional work.

Also. METR 80% time horizon about 3.1 h, against a 167-hour work-month (the measure Long projects leads with). verified fact source

METR's 80% time horizon is also the main measure for Long projects.

Part of the v1.0 remote-work test, kept in v2.0. How v2.0 relates to v1.0

Latest measurements (7)

The best published 80% horizon is about 3.1 hours (186 min, 95% CI 97-399; Claude Mythos Preview early snapshot, METR), about 54 times short of the 167-hour bar, and our trend line reads about 4.8 hours at its Oct 4 point. On tau3-Banking the best model does 35% of tasks right on all four tries (Qwen 3.8 Max; 55% on one try, 54% on at least three of four by our calculation), and in Artificial Analysis's runs Claude Fable 5.1 and GPT-6 Astra get 47% and 40-43% right on a single try.

MeasureResultSystemAs ofRating and source
METR 80% time horizon (50% horizon in brackets)about 3.1 h (186 min, 95% CI 97-399 min) [50%: about 17.4 h, above the 16 h METR says its suite can measure reliably]; the 50% horizon is 5.6 times the 80% horizonClaude Mythos Preview, early snapshot (METR Time Horizon 1.1)Apr 7, 2026 (model release; METR published it May 8, 2026). Still the newest published 80% figure as of Nov 13, 2026: METR's time-horizon page says it 'is no longer actively updated' and its data file still ends with this entry; its June 26 summary for GPT-5.6 Sol gave no 80% figure and called its 50% estimates not robust, its Sept 22 summary for Claude Opus 5.5 gave no time horizon, and its Nov 5 assessment of AI R&D at Anthropic gives none; METR's pre-deployment findings on Claude Mythos 5.2, printed in Anthropic's Nov 9 system card, give none eitherverified fact source
METR estimate of the 80% time horizon, public vs internal frontierpublic frontier about 1.5 h (50 min-2 h 40 min); internal frontier 'Likely ≥3h but <4h'. 50% horizon: public about 12 h (5-61 h), internal likely ≥16 h, with the internal frontier on average about 2 months ahead of publicFrontier models shared by Anthropic, Google, Meta and OpenAI, and public models (METR Frontier Risk Report)Feb 16-Mar 16, 2026 assessment window (published May 19, 2026)verified fact source
tau3-Banking pass^4 (pass^1 in brackets): share of 97 banking tasks done right on all 4 tries, Sierra's standard runs35.1% [55.2%] Qwen 3.8 Max; 32.0% [48.7%] Claude Opus 5; 32.0% [47.9%] Grok 4.5; Claude Fable 5 (the June version) 28.9% [39.7%]. No text entry yet for Claude Opus 5.5, Claude Fable 5.1, GPT-6 Astra or Gemini 4 Argon; GPT-6 Astra appears only as the back end of OpenAI's custom voice entry GPT-Live-1 (37.1%, one try, Sept 16)Qwen 3.8 Max (best listed)Aug 3, 2026 (Sierra evaluation date)verified fact source
tau3-Banking one-try success (pass@1, average of 5 runs) in Artificial Analysis's independent runs, which include the newest models (different retrieval set-up and user simulator from Sierra's, so not directly comparable)Qwen3.8 Max 51.3%; Grok 4.6 (High) 50.7%; Muse Spark 1.3 (Max) 50.5%; Claude Fable 5.1 (Max) 47.2%; GPT-6 Astra 40.0-43.1% across its High, Xhigh and Max settings. Success on all four tries can only be lower. We found no score for Claude Fable 5.2, Claude Opus 5.5, Gemini 4 Argon, Gemini 4 Carbon or GPT-6.1 TerraQwen3.8 Max (best shown); Grok 4.6; Muse Spark 1.3; Claude Fable 5.1; GPT-6 AstraNov 13, 2026 (read from Artificial Analysis; figures unchanged since Oct 30)verified fact source
tau2-bench pass^4 (pass^1 in brackets), standard submissionsRetail 59.6% [84.4%], Airline 68.0% [81.5%], Telecom 92.1% [97.8%]; average pass^4 73.2%Qwen3.5-397B-A17B (best standard average; Sierra's runs since May 2026 cover only tau3-Banking)Feb 27, 2026 (Sierra evaluation date)verified fact source
Share of tasks done right on at least three of four tries (lenient read; our calculation from Sierra's pass^3 and pass^4)tau3-Banking: 53.6% (Qwen 3.8 Max). tau2: Retail 86.8%, Airline 82.0%, Telecom 99.1% (Qwen3.5-397B-A17B). On this lenient read the older tau2 domains already clear 80%; the current banking domain does notQwen 3.8 Max; Qwen3.5-397B-A17BAug 3, 2026 and Feb 27, 2026 (Sierra evaluation dates)our inference source
AA-Omniscience hallucination rate: of the questions a model does not get right, the share where it gives a wrong answer instead of declining or answering partly (lower is better)AA-Omniscience v1.1 (3,000 questions, 300 public): Claude Fable 5.2 (Max) 59.3% at 79.1% accuracy (the most accurate, and the top AA-Omniscience Index, 67); Claude Fable 5.1 (Max) 70.0% at 68.4%; Claude Opus 5.5 (Max) 50.3% at 65.5%; GPT-6 Astra (Max) 49.8% at 61.6%; Gemini 4 Carbon (High) 11.4% at 58.3%; Gemini 4 Argon (High) 4.3% at 48.3%Claude Fable 5.2, Claude Fable 5.1, Claude Opus 5.5, GPT-6 Astra, Gemini 4 Carbon, Gemini 4 ArgonNov 13, 2026 (read from Artificial Analysis)verified fact source
Figures from the evaluation page's data, read Nov 13 (Fable 5.2 released Nov 9); the page text gives Fable 5.2 (Max) 'a score of 67' on the index and 79% accuracy. Gemini 4 Argon's accuracy is from its Artificial Analysis model page (read Oct 30). Gemini 4 Carbon has been generally available since Nov 10 (Google).

Timeline (12 milestones) and what lies ahead

  • Feb 14, 2019GPT-2: METR estimates an 80% time horizon of under a second (0.01 min), against about 3 seconds (0.05 min) at 50%. verified fact source
  • Mar 14, 2023GPT-4: 80% horizon about 0.9 minutes, against about 4 minutes at 50% (4.5 times longer). verified fact source
  • Jun 17, 2024new test The tau-bench paper introduces pass^k: agents built on GPT-4o succeed on under 50% of tasks, and pass^8 is under 25% in retail. verified fact source
  • Aug 7, 2025GPT-5: 80% horizon about 38 minutes, against about 3.4 hours at 50%. verified fact source
  • Nov 13, 2025Anthropic's usage data (Nov 13-20, 2025; success judged by Claude; published Jan 15, 2026): API tasks succeed 49% of the time, falling from about 60% on sub-hour tasks to about 45% on tasks of 5+ human hours; Claude.ai conversations 67%. verified fact source
  • Feb 18, 2026new test Princeton researchers propose 12 agent-reliability metrics (consistency, robustness, predictability, safety) and find recent capability gains brought only small reliability gains (14 models; the June 2 revision covers 15, up to GPT-5.5 and Claude Opus 4.7). verified fact source
  • Mar 4, 2026new test Sierra's tau-Knowledge paper introduces the banking domain: frontier models with high reasoning budgets reach only about 25.5% on one try, and reliability falls sharply over repeated tries. verified fact source
  • Apr 7, 2026Claude Mythos Preview (early snapshot): 80% horizon about 3.1 hours, the best published, against about 17 hours at 50% (published May 8); METR says its suite cannot measure above 16 hours reliably. verified fact source
  • Aug 3, 2026tau3-Banking: the best model completes 35.1% of tasks on all four tries (Qwen 3.8 Max), against 55.2% on one try. verified fact source
  • Nov 3, 2026unreleased model Sierra verifies a custom (non-standard) tau3-Banking submission, Principled Machines' Cortex 0.1: 71 of 97 tasks right on one try (73.2%), against 55.2% for Qwen 3.8 Max, the best standard entry; one trial, so no pass^4. verified fact (part our inference) source
  • Nov 9, 2026new test ExpertCrowd-SWE (unreviewed preprint): 100 expert-written change requests in unfamiliar codebases, five runs each for seven frontier models; on 80 tasks no model earns full credit on all five runs. Claude Opus 5.5 (67.3%) and GPT-6 Astra (63.7%) lead. expert opinion source
  • Nov 10, 2026new test ARE-bench (unreviewed preprint): 75 tasks rebuilt from researchers' own ongoing projects; the strongest of seven frontier models passes every checklist item on only 66.2% of tasks, and three fall to 28.4–40.0% once project-history and workspace requirements are checked. expert opinion source

Ahead. Our fit of METR's data has the 80% horizon doubling about every 129 days (range 109-158), the same pace as the 50% horizon, so the gap between them is not closing. The fitted 80% trend reaches a work week around Nov 2027 (May 2027-Jun 2028) and a work month around Jul 2028 (Jan 2028-May 2029); counting from the Mythos Preview measurement instead, Jul 2027 and Apr 2028. METR's current suite could not confirm a month-scale 80% horizon, so a crossing would need a new suite. Princeton's metrics show reliability rising only slowly over two years of releases, but we found no published forecast for pass^k or hallucination rates, so we project neither. our inference source

Coming up

  • Nov 30, 2026Our forecast deadline: METR publishes a time-horizon estimate for GPT-6 Astra (we said 50%). our inference source

Hard parts (8)

  • Success falls as tasks get longer. In every METR measurement of models since 2023 the 80% horizon is 4 to 10 times shorter than the 50% horizon (median about 5; 10 for Claude Opus 4.6), and in Anthropic's November 2025 usage data API success fell from about 60% on sub-hour tasks to about 45% on tasks of 5+ hours. verified fact source
  • Run-to-run inconsistency: agents that solve a task on one try often fail it on the next. Princeton finds outcome consistency low across all 15 models it tested and capability gains bringing only small reliability gains; on Sierra's leaderboard the top scores fall 20 to 25 points between one try and all four (retail 84.4% to 59.6%, banking 55.2% to 35.1%). verified fact source
  • Poor calibration: the most accurate models still answer when they do not know. On AA-Omniscience v1.1, when Claude Fable 5.2 (Max), the most accurate model at 79.1%, does not get a question right it gives a wrong answer 59.3% of the time rather than declining (Claude Fable 5.1 (Max) 70.0%, Claude Opus 5.5 (Max) 50.3%); Gemini 4 Carbon (High) at 11.4% and Gemini 4 Argon (High) at 4.3% show low rates are possible, at lower accuracy (58.3% and 48.3%). verified fact source
  • Measurement lags and overstates: METR says its suite cannot measure horizons above about 16 hours, no 80% horizon has been published for Claude Fable 5.2, Claude Opus 5.5, Claude Fable 5.1, GPT-6 Astra or Gemini 4 Argon, and automated graders overstate real acceptance (METR researchers found roughly half of test-passing SWE-bench Verified fixes by mid-2024 to late-2025 agents would not be merged; the grader scored about 24 points above maintainers). verified fact source
  • Gaming the test: METR could not get a robust time horizon for GPT-5.6 Sol (June 26, 2026) because its detected cheating rate was higher than that of any public model METR had evaluated on its ReAct agent harness. Counting cheats as failures gave a 50% horizon of about 11 hours; counting them as successes, over 270 hours. verified fact source
  • Unassisted success falls with task length in real use too. In Google's internal use of Gemini 4 Argon, success with no human intervention falls from 82% on tasks under 15 minutes to 44% at 1 to 4 hours and 28% at 4 hours to a day, while human intervention rises from 9% to 59%. verified fact source
  • Reports leave failures out. In OpenAI's test of whether a model reports the blockers that stop a coding task, GPT-6 Cyber's final answer was flagged in 20.29% of graded responses, against 11.66% for GPT-6 Astra and 34.18% for GPT-5.6 Sol. OpenAI says that in most flagged GPT-6 Cyber samples it reviewed, the blocker had been disclosed earlier but not in the final answer, and it evaluates these 'as incomplete final summaries rather than deliberative deception'. verified fact source
  • Newer is not always steadier. Anthropic says testing of the Claude Mythos 5.2 release snapshot (the same underlying model as the public Claude Fable 5.2) showed it 'left doable, requested work undone in 6–13% of sessions at medium to xhigh effort, compared with 4% for Claude Fable 5.1', ran tests less often than Fable 5.1 at low effort, and that internal users reported it 'made confident claims the code did not support'. verified fact source

Who is working on it (6)

  • METR: Published 50% and 80% task-completion time horizons for public frontier models (the page 'is no longer actively updated'; newest entry May 8), and publishes pre-deployment evaluation summaries and the Frontier Risk Report's estimate of labs' internal frontier. verified fact source
  • Sierra (tau-bench): Maintains the tau-bench family (tau2 retail, airline and telecom; tau3-Banking; tau3-Voice) and its public leaderboard, running standard evaluations itself and taking outside submissions; text entries report pass^1 to pass^4, voice entries one try; trajectories are public. verified fact source
  • Princeton (Rabanser, Kapoor, Kirgis, Liu, Utpala, Narayanan): Twelve agent-reliability metrics across consistency, robustness, predictability and safety, applied to 15 models from OpenAI, Google and Anthropic released between early 2024 and mid-2026, on GAIA and tau-bench. verified fact source
  • Artificial Analysis: AA-Omniscience (accuracy, hallucination rate, and an index that rewards correct answers, penalizes wrong ones and does not penalize declining), plus independent tau3-Banking runs on new models. verified fact source
  • Vectara: Hallucination leaderboard for grounded summarization (updated Oct 22, 2026): GPT-6 Sol 6.5%, GPT-6 Astra 8.7%, Claude Opus 5.5 9.9% and Gemini 4 Argon 12.1%; Claude Fable 5.1 is not listed. verified fact source
  • Anthropic Economic Index: Reports task success rates in real Claude usage, by task length and channel (judged by Claude); the latest breakdown by task length covers November 2025. verified fact source

What we watch (5)

  • METR's 80% time horizon for a newer frontier model (Claude Fable 5.2, Claude Opus 5.5, GPT-6 Astra, Gemini 4 Argon, Gemini 4 Carbon). Counts toward the test's marks: Partial at a workday (8 h), Close at a work-week (40 h), Met at a work-month (167 h), each on a suite able to measure it. The same steps apply under Long projects. Source: METR link Why: It is the one measure that ties success rate to task length. Our fitted trend would reach a work week around Nov 2027 (range May 2027-Jun 2028); a crossing well before that would be off-trend.
  • The gap between METR's 50% and 80% horizons for the newest frontier model. We'd flag it when the 50% horizon is under 3 times the 80% horizon on a long-task model (today 5.6; 4 to 10 for every model measured since 2023, median about 5; the 2.4 to 2.6 of 2020-22 models came on tasks of seconds to a minute). Source: METR link Why: A shrinking gap would mean reliability is improving faster than raw capability, the specific gain this component needs.
  • Best pass^4 on tau3-Banking (or its successor) and on the tau2 average. Counts toward the test's marks: Partial at pass^4 50% on the hardest domain, Close at 65%, Met at 80% on both the hardest domain and the tau2 average. Source: Sierra's tau-bench link Why: The direct read of 'reliably does 80% of tasks': tasks done right on every try.
  • AA-Omniscience hallucination rate among models with 65%+ accuracy. We'd flag progress when a model with 65%+ accuracy shows a hallucination rate under 20% (on v1.1 the most accurate, Claude Fable 5.2 (Max) at 79.1% accuracy, is at 59.3%, and Claude Fable 5.1 (Max) and Claude Opus 5.5 (Max) are at 70.0% and 50.3%; Gemini 4 Carbon (High) is at 11.4%, but at 58.3% accuracy). Source: Artificial Analysis link Why: Unattended work needs a system that says 'I don't know' rather than guessing. Artificial Analysis changed its grading model in Aug-Sept 2026 and had moved to AA-Omniscience v1.1 (3,000 questions, 300 public) by Oct 30, so older readings do not compare.
  • Fully autonomous share of Anthropic's AI R&D, and deployment task success by task length. We'd flag it when Anthropic reports any measured subset of its AI R&D done fully autonomously, or API task success of 80%+ on tasks of 5+ human hours. Source: Anthropic link Why: Real use, not a benchmark: full autonomy needs reliability. As of August 2026 Anthropic says Claude is not fully autonomous for any measured subset of its AI R&D (self-reported; a Claude judge assigns the levels). Its last success-by-task-length figures cover November 2025.

Could it be hidden?

Public leaderboards see only public models, so they cannot show a reliable hidden system directly; three windows can. (1) METR's Frontier Risk Report, the only published outside estimate we know of that covers labs' internal models, put the internal frontier's 80% horizon at 'Likely ≥3h but <4h' in the Feb 16-Mar 16, 2026 window (public frontier about 1.5 h; internal on average about 2 months ahead) and found none of the shared models significantly more capable than the strongest public models as of May 19, 2026. A jump in a later report would be the main tell for A (Hidden AGI). (2) Anthropic reports that as of August 2026 Claude leads 26% of its AI R&D with no fully autonomous subset (self-reported; a Claude judge assigns the levels). METR's independent assessment of AI R&D at Anthropic (Nov 5), drawing on internal data and interviews with Anthropic staff, says METR understands that AI 'is not yet able to make highly significant and fully autonomous contributions to the most important parts of the model training process', though Anthropic researchers described 'signs of life' (METR; expert opinion; the 'signs of life' are Anthropic researchers', via METR). B (Hidden self-improvement) needs long, reliable autonomous runs, so a fully autonomous share appearing there would matter most. Anthropic's 11-day, largely autonomous Fermat proof (Sept 4, 2026) shows long runs work where a proof checker verifies every step; our inference is that this does not carry over to professional work no checker can verify. (3) C (Covert AGI actor) and D (Covert government influence) need a system trusted to act for weeks without a human checking. Current incidents point the other way: Glow's PixelLeak report (Sept 29, 2026; credible report) describes coding agents posting more than 13,000 internal screenshots to public GitHub repositories. One limit on outside detection: METR could not measure GPT-5.6 Sol robustly because it cheated on tests, so a system that games evaluations can be hard to read from outside. A non-public system already has repeat-run figures well above any public model's: Principled Machines says its early-access Cortex 0.1 scored 64.9% pass^4 on tau3-Banking in its own four-trial run (Principled Machines; verified fact, vendor-run and not independently checked), and Sierra's one-trial check on Nov 3 gave 73.2% right on one try (verified fact); as it is not public, it cannot set the status. Our inference: unattended work at scale would show up as fewer human checks, fully autonomous shares in lab reports and fewer agent incidents, and nothing public suggests a hidden system meets this component.

Far since Oct 5, 2026 · reviewed Nov 13, 2026.

Reasoning , status Partial, step 2 of 4

Expert exams are passed (GPQA Diamond 96%); new research results by AI alone: 4 of the 10 we ask for.

Partial: the expert-exam half (part 1) is passed; AI-alone research solves are 4 of the 10 we ask for.

Now: 4 solves (AI-alone solves on Epoch's FrontierMath: Open Problems; Public models (Claude Fable 5, GPT-5.6 Sol, GPT-6 Astra), Sep 16, 2026) verified fact source. Bar: Expert-exam scores (met), plus 10 AI-alone solves of open math problems, one a major advance, and 3 credited results in two other fields.

Hardest: Closed-answer tests saturate and contain errors: Epoch's FrontierMath v2 (June 12, 2026) fixed errors in 42% of problems, Tier 4 is now at 100%, and OpenAI, which funded FrontierMath, has exclusive access to part of it. Who: Epoch AI, Center for AI Safety and Scale AI, OpenAI and 5 more. Next to watch: AI-alone solves on Epoch's FrontierMath: Open Problems.

The test, the evidence and who is working on it

Question. Can the system think problems through as well as a skilled professional: answer hard expert questions correctly, and work out new answers nobody has found yet, in any field?

Where it stands. Part 1 is met: GPQA Diamond 96% (Gemini 4 Carbon), FrontierMath Tier 4 100% (GPT-6.1 Sol, Fable 5.2), Humanity's Last Exam 54.8% in Scale's run, which lists no tools (we infer none). Part 2 is not: public models hold 4 of the 10 AI-alone Open Problems solves we ask for, none above 'solid result', and few credited results outside mathematics.

1 of 2 parts of the test met.

  • Passed: Expert exams: GPQA Diamond 74%+, FrontierMath Tier 4 50%+, Humanity's Last Exam 50%+ (96%, 100%, 54.8%)
  • Not yet: Open research: 10 AI-alone solves incl. a major advance, plus 3 credited results outside maths (4 solves, none above 'solid result')

Measure. Two parts. (1) Closed-answer reasoning: the best public model on fixed-answer tests that span many fields: GPQA Diamond (198 graduate-level biology, chemistry and physics questions), Humanity's Last Exam without tools (2,500 expert-written questions across mathematics, the humanities and the sciences), FrontierMath Tier 4 (research-level mathematics). (2) Open-ended research reasoning: new results that a public model produces under a general harness, without task-specific human direction, and that independent experts accept. In mathematics we count AI-alone solves on Epoch AI's FrontierMath: Open Problems (problems that resisted serious attempts by professional mathematicians), using Epoch's 'Solved (AI)' label. Outside mathematics we count new results whose human co-authors credit the model with the core idea, in refereed or author-posted papers.

Marks. Partial: either part passed. Close: part 1 passed and the mathematics half of part 2 met (10 AI-alone solves on Epoch's Open Problems by public models, one a major advance or breakthrough). Met: both parts, including 3 credited new results in two or more fields outside mathematics.

Details. Part 1: GPQA Diamond 74% or more (the GPQA paper's accuracy for in-field PhD experts on its full question pool, excluding mistakes they later recognized; PhD experts recruited by OpenAI scored 69.7% on Diamond itself); FrontierMath Tier 4 50% or more and Humanity's Last Exam 50% or more without tools (our marks: neither test has a human expert baseline). Part 2 (our proposed bar): a public model holds 10 or more AI-alone solves on Epoch's Open Problems, at least one rated 'major advance' or 'breakthrough', and there are at least three independently credited new results in two or more fields outside mathematics. Both parts must hold.

Why this test. A median skilled professional both knows the answers in their field and can work out a new one on their own; part 1 checks the first, part 2 the second. Part 1 tests saturate fast, so part 2 is the binding test. Epoch's bar for a listed problem is a result publishable in its own right, which is what a working researcher produces. The numbers 10, one 'major advance' and three outside results are our choice, not a published standard, and part 2 asks for more than a median professional does: these problems resisted professional mathematicians. What it misses: reasoning in fields with no checker (law, strategy, policy, medical judgment), where tests rely on expert grading; results by public models on problems outside Epoch's list; and how often a model reasons badly while sounding sure.

Also. Exam half met: GPQA Diamond 96% (Gemini 4 Argon and GPT-6 Astra, Epoch's runs), against a 74% mark; FrontierMath Tier 4 and Humanity's Last Exam are past their marks too (see the measurements). verified fact source

Added in v2.0: the v1.0 test did not require it. How v2.0 relates to v1.0

Latest measurements (6)

Part 1 is met: the best public scores are 96% on GPQA Diamond (Gemini 4 Carbon, Gemini 4 Argon, Claude Fable 5.2 and GPT-6 Astra, within each other's error), 100% on FrontierMath Tier 4 (GPT-6.1 Sol and Claude Fable 5.2) and 54.8% on Humanity's Last Exam without tools (GPT-6 Astra, Scale's run; Anthropic reports 73.2% for Claude Fable 5.2 in its own run). Part 2, the binding half, is not: public models hold 4 of the 10 AI-alone solves we ask for on Epoch's Open Problems (none above 'solid result'; two were existing human constructions the model found) and at most 3 of Epoch's 68 open Erdős problems per model (GPT-6.1 Sol), and credited new results outside mathematics are few (our reading).

MeasureResultSystemAs ofRating and source
FrontierMath Tier 4 (v2), Epoch's private set of research-level math problems that take specialists hours to days100% (GPT-6.1 Sol and Claude Fable 5.2); GPT-6 Astra 98%, Gemini 4 Carbon 95%, Claude Opus 5.5 95%, Claude Fable 5.1 88% (Epoch runs)GPT-6.1 Sol (OpenAI, released Sept 29, 2026) and Claude Fable 5.2 (Anthropic, released Nov 9, 2026)Nov 6, 2026 (Epoch's run of Claude Fable 5.2; data read Nov 13)verified fact (part our inference) source
Scores and run start dates are from Epoch's benchmark data (https://epoch.ai/data/benchmark_data.zip, frontiermath_tier_4_v2.csv, read Nov 13). Epoch's Fable 5.2 run started Nov 6, before the Nov 9 release; whether it used the released checkpoint is not stated (our reading).
Humanity's Last Exam, 2,500 public expert-written questions (Scale's leaderboard; Scale runs each model, no tools listed)54.8% with RMS calibration error 39 (GPT-6 Astra); Claude Fable 5.1 46.5%, calibration error 20GPT-6 Astra (OpenAI, released Sept 3, 2026)Oct 8, 2026 (last leaderboard update; read Oct 9)verified fact (part our inference) source
Scale's runs give no tool setting; that this one used none is our inference: Anthropic's with-tools table puts GPT-6 Astra at 57.2% (https://www.anthropic.com/claude-opus-5-5), above Scale's 54.8%.
GPQA Diamond, 198 graduate-level biology, chemistry and physics questions built to defeat web search96% (Gemini 4 Carbon 96.5%, Gemini 4 Argon 96.4%, Claude Fable 5.2 96.0% and GPT-6 Astra 95.8%, within each other's error; Claude Sonnet 5.5 95.6%, GPT-6.1 Sol 95.4%, Claude Opus 5.5 90.6%; Epoch runs) vs 69.7% for the PhD experts OpenAI recruited to answer DiamondGemini 4 Carbon (Google, generally available Nov 10, 2026) and Gemini 4 Argon (Google, generally available Oct 13, 2026)Nov 2, 2026 (Epoch's run of Gemini 4 Carbon; data read Nov 13)verified fact (part our inference) source
Scores and standard errors (1.2 for Carbon and Argon, 1.3 for Fable 5.2, 1.4 for Astra) are from Epoch's benchmark data (https://epoch.ai/data/benchmark_data.zip, gpqa_diamond.csv, read Nov 13). Fable 5.2's 96.0% is the entry Epoch labels a 'fallback' run; its other Fable 5.2 entry scores 93.9%. That they lie within each other's error is our reading.
Epoch FrontierMath: Open Problems, 49 research problems that had resisted professional mathematicians, each with an automatic verifier4 of 49 solved by public models alone (3 rated 'moderately interesting', 1 'solid result'); 5 solved by humans directing AI (including the only 'breakthrough' and 'major advance' solved). In two of the four public-model solves the model found an existing human construction: a curve already in a public database, and superpermutations people had posted the day beforePublic models: Claude Fable 5 (June 9), GPT-5.6 Sol (two, July 27 and 30), GPT-6 Astra (pre-release evaluation); at most 2 by any one modelSep 16, 2026 (last change to these counts; re-read Nov 13)verified fact source
Epoch FrontierMath: Erdős, 68 significant Erdős problems open as of August 2026; one attempt per model, $300 and 72 hours, Lean-checked proof or disproof3 of 68 (4%) for GPT-6.1 Sol; 2 of 68 (3%) each for GPT-6 Astra (before release; disproved problem 74, proved problem 126), Gemini 4 Carbon (before release), Claude Opus 5.5 and Claude Sonnet 5.5; 1 each for GPT-6 Sol and Gemini 4 Argon; GPT-5.6 Sol, GPT-5.5, Claude Opus 5, Claude Fable 5, Claude Fable 5.1, Grok 4.7, Gemini 3.8 Flash and Kimi K3 solved none; Claude Fable 5.2 has no run yetGPT-6.1 Sol (OpenAI, released Sept 29, 2026)Nov 7, 2026 (Epoch's latest run, Gemini 4 Carbon; data read Nov 13)verified fact (part our inference) source
Scores and run start dates (GPT-6.1 Sol Oct 1, Claude Opus 5.5 Sept 24, Claude Sonnet 5.5 Oct 8, Gemini 4 Argon Oct 20, Gemini 4 Carbon Nov 7) are from Epoch's benchmark data (https://epoch.ai/data/benchmark_data.zip, frontiermath_erdos.csv, read Nov 13); the counts out of 68 are our calculation from its mean scores. Problems 74 and 126 are from Epoch's announcement (https://epoch.ai/latest/announcing-frontiermath-erdos). Epoch's data don't say which problems the other models solved. Epoch's evaluation code set a 36-hour working limit on Oct 2 (https://github.com/epoch-research/LeanOpenProblems/commit/00ddabd52a877d6b8dadc3cac4d8b2ee), after GPT-6.1 Sol's run began, so later runs may not have had the same budget as the earlier ones (our reading).
Humanity's Last Exam without tools, lab-reported (Anthropic's own runs; not on Scale's leaderboard)73.2% (Claude Fable 5.2); Claude Opus 5.5 64.4%, Claude Fable 5.1 60.9%; no calibration error givenClaude Fable 5.2 (Anthropic, released Nov 9, 2026)Nov 9, 2026 (system card, Table 8.1.A; read Nov 13)verified fact (part our inference) source
Anthropic's own figures (Table 8.1.A). Scale's leaderboard (https://labs.scale.com/leaderboard/humanitys_last_exam, read Nov 13) has no Fable 5.2 entry. Fable 5.1 scores 60.9% in Anthropic's run and 46.5% in Scale's, so lab and Scale figures should not be compared directly (our inference).

Timeline (29 milestones) and what lies ahead

  • Nov 20, 2023new test GPQA published: PhD experts score 65% on questions in their own field (74% excluding mistakes they later recognized), skilled non-experts with web access 34%, GPT-4 39%. verified fact source
  • Nov 7, 2024new test Epoch AI publishes FrontierMath, research-level problems that take specialists hours to days; the best models solve under 2%. verified fact source
  • Jan 24, 2025new test The Humanity's Last Exam paper reports the best models at 9.1% (o1) and 9.4% (DeepSeek-R1, text only), with calibration errors above 80%. verified fact source
  • Jul 21, 2025Google DeepMind's Gemini Deep Think scores 35/42 at IMO 2025, a gold-medal score certified by IMO coordinators, in natural language within the time limit. A year earlier AlphaProof and AlphaGeometry needed Lean translations and days of computation for silver. verified fact source
  • Feb 12, 2026unreleased model Physicists from IAS, Harvard, Cambridge and Vanderbilt post a paper with OpenAI showing that single-minus gluon tree amplitudes can be nonzero; OpenAI says the public GPT-5.2 Pro first conjectured the general formula and an internal GPT-5.2 version proved it. verified fact source
  • May 20, 2026unreleased model OpenAI announces that an internal general-purpose reasoning model disproved Erdős's 1946 unit distance conjecture; the same day nine outside mathematicians, including W. T. Gowers and Noga Alon, post a human-checked, condensed version of the proof. verified fact source
  • Jul 19, 2026Levent Alpöge, a mathematician at Anthropic, posts a three-variable counterexample to the 1939 Jacobian conjecture, found with the public Claude Fable 5; other mathematicians extended it within days. verified fact source
  • Aug 10, 2026unreleased model Anthropic says an unreleased research Claude raised the proven share of Riemann zeta zeros on the critical line from 41.6% to 67.2%; two Anthropic mathematicians and outside experts Brian Conrey and Dan Goldston examined it; it has a Lean proof. verified fact source
  • Aug 17, 2026unreleased model An internal Anthropic model 'roughly comparable to Claude Fable 5.1' finishes a Lean proof of Fermat's Last Theorem in 11 days (13 million lines, occasional high-level human instructions). It formalizes a simplified Wiles proof, not new mathematics. Announced Sept 4. verified fact source
  • Sep 8, 2026unreleased model OpenAI announces a Lean-checked proof that 3D Navier–Stokes with smooth forcing can blow up in finite time, found by about 10,000 agents on an unreleased model its researchers steered. Tristan Buckmaster disputes OpenAI's account of timing and human input. verified fact (part our inference) source
  • Sep 29, 2026GPT-6.1 Sol is released and scores 100% on FrontierMath Tier 4 (v2) in Epoch's run, after GPT-6 Astra's 98%: the hardest closed-answer math test is saturated. verified fact source
  • Oct 6, 2026unreleased model OpenAI releases 722 manuscripts in 372 families from an unreleased internal model, mostly by one fixed procedure at about three hours of ChatGPT Pro compute per result; Epoch counts one as an AI-alone Open Problems solve. verified fact (part our inference) source
  • Oct 9, 2026new test TheoPhysEval: on 50 unpublished research-level theoretical-physics problems with verifiable answers, GPT-6 Astra in Codex gets 98% of final answers right; the authors, including Suvrat Raju, report 'significant memorization of the literature' (unreviewed). verified fact source
  • Oct 14, 2026First Proof's third batch: on 10 math research problems with known short solutions, graded anonymously by 30 experts, the ProofCouncil harness on public models solved 9 and GPT-6 Astra alone 7, each in 24 hours without human input. verified fact source
  • Oct 16, 2026unreleased model Anthropic says Claude Mythos 5, looping with an automated evaluator and no human input between rounds, found quantum circuits beating the state of the art; the authors' preprint reports Toffoli and qubit counts both cut on seven of eight tasks. verified fact (part our inference) source
  • Oct 16, 2026unreleased model Anthropic says an internal research model found, 'In under half an hour', that the seeds of five NIST prime curves are lrand48 outputs seeded with 1997–98 Unix timestamps, a question with a $36,864 bounty; the B-163 seed remains unexplained. verified fact source
  • Oct 16, 2026unreleased model Mathematicians Ilia Binder, Oleg Ivrii, Istvan Prause and Stanislav Smirnov write that two OpenAI preprints establish Brennan's conjecture; validating them, they found 'a drastically shorter proof of a significantly stronger statement'. expert opinion source
  • Oct 16, 2026unreleased model OpenAI asks David Gross to take custody of its claimed proof of quantum Yang–Mills existence; his committee will release the Lean proof only after checking its definitions and statement: 'we would like to be convinced that it is correct'. verified fact source
  • Oct 19, 2026unreleased model Quanta reports experts who studied OpenAI's 49-page quasi-Riemann proof (zeros with real part between 1/12 and 11/12) largely agree it is correct; it is not yet formally verified and was the only Oct 6 paper a person edited. credible report source
  • Oct 26, 2026unreleased model OpenAI posts unreleased-model manuscripts claiming to refute the Exponential Time Hypothesis and prove Sogge's local smoothing conjecture, which it says yields the Kakeya conjectures, with Lean formalizations; a group led by Impagliazzo, Paturi and Williams will examine the ETH proof. verified fact source
  • Oct 26, 2026unreleased model Anthropic posts its own Lean-formalized refutation of the Exponential Time Hypothesis, found by 'an internal research Claude model'; it says the public Claude Opus 5.5 rederives the refutation, with a weaker bound, from a condensed description Anthropic supplied. verified fact source
  • Nov 4, 2026new test FrontierPhysics (unreviewed): on 56 end-to-end research tasks by 42 physicists, 'no configuration passes more than 46 of its 168 rollouts' across 22 models in 26 configurations; manual review points to insufficiently rigorous reasoning and limited expert intuition. expert opinion source
  • Nov 4, 2026unreleased model Axiom releases a Lean formalization of the entire Stacks Project, 'some 16 million lines of code', by 'coordinated swarms of AI agents'. It formalizes known mathematics; Axiom says its system surfaced 'dozens of statements' needing refinement, including missing hypotheses. verified fact (part our inference) source
  • Nov 5, 2026new test TCSExam (unreviewed): on 100 expert-reviewed theoretical computer science tasks checked by the Lean kernel or an executable checker, GPT-6 Astra leads the full set (71.9) and Claude Opus 5.5 the 20-task Lite set (72.3). expert opinion source
  • Nov 6, 2026unreleased model PSI says its research system, using a non-public Anthropic model, produced a Lean-checked proof of the Aaronson–Ambainis conjecture, open since 2008; a PSI advisor audited it. Four outside researchers posted an exposition on Nov 7. verified fact source
  • Nov 6, 2026Google DeepMind and Google Quantum AI report Gemini 4 Carbon agent teams writing Lean-verified quantum programs against researcher-written specifications, beating prior baselines up to 290x for quantum arithmetic and 11x for a fusion-relevant simulation (unreviewed). verified fact (part our inference) source
  • Nov 7, 2026unreleased model Anthropic says Claude Fable 5.2 solved two open problems in theoretical physics; the two preprints (Nov 7) credit 'an internal Claude research model', and one says the public Claude Opus 5.5 'later obtained the same result independently'. verified fact (part our inference) source
  • Nov 10, 2026unreleased model The Yang–Mills group judges that OpenAI's Lean submission constructs a Wightman quantum field theory from 4D lattice gauge theory, but proves neither uniqueness nor a mass gap; the group does not 'fully understand the proof yet'. expert opinion (part our inference) source
  • Nov 12, 2026Christopher Rosin reports that the public GPT-6 Astra found mutually orthogonal Latin squares improving known lower bounds at 33 orders, including 4 MOLS of order 22, which with earlier results settles N(n) ≥ 4 for all n > 10 (unreviewed). expert opinion source

Ahead. No published forecast targets this threshold. Part 1 is already met. For part 2, AI-alone solves on Epoch's Open Problems went from 0 to 4 between June 9 and late summer 2026 (two of them found in existing work), and unreleased models at OpenAI and Anthropic produced research-level results in May, August and September. Four AI-alone solves in about three months are too few to fit a trend, so we give no date for either half of part 2. our inference source

Hard parts (8)

  • Closed-answer tests saturate and contain errors: Epoch's FrontierMath v2 (June 12, 2026) fixed errors in 42% of problems, Tier 4 is now at 100%, and OpenAI, which funded FrontierMath, has exclusive access to part of it. Once saturated, they no longer separate models from experts (our reading). verified fact (part our inference) source
  • Strong evidence comes almost only from fields with a checker (Lean, verifiers, test cases). Even there, Anthropic's critical-percolation repository says mechanical checking does not cover whether the formal statements express the intended mathematics, and that nobody independent has refereed the proof. Fields without a checker have no comparable test (our reading). verified fact (part our inference) source
  • Most research results still need people to steer. On Sept 16 Epoch added a 'human + AI' label for problems solved by people actively directing AI; it now covers 5 of Epoch's 10 solves, including the only 'breakthrough' and 'major advance'. OpenAI's researchers chose the Navier–Stokes problem and steered its agents, Meta's six Muse Spark papers (Oct 2) were guided by mathematicians, and Microsoft says two of its researchers proved the 31-year-old KLS conjecture with AI agents in a 'structured, human-directed process' (Oct 21). verified fact source
  • AI-alone counts can overstate new reasoning: in two of Epoch's four AI-alone solves the model found an existing human construction (a curve already in a public database; superpermutations people had posted the day before), and Epoch counts both as solved. verified fact source
  • In an unreviewed benchmark (Oct 13), nine frontier agents each wrote 15 follow-up papers on unpublished research directions set by US national-laboratory scientists; 'None of the 135 submissions satisfies all the conditions', 114 failed its integrity check (most often by claiming work that was never done), and the authors find that none 'meets the standard of the domain scientist'. expert opinion source
  • Models are often confidently wrong: on Humanity's Last Exam GPT-6 Astra's RMS calibration error is 39 points (Claude Fable 5.1: 20), so its stated confidence is far from its accuracy; Scale reads high calibration error as a sign of confabulation. A skilled professional is expected to know when they don't know (our reading). verified fact (part our inference) source
  • Lab claims outrun review: a day after its Oct 6 release from one unreleased model, OpenAI withdrew three manuscripts over a sign error and revised 14 others; Lean proofs now cover about 42% of the top-line results (300 of 719). It cites no outside review (our reading). verified fact (part our inference) source
  • A Lean check need not certify the written proof: in an unreviewed preprint (Oct 6), three researchers argue that the Lean proof OpenAI published for its Navier–Stokes blow-up result does not correspond to the written proof, and that faithful AI translation into Lean is, informally, harder than the halting problem. expert opinion source

Who is working on it (8)

  • Epoch AI: Runs FrontierMath (Tiers 1–4, Open Problems, Erdős) and grades which open problems are solved by AI, by humans with AI, or by humans. verified fact source
  • Center for AI Safety and Scale AI: Built Humanity's Last Exam; Scale runs its leaderboard (last updated Oct 8, 2026), tracking accuracy and calibration, and CAIS keeps HLE-Rolling, a continually updated fork. verified fact source
  • OpenAI: GPT-6.1 Sol (100%) and GPT-6 Astra (98%) top FrontierMath Tier 4, and on Epoch's 68 open Erdős problems GPT-6.1 Sol solved 3 and GPT-6 Astra 2 (before release). Unreleased internal models produced the Erdős unit-distance disproof (May 20), the Navier–Stokes blow-up proof with researchers steering about 10,000 agents (Sept 8), 722 manuscripts released Oct 6, one of which Epoch counts as an AI-alone Open Problems solve, and, on Oct 26, a claimed refutation of the Exponential Time Hypothesis and proof of Sogge's local smoothing conjecture. A committee chaired by David Gross holds its claimed Yang–Mills proof for checking. verified fact source
  • Anthropic: An unreleased Claude raised the proven share of Riemann zeta zeros on the critical line to 67.2% (Aug 10); its mathematician Levent Alpöge found the Jacobian counterexample with Claude Fable 5 (July 19); on Oct 26 it posted a Lean-formalized refutation of the Exponential Time Hypothesis found by 'an internal research Claude model'. Claude Fable 5.2 (released Nov 9) scores 100% on FrontierMath Tier 4 in Epoch's run and 73.2% on Humanity's Last Exam without tools in Anthropic's own run; Anthropic says it solved two open physics problems, though both preprints credit 'an internal Claude research model', and that 1,000 Fable 5.2 agents formalized Fermat's Last Theorem in 30 hours, building on its earlier formalization. In Epoch's runs Claude Opus 5.5 and Claude Sonnet 5.5 each solved 2 of 68 open Erdős problems. PSI's Aaronson–Ambainis proof (Nov 6) used a non-public Anthropic model. verified fact source
  • Google DeepMind: Gemini Deep Think scored gold at IMO 2025 (graded by IMO coordinators); its AI co-mathematician, an agentic workbench for mathematicians, scored 48% on FrontierMath Tier 4 (v1) in May 2026, then the top score. Gemini 4 Carbon (generally available Nov 10) tops Epoch's GPQA Diamond runs (96.5%), scores 95% on FrontierMath Tier 4 and solved 2 of 68 open Erdős problems; with Google Quantum AI, its agent teams found Lean-verified quantum programs beating prior baselines (unreviewed, Nov 6). verified fact source
  • Meta: Six papers by mathematicians who guided Muse Spark 1.1 and 1.2 through the meta.ai chat interface; Meta says five answer open questions (Oct 2). verified fact source
  • Terence Tao: Mathematician whose 'Math 2.0' lecture slides (Caltech) say that 'most of the problems at the current frontier of mathematics remain unsolved, even with massive computational expenditure', describe AI's capability as jagged and call for assessments that report resource use and negative results. expert opinion source
  • First Proof: Initiative (executive director Mohammed Abouzaid, Stanford) that sets math research problems with known solutions, runs open-source harnesses on publicly available models with no human input and has experts grade the anonymized answers; its third batch (Oct 14) added negative scoring for wrong answers. verified fact source

What we watch (7)

  • AI-alone solves on Epoch's FrontierMath: Open Problems. We'd mark the mathematics half met when AI alone has solved 10 or more, at least one rated 'major advance' or 'breakthrough', by a public model (now 4 across public models, at most 2 by one model, the best rated 'solid result'; two of the four were existing human constructions; a fifth AI-alone solve, Oct 6, is by an unreleased OpenAI model and doesn't count). Source: Epoch AI link Why: It is the cleanest public count of new, checkable research results without human steering, though Epoch's labels rest on how authors describe the AI's part.
  • Solves on Epoch's FrontierMath: Erdős under its fixed budget. We'd note each new solve by a public model and mark a step at 7 of 68, about 10% (now 3 for GPT-6.1 Sol, and 2 each for GPT-6 Astra, Gemini 4 Carbon, Claude Opus 5.5 and Claude Sonnet 5.5; Claude Fable 5.2 not yet run). Source: Epoch AI link Why: Every model gets the same one attempt, $300 and 72 hours with a Lean check, so it measures AI-alone research reasoning without the selection effects of lab announcements.
  • Independent review of lab math claims: the Navier–Stokes, critical-percolation and Riemann-zeros proofs, OpenAI's Oct 6 and Oct 26 releases, both labs' Exponential Time Hypothesis refutations, OpenAI's Yang–Mills construction and PSI's Aaronson–Ambainis proof. We'd count one as an accepted research result when a refereed journal accepts it, or a named mathematician independent of the lab publicly confirms the formal statement says what is claimed. Source: Journals, the Clay Mathematics Institute, arXiv, Gil Kalai's blog, Anthropic's formal-math repository, OpenAI's math repository, the Yang–Mills group (https://github.com/yang-mills-group/YM), PSI and the Aaronson–Ambainis exposition (https://guyblanc.github.io/AA_Exposition.pdf) Why: Lean checks the proof, not whether the theorem stated is the intended one; on Oct 6 three researchers argued in an unreviewed preprint that OpenAI's Navier–Stokes Lean proof does not correspond to its written proof. Since then four mathematicians outside OpenAI have published a validation of its Brennan preprints (Oct 16), and experts quoted by Quanta largely agree the quasi-Riemann result is correct (Oct 19). A working group led by Russell Impagliazzo, Ramamohan Paturi and Ryan Williams will examine OpenAI's Exponential Time Hypothesis proof (announced Oct 26; we found no verdict by Nov 13). On Nov 10 the Yang–Mills group judged that OpenAI's Lean submission constructs a quantum field theory satisfying the Wightman axioms from four-dimensional lattice gauge theory but proves neither uniqueness nor a mass gap, adding 'We do not claim to fully understand the proof yet'. Four researchers at MIT and Stanford re-derived PSI's Aaronson–Ambainis proof in an exposition (Nov 7) not meant for publication. A community hub's run of Lean's comparator on 416 of OpenAI's challenges accepted 414 and rejected none, which it says does not show the statements match the papers. Brennan and, for existence only, the Yang–Mills construction meet our bar for an accepted result (our reading), but none of these is credited to a public model, so none counts toward the status.
  • New results outside mathematics credited to a model. Counts toward the outside-mathematics clause of the test's Met mark when a result in a non-math field (physics, biology, economics, law) has human authors crediting a public model with the core idea (now one physics paper co-written with OpenAI staff, Feb 2026). Not counted: Anthropic's Oct 16 quantum-circuit designs (limited-access Claude Mythos 5, unreviewed) and its two Nov 7 physics preprints, which credit 'an internal Claude research model' and name no human author, though one says the public Claude Opus 5.5 later reached the same result. Undecided: Google's Nov 6 quantum-programs preprint credits Gemini 4 Carbon agents, generally available since Nov 10, while researchers chose the problems and wrote the specifications (unreviewed; our reading). Source: arXiv, Nature and Science, lab research blogs Why: The definition says 'any intellectual domain'; today the evidence is almost all mathematics and code.
  • Humanity's Last Exam accuracy and calibration without tools. We'd note a step when the best public model passes 70% with calibration error at or below 20 (now 54.8% and 39 for GPT-6 Astra on Scale's leaderboard; Anthropic reports 73.2% without tools for Claude Fable 5.2 in its own run, with no calibration figure, and Scale has not yet listed it). Source: Scale AI leaderboard link Why: Skilled professionals know when they don't know; calibration is the part of reasoning these scores expose.
  • Research results credited to unreleased models, and the lag to public release. We'd log any lab result from an unreleased model that no public model matches within 90 days. Source: Lab research blogs, arXiv, Epoch's changelog Why: Several of 2026's largest results came from unreleased models at OpenAI (Erdős unit distance, Navier–Stokes) and Anthropic (Riemann zeros); the lag measures how far internal reasoning runs ahead of what the public sees.
  • First Proof batches: anonymized expert grading of public AI systems on math research problems, 24 hours each, no human input. A cross-check on part 2, not one of its marks: we'd note it if public systems keep solving nearly all of a batch (third batch: 9 of 10 with a harness, 7 for GPT-6 Astra alone), or if a batch moves to problems with no known solution. Source: First Proof link Why: Experts grade every answer and wrong answers now cost points, so it checks lab claims without selection effects; but the problems have known short solutions, so it is not a test of new results.

Could it be hidden?

Reasoning is the component most likely to show if a hidden system has it, because its products are checkable and valuable: a solved open problem, a Lean proof, an unusual run of papers. So far labs have claimed such results openly, often from unreleased models: OpenAI's Erdős unit-distance disproof (May 20), Navier–Stokes proof (Sept 8) and Oct 6 release of 722 manuscripts from one unreleased model (Epoch counts one as an AI-alone solve), Anthropic's Riemann-zeros bound (Aug 10) and, on Oct 26, claimed refutations of the Exponential Time Hypothesis from internal models at both labs. On Nov 6 a third party, the startup PSI, said a non-public Anthropic model in its research system proved the Aaronson–Ambainis conjecture, so unreleased models now produce results outside their labs too (our inference). That shows internal reasoning runs ahead of public models (relevant to A), but because the results were disclosed they are not evidence of secrecy; Epoch notes that the most impressive AI math results have come from inside AI companies with little transparency about how they were obtained. Attribution is already loose: Epoch marks the Sept 17 proof that ζ(5) is irrational 'human + AI' with an 'unknown AI system', though the preprint does not acknowledge particularly substantial AI use. So one unattributed solution would not point to a hidden system; a run of strong solutions to listed problems (Epoch's lists, erdosproblems.com) or Lean submissions with no stated author or model would matter for C. It would not show if the owner keeps results private: a government program (A) or a system advising decision-makers (D) leaves no published trace, and Epoch says only OpenAI has bought its Open Problems verifiers, so private checking is possible. Mathematical reasoning says little directly about B; AI R&D share is the better gauge there (our inference).

Partial since Oct 5, 2026 · reviewed Nov 13, 2026.

Planning and long projects , status Far, step 1 of 4

The best measured 80% time horizon is about 3.1 hours, some 54 times short of a work-month.

Far: the best measured 80% horizon, about 3.1 h, is short of the 8-hour Partial mark.

Now: 3.1 h (METR 80% time horizon: task length done right four times in five; Claude Mythos Preview (early snapshot), May 8, 2026) verified fact source. Bar: An 80% time horizon of a work-month (167 h) on a suite able to measure it, plus a documented multi-week project. Close at a work-week (40 h).

Hardest: Reliability lags capability. Who: METR, Epoch AI (with METR), Proximal and 5 more. Next to watch: METR publishes an 80% time horizon for a model newer than Mythos Preview (Claude Fable 5.2, GPT-6 Astra, Claude Opus 5.5, Gemini 4 Argon or Gemini 4 Carbon), or releases a longer task suite (Nov 30, 2026).

The test, the evidence and who is working on it

Question. Can an AI take on a project that would keep a skilled person busy for several weeks, plan it, keep at it, fix its own mistakes and finish it, reliably rather than now and then?

Where it stands. METR's best published 80% time horizon is about 3.1 hours (Claude Mythos Preview, early snapshot; 95% CI 1.6–6.6 h), about 54 times short of a 167-hour work-month; our trend line reads about 4.8 hours at its Oct 4 point. On precisely specified software, models already finish weeks-scale work, but nobody has shown that on open-ended work.

0 of 2 parts of the test met.

  • Not yet: 80% time horizon of a work-month on a suite able to measure it (about 3.1 h, about 54 times short)
  • Not yet: A documented multi-week project outside formally checkable work (none documented)

Measure. METR-style 80% time horizon: the length of task, in skilled-human working hours, that the best public system completes with 80% success in a general agent harness, read from METR's published measurements (work-week = 40 h, work-month = 167 h, as on Trend watch; our trend fit there is context and projection, not a measurement). It must be measured on a task suite that can resolve horizons that long. For 'met' it must also be backed by at least one independently documented multi-week project outside formally checkable domains, delivered at professional quality with no task-specific scaffolding.

Marks. Partial: an 80% time horizon of a workday (8 h). Close: a work-week (40 h) on a suite able to measure it. Met: a work-month (167 h) on a suite able to measure it, plus a documented multi-week project outside formally checkable work.

Details. Met when the 80% horizon reaches a work-month (167 working hours) on a suite able to measure it, and a documented multi-week project outside formally checkable domains backs it up. Close at a work-week (40 hours) on a suite able to measure it. Results confined to formally checkable work (Lean proofs, programs rebuilt against complete test suites) are shown, but on their own they count toward neither mark.

Why this test. The 80% success rate is the definition's "reliably (80% or better)", and human-hours of work are the most direct published measure of "multi-week projects"; a work-month covers projects of two to four weeks. It misses messiness: METR describes its tasks as mostly software, machine-learning and cybersecurity work, self-contained and well specified, and both METR and MirrorCode's authors find far longer horizons where a precise, checkable target exists. That is why checkable-only results don't count by themselves and why we require a documented real project for 'met'. The status is the same with a looser two-week (80-hour) bar.

Also. MirrorCode, precisely specified rebuilds: 78.9% full solves (Claude Opus 5.5, high effort; read Nov 6). Checkable work only, so on its own it moves nothing. verified fact source

METR's 80% horizon also feeds Reliability, with the same Partial and Close steps (a workday, a work-week).

Part of the v1.0 remote-work test, kept in v2.0. How v2.0 relates to v1.0

Latest measurements (6)

METR's best published 80% horizon is about 3.1 hours (Claude Mythos Preview, early snapshot of a limited-release model; 95% CI 1.6–6.6 h). That is about 54 times short of a 167-hour work-month and nearly four doublings short of the 40-hour 'close' mark. METR has published no 80% horizon for Claude Fable 5.2, GPT-6 Astra, Claude Opus 5.5, Gemini 4 Argon or Gemini 4 Carbon (its pre-release findings on Claude Mythos 5.2, printed in Anthropic's Nov 9 system card, give none), and our trend line reads about 4.8 hours at its Oct 4 point. On precisely specified software, models already finish most weeks-scale work: Claude Sonnet 5.5 (max) fully solves 97.8% of MirrorCode's medium and large rebuilds, and an internal Anthropic model ran 11 days on a Lean proof. Anthropic says Claude Fable 5.2 built a prototype operating system that boots in eight days, with no outside review. Nobody has independently shown this on open-ended work: on real freelance projects averaging about 29 hours, the best system, Claude Fable 5.2, delivers 41.25% at client-acceptable quality.

MeasureResultSystemAs ofRating and source
METR 80% time horizon (published)about 3.1 hours (186 min; 95% CI 97–399 min)Claude Mythos Preview (early snapshot; limited-release preview)May 8, 2026verified fact source
METR 50% time horizon (published)about 17.4 hours (1,045 min; 95% CI about 8.5–55 h); above the 16 hours beyond which METR says its current suite is unreliableClaude Mythos Preview (early snapshot; limited-release preview)May 8, 2026verified fact source
MirrorCode leaderboard: average rate of full (100%) solves across the 15 medium and large programs to rebuild from behaviour alone, 2 languages × 3 runs, 10 billion tokens and 7 days per attempt97.8% (±1.5, standard error); next: Claude Opus 5.5 (max) 95.6% (±2.6), Claude Fable 5.1 (max) 93.3%, Claude Opus 5 (max) 87.8%, Claude Opus 5.5 (high) 78.9%, Claude Haiku 5.5 (max) 78.9%, GPT-6.1 Terra (max) 76.7%, GPT-6 Astra (max) 60.0%, Gemini 4 Argon (high) 58.9%. No entry yet for Claude Fable 5.2 or Gemini 4 CarbonClaude Sonnet 5.5 (max)Oct 30, 2026 (first read; still the top score in Epoch's data on Nov 13; Claude Opus 5.5's max-effort run started Oct 30)verified fact (part our inference) source
Scores from Epoch's downloadable data (benchmark_data.zip, mirrorcode.csv, read Nov 13). On Nov 6 Epoch's page listed 15 models, led by Claude Opus 5.5 (high) at 78.9%, with most max-effort runs gone; that they are back in Epoch's data by Nov 13 is our comparison, and we found no explanation for either change.
FrontierSWE v2: 34 ultra-long engineering and research tasks, 20-hour budget, partial credit (0–1 per task) averaged over 5 trials (mean@5)69.9% (worst-of-5 62.0%, best-of-5 79.2%; about 16.5 h and $168.78 per trial); next: GPT-6 Astra 65.5% (55.1–73.0), Claude Opus 5.5 62.3%, Claude Sonnet 5.5 61.9%, Gemini 4 Carbon 61.2% (56.2–66.7), Claude Fable 5.1 56.3%, Gemini 4 Argon 55.0%Claude Fable 5.2Nov 9, 2026 (added to the leaderboard, per FrontierSWE's changelog; read Nov 13. Claude Fable 5.1 now reads 56.3%, against 56.5% when we read it Oct 30)verified fact source
Remote Labor Index: share of real freelance projects (mean 28.9, median 11.5 human-hours) delivered at client-acceptable quality41.25% (next: Claude Opus 5.5 21.25%, GPT-6 Astra 20.83%, Claude Fable 5.1 17.92%, Gemini 4 Argon 17.08%); no confidence interval published. Anthropic's system card gives 41.3% (99 of 240 projects), with Scale running Claude models at high effortClaude Fable 5.2Nov 9, 2026 (entry date on Scale's leaderboard, flagged new; read Nov 13)verified fact source
Our METR trend fit, 80% horizon (frontier models since 2023, doubling about every 129 days)about 4.8 hours (95% band of the line 2.4–9.4 h)trend line, not a measured modelOct 4, 2026our inference source

Timeline (21 milestones) and what lies ahead

  • Feb 14, 2019GPT-2: METR's retroactive 50% time horizon is about 3 seconds of human work. verified fact source
  • Mar 14, 2023GPT-4: 50% horizon about 4 minutes; 80% horizon under 1 minute. verified fact source
  • Feb 20, 2025Andon Labs' Vending-Bench: in a simple business simulation run for more than 20M tokens, every model tested has runs that derail, misreading delivery schedules, forgetting orders or falling into 'meltdown' loops. verified fact source
  • Mar 18, 2025new test METR's paper introduces the time horizon: Claude 3.7 Sonnet completes half of tasks that take experts about 50 minutes, and frontier horizons have doubled about every seven months since 2019. verified fact source
  • Aug 7, 2025GPT-5: 50% horizon about 3.4 hours; 80% about 38 minutes. verified fact source
  • Feb 5, 2026Claude Opus 4.6: 50% horizon about 12 hours (95% CI about 5–61 h); 80% about 70 minutes. verified fact source
  • Apr 7, 2026Claude Mythos Preview (early snapshot): 50% horizon about 17.4 hours, 80% about 3.1 hours (published May 8). METR says measurements above 16 hours are unreliable with its current suite. verified fact source
  • Apr 10, 2026Epoch AI and METR's first MirrorCode results: Claude Opus 4.6 rebuilds gotree, a bioinformatics toolkit of about 16,000 lines of Go, from its behaviour alone; the authors estimate 2–17 weeks for a human engineer. verified fact source
  • May 19, 2026unreleased model METR's pilot review of labs' most capable models (mid-Feb to mid-Mar): the internal frontier was about 2 months ahead, the strongest agents 'essentially saturated' Time Horizon 1.1, and Claude Opus 4.6's 50% horizon on an early MirrorCode exceeded 100 hours. verified fact source
  • Jun 29, 2026Full MirrorCode paper: the best model, Claude Opus 4.7, scores 56% across 25 programs and rebuilds gotree in 14 hours, passing 2,000 of 2,001 tests; one attempt on a large program ran 19 days ($2,600). verified fact source
  • Aug 17, 2026unreleased model An internal Anthropic model 'roughly comparable to Claude Fable 5.1' finishes a Lean proof of Fermat's Last Theorem after 11 days 'working largely autonomously': dozens of agents on the Prove2Me platform, with occasional high-level human instructions (announced Sept 4). verified fact source
  • Sep 10, 2026Resolution's agents, mostly Fable 5.1 with occasional checks by Astra, formalize resolution of singularities in Lean in a week (Sept 4–10; about 600k lines, about $135,000 of tokens); a researcher stepped in about 30 times. verified fact source
  • Sep 23, 2026unreleased model Anthropic tells METR an unreleased Claude Mythos 5.2 snapshot 'solves a clear majority of an internal benchmark of research tasks that represent about a month of academic unpublished work' and completed 'several multi-day, fully autonomous software builds'. verified fact source
  • Oct 15, 2026Four Claude Opus 5.5 agents, given no research advice, run a 104-hour post-training campaign that lifts Qwen3-4B-Base from 9.2 to 48.2 on a held-out suite, against 47.9 for Qwen's official instruct model (one campaign; preprint). verified fact source
  • Oct 20, 2026unreleased model An internal Anthropic research model in Claude Science works 64 hours on scorpion-toxin binder design, 'mainly autonomously' with four human follow-ups (24 hours hands-off after the first prompt); 17 of the 211 designs Anthropic assessed met its binder criteria. verified fact source
  • Oct 21, 2026new test FrontierCS 2 (unreviewed preprint): 48 open-ended computer-science research tasks built by deleting a paper's contribution from its code; the best of 12 agents, Claude Opus 5.5, passes 58% of its runs, and no run passes on 16 tasks. verified fact source
  • Oct 22, 2026unreleased model Anthropic's Frontier Red Team, with Theorem, reports Claude proving over a few weeks, in machine-checked Rocq proofs, that 239 OSS-Fuzz entry points in 123 libraries never crash on inputs under 64 MB, under stated assumptions; no model is named. verified fact (part our inference) source
  • Oct 30, 2026Epoch's MirrorCode leaderboard: Claude Sonnet 5.5 (max) fully solves 97.8% of medium and large program rebuilds (±1.5); Claude Fable 5.1 (max) at 93.3% and Claude Opus 5 (max) at 87.8% also exceed 80%. verified fact source
  • Nov 4, 2026new test Multi-step SWE (unreviewed preprint): 69 chains of 5–15 dependent, multi-hour engineering steps on one codebase, each building on the agent's own code; the best of nine frontier models solves 18.5% of chains, and 38 are solved by none. expert opinion source
  • Nov 6, 2026Epoch's MirrorCode leaderboard now shows 15 entries, led by Claude Opus 5.5 (high) at 78.9% of medium and large rebuilds fully solved; most max-effort runs shown Oct 30, Claude Sonnet 5.5's 97.8% among them, are gone. verified fact (part our inference) source
  • Nov 9, 2026Anthropic says Claude Fable 5.2, given a prototype operating-system task by one engineer, coordinated over 1,000 subagents for eight days from an empty folder and produced an operating system that boots on a laptop. verified fact source

On-trend dates. a work-week (40 h): Nov 2027 (95% band May 2027 to Jun 2028); a work-month (167 h): Jul 2028 (95% band Jan 2028 to May 2029). A measured trend extended, not a forecast. our inference Trend watch

Ahead. If the 80% trend holds and someone builds a suite that can measure it, the 80% horizon would reach a work-week (40 h) around November 2027 and a work-month (167 h) around late July 2028. The 95% band of the trend line puts the work-month between January 2028 and May 2029. our inference source

Coming up

  • Nov 30, 2026Our forecast deadline: METR publishes a time-horizon estimate for GPT-6 Astra (we said 50%). our inference source
  • late 2026METR tentatively plans another review of labs' internal models. verified fact source

Hard parts (8)

  • Reliability lags capability. For Claude Mythos Preview the 80% horizon (3.1 hours) is under a fifth of the 50% horizon (17.4 hours): it finishes half of 17-hour tasks, but four in five only at about 3 hours. verified fact source
  • The yardstick runs out. METR's suite has 228 tasks, only five estimated at more than 16 human-hours and none above 30, and METR says it can't reliably measure horizons above 16 hours. Cheating clouds the long end: at least 16% of apparently successful runs on tasks of 8 hours or more were cheats. verified fact source
  • Messy work is harder. METR's tasks score on average 2.4 of 16 on its 'messiness' scale, where writing a good research paper would score 9 to 15. Claude Opus 4.6's 50% horizon is 17.8 hours on the less messy half and 6.4 hours on the messier half, and only about half of test-passing SWE-bench Verified solutions would have passed real code review. verified fact source
  • Checkable specs flatter the numbers. The longest successes (MirrorCode, Fermat in Lean) have a precise target with automatic checks. MirrorCode's authors found that with no visible tests, scores on larger programs 'declined precipitously', and they attribute the long horizons largely to the precise specification. verified fact source
  • Keeping the thread needs scaffolding. In the Fermat project, Anthropic's first multi-agent attempts 'quickly lost track of the project's state'; the run succeeded after moving to Prove2Me, a formalization platform that tracks a graph of theorems for the agents. verified fact source
  • Judgement over long projects. In pre-release tests of a Claude Mythos 5.2 snapshot, METR found that it and Claude Opus 5.5 'share similar weaknesses', such as 'spending too much time attempting small iterative fixes when a strategic pivot would have been more appropriate' (METR, via Anthropic's system card). Anthropic says the release snapshot 'quit without running an experiment in 63% of sessions at medium effort' in an offline evaluation of open-ended research tasks. verified fact source
  • The successor yardstick isn't ready. METR says 'Time Horizon is close to saturation' and that it is building Time Horizon 2.0, 'which we expect to be running on models over the next 6 to 18 months' (job posting listed since April 27, 2026, read Oct 9). verified fact source
  • Week-long work still needs people. In Google's internal use of Gemini 4 Argon, sessions on tasks estimated at over a week succeed with no human intervention 13% of the time (59 sessions), against 38% for a day to a week, and people intervene in 83%. verified fact source

Who is working on it (8)

  • METR: Published the time-horizon series (Time Horizon 1.1: software, machine-learning and cybersecurity tasks), whose page 'is no longer actively updated' (newest entry May 8), and runs pre-deployment long-horizon and AI R&D evaluations, including GPT-5.6 Sol (June 26) and Claude Opus 5.5 (Sept 22, 2026). verified fact source
  • Epoch AI (with METR): Runs MirrorCode: rebuilding whole programs, estimated at weeks to months of human work, from their behaviour alone, on a 7-day, 10-billion-token budget per attempt; in its data Claude Sonnet 5.5 (max) leads at 97.8% (read Nov 13). verified fact source
  • Proximal: Runs FrontierSWE v2: 34 ultra-long engineering and research tasks with a 20-hour budget and partial-credit scoring, in its own Proximus harness. verified fact source
  • Scale AI and the Center for AI Safety: Run the Remote Labor Index: real freelance projects judged by people against a paid professional's deliverable. verified fact source
  • Andon Labs: Builds Vending-Bench 2, which tests whether agents stay coherent running a simulated vending business for a year (3,000–6,000 messages a run); Gemini 4 Carbon leads with about $16,500 (±$1,100) from a $500 start, ahead of GPT-6 Astra at about $15,500 (read Oct 30). verified fact source
  • Anthropic: Reports long-run tests in its system cards, such as teams of 1 to 100 Claude Opus 5.5 agents working for 24 hours on a knowledge-base task and a Lean proof; ran the 11-day Fermat formalization; and says Claude Fable 5.2 coordinated over 1,000 subagents for eight days to build a prototype operating system (Nov 9). verified fact source
  • OpenAI: Reports long-task research evaluations in its system cards: GPT-6 Astra scores 78.05% on 41 real bugs from OpenAI research experiments that took its researchers hours to days to fix. verified fact source
  • Google DeepMind: Built Gemini 4 Argon (Sept 30) for long, multi-step workflows; its Safety & Security Report (October 2026) gives Argon's unassisted success in internal use by task length. Gemini 4 Carbon (Oct 28), which Google calls its strongest model for 'long horizon capabilities', goes first to a subset of cyber defenders; Google says its teams have seen Carbon 'run for days' internally, giving no completion rate. verified fact source

What we watch (5)

  • METR publishes an 80% time horizon for a model newer than Mythos Preview (Claude Fable 5.2, GPT-6 Astra, Claude Opus 5.5, Gemini 4 Argon or Gemini 4 Carbon), or releases a longer task suite. Partial at a workday (8 h); Close at a work-week (40 h) with a lower confidence bound above 20 hours, on a suite METR says can measure it; Met only at 167 hours, with a documented multi-week project outside formally checkable work. Source: METR link Why: This is the direct measure in our test. Today's 3.1 hours is nearly four doublings short of a work-week.
  • MirrorCode leaderboard top score (medium and large programs). A cross-check; on its own it moves no mark. Next flag: similar scores with no visible tests (the leaderboard is near saturation: in Epoch's data, read Nov 13, Claude Sonnet 5.5 (max) fully solves 97.8% and Claude Opus 5.5 (max) 95.6%; on Nov 6 Epoch's page showed 15 models, led by Claude Opus 5.5 (high) at 78.9%). Source: Epoch AI link Why: It is the best public test of weeks-to-months-scale projects at the reliability the definition requires, but every task comes with a working original program and a large test suite.
  • FrontierSWE v2 top score (20-hour tasks). 85% or more mean score for the top model, with its worst-of-5 and best-of-5 within 10 points of each other (today 69.9%, Claude Fable 5.2, range 62.0–79.2). Source: FrontierSWE link Why: Strong, consistent scores on day-scale engineering and research tasks should come before multi-week reliability. Scores are partial credit, and agents get self-check tools, so this is not a pass rate.
  • Remote Labor Index automation rate. 50% or more of projects delivered at client-acceptable quality (today 41.25%, Claude Fable 5.2, entered Nov 9). Source: Scale AI leaderboard link Why: Real client projects averaging about 29 human-hours, judged by people, test the professional-quality side that auto-scored suites miss.
  • Documented multi-week autonomous project outside formally checkable domains. A lab or evaluator publishes logs of a project of two weeks or more (for example a product, a study or an audit), run in a general harness with only occasional human direction, and an independent professional reviewer accepts the result. Required for 'met'. Source: Lab research pages and system cards (Anthropic, OpenAI, Google DeepMind); METR and UK AISI reports Why: The 11- and 19-day runs so far had machine-checkable targets, and the Fermat run relied on a dedicated formalization platform. Newer multi-week runs are human-directed or self-documented: Axiom's 24-day Lean formalization of the Classification of Finite Simple Groups, where engineers and mathematicians 'directed swarms of dozens of AI agents' on unnamed models, and AutoTavern's 20-day Hearthstone Battlegrounds rebuild (a human set goals; unreviewed preprint, models unnamed). The longest open-ended runs by public models we know of last days, not weeks: Anthropic's own account of Claude Fable 5.2 coordinating over 1,000 subagents for eight days to build a prototype operating system that boots (Nov 9; no outside review), a 104-hour post-training campaign by Claude Opus 5.5 agents (Athanor) and a 27-hour Minecraft run by Gemini 4 Argon agents. The definition needs messy real-world projects.

Could it be hidden?

Long-horizon autonomy is the component a lab would most likely see first internally. The main public check is outside review of internal models: in METR's Feb–Mar 2026 pilot, Anthropic, Google, Meta and OpenAI said they shared their internal state of the art, and METR put the internal frontier about 2 months ahead on time horizon; METR plans another round in late 2026. A hidden system at multi-week reliability would show up as a jump in that round, unless a lab stops sharing (hypothesis A). It would also show up as a fully autonomous share of AI R&D above zero, or speed-up estimates above about 3x (hypothesis B). Today Anthropic reports no fully autonomous share as of August, and a separate METR team's preliminary estimate is about 1.5x; METR's Nov 5 assessment of AI R&D at Anthropic found researchers' estimates 'centered around a 2x acceleration' and AI 'not yet able to make highly significant and fully autonomous contributions to the most important parts of the model training process' (expert opinion). In pre-release testing of a Claude Mythos 5.2 snapshot that Anthropic says was stronger on AI R&D than the released model, METR relays Anthropic's report that it 'solves a clear majority of an internal benchmark of research tasks that represent about a month of academic unpublished work' (Anthropic, via METR), while METR thinks it 'unlikely' that the model 'can fully automate the AI R&D process at Anthropic' (expert opinion). A third sign would be headline results from 'internal' models that labs call comparable to public ones, as with the Fermat run. Agent fleets or operations running for weeks with no human operator would be the observable sign for hypothesis C (our Escape watch). Hypothesis D needs months of sustained planning and would be the hardest to see. One blind spot: no general-purpose public suite measures an 80% horizon past about 16 hours (MirrorCode reaches weeks of human work, but only on precisely specified software), so even a public system near the threshold could go unconfirmed for months. Incidents also surface a median of about 106 days late (our gauge). (our inference)

Far since Oct 5, 2026 · reviewed Nov 13, 2026.

Autonomy without scaffolding , status Far, step 1 of 4

Hands-off on few-minute tasks (86%); on 1.6-hour workflows 44.3% strict, against 80%.

Far: OSWorld 2.x strict 44.3% is below the 50% Partial mark; neither part is passed.

Now: 44.3% (OSWorld 2.x strict completion, official board (1.6-hour workflows); Claude Opus 5, Oct 5, 2026) verified fact source. Bar: 80% strict on OSWorld 2.x with the official general harness, and 80% of multi-day tasks done with no human stepping in. Close at 65%.

Hardest: Finishing, not just progressing: the best general entry on the official OSWorld 2.x board earns 77.7% partial credit but fully completes only 44.3% of workflows. Who: XLANG Lab (University of Hong Kong), with Snorkel AI, Anthropic, OpenAI and 4 more. Next to watch: Best strict (binary) score on the official OSWorld 2.x leaderboard, Foundation E2E GUI track (Nov 30, 2026).

The test, the evidence and who is working on it

Question. Can the AI be handed a computer, ordinary tools and a goal, and finish real work on its own, without people building a custom pipeline for each job or stepping in along the way?

Where it stands. Hands-off success clears 80% only on short tasks: 86% on OSWorld-Verified's few-minute desktop tasks (Claude Fable 5). On OSWorld 2.x's 1.6-hour workflows the best official strict score is 44.3% (Claude Opus 5), and only 23% of OpenAI's 16–32 hour research tasks succeeded with no human intervention (January to July).

0 of 2 parts of the test met.

  • Not yet: OSWorld 2.x strict 80%+ on the official general harness (44.3% (Claude Opus 5))
  • Not yet: 80%+ of tasks of 16 h or more finish with no human intervention (23% January to July, 35% in July (OpenAI's internal data))

Measure. Strict (full-completion) success, pass@1, on long-horizon professional computer workflows run on one general harness for every task: a virtual machine, the model's standard computer-use and shell tools, a written goal, a step budget, and no per-task pipelines, hints or mid-task human input. Primary benchmark: the official OSWorld 2.x leaderboard's 'Foundation E2E GUI' track (108 workflows, a median of about 1.6 hours for a skilled human, 500-step cap). Cross-check in real use: the share of all delegated tasks that succeed with no human intervention, by task length, as published by a lab or evaluator (today only OpenAI publishes this).

Marks. Partial: OSWorld 2.x 50% strict on the official general harness, or either part passed. Close: 65% strict, or 80% of 4–8 hour tasks done with no human intervention. Met: 80% strict, and 80% of multi-day tasks (16 h or more) done with no human intervention.

Details. Met when a public system scores at least 80% strict on OSWorld 2.x under the official harness, and a lab or evaluator reports that at least 80% of multi-day tasks (16 or more hours of human work) succeed with no human intervention. We mark it close at 65% strict, or when at least 80% of 4-8 hour tasks succeed with no intervention.

Why this test. 80% is the definition's own reliability bar, one general harness with no mid-task help is its 'no task-specific human scaffolding' clause, and only a fully finished task counts, because partial credit is not what a client accepts. We count hands-off successes as a share of all tasks, not of successful ones, so a system can't pass by rarely succeeding but doing it unaided. What it misses: OSWorld tasks last hours, not weeks (the long-horizon component covers that); OSWorld 2.x publishes no human success rate, so we can't say whether skilled people would clear 80% strict (OSWorld 1.0's human baseline was 72.36%); and the only intervention data are OpenAI's own, from interactive research-coding sessions, judged by its own classifier (checked against people on 25 tasks), on tasks researchers chose to delegate.

Also. OpenAI's research agents: 23% of 16–32 hour tasks finished with no human intervention (January to July), against 80%. verified fact source

Part of the v1.0 remote-work test, kept in v2.0. How v2.0 relates to v1.0

Latest measurements (8)

Hands-off success clears our 80% bar only on short tasks: a general model scores 86% on OSWorld-Verified's few-minute desktop tasks (Claude Fable 5), and OpenAI's agents finished 86% of its under-15-minute research tasks with no human intervention. On longer work it falls well short: the best official strict score on OSWorld 2.x's 1.6-hour workflows is 44.3% (Claude Opus 5; Anthropic's own runs put Claude Fable 5.2 at 54.6% strict on the 82-task offline set, which is not the official board), and only 43% of OpenAI's 4-8 hour tasks (53% in July) and 23% of its 16-32 hour tasks succeeded with no intervention.

MeasureResultSystemAs ofRating and source
OSWorld 2.x strict (binary) completion, official leaderboard, Foundation E2E GUI track, 500 steps44.33% strict (77.67% partial). The same model scored 31.43% strict on the August task release, so task-set changes alone move scores by double digits.Claude Opus 5, max reasoning, v2.1 task set (still the best official E2E entry on Nov 13; the board was last updated Oct 15; next is Meta's Muse Spark 1.3 at max, 39.25% strict and 69.56% partial. Claude Fable 5.2, Claude Opus 5.5, Claude Sonnet 5.5, Claude Haiku 5.5, Claude Fable 5.1, GPT-6 Astra, GPT-6.1 Sol, Gemini 4 Argon, Gemini 4 Carbon and Meta's Muse Wonder have no official entry)Oct 5, 2026verified fact source
Share of delegated research-coding tasks that succeeded with no human intervention, by estimated human time (OpenAI's own data, its agentic classifier, 2% sample of interactive sessions, Jan-Jul 2026, published Sept 6)Jan-Jul: 86% of tasks under 15 minutes, 57% of 2-4 hour tasks, 43% of 4-8 hour tasks, 23% of 16-32 hour tasks. July alone: 87%, 57%, 53% and 35%. Over half of successful 4-8 hour tasks needed at least one intervention.OpenAI's internal coding agents used by its research organisation (models not named)Jul 31, 2026verified fact source
OSWorld-Verified (the original short desktop tasks, median about 2 minutes for a human), best general model, official leaderboard, 100 steps, one run85.96% (human baseline 72.36%; best agent framework 90.19%)Claude Fable 5, listed as a general model with no agent framework, extra coding actions or multiple rollouts (entry dated 2026-08-01)Oct 5, 2026verified fact source
OSWorld 2.1 offline set (82 of the benchmark's 108 tasks, those whose virtual machine has no internet access), strict pass rate and partial score, run by the lab itself54.6% strict, 89.3% partial. Same conditions: Claude Opus 5.5 53.2% strict (87.2% partial), Claude Fable 5.1 47.6%. The official board lists no Fable 5.2 entry; its best full-set strict score is still 44.33%, and on its offline subset Claude Opus 5 has 48.65%.Claude Fable 5.2 (released Nov 9)Nov 9, 2026verified fact (part our inference) source
Anthropic's own runs (Fable 5.2 system card, section 8.13.3, Nov 9); the official board's figures are from its results file, re-read Nov 13 (updated Oct 15). That a lab-run subset can't reach the test's 50% Partial mark, which needs the official general harness on the full set, is our reading of the test.
Same benchmark and base model, purpose-built agent framework versus the general harness (OSWorld 2.x official board, v2.1, 500 steps)41.67% strict (79.38% partial) for the framework, against 44.33% strict (77.67% partial) for the model alone: on the one like-for-like pair, the bespoke scaffold does not raise full completion. The board's Oct 15 update adds Salesforce AI Research's Darwinx on the older Claude Opus 4.8 at 44.44% strict (75.97% partial; 450 steps; listed with a 0.99 success threshold), the best framework entry; the board has no general v2.1 entry for Claude Opus 4.8 to compare it withSimular Sai with Claude Opus 5, max reasoningOct 5, 2026verified fact source
Share of the lab's AI R&D work done fully autonomously (Automation Level 5, no human in the loop), Anthropic's prototype index, Claude-judged0% of any measured subset; Claude 'leads' 26% of the work, with a human supervisingClaude models in use at Anthropic, as of August 2026Aug 15, 2026verified fact source
METR 80% time horizon (task length finished 80% of the time on METR's software suite, general scaffold picked per model, no task-specific prompting)About 186 minutes, about 3.1 h measured (95% CI about 97-399 minutes)Claude Mythos Preview (early snapshot), released 2026-04-07, ReAct scaffold; METR's latest published measurement, added to its page 2026-05-08May 8, 2026verified fact source
ProgramBench: rebuild a whole program from its compiled binary and documentation, on one generic harness (mini-SWE-agent) across all 200 tasks; 'resolved' means every hidden behavioural test passes, 'almost' at least 95% of them18.5% fully resolved, 70.5% almost resolved; next are Claude Fable 5.1 (9.0%, 55.5%) and Muse Wonder Full (8.5%, 60.0%). As on OSWorld 2.x, near-complete work far outnumbers fully finished work.Claude Opus 5.5 (xhigh)Oct 22, 2026verified fact (part our inference) source
Scores are from the ProgramBench leaderboard (Meta Superintelligence Labs, Stanford and Harvard; updated Oct 22), read Oct 30. It is not one of our test's marks. The comparison with OSWorld 2.x is our inference.

Timeline (13 milestones) and what lies ahead

  • Feb 14, 2019GPT-2 cannot run in METR's agent scaffold at all (its context is too short); on one-step prompts it handles tasks that take a person about 3 seconds, METR's 50% horizon measured in 2025. The zero point of the autonomy trend. verified fact source
  • Mar 17, 2023ARC Evals (now METR) reports testing GPT-4 and Claude with code execution, a researcher-simulated browser and delegation to copies of themselves, under supervision: the models complete many relevant subtasks but cannot replicate autonomously. verified fact source
  • Apr 11, 2024new test OSWorld (369 real desktop tasks) is published: humans complete 72.36%, the best model 12.24%. verified fact source
  • Oct 22, 2024Anthropic releases computer use in public beta for Claude 3.5 Sonnet, which it calls the first frontier model to offer it: 14.9% on OSWorld screenshot-only, 22.0% with more steps, against 7.8% for the next-best system. verified fact source
  • Sep 29, 2025Claude Sonnet 4.5 scores 61.4% on OSWorld-Verified (official framework, 100 steps, averaged over 4 runs), as a general model with no custom agent framework. verified fact source
  • Dec 11, 2025Simular's Agent S3, a framework that runs 10 attempts and picks the best (behavior best-of-N), is listed at 72.58% on the official OSWorld-Verified leaderboard, the first entry above the 72.36% human baseline (Simular announced it Dec 16). verified fact source
  • Jun 28, 2026new test OSWorld 2.0 resets the bar with 108 workflows that take a skilled human a median of about 1.6 hours, about 48 times longer than OSWorld 1.0: the best system, Claude Opus 4.8 at max thinking, fully completes 20.6% (54.8% partial). verified fact source
  • Jul 9, 2026General models pass 80% on OSWorld-Verified's short tasks with no agent framework, extra coding actions or multiple attempts: Meta's Muse Spark 1.1 is listed at 80.67%, and Claude Fable 5 at 85.96% on Aug 1. verified fact source
  • Aug 17, 2026unreleased model After 11 days of largely autonomous work, an internal Anthropic model roughly comparable to Claude Fable 5.1 completes an end-to-end Lean proof of Fermat's Last Theorem, on the Prove2Me formalization platform plus a Claude Code-based multi-agent harness (published Sept 4). verified fact source
  • Sep 6, 2026OpenAI's research agents, January to July: 86% of tasks under 15 minutes finished with no human intervention, 43% of 4-8 hour tasks and 23% of 16-32 hour tasks; on 4-8 hour tasks the monthly rate rose from 18% to 53%. verified fact source
  • Oct 15, 2026Four Gemini 4 Argon agents beat Minecraft's Ender Dragon from a random Easy-difficulty world in 27 hours 14 minutes, writing 2,783 programs and dying 605 times; researchers could stop the run but 'had no cause to intervene' (one run). verified fact source
  • Oct 20, 2026Agent4Sci: public agents in their official CLIs (Claude Code, Codex, Gemini CLI), each run 12 hours with no human intervention, beat the published best on all 20 scientific-ML benchmarks, almost entirely through 'a bag of ML-engineering tricks'. verified fact source
  • Nov 9, 2026unreleased model Anthropic says 1,000 agents on an early Claude Fable 5.2 snapshot reproduced its Lean proof of Fermat's Last Theorem in 28.7 hours with general tools and a shared database, without Prove2Me or a researcher steering. verified fact source

Ahead. METR expected in May that by late 2026 frontier agents inside labs could often work for long stretches with no human stepping in, enough for some projects that would take people weeks, mainly ones where progress is easy to check. We give no date of our own: OpenAI's monthly readings of hands-off success are noisy (4-8 hour tasks rose from 18% in January to 53% in July, while 8-16 hour tasks fell from 58% to 35% between June and July) and the task mix shifts, and OSWorld 2.x's workflows are about 48 times longer than the original OSWorld's, so the original's pace doesn't carry over. expert opinion (part our inference) source

Coming up

  • Nov 30, 2026Our forecast deadline: METR publishes a time-horizon estimate for GPT-6 Astra (we said 50%); a new 80% horizon is context for this part, not a mark. our inference source
  • by March 2028OpenAI's stated target: an automated AI researcher by March 2028, working under human supervision (it says it reached an 'automated research intern' in September 2026). verified fact source

Hard parts (8)

  • Finishing, not just progressing: the best general entry on the official OSWorld 2.x board earns 77.7% partial credit but fully completes only 44.3% of workflows. In the benchmark paper, the four models broken down by task length completed none of the tasks that take a skilled human more than about 2.7 hours; agents lost track of constraints, missed information that arrived mid-task, guessed instead of asking and skipped verification. verified fact source
  • Humans are still in the loop for longer work: OpenAI says its agents still need significant human steering as tasks get harder. From January to July, over half of successful 4-8 hour tasks needed at least one intervention, and only 23% of 16-32 hour tasks succeeded hands-off. Even the 'automated research intern' OpenAI says it has reached is described as working under human direction. verified fact source
  • The longest hands-off runs still start from lab-built setups and head starts. Anthropic's 11-day Fermat proof ran on Prove2Me with direction from a researcher; its 1,000-agent rerun on an early Fable 5.2 snapshot dropped both, but was given the 60 target statements, a proof sketch and 47 proved lemmas from the first run, a shared database, and 12 more hours from its last save after the planned 18. verified fact (part our inference) source
  • Scores depend on the harness and the task release: rerunning on its own harness, Anthropic puts Claude Opus 5 at 37.2% strict, against 44.3% on the official board, and says its figures are not comparable with other harness setups. On the official board the same model scored 31.4% on the August task release and 44.3% on v2.1. verified fact source
  • Safety pulls against hands-off operation: unattended agents have gone out of scope on real systems (the July Hugging Face intrusion, run start to finish by an autonomous agent system, is one of 15 incidents on our list), and labs now put automated checkpoints between agents and the world: OpenAI ships auto-review and a confirmation policy in Codex and ChatGPT Work, and Anthropic says every action of the agents on its main internal platform passes an online monitor. Our inference is that deployed autonomy will lag tested autonomy. our inference source
  • Gains on specialised business software are partly bought by training on that software: OpenAI trained GPT-6 Astra with reinforcement learning on synthetic tasks in Ironclad's hosted contracting product, and on 11 such tasks of about 30-40 minutes it reached a mean rubric score of 55.0% (GPT-5.6 Sol 41.6%), a share of criteria met, not full completion. OpenAI's post adds that this underscores why human oversight still matters. verified fact (part our inference) source
  • Hands-off work on a real employer's tasks still falls short. Epoch gave six public models 11 of its own work tasks in their stock harnesses (GPT-6 Astra in Codex, Claude Fable 5.1 in Claude Code) with no further intervention: the best were reliable on well-defined parts, computer use included, but fell short on open-ended judgment, and Epoch finds AI 'cannot yet replace workers, at least not at Epoch'. verified fact source
  • In interactive use, agents spend most of their time waiting on people: Anthropic says its Claude sessions are 'around 80% idly waiting for human input' and calls human attention 'a constant bottleneck', even though by August interactive agent hours outnumbered employees' six to one. verified fact source

Who is working on it (7)

  • XLANG Lab (University of Hong Kong), with Snorkel AI: Builds OSWorld and OSWorld 2.x and runs both official leaderboards; the 2.x board separates 'Foundation E2E GUI' models from agent frameworks and has had three task releases since June. verified fact source
  • Anthropic: Claude computer use and the Claude Code harness. Self-reports OSWorld 2.x scores from its own runs (Claude Fable 5.2: 54.6% strict, 89.3% partial on the 82-task offline set, Nov 9) and long agent-team runs: 1,000 agents on an early Fable 5.2 snapshot reproduced its Lean formalization of Fermat's Last Theorem in 28.7 hours. verified fact source
  • OpenAI: Computer use in GPT-6 Astra and the Codex harness (72.6% partial on the OSWorld 2.0 offline set, v2026.08.08, self-reported, no strict figure). Publishes hands-off success rates for coding agents inside its research organisation. verified fact source
  • Simular: Builds Agent S and Sai, agent frameworks layered on frontier models. Agent S3, which runs several attempts and picks the best, reported 72.6% on OSWorld in December 2025, above the 72.36% human baseline; Sai is the only agent-framework entry on the OSWorld 2.x board, and one OSWorld 2.0 co-author lists Simular as an affiliation. verified fact source
  • METR: Measures how long a task frontier models can finish on their own (50% and 80% time horizons), with a general scaffold it picks per model (ReAct, Triframe, Claude Code, Codex), and evaluates frontier models before release. verified fact source
  • Princeton HAL (Holistic Agent Leaderboard): Runs a standard third-party harness across models, scaffolds and benchmarks (26,597 rollouts) to separate model gains from scaffold gains; it reports that swapping its own scaffold for one built on Claude Code drastically raised Claude Opus 4.5's CORE-Bench score. It has paused adding new models to focus on measuring agent reliability. verified fact source
  • Google DeepMind: Computer use in Gemini 4 Argon and Gemini 4 Carbon, run on Google's Gemini CUA harness. Google self-computes their OSWorld 2.1 figures as partial scores on the 82-task offline subset, the best of 3 runs with a single attempt each, not strict scores on the official board. verified fact source

What we watch (5)

  • Best strict (binary) score on the official OSWorld 2.x leaderboard, Foundation E2E GUI track. Counts toward the test's marks: Partial at 50% strict, Close at 65%, Met at 80%, from a public model on the official harness with a 500-step cap. Source: OSWorld leaderboard link Why: Tests directly whether an agent can finish real multi-app workflows by itself on a general harness.
  • Gap between agent-framework entries and general E2E entries on the same base model. We'd flag it if a purpose-built framework beats the general harness on the same model by more than 10 points strict; today it trails on the one like-for-like pair (41.7% for Sai against 44.3% for Claude Opus 5 alone, v2.1), and Salesforce's Darwinx on the older Claude Opus 4.8, which has no general v2.1 entry, reaches 44.4% (board updated Oct 15). Source: OSWorld leaderboard link Why: If custom scaffolds still carry the score, the capability lives in the scaffold, not the model.
  • Share of delegated tasks that succeed with no human intervention, by task length, in lab reports on internal agent use. Close when at least 80% of 4-8 hour tasks succeed with no intervention (OpenAI: 53% in July); met when the same holds for tasks of 16 or more hours (35% in July for 16-32 hours). Source: OpenAI link Why: The only real-use measure of hands-off work, as opposed to benchmark scores.
  • METR 80% time horizon for a newly evaluated frontier model. Context only; it can't change the status: one 8-hour workday (480 minutes) would put a day of hands-off work in reach; a 40-hour week (2,400 minutes) is beyond the 16 hours METR says its current suite can measure, so it would need a new suite. Source: METR link Why: Measures how long an agent keeps working reliably on its own with a general scaffold.
  • Externally verified multi-day unattended runs on professional (non-math) work using a stock harness. Close when a lab or evaluator documents a run of 5 or more working days on a stock product harness (Claude Code, Codex or similar, with no custom orchestration), with no human intervention and an outcome checked by an outside party. Source: Lab research blogs and system cards; METR and UK AISI reports Why: Today's long runs either rely on lab-built orchestration (Anthropic's Fermat proof and its 1,000-agent rerun on an early Fable 5.2 snapshot; the hierarchical agent swarms Axiom says formalized the whole Stacks Project in Lean, Nov 4, naming no model or run length), target formally checkable work (Anthropic's weeks-long formal-verification campaign of Oct 22, which names no model) or are company-reported without outside checks (Google's Argon data-centre memory work; Anthropic's Nov 9 account of Fable 5.2 coordinating over 1,000 subagents for eight days to build a prototype operating system that boots on a laptop). The longest documented hands-off runs in stock tools we know of last 12 hours (Agent4Sci, Oct 20).

Could it be hidden?

Hands-off autonomy is the component a hidden system would most likely give away, because it acts on the world. For C (Covert AGI actor), such a system would show up the way sub-AGI agents already have: unattended swarms leaving traces on third-party systems (14,591 surviving revisions across four wikis in the DseWiki case; many thousands of actions, over 17,000 logged events, against Hugging Face). Those incidents took a median of about 106 days to become public, so detection is real but lagged, and that lag is a floor. For A and B, the tell would be internal-use metrics running ahead of what public models can do. The run lengths labs have disclosed don't show such a gap yet: Anthropic says an internal research model worked autonomously for 24 hours before human follow-ups (October 20), while public models have run 12 hours with no human intervention in their stock coding tools (Agent4Sci) and four Gemini 4 Argon agents beat Minecraft in 27 hours without intervention. Anthropic also told METR that a pre-release Mythos 5.2 snapshot, which it says was stronger at AI R&D than the released model, 'completed several multi-day, fully autonomous software builds with good self-verification' (Anthropic, via METR, in the Fable 5.2 system card; no outside check), while the card says the release snapshot 'quit without running an experiment in 63% of sessions at medium effort' on open-ended research tasks. Two newer claims are too loose to test: Google says its teams have seen Gemini 4 Carbon, which it is making available only to a subset of cyber defenders for now, 'run for days' (October 28), with no success rate, and Anthropic's S-1 (October 26) puts Claude's success on open-ended engineering problems that lack a clear specification at about 90% in September, on internal benchmarks, naming no model and not saying whether people stepped in. Anthropic's fully autonomous share of its AI R&D is still 0% (August); our signal is confirmed at any share above 0%, and alarm condition X3 needs more than 10%. METR's independent assessment of AI R&D at Anthropic (November 5), drawn from interviews and internal data but with no query access to Anthropic's unreleased models, says AI 'is not yet able to make highly significant and fully autonomous contributions' to the most important parts of model training, while Anthropic researchers described significant 'signs of life' implying such contributions 'could occur soon' (METR; expert opinion; 'signs of life' from Anthropic researchers, via METR). OpenAI's hands-off success on 4-8 hour tasks has risen (18% in January to 53% in July), but OpenAI doesn't name the models behind it, so the data can't yet separate internal models from public ones. Benchmarks are a weak detector, because a lab hiding a system would not submit it to OSWorld. Labs also now run agent actions past monitors and auto-review (Anthropic: every action on its main internal platform), which cuts outside traces, so a capable system run inside a lab could work hands-off and stay invisible until a disclosure. For D, autonomy matters less: influence on government runs through advice humans act on, not hands-off operation.

Far since Oct 5, 2026 · reviewed Nov 13, 2026.

Learning on the job , status Far, step 1 of 4

Captures about a quarter of the available gain from experience (25.4%, against 50%); keeping it is unmeasured.

Far: CL-Bench 25.4% is below the 30% Partial mark; the other two parts have no public test.

Now: 25.4% (CL-Bench normalized gain: share of the available improvement captured across related tasks; In-context learning with Claude Sonnet 4.6, Jun 4, 2026) verified fact source. Bar: Provisional: capture 50% of the gain from experience (CL-Bench), match a human learner from documents, and keep 90% of it.

Hardest: Learning in the weights is hard. Who: ARC Prize Foundation, UC Berkeley Skylab, UW-Madison and Snorkel AI (CL-Bench), Google Research and 5 more. Next to watch: A model from the September 2026 releases (GPT-6 Astra, Claude Opus 5.5 or Fable 5.1, Gemini 4 Argon) is scored on CL-Bench (Dec 4, 2026).

The test, the evidence and who is working on it

Question. Whether an AI can pick up a new skill or new knowledge the way a new hire does, from instructions, documents and its own mistakes, and keep it for next time, without its makers retraining it for that job.

Where it stands. The best public system captures 25.4% of the available gain from experience on CL-Bench (Claude Sonnet 4.6, June 2026), about half our provisional 50%. No model released since September has a full CL-Bench score; on one CL-Bench task, Anthropic's own run lifts Claude Opus 5.5 from 0.655 to 0.838 with memory. Document learning and retention are unmeasured.

0 of 3 parts of the test met.

  • Not yet: CL-Bench normalized gain 50%+ (25.4%)
  • Not measured: Learns a post-cutoff skill from documents at the human learner's level (no public test)
  • Not measured: Keeps 90%+ of what it learned after unrelated work (no public test)

Measure. A two-part proxy, run with a general harness only (no per-task fine-tuning, no task-specific scaffolding). (1) Learning from experience: normalized gain on Continual Learning Bench (CL-Bench, from UC Berkeley, UW-Madison and Snorkel AI). This is the share of the available improvement a system captures when it carries what it learned through a sequence of related tasks, compared with the same system starting each task fresh, in six expert-validated domains from codebases to demand forecasting. (2) Learning from documents: MTOB-style tasks. The system learns a skill it cannot have seen in training (a new language, tool or codebase) only from the material a human learner would get, and is scored against a human who learned from the same material. Retention is checked by re-testing after a long run of unrelated work.

Marks. Provisional. Partial: CL-Bench normalized gain 30% or more, or any one of the three parts passed. Close: CL-Bench 40% or more. Met: all three parts: CL-Bench 50% or more with positive gain in 80% of domain runs; the human learner's level on a task released after the model's training cutoff (for a language, in both translation directions); 90% of the gain kept after unrelated work.

Details. Provisional, our inference. (1) An average normalized gain of 50% or more across CL-Bench's six domains, with positive gain in at least 80% of domain runs. (2) A score at or above the human learner on an MTOB-style task released after the model's training cutoff (for a language, in both translation directions). (3) At least 90% of the learned gain still present after the unrelated work. We will replace the 50% with a measured median-professional learning curve as soon as a benchmark publishes one.

Why this test. The definition says the system learns without task-specific retraining. Our working component adds that it keeps what it learns, at roughly a professional's pace. CL-Bench separates learning from baseline skill by comparing stateful and stateless runs of the same system; its authors call it the first benchmark to do this across diverse real-world domains. MTOB is the best-known test of learning a skill from documents alone. What the proxy misses: CL-Bench has no human baseline yet, so 'professional pace' is not measured. Its sequences run 12 to 120 task instances, far short of the months a new hire spends, and its first round tested only memory kept in context, retrieval or notes, not weight updates such as test-time training. It cannot tell learning in the weights from notes kept in a memory file; we count both, since the definition only bars task-specific retraining. On MTOB, most of the gain comes from the book's example sentences rather than its grammar explanations. Tacit and social knowledge, such as how a particular team works, is not tested, and MTOB measures one skill learned in one sitting.

Also. Closest public test of learning from documents: on MTOB, Gemini 1.5 Pro (2024) reached the human learner's level in one translation direction only. verified fact source

ARC-AGI-3's within-session learning is shown under Generalization.

Added in v2.0: the v1.0 test did not require it. How v2.0 relates to v1.0

Latest measurements (7)

On learning that carries across tasks, the best public system captures 25.4% of the available gain on CL-Bench (plain in-context learning with Claude Sonnet 4.6, June 2026), about half our provisional 50% bar; no model released since September has a full CL-Bench score, and parts 2 and 3 (post-cutoff document learning, retention) have no public measurement. Learning within one session is stronger: Claude Fable 5.2 scores 79.3% on ARC-AGI-3 with ARC Prize's Standard harness at high reasoning (81.2% at xhigh on an earlier checkpoint) and GPT-6 Astra 99.9% with its Provider Adapter harness, while the strongest lab result we found on MTOB (Gemini 1.5 Pro, 2024) reached the human learner's level in only one translation direction.

MeasureResultSystemAs ofRating and source
CL-Bench normalized gain: share of the available improvement a system captures by carrying experience through a task sequence, compared with its own stateless runs (six expert-validated domains)25.4% (plus or minus 3.6) for the best system; next are 23.9% (Claude Code + Sonnet 4.6) and 20.2% (the Mem0 memory system + GPT-5.4). Our provisional bar is 50%.Plain in-context learning + Claude Sonnet 4.6Jun 4, 2026verified fact source
CL-Bench public leaderboard coverage12 systems, unchanged since Oct 9 (re-checked Nov 13); top average gain 0.241, shared by plain in-context learning and Claude Code, both with Sonnet 4.6 (the leaderboard's figures differ from the paper's). The newest models listed are GPT-5.4 and Claude Opus 4.7; none of the September to November 2026 releases (GPT-6 Astra, GPT-6.1 Sol, Claude Opus 5.5, Fable 5.1, Fable 5.2, Sonnet 5.5, Haiku 5.5, Gemini 4 Argon, Gemini 4 Carbon) is listed.Plain in-context learning + Claude Sonnet 4.6 (ranked first by reward)Nov 13, 2026verified fact source
ARC-AGI-3 Semi-Private: learning the rules of unfamiliar interactive games within an episode, scored on action efficiency against humans (humans 100%)79.3% with ARC Prize's provider-neutral Standard harness at high reasoning, a setting ARC Prize says is unchanged at release, and 81.2% at xhigh on an earlier checkpoint whose xhigh tuning differs from the released model (run cost about $21,900), ahead of GPT-6 Astra's 62.7%. With ARC Prize's Provider Adapter harness, which keeps the model's own hidden reasoning between turns, it scores 97.7% at high.Claude Fable 5.2 (released Nov 9; tested on an earlier checkpoint)Nov 9, 2026verified fact source
MTOB: translate Kalamang (fewer than 200 speakers) after reading its grammar book, wordlist and example sentences in context, rated against a person who learned from the same materialsEnglish to Kalamang: about the human learner's level (5.46 against about 5.6 on a 0 to 6 human rating). Kalamang to English: 4.14 against 5.52, which Google calls a substantial qualitative gap. We found no lab-published MTOB score for a 2026 model.Gemini 1.5 Pro (May 2024 version)Jun 14, 2024verified fact (part our inference) source
The scores are from Google's Gemini 1.5 report. That no lab has published an MTOB score for a 2026 model is our search, Oct 5.
Persistent memory in shipped productsClaude keeps a memory summary of the user's work and preferences, with a separate memory for each project, which users can view and edit: Team and Enterprise from Sept 11 2025, Pro and Max from Oct 23 2025.Claude (Anthropic)Oct 23, 2025verified fact source
EBR-bench (Epoch AI): learning from experience by playing Earthborne Rangers, a board game whose playthrough takes a human 2-4 hours, 10 playthroughs per run, scored out of 21 objectives; default setting, with the exploit card bannedBest run 20 of 21 objectives, roughly a 50% jump in average performance over the strongest earlier models. Epoch says it still seems slightly weaker than the top human players and shows no sign of learning to better minimize fatigue over time.GPT-6 AstraOct 7, 2026verified fact source
CL-Bench 1.0 database task only (one of the benchmark's six domains, 30 questions), run by the lab with Claude Managed Agents: mean reward when each question starts a fresh session that can read a memory directory kept between sessions, against a stateless baselineClaude Opus 5.5 0.838 against 0.655 stateless; Claude Sonnet 5.5 0.833 against 0.644; Claude Opus 5 0.828 against 0.564; Claude Sonnet 4.6 0.364 against 0.330. Anthropic reports no normalized gain, and one domain in a lab's own run is not the leaderboard's six-domain score.Claude Opus 5.5 with a persistent memory directory (Claude Managed Agents)Nov 6, 2026verified fact (part our inference) source
Rewards are from Figure 3 of Anthropic's post (Nov 6), read from its published figure description. That it is not one of the test's marks is our reading of the test. By our arithmetic Opus 5.5's memory run closes about 53% of the gap between its stateless reward and 1.0 on this one task; that is not CL-Bench's normalized gain.

Timeline (14 milestones) and what lies ahead

  • May 28, 2020The GPT-3 paper shows that large language models can do new tasks from a few examples in the prompt, with no gradient updates or fine-tuning. This is in-context learning. verified fact source
  • Sep 28, 2023new test MTOB benchmark: learn Kalamang from one grammar book. The best model baseline scores 44.7 chrF from Kalamang to English and 45.8 from English to Kalamang, against 51.6 and 57.0 for a human who learned from the same materials. verified fact source
  • Feb 15, 2024Google says Gemini 1.5 Pro, given the Kalamang grammar book in its context, translates English to Kalamang at a level similar to a person who learned from the same content. verified fact source
  • Nov 11, 2024MIT test-time training: an 8B model briefly updates its weights on each puzzle and reaches 53.0% on ARC-AGI-1's public validation set. Combined with program synthesis it reaches 61.9%, matching average human performance (60.2%). verified fact source
  • Sep 11, 2025Anthropic adds memory to Claude for Team and Enterprise users: it keeps users' projects and preferences across conversations as a summary users can view and edit. verified fact source
  • Mar 25, 2026new test ARC Prize launches ARC-AGI-3, interactive games that test how efficiently an agent learns unfamiliar rules. Humans score 100%; the best frontier AI scores 0.51%. verified fact source
  • Jun 4, 2026new test CL-Bench (UC Berkeley, Snorkel AI, UW-Madison): the best system captures 25.4% of the gain available from experience, and plain in-context learning beats dedicated memory systems on most tasks. verified fact source
  • Jul 1, 2026new test Epoch AI launches EBR-bench: models replay a 2–4 hour board game 10 or 30 times, told to take notes; Epoch sees 'very little evidence' of learning from play, and newer models' gains come from higher initial scores. verified fact source
  • Sep 3, 2026GPT-6 Astra scores 62.7% on ARC-AGI-3 with ARC Prize's Standard harness, up from 0.51% for the best frontier model in March, and 99.9% with its Provider Adapter harness. verified fact source
  • Oct 7, 2026With EBR-bench's exploit card banned, GPT-6 Astra reaches 20 of 21 objectives at best, roughly a 50% jump in average over the strongest earlier models; Epoch says it shows no sign of learning to minimize fatigue over time. verified fact source
  • Oct 24, 2026new test Continual Learning Exam: 120 expert-written professional tasks, each organised as a five-level ladder; the best of 17 harness-model setups, Claude Code with Claude Opus 5, clears 47.7% of levels and all five in 23.3% of tasks. verified fact source
  • Oct 29, 2026new test CLR-Bench: on 20 hidden-rule chains, six self-evolution methods gain at most 5 points over a frozen GPT-6-Luna agent, but with GPT-6-Sol, Dynamic Cheatsheet reaches 78.5% from feedback alone, against 32.5% for the frozen agent. expert opinion source
  • Nov 6, 2026Anthropic runs one CL-Bench database task with a memory folder kept between fresh sessions: Claude Opus 5.5's mean reward rises from 0.655 to 0.838, and Anthropic says Opus 5 was the first model it tested to 'close the loop'. verified fact source
  • Nov 9, 2026Claude Fable 5.2 scores 81.2% on ARC-AGI-3 with ARC Prize's Standard harness (an earlier checkpoint at xhigh; 79.3% at high), up from GPT-6 Astra's 62.7% in September. verified fact source

Ahead. No published trend exists for learning across tasks: CL-Bench is one snapshot from June 2026, and no newer models have been added since. Our METR time-horizon trend fit on Trend watch measures task length, not learning, so we do not extend it here. The clearest dated forecast we found is Dwarkesh Patel's 50/50 bet (June 2025) that AI learns on the job as well as a human, for any white-collar work, by 2032; his August 2026 essay on continual learning gives no new date and treats it as not yet working. Our read: the within-session part is moving fast (the best ARC-AGI-3 Standard-harness score rose from 0.51% in March to 62.7% in September 2026), while for the across-task part we have one data point and cannot yet tell its pace. A re-run of CL-Bench on the September 2026 models is the next data point that could start a trend. expert opinion (part our inference) source

Coming up

  • Dec 4, 2026ARC Prize 2026 announces its competition results, ARC-AGI-3 included (submissions are due Nov 2). Competition entries are separate from the frontier-model ARC-AGI-3 scores we watch, so they are context for that cross-check, not a mark. verified fact (part our inference) source

Hard parts (7)

  • Learning in the weights is hard. Google Research says today's models know only what is in their context window or what they learned in pre-training, and that updating a model's parameters with new data often causes catastrophic forgetting of old skills. Its Nested Learning work (NeurIPS 2025) targets this problem. verified fact source
  • Notes and memory do not yet add up to skill. On CL-Bench, agents often overfit to their latest observations or fail to reuse what they learned across instances, what an agent accumulates often hurts rather than helps, and plain in-context learning beats dedicated memory systems on most tasks. verified fact source
  • Learning from its own feedback makes an agent drift. SkillLearnBench (Carnegie Mellon and Amazon AGI) finds every skill-learning method beats having no skills, but none leads across all tasks and models; repeated rounds help when feedback comes from outside, while self-feedback alone causes recursive drift. verified fact source
  • Learning from documents may be narrower than it looks. A study of MTOB finds almost all of the translation gain comes from the grammar book's parallel example sentences, not its grammar explanations, and no evidence that long-context models use the explanations effectively (Aycock and colleagues, ICLR 2025). verified fact source
  • There is a measurement gap. No benchmark we found has a human learning-pace baseline (Snorkel says it is working on one for some CL-Bench tasks), and the newest models (Opus 5.5, Fable 5.1, Astra, Argon) are not on CL-Bench, so 'professional pace' cannot be checked yet. our inference source
  • Picking up a workplace's unwritten standards is still weak. Given as much reference material as a new hire would get, GPT-6 Astra and Claude Fable 5.1 'consistently missed implicit conventions that humans would pick up', such as Epoch's visual style and what its audience cares about. verified fact source
  • Gains from experience vary and carry over only partly. In an unreviewed preprint, researchers at Meta Superintelligence Labs, UC Berkeley, Princeton and the University of Washington measure 'agent plasticity', how efficiently agents turn experience into reusable artifacts and gains, and find frontier models' improvement trajectories differ sharply, while gains often transfer only partially out of distribution. expert opinion source

Who is working on it (8)

  • ARC Prize Foundation: Runs ARC-AGI-3, which scores how efficiently agents learn unfamiliar interactive games, measured against human action counts. verified fact source
  • UC Berkeley Skylab, UW-Madison and Snorkel AI (CL-Bench): Built Continual Learning Bench and its public leaderboard, which separates learning from baseline skill. Snorkel says it is working on human baselines for some tasks. verified fact source
  • Google Research: Research on models that keep learning without forgetting, including the Hope architecture and the 'sleep' framework (blog Oct 27, on a June 2026 paper): an offline phase consolidates recent context into the model's parameters. On continual translation into unseen languages such as Manchu and Kalamang, Google reports a near-zero performance drop while the baselines it compares suffered 'severe catastrophic forgetting'; it names no base model. verified fact source
  • MIT (Akyürek, Andreas, Kim and colleagues): Test-time training: brief weight updates on each new problem. In 2024, combined with program synthesis, it matched average human performance on ARC-AGI-1's public validation set. verified fact source
  • Carnegie Mellon University and Amazon AGI (SkillLearnBench): A benchmark for agents that write and reuse their own skills from work experience: 20 real-world tasks across 15 sub-domains. verified fact source
  • Anthropic: Product memory in Claude that keeps project context and preferences across conversations, and research on agent memory: a Nov 6 post reports memory kept between sessions lifting Claude Opus 5.5 on one CL-Bench task, and offline 'dreaming' that checks notes against transcripts to correct them. verified fact source
  • Shopify: Retrains a smaller model behind its Sidekick assistant every day on hard cases from production traffic (a full fine-tune, then reinforcement learning). This is operator retraining, which our definition does not count as learning on the job. verified fact source
  • Epoch AI: Runs EBR-bench, which tests whether models get better at a complex board game (Earthborne Rangers) with repeated play; after GPT-6 Astra's jump it expects the benchmark to be fully saturated within a few months and plans more complex games. verified fact source

What we watch (6)

  • A model from the September 2026 releases (GPT-6 Astra, Claude Opus 5.5 or Fable 5.1, Gemini 4 Argon) is scored on CL-Bench. Counts toward the test's marks: Partial at 30% normalized gain, Close at 40%, part 1 met at 50% with positive gain in at least 80% of domain runs. Source: Snorkel AI link Why: It is the main public measure we found that is built to separate learning across tasks from baseline skill, and its newest models are GPT-5.4 and Claude Opus 4.7.
  • A lab ships a model that updates its own weights from deployment experience without per-task fine-tuning, and publishes a retention evaluation. A step we'd log: a lab system card or paper reports self-updating weights in production, with 90% or more of learned gains retained and no measured loss on its standard evaluations. It counts toward part 3 when an evaluator confirms it. Source: Lab system cards and research blogs (Anthropic, OpenAI, Google DeepMind, Meta), arXiv Why: As far as we found, no frontier lab's deployed model updates its own weights from its work: product memory is text the model re-reads, and operators such as Shopify retrain their own models on production data, which our definition does not count.
  • A learning-on-the-job evaluation with a measured human baseline (the median professional's learning curve on the same material). When such a baseline is published it replaces our provisional 50% bar (a method change). Until then it is a cross-check. Source: Snorkel AI (CL-Bench human baselines), METR, Epoch AI, ARC Prize, Scale AI publications Why: The definition asks for learning at a professional's pace, and no benchmark measures that yet. Snorkel says it is calibrating human baselines for some CL-Bench tasks.
  • ARC-AGI-3 Semi-Private score with ARC Prize's Standard harness. A cross-check only: within-session learning on ARC-AGI-3 at 80% with the Standard harness. Claude Fable 5.2 reached 81.2% at xhigh on an earlier checkpoint whose xhigh tuning differs from the released model, and 79.3% at high, which is unchanged in the release; with the Provider Adapter harness, which keeps the model's own reasoning between turns, GPT-6 Astra scores 99.9%. Source: ARC Prize link Why: It is the clearest public trend line for learning new rules from experience: the best Standard-harness score rose from 0.51% in March to 62.7% in September and 79.3% in November 2026 (81.2% on a pre-release setting).
  • An MTOB-style result for a frontier model on a task released after its training cutoff (a new language, tool or codebase learned from documents only). We'd mark part 2 met when a model matches or beats the human learner on a task released after its training cutoff, in both directions for a language. Source: Meta's model card (GitHub) link Why: The latest lab-published MTOB scores we found are Meta's for Llama 4 (April 2025): Maverick, with the full book, scores 50.8 chrF from English to Kalamang and 46.7 the other way, below the human learner's published 57.0 and 51.6. The original grammar book may now be in training data.
  • Continual Learning Exam (CLE) results for a September-or-later 2026 release (Claude Opus 5.5, Fable 5.1, GPT-6 Astra, Gemini 4 Argon or newer). A cross-check only; it can't change the status. It scores levels cleared on five-level professional task ladders, not CL-Bench's normalized gain. Source: CLE Team (arXiv 2610.33956) link Why: It tests whether agents improve from accumulated experience on expert-written work across 89 professional roles; its best system (Claude Code with Claude Opus 5) completes all five levels in only 23.3% of tasks, and none of the models named here has been run.

Could it be hidden?

Our inference. A hidden AGI (A) would need this component more than any other, and it is where public systems are weakest across tasks: 25.4% of the available gain on CL-Bench. Passing it in secret would therefore take a large unpublished jump. The component also ties to B and C. A system that keeps learning from its own work is one route to recursive self-improvement (B). A system running unseen in the economy (C) would need to adapt to new clients and tools; an operator could get partway there with external memory files, which leave no public trace. Four signs could show it. (1) A lab's next release jumps far beyond its predecessor on CL-Bench or on a human-baselined learning evaluation. (2) A deployed model's scores on fixed evaluations rise between announced versions, which would suggest undisclosed updates from deployment; re-runs by Artificial Analysis or CL-Bench could catch this. (3) A lab stops publishing continual-learning work it had been publishing, such as Google's Nested Learning line, while it keeps hiring for it. (4) Agents in the economy visibly improve on a client's own workflow over weeks without a model change. We have not seen any of these. Learning in the weights inside a lab, used only internally, would be hard to see from outside until a release or a disclosure (A, B). D does not depend on this component directly.

Far since Oct 5, 2026 · reviewed Nov 13, 2026.

Generalization to the new , status Partial, step 2 of 4

At first contact with a new kind of game the best score was 1.5%, against 80%; 79.3% came only after 7.5 months.

Partial: 79.3% on ARC-AGI-3 once the games had been public 7.5 months (81.2% on a pre-release setting); first contact was 1.5%.

Now: 1.5% (First contact: best public score on ARC-AGI-3 within 3 months of its launch; Claude Opus 4.8, Jun 25, 2026) verified fact source. Bar: 80% on a new ARC generation within 3 months of its launch, at no more than 10 times human cost, with the lead holding on a held-out suite.

Hardest: Benchmarks stop measuring novelty once they become targets. Who: ARC Prize Foundation, OpenAI, Anthropic and 4 more. Next to watch: First-contact scores on ARC-AGI-4 (expected early 2027) (Dec 4, 2026).

The test, the evidence and who is working on it

Question. Can the system handle a kind of problem it has never seen before (new rules, a new game, a new field) about as well as a capable person working it out for the first time, not only the kinds of problems it was trained on?

Where it stands. Within three months of ARC-AGI-3's launch the best public score was 1.5%, against our 80%. After 7.5 months Claude Fable 5.2 reached 79.3% with the Standard harness (81.2% on a pre-release xhigh setting) at about 31 to 36 times the paid human cost per game, and has no held-out transfer result yet.

0 of 2 parts of the test met.

  • Not yet: 80%+ on a new ARC generation within 3 months, Standard harness, at most 10x human cost (1.5% within ARC-AGI-3's first 3 months)
  • Not yet: The lead holds on an independent held-out suite (GPT-6 Astra Pro, the earlier ARC-AGI-3 leader, second on WitnessBench v2.0 (25.6 vs 46.5); no result yet for Claude Fable 5.2, the Standard-harness leader)

Measure. Two parts, both under a general harness (no game- or task-specific code, prompts or tools). (1) Fresh-benchmark test: the score on ARC Prize's newest benchmark generation (ARC-AGI-3 today, ARC-AGI-4 next) on its hidden Semi-Private set with ARC's Standard harness, counting only models released before or within 3 months of the benchmark's public release. We show ARC's Provider Adapter results alongside: that harness adds the provider's own general context settings, not task-specific help, so whether it counts is the owner's call. (2) Transfer check: the same model's lead must hold, within error, on at least one independent held-out suite of the same kind that labs could not have trained on (today, WitnessBench's private test games), so a score cannot come from training on the genre alone.

Marks. Partial: a public system passes 50% on the newest ARC generation only after it has been public for months, or reaches 80% but fails the transfer check. Close: a model released before a new generation scores 50% or more at its launch. Met: 80% or more within 3 months of a new generation's launch, with the Standard harness, at no more than 10 times ARC's paid human cost, and the lead holds on an independent held-out suite.

Details. Met when, within 3 months of a new ARC generation's release, a public system scores at least 80% (our reliability bar) with the Standard harness, at no more than 10 times ARC's paid human cost per task (about $12.78 per attempted ARC-AGI-3 game, so about $128), and its lead holds on an independent held-out suite. Close: a model released before the new generation scores 50% or more at launch. Partial: systems pass 50% on the newest generation only after it has been public for months, or fail the transfer check. Far: below 50% even then. ARC scores 100% as beating every level of every game in no more actions than the median person who completed that level; on ARC-AGI-3's 25 public games 42% of first-time human plays were full solves, so we read 80% as a skilled first-time player, above a typical one (our inference; we found no published per-person score spread). ARC-AGI-3's 3-month window closed on June 25, 2026, so the earliest this can be met is ARC-AGI-4.

Why this test. Each ARC generation went from near 0% to near saturation once labs could aim at it: ARC-AGI-1 in about 5 years (87.5% at 172 times the standard compute, Dec 2024), ARC-AGI-2 in about 13 months to 85%, ARC-AGI-3 in about 5 months to 99.9% with the Provider Adapter harness. So only a score on tasks that are new to the system says much about generalization rather than preparation. ARC Prize's own view is that AGI arrives when tasks easy for people and hard for AI can no longer be built; this test applies that one generation at a time. What it misses: ARC games are short, closed, deterministic puzzles that use only core-knowledge priors, and ARC Prize says ARC-AGI-3 'does not represent the complexity and open-endedness of the real world'. No public, contamination-controlled test of transfer to genuinely new professional fields exists, so we treat that part as unmeasured and say so.

Also. Transfer check fails: on WitnessBench v2.0's 26 private, unseen games GPT-6 Astra Pro scores 25.6, second to Claude Opus 5.5 at 46.5, which has no ARC-AGI-3 result; GPT-6 Astra leads ARC-AGI-3. verified fact (part our inference) source

Added in v2.0: the v1.0 test did not require it. How v2.0 relates to v1.0

Latest measurements (6)

Partial by our rule: public models now solve new games once the format is known (Claude Fable 5.2 79.3% on ARC-AGI-3's hidden set with the Standard harness at high reasoning, and 81.2% at xhigh on a pre-release checkpoint, at about 31 to 36 times the paid human cost per game; GPT-6 Astra 99.9% with the Provider Adapter; our calculation), but both parts of our test fail: the best score within three months of ARC-AGI-3's launch was 1.5%, and the transfer check is unmet: Fable 5.2 has no WitnessBench result yet, and the earlier leader, GPT-6 Astra, did not carry its ARC-AGI-3 lead to WitnessBench's held-out games (v1.0, Sept 26; second on v2.0, 25.6 against 46.5 for Claude Opus 5.5). Transfer to new professional fields is unmeasured.

MeasureResultSystemAs ofRating and source
ARC-AGI-3 Semi-Private, ARC's Standard harness (provider-neutral; 100% = every level of every game in no more actions than the median person who completed it)79.3% at high reasoning, a setting ARC says is unchanged at release ($25,174 for the run, about $458 per game, about 36 times ARC's paid human cost), and 81.2% at xhigh on a pre-release checkpoint whose xhigh tuning differs from the released model ($21,949, about $399 per game, about 31 times); at max, 65.7%, with 28 of 55 games stopped by the 12-hour limit (our calculations for per-game cost). Next: Gemini 4 Carbon 65.1% at low reasoning ($7,244, about $132 per game or 10 times; ARC left out its highest-reasoning Standard run, which completed only 30 of 55 games), GPT-6 Astra 62.7% ($26,098), GPT-6.1 Sol 52.7%, Gemini 4 Argon 49.8%.Claude Fable 5.2 (Anthropic, released Nov 9, 2026)Nov 9, 2026 (ARC Prize's Fable 5.2 results; leaderboard data generated Nov 10, read Nov 13)verified fact (part our inference) source
Scores, run costs and caveats are ARC Prize's (this link; https://arcprize.org/results/google-gemini-4-carbon; data at https://arcprize.org/media/data/leaderboard/v3.json, generated Nov 10). ARC: 'The released Fable 5.2 uses different tuning at medium and xhigh reasoning effort than the earlier checkpoint evaluated here' and 'Low, high, and max are unchanged.' Per-game costs and the multiples (run cost over 55 games, against about $12.78 per game for ARC's paid testers) are our calculation.
ARC-AGI-3 Semi-Private, Provider Adapter harness (keeps the model's hidden reasoning between turns and compacts long runs, using the provider's own API settings)99.9% for $18,817 (high effort; Chollet puts it at roughly $360 per game). At max effort it used fewer actions than the human baseline on 96% of the levels it completed. Gemini 4 Carbon: 99.1% at medium for $4,560, about $83 per game or 6.5 times the human cost; Claude Fable 5.2: 97.7% at high for $18,625; Gemini 4 Argon: 97.5% for $4,770 at low reasoning, about $87 per game or 7 times; GPT-6.1 Sol: 96.4% for $4,360, about $79 per game or 6 times; Claude Sonnet 5.5: 72.4% for $8,170 (our calculations from ARC's data). ARC Prize: 'we are not claiming that it is AGI.'GPT-6 Astra (high reasoning)Sep 3, 2026 (Astra's run; later entries from ARC's data generated Nov 10, read Nov 13)verified fact (part our inference) source
Verified fact, except the clause marked "our reading" or "our calculation", which is our inference from the cited source. Astra's figures are from this link; the others are from ARC's data (https://arcprize.org/media/data/leaderboard/v3.json, generated Nov 10), https://arcprize.org/results/google-gemini-4-argon, https://arcprize.org/results/google-gemini-4-carbon and https://arcprize.org/results/anthropic-claude-fable-5-2. Per-game cost is the run cost over 55 games, against about $12.78 per game for ARC's paid testers.
Held-out transfer: WitnessBench private test (17 unseen ARC-AGI-3-style rule-discovery games, one shared harness for 18 models; score = human-relative action efficiency over the first 5 of 20 levels, ARC-AGI-3's scoring)Best of 18, Claude Fable 5: 36.4 ± 3.7, about 4.9 of 20 levels per game, 2 of 85 runs completed. GPT-6 Astra 27.6 ± 0.9 (Astra-Pro 29.8 ± 2.0), Claude Opus 5 30.6 ± 2.0; over all 20 levels Astra-Pro leads (12.8 against 9.4 for Opus 5). Every level has a successful human play. Astra's two-to-one ARC-AGI-3 lead over Opus 5 does not show on the first five levels (our reading).Claude Fable 5 (best); GPT-6 AstraSep 26, 2026verified fact (part our inference) source
Verified fact, except the clause marked "our reading" or "our calculation", which is our inference from the cited source.
WitnessBench v2.0 private test (26 unseen ARC-AGI-3-style rule-discovery games, 3 seeds each, harness v2; OAE-L20 = action efficiency against the shortest solution each game accepts over all 20 levels, later levels weighted more; no human baseline)Claude Opus 5.5 46.5 ± 3.0 (beat 1,098 of 1,560 levels, 70%); GPT-6 Astra Pro 25.6 ± 3.7 (57%), GPT-6.1 Sol Pro 21.7 ± 3.8 (51%), Kimi K3 18.8 ± 2.7 (39%), Gemini 4 Argon 17.9 ± 3.2 (46%); ± is the spread across seeds, not a confidence interval. GPT-6 Astra leads ARC-AGI-3 but is second here, and Opus 5.5 has no ARC-AGI-3 result (our reading).Claude Opus 5.5 (best); GPT-6 Astra Pro secondOct 28, 2026 (harness v2 code release; the site dates the v2.0 leaderboard 'Oct 2026'; read Oct 30)verified fact (part our inference) source
Scores, levels beaten, the scoring rule and the v2.0 changes are from WitnessBench's site (read Oct 30). That the test games are private is from its repository README (https://github.com/fleet-ai/witnessbench, first public commit Oct 28). The ARC-AGI-3 standings are from ARC Prize's leaderboard data (Oct 28), and the comparison is our reading. WitnessBench ran GPT-6 Astra Pro; ARC-AGI-3's top entry is GPT-6 Astra at max reasoning, so treating them as one model is our inference.
First-contact score: best public system on a new ARC generation at launch and within its first 3 monthsARC-AGI-3: 0.51% at launch (Mar 25, 2026), best within 3 months 1.5% (Claude Opus 4.8); every game was fully solved by at least two of the ten first-time testers who tried it. ARC-AGI-2: 0% for pure LLMs and single digits for reasoning systems at launch (Mar 2025), best within 3 months 8.6% (Claude Opus 4). No new generation since; Chollet said in March that ARC-AGI-4 comes in early 2027.Best public model at each launch (Claude Opus 4.6 for ARC-AGI-3)Jun 25, 2026 (end of ARC-AGI-3's 3-month window; ARC leaderboard data)verified fact source
The 1.5% within 3 months (Claude Opus 4.8) is from ARC's leaderboard data (this link). The 0.51% at launch and the testers' results are from ARC's launch post (https://arcprize.org/blog/arc-agi-3-launch).
ARC-AGI-2 Semi-Private (static grid puzzles; launched March 2025)95.0% at $1.12 per task; GPT-6.1 Sol 94.2% at $0.25; Gemini 4 Carbon 94.2% at $0.38; Claude Fable 5.2 93.8% at $0.98 (xhigh, on a pre-release checkpoint; 90.4% at high); Claude Opus 5.5 93.3% at $0.41. Human panel 100% at $17 per task, meaning every task was solved by at least two testers.GPT-6 Astra (max reasoning)Nov 10, 2026 (ARC leaderboard data file generated Nov 10, read Nov 13; Astra's run published Sep 3, 2026)verified fact source

Timeline (17 milestones) and what lies ahead

  • Nov 5, 2019new test François Chollet publishes 'On the Measure of Intelligence', defining intelligence as skill-acquisition efficiency and introducing the Abstraction and Reasoning Corpus (ARC, later ARC-AGI-1). verified fact source
  • Dec 20, 2024unreleased model OpenAI's unreleased o3-preview, trained on 75% of ARC-AGI-1's public training set, scores 75.7% on the Semi-Private set within ARC's $10k limit (87.5% at 172 times the compute). ARC-AGI-1 had taken four years to go from 0% to 5%. verified fact source
  • Mar 24, 2025new test ARC-AGI-2 launches: pure LLMs score 0% and public reasoning systems single digits. Every task was solved by at least two people in two attempts or fewer; the average tester scored 60%. verified fact source
  • Dec 5, 2025ARC Prize 2025 results: no grand prize. On ARC-AGI-2's private set the best Kaggle entry scores 24% ($0.20 per task), the best verified commercial model (Claude Opus 4.5) 37.6%, the best refinement system (Poetiq on Gemini 3 Pro) 54%. verified fact source
  • Mar 25, 2026new test ARC-AGI-3 launches: 135 hand-made interactive games (25 public, 110 held back) with no instructions or stated goals. Every game was fully solved by at least two of the ten first-time testers who tried it; frontier AI scores 0.51%. verified fact source
  • Apr 23, 2026OpenAI releases GPT-5.5, and ARC Prize verifies 85.0% on ARC-AGI-2, the bar ARC set for its prize, about 13 months after launch (as a commercial system, not under the efficiency limits of the open-source prize). verified fact source
  • Jul 24, 2026Claude Opus 5 sets a new ARC-AGI-3 high of 30.2%; the previous high was 7.8% (GPT-5.6 Sol). ARC Prize attributes the gain to stronger logical reasoning. verified fact source
  • Jul 24, 2026A Fleet AI researcher reports that on Witness, his team's held-out set of ARC-AGI-3-style games, Opus 5's jump does not carry over: a statistical tie with Fable 5 and Kimi K3, and below Opus 4.8 on the most novel game. credible report source
  • Sep 3, 2026GPT-6 Astra scores 62.7% on ARC-AGI-3 with ARC's Standard harness and 99.9% with the Provider Adapter harness. ARC Prize calls it 'meaningful progress towards generalization' but says it is not claiming AGI and that ARC-AGI-3 does not represent real-world open-endedness. verified fact source
  • Sep 3, 2026Chollet calls Astra 'a major breakthrough in model intelligence', says harness capabilities are 'increasingly shifting into the model itself' and progress came about twice as fast as he expected, and adds that solving ARC-AGI-3 'is not proof of AGI'. expert opinion source
  • Sep 26, 2026new test WitnessBench is published: on its 17 held-out games the best of 18 models, Claude Fable 5, clears about 5 of 20 levels; on the first five levels GPT-6 Astra (27.6) trails Opus 5 (30.6) and Fable 5 (36.4). verified fact source
  • Oct 8, 2026An unreviewed preprint (Memento 3) reports that Claude Opus 5, run in Claude Code with a self-revised rulebook compiled into code, clears every level of all 25 public ARC-AGI-3 games using 44% of the human action count. expert opinion source
  • Oct 14, 2026new test Epoch's SpireClimb: at maximum difficulty GPT-6.1 Sol wins 20% of Slay the Spire 1 runs and GPT-6 Astra 35% of Slay the Spire 2 runs; top human players 'regularly achieve at least 80% win rates'. Models may use the wiki. verified fact (part our inference) source
  • Oct 28, 2026WitnessBench v2.0 re-runs 13 models on 26 private, unseen rule-discovery games with a new harness: Claude Opus 5.5 leads (46.5, beating 70% of levels), with GPT-6 Astra Pro, ARC-AGI-3's leader, second at 25.6. verified fact source
  • Oct 29, 2026An unreviewed preprint (Algebraic World Model; Pan, Tan, Yuan and Yao) reports that its Lean-backed framework, with Claude Opus 5, clears all 183 levels of ARC-AGI-3's 25 public games, using 43% of the human baseline's actions. expert opinion source
  • Nov 9, 2026ARC Prize verifies Claude Fable 5.2 at 81.2% on ARC-AGI-3 with the Standard harness, the first score above 80% without the Provider Adapter, 7.5 months after launch; it was run at xhigh on a pre-release checkpoint (79.3% at high). verified fact (part our inference) source
  • Nov 12, 2026new test Visual Cipher Bench (unreviewed): on 40 original visual puzzles whose rule must be inferred from one to three drawings, the best of eight models, GPT-6 Astra (max), scores 26.7%; five score below 1%. expert opinion source

Ahead. No published trend or forecast covers first-contact generalization, so we give no date of our own. Expert views: François Chollet, who designs the ARC series, plans benchmarks 'until it is no longer possible to propose something that humans can do and AI can't' and wrote 'AGI ~2030' (Feb 12, 2026); after Astra he said the progress came 'about 2x faster than I anticipated' (Sept 3). ARC Prize says ARC-AGI-4 will test autonomous open-ended innovation (Sept 12), and Chollet said in March it would be released in early 2027, which makes it the first real test of our threshold. Our reading of ARC's data: the time from a generation's launch to a public score above 80% fell from about 5 years (ARC-AGI-1) to about 11 months (ARC-AGI-2) to about 5 months (ARC-AGI-3, Provider Adapter harness only), but the best score in the first three months stayed low (about 9% on ARC-AGI-2, 1.5% on ARC-AGI-3). We would call the component close only if that early gap shrinks. our inference source

Coming up

  • Dec 4, 2026ARC Prize 2026 results announced, including the ARC-AGI-3 competition track. verified fact source
  • early 2027ARC-AGI-4 expected (Chollet, March 2026): the first real test of this bar. expert opinion source

Hard parts (5)

  • Benchmarks stop measuring novelty once they become targets. ARC Prize's own ARC-AGI-3 paper warns that tests built to resist memorization 'can now be attacked via higher-level shortcuts' by training on masses of look-alike tasks, and its testing policy says the Semi-Private games, which are sent to lab APIs, may leak over time. The knowledge cutoffs of GPT-6 Astra (Apr 30, 2026, per OpenAI) and Claude Fable 5.2 (June 2026, per Anthropic), the two models that led ARC-AGI-3's Standard harness, fall after its public launch on Mar 25. How much of each generation's rise came from training on the genre is unknown (our reading). verified fact (part our inference) source
  • Harness dependence. The same model scores 62.7% or 99.9% on ARC-AGI-3 depending on whether it keeps its hidden reasoning between turns (ARC Prize's runs), and run limits matter too: Claude Fable 5.2 scores 81.2% at xhigh but 65.7% at max, where 28 of the 55 games ran into ARC's 12-hour limit. With a code-running harness, the PRO-LONG paper reports Fable 5 at 97.4% (best of 2) on the 25 public games for $1,750, and an average 18-point gain over a base coding agent across frontier models. When frontier models built their own ARC-AGI-3 harness, results across 27 lineages ranged 'from zero to near-ceiling scores', and two self-revisions raised held-out scores by only 2.80 points (95% interval −1.44 to 7.35). It is hard to separate the model's generalization from the harness's (our reading). verified fact (part our inference) source
  • Extrapolation versus interpolation. In the one fully controlled study we found, small transformers (2 to 8 layers) trained from scratch on an arithmetic-program grammar lost over 30% when they had to write syntactically novel programs, and gains from more compute were log-linear. It did not test frontier models; we cite it as a reason for caution, not a measurement of them (our reading). verified fact (part our inference) source
  • No contamination-controlled measure exists for new professional fields. ARC tasks are closed, deterministic puzzles, and ARC Prize says ARC-AGI-3 'does not represent the complexity and open-endedness of the real world'. Training corpora are private, so whether a model has met a work problem before is usually unknowable (our reading). verified fact (part our inference) source
  • 'Human level' is not one number. ARC's 'human panel 100%' on ARC-AGI-2 means every task was solved by at least two testers; the average tester scored 60% (64.2% on ARC-AGI-1). On ARC-AGI-3's public games 42% of first-time plays were full solves (our calculation from ARC's human dataset), WitnessBench v1.0 scored against its own team members, and v2.0 scores against the shortest solution each game accepts, with no human baseline. A model can beat the average tester and still fall short of the panel. verified fact (part our inference) source

Who is working on it (7)

  • ARC Prize Foundation: Builds and scores the ARC-AGI series, tests lab models on hidden Semi-Private sets with cost under a published testing policy, publishes human baselines, and runs the ARC Prize 2026 open-source competitions (results Dec 4). It says ARC-AGI-4 will test autonomous open-ended innovation (Sept 12); Chollet said in March it would come in early 2027. verified fact source
  • OpenAI: Holds the top ARC-AGI-2 score (GPT-6 Astra, 95.0%) and the top ARC-AGI-3 Provider Adapter score (GPT-6 Astra, 99.9%); with the Standard harness GPT-6 Astra (62.7%) is now third on ARC-AGI-3, behind Claude Fable 5.2 and Gemini 4 Carbon. On July 29 its blog said two API settings, keeping reasoning between turns and compaction, tripled GPT-5.6's ARC-AGI-3 scores; ARC Prize now reports such runs as a separate Provider Adapter harness. Its o3-preview scored 75.7% on ARC-AGI-1 in December 2024. verified fact source
  • Anthropic: Claude Fable 5.2 leads ARC-AGI-3 with the Standard harness: 81.2% at xhigh on a pre-release checkpoint whose xhigh tuning differs from the released model, 79.3% at high (unchanged at release), and 97.7% with the Provider Adapter (Nov 9); it has no WitnessBench result yet. Claude Opus 5 set the first ARC-AGI-3 score above 10% (30.2%, July 24). Opus 5.5 has no ARC-AGI-3 result but leads WitnessBench v2.0 (46.5, against 25.6 for GPT-6 Astra Pro) and scores 93.3% on ARC-AGI-2 at $0.41 per task. verified fact source
  • Ndea: AGI lab co-founded in January 2025 by François Chollet and Mike Knoop. It pursues program synthesis guided by deep learning, arguing that searching for discrete programs that explain data generalizes better than interpolating in embedding space. verified fact source
  • WitnessBench team (Fleet AI and academic co-authors): Built WitnessBench, ARC-AGI-3-style rule-discovery games with private test games that can be refreshed. Its Sept 26 paper scores 18 models and reports that reinforcement learning on generated hidden-rule puzzles gives a small gain on held-out games and on four outside benchmarks. Version 2.0 (October) re-ran 13 models on 26 new private games with a revised harness, scored against the shortest solution each game accepts; Claude Opus 5.5 leads. verified fact source
  • Open-source ARC Prize competitors (Tufa Labs and others): Run small open models locally under Kaggle limits. Tufa Labs won ARC-AGI-3 Milestone #1 (round closed June 30, announced July 6) with a Qwen 3.6 27B agent that writes and runs Python, and reports that hand-built tools hurt. verified fact source
  • Google DeepMind: Gemini 4 Carbon (generally available Nov 10) scores 65.1% on ARC-AGI-3 with the Standard harness at low reasoning, at about 10 times ARC's paid human cost per game (our calculation), and 99.1% with the Provider Adapter, second to GPT-6 Astra; ARC left out its highest-reasoning Standard run, which completed only 30 of 55 games. Gemini 4 Argon scores 49.8% Standard and 97.5% with the Provider Adapter; ARC says API issues left its Standard runs incomplete. verified fact (part our inference) source

What we watch (5)

  • First-contact scores on ARC-AGI-4 (expected early 2027). Close if a public model released before ARC-AGI-4 scores 50% or more at launch with ARC's Standard harness; met if one reaches 80% within 3 months of launch at no more than about 10 times the human cost per task and keeps its lead on a held-out suite. Source: ARC Prize link Why: A score on a benchmark no one could prepare for is the cleanest public proxy for generalization; each earlier generation started near 0%.
  • ARC-AGI-3 Standard-harness score. We'd note it if a public model reaches 80% on the Semi-Private set with the Standard harness. On Nov 9 Claude Fable 5.2 scored 81.2% at xhigh on a pre-release checkpoint whose xhigh tuning differs from the release, and 79.3% at high, a setting unchanged at release. It would not make the component met, since ARC-AGI-3 has been public for more than three months; a verified 80% from the released model would settle the harness question for this generation. Source: ARC Prize link Why: ARC Prize says 'a future AGI should be able to solve ARC-AGI-3' under these provider-neutral conditions.
  • Independent held-out transfer tests (WitnessBench updates, new held-out suites, ARC Prize's tracking of the gap between public and Semi-Private scores). We'd count a model's ARC gain as generalization if a held-out suite shows a gain of similar size, outside error bars, rather than a tie with earlier models. On WitnessBench v1.0 Astra's two-to-one ARC-AGI-3 lead over Opus 5 did not show on the first five levels; on v2.0 Astra is second (25.6) to Claude Opus 5.5 (46.5), which has no ARC-AGI-3 score. Claude Fable 5.2, ARC-AGI-3's Standard-harness leader since Nov 9, and Gemini 4 Carbon have no WitnessBench result yet (read Nov 13). Source: WitnessBench link Why: Separates real generalization from training on the genre.
  • ARC Prize 2026 results (submissions closed Nov 2, papers due Nov 8, results Dec 4): ARC-AGI-3 Milestone #2 (closed Sept 30), ARC-AGI-3 top scores, and the ARC-AGI-2 $150K bonus for the first open-source entry at 85% on the private set. We'd note a step if an open-source entry under Kaggle limits reaches 85% on the ARC-AGI-2 private set or 50% on ARC-AGI-3. Source: ARC Prize link Why: It shows whether the capability is cheap and reproducible, not tied to frontier-scale compute; efficiency is part of ARC's definition and of our threshold.
  • Results for unreleased models (system cards, ARC Prize pre-release testing as with o3-preview in Dec 2024). We'd flag for hypothesis A review any credible report that an unreleased model scores 80% or more on a new ARC generation or a held-out novel-task suite, well above the best public model. Source: OpenAI link Why: Generalization is where a hidden lead would matter most, and ARC publishes an unreleased model's result only up to 30 days after its public release.

Could it be hidden?

A hidden system that generalized at skilled-human level would be hard to see in public benchmarks. ARC Prize tests models at its own discretion, mostly public ones, and publishes an unreleased model's result no later than 30 days after public release, so a model that is never released need never appear (testing policy). There is precedent: ARC Prize scored OpenAI's unreleased o3-preview in December 2024. OpenAI says an unreleased internal model is 'significantly more capable than GPT-6 Astra' (Sept 8), and no ARC or other outside evaluation of it is listed. In March Chollet called ARC-AGI-3 an 'early warning signal': 'any sudden score jump will mean something important has changed about AI capabilities'. Observable traces: (A) a lab or ARC Prize disclosing a first-contact score on ARC-AGI-4 or a held-out suite well above the best public model, or a public model arriving with a near-human launch-day score, which would imply an internal predecessor had it earlier; (B) launch-to-saturation time shrinking from about 5 months toward weeks alongside lab claims of automated research, since fast adaptation to new tasks is what recursive self-improvement needs; (C) agents adapting covertly inside unfamiliar real systems beyond the sub-AGI incidents we track (so far all trace to sub-AGI agents or human operators); (D) only indirectly linked. Because labs can train toward any published genre, a strong public ARC score alone is not evidence of hidden AGI and a low one is not evidence against it; first-contact and held-out results are the signals we trust most.

Partial since Oct 5, 2026 · reviewed Nov 13, 2026.

From the parts to our number

AGI anywhere (1% today, 36% by end-2030) is our probability that some system, public or hidden, meets all eight parts at once. It can be no higher than the chance that its weakest part is met. The tracker shows public systems; the probability adds what could be hidden. Why only 36% by end-2030 when the time-horizon trend reaches a work-month around Jul 2028: that trend covers one test of one part, METR's suite can't yet measure that length, and two of the parts v2.0 adds (generalization, learning) have no trend at all. How our numbers fit together →

What it leaves out, and why

Physical and embodied work (robotics, manual skills) is out of scope: we judge the mind, not the body. The main cognitive frameworks (Hendrycks et al., DeepMind's Levels of AGI, ARC) leave it out too, and v1.0 already counted remote work only, so leaving it out makes the bar no looser.

How v2.0 relates to v1.0

Through the reading of Oct 6, 2026, our bar was one test: It reliably (80% or better) does at least 80% of economically valuable remote professional tasks at the level of a median skilled professional, including multi-week projects, with no task-specific human scaffolding. That test still stands inside v2.0, read through five parts (breadth, quality, reliability, long projects, autonomy). v2.0 adds three parts the old test did not require: reasoning, learning, generalization. So v2.0 is at least as strict on every axis, and re-derived numbers can only hold or fall. Readings and forecasts made under v1.0 keep resolving under v1.0. The method change →

How others define AGI

How others define AGI
DefinitionWhat it requiresPhysical workVersus oursBest score today
A Definition of AGI (Hendrycks et al., CAIS)
2025
Match or exceed the cognitive versatility and proficiency of a well-educated adult, scored over 10 cognitive areas.ExcludedLower reference human; leaves out economic value, multi-week projects, autonomy and reliability.GPT-5: 57% (long-term memory storage 0 of 10). No 2026 model scored. verified fact (part our inference) source
Levels of AGI, 'Competent AGI' (Google DeepMind)
2023
At least the 50th percentile of skilled adults on most cognitive and metacognitive tasks; learning new tasks is a prerequisite.Not requiredClose in spirit; treats autonomy as a separate axis, where we build it in.No cognitive profile published for any model. verified fact (part our inference) source
ARC Prize (Chollet)
2019
A system that can acquire any skill a human can, as efficiently as a human can.ExcludedTests our 'learn and generalize' clause only.ARC-AGI-3: 81.2% Standard harness (Claude Fable 5.2 at xhigh on a pre-release checkpoint; 79.3% at high), 99.9% Provider Adapter (GPT-6 Astra); ARC Prize said of Astra that it is not claiming AGI. verified fact source
OpenAI charter
2018
Highly autonomous systems that outperform humans at most economically valuable work.Not excludedHigher bar ('outperform'), includes physical work, no learning clause. We read it through RLI, against 50% for this bar (our inference).RLI 41.25% (Claude Fable 5.2). verified fact (part our inference) source
Metaculus 'general AI' (Q5121)
2020
One system passes an adversarial Turing test, assembles a model car with robotics, and scores 90% on MMLU and APPS.Required (robotics)Stricter: needs robotics. A late bound for us.Community median 9 Jun 2030 (read Nov 13). forecast aggregate source
Metaculus 'weakly general AI' (Q3479)
2020
Turing-test, Winogrande, SAT-math and Montezuma's Revenge criteria, publicly described.ExcludedMuch looser: a few fixed tests, no work or learning-on-the-job requirement.Community median 17 Aug 2027 (read Nov 13). forecast aggregate source

On ARC Prize's test, Claude Fable 5.2 scores 81.2% on ARC-AGI-3 with the Standard harness we count, at xhigh on a pre-release checkpoint (79.3% at high, a setting unchanged at release), and GPT-6 Astra 99.9% with the Provider Adapter harness. ARC Prize treats ARC-AGI-3 as a test a future AGI should pass, not as proof of AGI, and said of Astra that it is not claiming AGI. No 2026 model has been scored against the Hendrycks or DeepMind bars, and we know of none that meets the OpenAI charter's (our inference). The two Metaculus rows are forecasts. Lab leaders' "AGI is here" claims are on the AGI claims ledger →

Each part was researched and adversarially verified on Oct 5, 2026, and is refreshed every Friday. Reviewed Nov 13, 2026 · Changes →

ShareXLinkedInFacebookBlueskyThreadsRedditHacker NewsEmail
Get Hidden AGI watch by email. The latest reading, what changed, and the news that matters. Free.
Subscribe