Anthropic’s new AI leads freelance benchmarks, yet fails 58% of the tasks
Anthropic's Fable 5.2 tops a freelance work benchmark at 41.3%, meeting professional standards on roughly two in five projects.
Fable 5.2 leads Scale's Remote Labor Index at 41.30%, ahead of Opus 5.5's 21.25% project pass rate. The RLI measures complete deliverables against professional reference work, providing a practical test of AI capabilities in freelance projects.
The benchmark, announced by Anthropic on November 9, evaluates AI performance on 240 projects across 23 freelance categories. Each project includes a brief, necessary files, and a professional human-reference deliverable for comparison.
The evaluation method uses trained human reviewers to assess whether the AI's work satisfies the brief at least as well as the reference deliverable, at a quality a reasonable client would accept.
Benchmark limitations
Fable 5.2's score still leaves most evaluated projects below the acceptance threshold. The original study excludes work requiring client interaction, teamwork, or physical labor, limiting how broadly the result applies to the wider remote-work economy.
Passing demands a deliverable that clears the professional reference standard.
Unresolved factors
Anthropic's release specifies 'High effort' but doesn't detail the precise RLI execution setup, spending per project, or amount of human support. The score alone doesn't establish job losses, deployment costs, or human-free execution capabilities.
The RLI provides a demanding definition of success for businesses assessing AI's usefulness: carrying an assignment through to an inspectable product that meets professional standards. Current limitations suggest AI still requires significant human oversight for most freelance work.
Related
