Which model leads on WeirdML?

GPT-6 Astra from OpenAI leads WeirdML with a score of 93.6. The median score across 116 tested models is 45.5.

Is WeirdML saturated?

No · the top score is 93.6 out of 100 (94%). There is still meaningful room for improvement on WeirdML.

Does WeirdML predict performance on other benchmarks?

Yes · WeirdML scores correlate 0.99 with Osworld 2 0 across 6 shared models. Models that do well on WeirdML tend to do well on Osworld 2 0.

How often is WeirdML data refreshed?

BenchGecko pulls updates daily. New model scores on WeirdML appear as soon as they are published by Epoch AI or the model provider.

Benchmark · CodeSettled

WeirdML

Name: WeirdML Benchmark
Creator: BenchGecko
License: https://creativecommons.org/licenses/by/4.0/

WeirdML · tests models on unusual and adversarial machine learning tasks that require creative problem-solving beyond standard patterns.

Updated 2026-09-04

Models tested

116

Top score

93.6

GPT-6 Astra

Median

45.5

min 1.7

Top-5 spread

σ 1.4

Settled

The Frontier

Best score over time · one chart, every benchmark

Chart type

Frontier on WeirdML rose from 25.1 to 93.6 in 22 months · +68.4 points · latest leader GPT-6 Astra from OpenAI.

Pink dots = frontier records · 15 totalClick to open model page

Full rankings

116 models tested · sorted by score

#	Model	Score	Price
1	GPT-6 Astra· OpenAI	93.6	$10.00
2	Claude Fable 5.1· Anthropic	92.9	$10.00
3	Claude Fable 5· Anthropic	91.9	$10.00
4	Claude Opus 5· Anthropic	91.8	$5.00
5	GPT-5.6 Sol· OpenAI	89.4	$2.00
6	Gemini 3.8 Flash· Google DeepMind	84.8	$0.75
7	Claude Opus 4.8· Anthropic	82.9	$5.00
8	Kimi K3· moonshotai	82.6	$0.45
9	GPT-5.3-Codex· OpenAI	79.3	$1.75
10	GPT-5.6 Terra· OpenAI	78.3	$2.00
11	Claude Opus 4.6· Anthropic	78.0	$5.00
12	GPT-5.4· OpenAI	77.7	$2.50
13	Claude Opus 4.7· Anthropic	76.4	$5.00
14	GLM 5.3· z-ai	75.4	$0.03
15	GPT-5.2· OpenAI	72.2	$1.75
16	Gemini 3.1 Pro Preview· Google DeepMind	72.1	$2.00
17	GLM 5.2· z-ai	70.1	$0.02
18	Gemini 3 Pro· Google DeepMind	69.9	n/a
19	Claude Sonnet 5· Anthropic	68.8	$2.00
20	Grok 4.6· xAI	67.3	$2.00
21	DeepSeek V4 Pro 0813· DeepSeek	66.2	$0.66
22	Claude Sonnet 4.6· Anthropic	66.1	$3.00
23	Claude Opus 4.5· Anthropic	63.7	$5.00
24	DeepSeek V4 Flash 0731· DeepSeek	63.0	$0.02
25	Gemini 3.5 Flash· Google DeepMind	62.6	$1.50
26	Gemini 3 Flash Preview· Google DeepMind	61.6	$0.50
27	GPT-5.6 Luna· OpenAI	60.9	$0.20
28	GPT-5.1· OpenAI	60.8	$1.25
29	GPT-5· OpenAI	60.7	$1.25
30	GPT-5 Pro· OpenAI	60.4	$15.00
31	GPT-5.4 Mini· OpenAI	60.3	$0.75
32	o3 Pro· OpenAI	58.2	$20.00
33	GPT-5.4 Pro· OpenAI	57.4	$30.00
34	GLM 5.1· z-ai	57.1	$0.97
35	Gemini 3.6 Flash· Google DeepMind	56.1	$0.75
36	Kimi K2.6· moonshotai	55.9	$0.95
37	Kimi K2.7 Code· moonshotai	54.1	$0.67
38	Gemini 2.5 Pro· Google DeepMind	54.0	$1.25
39	GPT-5 Mini· OpenAI	52.7	$0.25
40	o4 Mini· OpenAI	52.6	$1.10
41	o3· OpenAI	52.4	$2.00
42	Gemma 4 31B· Google DeepMind	52.3	$0.09
43	Grok 4.20· xAI	52.3	$1.25
44	Gemini 3.1 Flash Lite· Google DeepMind	52.2	$0.25
45	Grok 4.3· xAI	49.9	$1.25
46	GPT-5.4 Nano· OpenAI	49.2	$0.20
47	DeepSeek V4 Pro· DeepSeek	48.9	$0.96
48	GLM 5· z-ai	48.2	$1.00
49	gpt-oss-120b· OpenAI	48.2	$0.04
50	Claude Sonnet 4.5· Anthropic	47.7	$3.00
51	o1-preview· OpenAI	47.6	n/a
52	Grok 4.5· xAI	46.4	$2.00
53	Claude Sonnet 4· Anthropic	46.1	$3.00
54	o1· OpenAI	46.1	$15.00
55	Claude Opus 4.1· Anthropic	45.9	$15.00
56	Grok 4· xAI	45.7	n/a
57	DeepSeek V4 Flash· DeepSeek	45.6	$0.09
58	Kimi K2.5· moonshotai	45.6	$0.45
59	Claude Haiku 4.5· Anthropic	45.4	$1.00
60	Claude Opus 4· Anthropic	43.7	$15.00
61	Mistral Medium 3.5· Mistral AI	43.7	$1.50
62	o3 Mini· OpenAI	43.7	$1.10
63	Nemotron 3 Ultra· NVIDIA	43.5	$0.60
64	Grok 4 Fast· xAI	42.9	n/a
65	Kimi K2 Thinking· moonshotai	42.8	$0.60
66	Grok-3 mini· xAI	42.6	n/a
67	R1 0528· DeepSeek	41.6	$0.50
68	Qwen3 235B A22B Thinking 2507· Alibaba Qwen	41.0	$0.23
69	Gemini 2.5 Flash· Google DeepMind	41.0	$0.30
70	gpt-oss-20b· OpenAI	40.9	$0.02
71	DeepSeek V3.2 Exp· DeepSeek	39.5	$0.27
72	GPT-4.5· OpenAI	39.4	n/a
73	Kimi K2 0711· moonshotai	39.4	$0.57
74	GPT-4.1· OpenAI	39.0	$2.00
75	Gemini 3.5 Flash Lite· Google DeepMind	39.0	$0.30
76	Qwen3 235B A22B Instruct 2507· Alibaba Qwen	38.7	$0.09
77	DeepSeek V3.1· DeepSeek	38.4	$0.25
78	GPT-5 Nano· OpenAI	38.1	$0.05
79	GPT-4.1 Mini· OpenAI	37.6	$0.40
80	Qwen3 235B A22B· Alibaba Qwen	37.3	$0.46
81	Grok 3· xAI	37.2	n/a
82	MiniMax M2.7· minimax	37.0	$0.30
83	R1· DeepSeek	36.5	$0.70
84	o1-mini· OpenAI	36.3	n/a
85	DeepSeek V3· DeepSeek	36.1	$0.32
86	Gemini 2.5 Flash Lite· Google DeepMind	35.2	$0.10
87	Gemma 4 26B A4B · Google DeepMind	35.2	$0.07
88	Qwen3.6 35B A3B· Alibaba Qwen	34.5	$0.15
89	Claude 3.5 Sonnet· Anthropic	31.0	n/a
90	Claude 3.5 Haiku· Anthropic	30.7	n/a
91	Qwen3 30B A3B· Alibaba Qwen	29.8	$0.12
92	Gemini 2.0 Flash· Google DeepMind	25.8	$0.10
93	Gemini 2.0 Flash (Dec 2024)· Google DeepMind	25.8	n/a
94	GPT-4o (2024-11-20)· OpenAI	25.1	$2.50
95	Gemini 1.5 Flash (May 2024)· Google DeepMind	24.9	n/a
96	Llama 4 Maverick· Meta	24.5	$0.19
97	Grok-2 (Dec 2024)· xAI	22.2	n/a
98	Gemini 1.5 Pro (Feb 2024)· Google DeepMind	22.2	n/a
99	Gemini 1.5 Pro (May 2024)· Google DeepMind	22.2	n/a
100	Llama 3.1 405B· Meta	21.4	n/a
101	Claude 3 Opus· Anthropic	19.2	n/a
102	GPT-4.1 Nano· OpenAI	19.0	$0.10
103	Qwen2.5 72B Instruct· Alibaba Qwen	16.0	$0.36
104	Llama 3.3 70B Instruct· Meta	14.4	$0.10
105	Llama 3.3 70B Instruct (free)· Meta	14.4	$0.00
106	GPT-4 Turbo· OpenAI	12.4	$10.00
107	GPT-4o-mini· OpenAI	11.8	$0.15
108	GPT-4o-mini (2024-07-18)· OpenAI	11.8	$0.15
109	Qwen2-72B· Alibaba Qwen	11.3	n/a
110	Claude 3 Sonnet· Anthropic	10.2	n/a
111	Claude 3 Haiku· Anthropic	9.8	$0.25
112	Llama 3.1 70B Instruct· Meta	9.0	$0.40
113	Claude 2.1· Anthropic	7.1	n/a
114	GPT-3.5 Turbo (older v0613)· OpenAI	3.5	$1.00
115	Mixtral 8x22B Instruct· Mistral AI	3.2	$2.00
116	Llama 3.1 8B Instruct· Meta	1.7	$0.05