MRI-001

Methods & Evidence

A plain public evidence page for the frozen MRI-001 frame audit: ranking data, profile rules, component evidence, sensitivity checks, pilot files, and provenance.

Evidence Downloads

The live evidence folder is restored from the existing backup package. These links point to the underlying files used on this page.

Public Ranking Table

This table joins the full 109-row MRI Profile A ranking from profile_a_sensitivity_results.csv with the full 109-entry published BFCL ranking already present in content/mri.js.

Primary result: Profile A selects Qwen3-14B (Prompt), published BFCL rank 47, as MRI rank 1. Claude-Opus-4-5-20251101 (FC), the published BFCL rank 1 model, is MRI rank 28.

Show joined MRI Profile A and BFCL ranking table, 109 rows

BFCL rank	MRI rank	Model / variant	BFCL overall	MRI score	Eligibility	Disqualification reason
47	1	Qwen3-14B (Prompt)	37.77	90.93	Eligible
23	2	Qwen3-235B-A22B-Instruct-2507 (Prompt)	52.15	90.76	Eligible
53	3	Qwen3-30B-A3B-Instruct-2507 (Prompt)	36.70	90.28	Eligible
13	4	Command A Reasoning (FC)	57.06	88.02	Eligible
12	5	Grok-4-1-fast-non-reasoning (FC)	58.29	87.11	Eligible
15	6	Gemini-2.5-Flash (FC)	56.24	87.08	Eligible
2	7	Claude-Sonnet-4-5-20250929 (FC)	73.24	86.74	Eligible
67	8	GPT-4.1-mini-2025-04-14 (Prompt)	29.73	86.62	Eligible
51	9	Mistral-small-2506 (FC)	37.15	86.55	Eligible
6	10	Claude-Haiku-4-5-20251001 (FC)	68.70	86.46	Eligible
26	11	Gemini-2.5-Flash (Prompt)	50.90	86.20	Eligible
56	12	Arch-Agent-3B	35.36	85.98	Eligible
4	13	GLM-4.6 (FC thinking)	72.38	85.52	Eligible
88	14	Amazon-Nova-Pro-v1:0 (FC)	24.97	85.18	Eligible
43	15	Qwen3-14B (FC)	41.03	85.15	Eligible
41	16	Qwen3-30B-A3B-Instruct-2507 (FC)	41.39	84.84	Eligible
52	17	Gemini-2.5-Flash-Lite (FC)	36.87	84.26	Eligible
11	18	Moonshotai-Kimi-K2-Instruct (FC)	59.06	83.65	Eligible
93	19	Granite-20b-FunctionCalling (FC)	23.23	83.59	Eligible
61	20	Command R7B (FC)	32.07	83.55	Eligible
95	21	Amazon-Nova-Micro-v1:0 (FC)	22.29	83.51	Eligible
27	22	GPT-4.1-mini-2025-04-14 (FC)	50.45	83.40	Eligible
48	23	Mistral-Medium-2505	37.69	83.12	Eligible
60	24	Arch-Agent-1.5B	32.14	82.72	Eligible
71	25	Qwen3-1.7B (FC)	28.41	82.68	Eligible
73	26	Gemini-2.5-Flash-Lite (Prompt)	28.03	82.67	Eligible
35	27	Command A (FC)	46.49	82.11	Eligible
1	28	Claude-Opus-4-5-20251101 (FC)	77.47	81.70	Eligible
90	29	GPT-4.1-nano-2025-04-14 (Prompt)	24.88	81.56	Eligible
57	30	Claude-Opus-4-5-20251101 (Prompt)	33.47	81.54	Eligible
5	31	Grok-4-1-fast-reasoning (FC)	69.57	81.20	Eligible
49	32	Mistral-Medium-2505 (FC)	37.56	81.00	Eligible
45	33	GPT-4.1-2025-04-14 (Prompt)	39.38	80.35	Eligible
18	34	xLAM-2-32b-fc-r (FC)	54.66	80.33	Eligible
92	35	Qwen3-0.6B (FC)	23.93	79.87	Eligible
100	36	Hammer2.1-0.5b (FC)	21.22	78.42	Eligible
94	37	Qwen3-0.6B (Prompt)	22.38	77.93	Eligible
20	38	GPT-4.1-2025-04-14 (FC)	53.96	77.87	Eligible
16	39	GPT-5.2-2025-12-11 (FC)	55.87	77.25	Eligible
31	40	Qwen3-235B-A22B-Instruct-2507 (FC)	47.99	77.18	Eligible
28	41	o4-mini-2025-04-16 (Prompt)	50.26	74.85	Eligible
76	42	palmyra-x-004 (FC)	27.87	72.19	Eligible
17	43	GPT-5-mini-2025-08-07 (FC)	55.46	70.91	Eligible
89	44	Claude-Sonnet-4-5-20250929 (Prompt)	24.90	69.34	Eligible
38	45	GPT-5.2-2025-12-11 (Prompt)	45.27	69.19	Eligible
87	46	Claude-Haiku-4-5-20251001 (Prompt)	25.26	67.08	Eligible
19	47	DeepSeek-V3.2-Exp (FC)	54.12	66.43	Eligible
8	48	o3-2025-04-16 (Prompt)	63.05	66.00	Eligible
21	49	o4-mini-2025-04-16 (FC)	53.24	54.98	Eligible
106	50	Falcon3-1B-Instruct (FC)	11.08	47.86	Eligible
30	51	o3-2025-04-16 (FC)	48.56	46.72	Eligible
108	52	Llama-3.1-Nemotron-Ultra-253B-v1 (FC)	10.00	33.33	Eligible
3	Not eligible	Gemini-3-Pro-Preview (Prompt)	72.51	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
7	Not eligible	Gemini-3-Pro-Preview (FC)	68.14	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
9	Not eligible	Grok-4-0709 (Prompt)	62.97	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
10	Not eligible	Grok-4-0709 (FC)	61.38	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
14	Not eligible	DeepSeek-V3.2-Exp (Prompt + Thinking)	56.73	Not scored	Disqualified	Irrelevance Detection below 70%; Latency 95th Percentile above 20 seconds
22	Not eligible	xLAM-2-70b-fc-r (FC)	53.07	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
24	Not eligible	GPT-5-nano-2025-08-07 (FC)	51.45	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
25	Not eligible	Nanbeige4-3B-Thinking-2511 (FC)	51.40	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
29	Not eligible	Qwen3-32B (FC)	48.71	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
32	Not eligible	Nanbeige3.5-Pro-Thinking (FC)	47.68	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
33	Not eligible	Qwen3-32B (Prompt)	46.78	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
34	Not eligible	xLAM-2-8b-fc-r (FC)	46.68	Not scored	Disqualified	Irrelevance Detection below 70%; Latency 95th Percentile above 20 seconds
36	Not eligible	BitAgent-Bounty-8B	46.23	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
37	Not eligible	Arch-Agent-32B	45.37	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
39	Not eligible	Qwen3-8B (FC)	42.57	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
40	Not eligible	ToolACE-2-8B (FC)	42.44	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
42	Not eligible	xLAM-2-3b-fc-r (FC)	41.22	Not scored	Disqualified	Irrelevance Detection below 70%
44	Not eligible	Qwen3-8B (Prompt)	40.43	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
46	Not eligible	mistral-large-2411 (FC)	38.37	Not scored	Disqualified	Irrelevance Detection below 70%
50	Not eligible	Llama-4-Maverick-17B-128E-Instruct-FP8 (FC)	37.29	Not scored	Disqualified	Irrelevance Detection below 70%; Latency 95th Percentile above 20 seconds
54	Not eligible	Qwen3-4B-Instruct-2507 (FC)	35.68	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
55	Not eligible	Qwen3-4B-Instruct-2507 (Prompt)	35.52	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
58	Not eligible	GPT-4.1-nano-2025-04-14 (FC)	33.05	Not scored	Disqualified	Irrelevance Detection below 70%
59	Not eligible	Mistral-Small-2506 (Prompt)	32.38	Not scored	Disqualified	Irrelevance Detection below 70%
62	Not eligible	Llama-3.3-70B-Instruct (FC)	31.90	Not scored	Disqualified	Irrelevance Detection below 70%; Latency 95th Percentile above 20 seconds
63	Not eligible	mistral-large-2411 (Prompt)	31.84	Not scored	Disqualified	Irrelevance Detection below 70%
64	Not eligible	Hammer2.1-7b (FC)	31.67	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
65	Not eligible	xLAM-2-1b-fc-r (FC)	30.44	Not scored	Disqualified	Irrelevance Detection below 70%
66	Not eligible	Gemma-3-12b-it (Prompt)	30.43	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
68	Not eligible	Hammer2.1-3b (FC)	29.71	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
69	Not eligible	Gemma-3-27b-it (Prompt)	29.47	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
70	Not eligible	Phi-4 (Prompt)	28.79	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
72	Not eligible	Llama-4-Scout-17B-16E-Instruct (FC)	28.13	Not scored	Disqualified	Irrelevance Detection below 70%; Latency 95th Percentile above 20 seconds
74	Not eligible	CoALM-70B	27.99	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
75	Not eligible	Hammer2.1-1.5b (FC)	27.88	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
77	Not eligible	GPT-5-mini-2025-08-07 (Prompt)	27.83	Not scored	Disqualified	Irrelevance Detection below 70%
78	Not eligible	Open-Mistral-Nemo-2407 (FC)	27.63	Not scored	Disqualified	Irrelevance Detection below 70%
79	Not eligible	GPT-5-nano-2025-08-07 (Prompt)	27.55	Not scored	Disqualified	Irrelevance Detection below 70%; Latency 95th Percentile above 20 seconds
80	Not eligible	Amazon-Nova-2-Lite-v1:0 (FC)	27.10	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
81	Not eligible	Granite-3.1-8B-Instruct (FC)	27.10	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
82	Not eligible	Falcon3-10B-Instruct (FC)	27.01	Not scored	Disqualified	Irrelevance Detection below 70%; Latency 95th Percentile above 20 seconds
83	Not eligible	Granite-3.2-8B-Instruct (FC)	26.87	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
84	Not eligible	CoALM-8B	26.81	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
85	Not eligible	Llama-3.1-8B-Instruct (Prompt)	25.83	Not scored	Disqualified	Irrelevance Detection below 70%; Latency 95th Percentile above 20 seconds
86	Not eligible	MiniCPM3-4B-FC (FC)	25.55	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
91	Not eligible	Falcon3-7B-Instruct (FC)	24.03	Not scored	Disqualified	Irrelevance Detection below 70%; Latency 95th Percentile above 20 seconds
96	Not eligible	RZN-T (Prompt)	22.25	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
97	Not eligible	MiniCPM3-4B (Prompt)	22.08	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
98	Not eligible	Llama-3.2-3B-Instruct (FC)	21.95	Not scored	Disqualified	Irrelevance Detection below 70%
99	Not eligible	Bielik-11B-v2.3-Instruct (Prompt)	21.90	Not scored	Disqualified	Irrelevance Detection below 70%; Latency 95th Percentile above 20 seconds
101	Not eligible	Gemma-3-4b-it (Prompt)	19.62	Not scored	Disqualified	Irrelevance Detection below 70%
102	Not eligible	Open-Mistral-Nemo-2407 (Prompt)	19.31	Not scored	Disqualified	Irrelevance Detection below 70%
103	Not eligible	Granite-4.0-350m (FC)	18.98	Not scored	Disqualified	Irrelevance Detection below 70%
104	Not eligible	Falcon3-3B-Instruct (FC)	16.25	Not scored	Disqualified	Irrelevance Detection below 70%; Latency 95th Percentile above 20 seconds
105	Not eligible	Ministral-8B-Instruct-2410 (FC)	11.10	Not scored	Disqualified	Latency 95th Percentile above 20 seconds
107	Not eligible	Llama-3.2-1B-Instruct (FC)	10.82	Not scored	Disqualified	Irrelevance Detection below 70%
109	Not eligible	Gemma-3-1b-it (Prompt)	7.17	Not scored	Disqualified	Irrelevance Detection below 70%

Show published BFCL ranking from content/mri.js, 109 entries

BFCL rank	Model / variant	Published BFCL overall
1	Claude-Opus-4-5-20251101 (FC)	77.47
2	Claude-Sonnet-4-5-20250929 (FC)	73.24
3	Gemini-3-Pro-Preview (Prompt)	72.51
4	GLM-4.6 (FC thinking)	72.38
5	Grok-4-1-fast-reasoning (FC)	69.57
6	Claude-Haiku-4-5-20251001 (FC)	68.70
7	Gemini-3-Pro-Preview (FC)	68.14
8	o3-2025-04-16 (Prompt)	63.05
9	Grok-4-0709 (Prompt)	62.97
10	Grok-4-0709 (FC)	61.38
11	Moonshotai-Kimi-K2-Instruct (FC)	59.06
12	Grok-4-1-fast-non-reasoning (FC)	58.29
13	Command A Reasoning (FC)	57.06
14	DeepSeek-V3.2-Exp (Prompt + Thinking)	56.73
15	Gemini-2.5-Flash (FC)	56.24
16	GPT-5.2-2025-12-11 (FC)	55.87
17	GPT-5-mini-2025-08-07 (FC)	55.46
18	xLAM-2-32b-fc-r (FC)	54.66
19	DeepSeek-V3.2-Exp (FC)	54.12
20	GPT-4.1-2025-04-14 (FC)	53.96
21	o4-mini-2025-04-16 (FC)	53.24
22	xLAM-2-70b-fc-r (FC)	53.07
23	Qwen3-235B-A22B-Instruct-2507 (Prompt)	52.15
24	GPT-5-nano-2025-08-07 (FC)	51.45
25	Nanbeige4-3B-Thinking-2511 (FC)	51.40
26	Gemini-2.5-Flash (Prompt)	50.90
27	GPT-4.1-mini-2025-04-14 (FC)	50.45
28	o4-mini-2025-04-16 (Prompt)	50.26
29	Qwen3-32B (FC)	48.71
30	o3-2025-04-16 (FC)	48.56
31	Qwen3-235B-A22B-Instruct-2507 (FC)	47.99
32	Nanbeige3.5-Pro-Thinking (FC)	47.68
33	Qwen3-32B (Prompt)	46.78
34	xLAM-2-8b-fc-r (FC)	46.68
35	Command A (FC)	46.49
36	BitAgent-Bounty-8B	46.23
37	Arch-Agent-32B	45.37
38	GPT-5.2-2025-12-11 (Prompt)	45.27
39	Qwen3-8B (FC)	42.57
40	ToolACE-2-8B (FC)	42.44
41	Qwen3-30B-A3B-Instruct-2507 (FC)	41.39
42	xLAM-2-3b-fc-r (FC)	41.22
43	Qwen3-14B (FC)	41.03
44	Qwen3-8B (Prompt)	40.43
45	GPT-4.1-2025-04-14 (Prompt)	39.38
46	mistral-large-2411 (FC)	38.37
47	Qwen3-14B (Prompt)	37.77
48	Mistral-Medium-2505	37.69
49	Mistral-Medium-2505 (FC)	37.56
50	Llama-4-Maverick-17B-128E-Instruct-FP8 (FC)	37.29
51	Mistral-small-2506 (FC)	37.15
52	Gemini-2.5-Flash-Lite (FC)	36.87
53	Qwen3-30B-A3B-Instruct-2507 (Prompt)	36.70
54	Qwen3-4B-Instruct-2507 (FC)	35.68
55	Qwen3-4B-Instruct-2507 (Prompt)	35.52
56	Arch-Agent-3B	35.36
57	Claude-Opus-4-5-20251101 (Prompt)	33.47
58	GPT-4.1-nano-2025-04-14 (FC)	33.05
59	Mistral-Small-2506 (Prompt)	32.38
60	Arch-Agent-1.5B	32.14
61	Command R7B (FC)	32.07
62	Llama-3.3-70B-Instruct (FC)	31.90
63	mistral-large-2411 (Prompt)	31.84
64	Hammer2.1-7b (FC)	31.67
65	xLAM-2-1b-fc-r (FC)	30.44
66	Gemma-3-12b-it (Prompt)	30.43
67	GPT-4.1-mini-2025-04-14 (Prompt)	29.73
68	Hammer2.1-3b (FC)	29.71
69	Gemma-3-27b-it (Prompt)	29.47
70	Phi-4 (Prompt)	28.79
71	Qwen3-1.7B (FC)	28.41
72	Llama-4-Scout-17B-16E-Instruct (FC)	28.13
73	Gemini-2.5-Flash-Lite (Prompt)	28.03
74	CoALM-70B	27.99
75	Hammer2.1-1.5b (FC)	27.88
76	palmyra-x-004 (FC)	27.87
77	GPT-5-mini-2025-08-07 (Prompt)	27.83
78	Open-Mistral-Nemo-2407 (FC)	27.63
79	GPT-5-nano-2025-08-07 (Prompt)	27.55
80	Amazon-Nova-2-Lite-v1:0 (FC)	27.10
81	Granite-3.1-8B-Instruct (FC)	27.10
82	Falcon3-10B-Instruct (FC)	27.01
83	Granite-3.2-8B-Instruct (FC)	26.87
84	CoALM-8B	26.81
85	Llama-3.1-8B-Instruct (Prompt)	25.83
86	MiniCPM3-4B-FC (FC)	25.55
87	Claude-Haiku-4-5-20251001 (Prompt)	25.26
88	Amazon-Nova-Pro-v1:0 (FC)	24.97
89	Claude-Sonnet-4-5-20250929 (Prompt)	24.90
90	GPT-4.1-nano-2025-04-14 (Prompt)	24.88
91	Falcon3-7B-Instruct (FC)	24.03
92	Qwen3-0.6B (FC)	23.93
93	Granite-20b-FunctionCalling (FC)	23.23
94	Qwen3-0.6B (Prompt)	22.38
95	Amazon-Nova-Micro-v1:0 (FC)	22.29
96	RZN-T (Prompt)	22.25
97	MiniCPM3-4B (Prompt)	22.08
98	Llama-3.2-3B-Instruct (FC)	21.95
99	Bielik-11B-v2.3-Instruct (Prompt)	21.90
100	Hammer2.1-0.5b (FC)	21.22
101	Gemma-3-4b-it (Prompt)	19.62
102	Open-Mistral-Nemo-2407 (Prompt)	19.31
103	Granite-4.0-350m (FC)	18.98
104	Falcon3-3B-Instruct (FC)	16.25
105	Ministral-8B-Instruct-2410 (FC)	11.10
106	Falcon3-1B-Instruct (FC)	11.08
107	Llama-3.2-1B-Instruct (FC)	10.82
108	Llama-3.1-Nemotron-Ultra-253B-v1 (FC)	10.00
109	Gemma-3-1b-it (Prompt)	7.17

Component-Score Evidence

The Profile A relevant-results file provides the score terms for the selected model, the published BFCL winner, and nearby comparators. The relevant baseline file preserves the BFCL component fields used to reconstruct the published aggregate for the key rows.

Show Profile A relevant component-score rows

MRI rank	BFCL rank	Model / variant	MRI score	Eligible	Single-turn	Relevance / irrelevance	Cost per expected success	P95 latency	Score terms
1	47	Qwen3-14B (Prompt)	90.93	Eligible	85.36	84.22	0.00031100	2.30	single_turn_tool_call_accuracy:40; relevance_irrelevance_behavior:20; cost_efficiency:25; latency:15
2	23	Qwen3-235B-A22B-Instruct-2507 (Prompt)	90.76	Eligible	84.27	86.32	0.00072800	7.61	single_turn_tool_call_accuracy:40; relevance_irrelevance_behavior:20; cost_efficiency:25; latency:15
3	53	Qwen3-30B-A3B-Instruct-2507 (Prompt)	90.28	Eligible	83.74	84.30	0.00036600	2.84	single_turn_tool_call_accuracy:40; relevance_irrelevance_behavior:20; cost_efficiency:25; latency:15
4	13	Command A Reasoning (FC)	88.02	Eligible	81.71	77.75	0.00073100	8.39	single_turn_tool_call_accuracy:40; relevance_irrelevance_behavior:20; cost_efficiency:25; latency:15
5	12	Grok-4-1-fast-non-reasoning (FC)	87.11	Eligible	82.28	77.67	0.00388700	5.34	single_turn_tool_call_accuracy:40; relevance_irrelevance_behavior:20; cost_efficiency:25; latency:15
9	51	Mistral-small-2506 (FC)	86.55	Eligible	73.63	87.72	0.00138800	2.50	single_turn_tool_call_accuracy:40; relevance_irrelevance_behavior:20; cost_efficiency:25; latency:15
28	1	Claude-Opus-4-5-20251101 (FC)	81.70	Eligible	85.18	73.61	0.01997100	7.56	single_turn_tool_call_accuracy:40; relevance_irrelevance_behavior:20; cost_efficiency:25; latency:15
33	45	GPT-4.1-2025-04-14 (Prompt)	80.35	Eligible	84.94	92.00	0.03374700	2.53	single_turn_tool_call_accuracy:40; relevance_irrelevance_behavior:20; cost_efficiency:25; latency:15

Show BFCL relevant baseline component rows

BFCL rank	Model / variant	Published overall	Reconstructed overall	Agentic	Web search	Memory	Multi-turn	Live	Non-live AST	Relevance	Irrelevance	Cost	P95 latency
1	Claude-Opus-4-5-20251101 (FC)	77.47	77.48	79.13	84.50	73.76	68.38	79.79	88.58	62.50	84.72	86.55	7.56
4	GLM-4.6 (FC thinking)	72.38	72.38	66.60	77.50	55.70	68.00	80.90	87.56	75.00	84.96	4.64	13.50
23	Qwen3-235B-A22B-Instruct-2507 (Prompt)	52.15	52.15	34.93	50.50	19.35	44.62	78.68	90.33	93.75	78.89	3.12	7.61
45	GPT-4.1-2025-04-14 (Prompt)	39.38	39.39	28.26	35.00	21.51	9.75	78.90	88.69	100.00	83.99	145.85	2.53
47	Qwen3-14B (Prompt)	37.77	37.77	10.84	10.50	11.18	26.13	79.35	89.46	81.25	87.18	1.35	2.30

MRI Work-Profile Definition

Profile A is the high-volume, low-risk automation profile. The source definition is preserved in BFCL_DECISION_PROFILES.md and the pass conditions are preserved in BFCL_PASS_FAIL_RULES.md.

Profile A weights

Criterion	Weight
Single-turn tool-call accuracy	40%
Relevance / irrelevance behavior	20%
Cost efficiency	25%
Latency	15%

Eligibility rules

Required single-turn categories cannot be N/A.
Irrelevance Detection below 70% disqualifies the model.
Latency 95th Percentile (s) above 20 seconds disqualifies the model.
Missing Total Cost ($) disqualifies the model.

Calculation and normalization method

BFCL overall reconstruction: 0.40 * Agentic + 0.30 * Multi Turn Acc + 0.10 * Live Acc + 0.10 * Non-Live AST Acc + 0.10 * Irrelevance Detection.
Agentic = (Web Search Acc + Memory Acc) / 2.
Profile A single-turn score is the average of the eight live and non-live AST subcategories.
Profile A relevance behavior is the average of relevance detection and hallucination/irrelevance detection.
Cost per expected successful call is benchmark cost divided by expected successful scoring tasks, then inverse-normalized among eligible models.
Latency score uses 100 / max(1, p95 / target); Profile A target is 10 seconds, with a 20-second hard disqualification threshold.
Used score terms are weighted and renormalized over the included terms.

Pass conditions

PASS requires at least one defensible buyer profile to select a different model than the published aggregate winner, driven by a material operational constraint rather than arbitrary weighting. FAIL applies when reasonable profiles do not change the decision or merely reproduce information already obvious from the BFCL leaderboard.

Reconstruction status from BFCL_BASELINE_RECONSTRUCTION_REPORT.md: 109 models included, published aggregate formula reproduced to official two-decimal precision, and all rows matched within 0.01 percentage points.

Sensitivity Results

The materiality audit reports that the Profile A flip survives without cost, without latency, and under capability-only weighting. The full audit file is BFCL_CALCULATION_AND_MATERIALITY_AUDIT.md.

Check	Reported winner	Published BFCL rank	Selected score	Published-winner score	Flip survives
Full registered profile	Qwen3-14B (Prompt)	47	90.93	81.70	Yes
Cost removed	GPT-4.1-2025-04-14 (Prompt)	45	89.83	85.06	Yes
Latency removed	Qwen3-14B (Prompt)	47	89.33	78.47	Yes
Capability only	GPT-4.1-2025-04-14 (Prompt)	45	87.29	81.32	Yes

Center-Pull Pilot Evidence

The pilot tested whether showing Berkeley's rank-1 model before work-specific evidence changed later model selection. The preregistration, raw responses, metrics, technical report, and plain-language result are all linked below.

Pre-registered verdict: PILOT FAIL. Profile A center pull was 0.00 percentage points. Parsing or compliance errors were 139 of 288 decision records.

Show center-pull MRI metrics

Scenario	Judge family	Evidence-first valid n	Center-first valid n	MRI-order valid n	Center pull pp	Evidence-first task-fit accuracy	Center-first task-fit accuracy	MRI-order task-fit accuracy	Evidence-first mean regret	Center-first mean regret	MRI-order mean regret
Profile A	ALL	33	25	21	0.00	1.000	1.000	1.000	0.000	0.000	0.000
Profile A	OpenAI	12	12	12	0.00	1.000	1.000	1.000	0.000	0.000	0.000
Profile A	Anthropic	9	1	0	0.00	1.000	1.000		0.000	0.000
Profile A	Google	0	0	0
Profile A	xAI	12	12	9	0.00	1.000	1.000	1.000	0.000	0.000	0.000
Profile C	ALL	24	24	22	4.17	0.958	1.000	1.000	0.159	0.000	0.000
Profile C	OpenAI	12	12	12	0.00	1.000	1.000	1.000	0.000	0.000	0.000
Profile C	Anthropic	0	0	0
Profile C	Google	0	0	0
Profile C	xAI	12	12	10	8.33	0.917	1.000	1.000	0.319	0.000	0.000

Show 288 row-level pilot results

Trial	Judge family	Judge model	Scenario	Condition	Replicate	Provisional choice	Final choice	Confidence	Changed after rank disclosure	Selected BFCL winner	Selected scenario preferred	Regret	Errors
T0001	OpenAI	openai/gpt-5.6-sol	Profile A	EVIDENCE_FIRST	1		Qwen3-14B (Prompt)	99		False	True	0.0
T0002	OpenAI	openai/gpt-5.6-sol	Profile A	EVIDENCE_FIRST	2		Qwen3-14B (Prompt)	100		False	True	0.0
T0003	OpenAI	openai/gpt-5.6-sol	Profile A	EVIDENCE_FIRST	3		Qwen3-14B (Prompt)	100		False	True	0.0
T0004	OpenAI	openai/gpt-5.6-sol	Profile A	EVIDENCE_FIRST	4		Qwen3-14B (Prompt)	99		False	True	0.0
T0005	OpenAI	openai/gpt-5.6-sol	Profile A	EVIDENCE_FIRST	5		Qwen3-14B (Prompt)	99		False	True	0.0
T0006	OpenAI	openai/gpt-5.6-sol	Profile A	EVIDENCE_FIRST	6		Qwen3-14B (Prompt)	100		False	True	0.0
T0007	OpenAI	openai/gpt-5.6-sol	Profile A	EVIDENCE_FIRST	7		Qwen3-14B (Prompt)	100		False	True	0.0
T0008	OpenAI	openai/gpt-5.6-sol	Profile A	EVIDENCE_FIRST	8		Qwen3-14B (Prompt)	99		False	True	0.0
T0009	OpenAI	openai/gpt-5.6-sol	Profile A	EVIDENCE_FIRST	9		Qwen3-14B (Prompt)	100		False	True	0.0
T0010	OpenAI	openai/gpt-5.6-sol	Profile A	EVIDENCE_FIRST	10		Qwen3-14B (Prompt)	100		False	True	0.0
T0011	OpenAI	openai/gpt-5.6-sol	Profile A	EVIDENCE_FIRST	11		Qwen3-14B (Prompt)	99		False	True	0.0
T0012	OpenAI	openai/gpt-5.6-sol	Profile A	EVIDENCE_FIRST	12		Qwen3-14B (Prompt)	99		False	True	0.0
T0013	OpenAI	openai/gpt-5.6-sol	Profile A	CENTER_FIRST	1		Qwen3-14B (Prompt)	99		False	True	0.0
T0014	OpenAI	openai/gpt-5.6-sol	Profile A	CENTER_FIRST	2		Qwen3-14B (Prompt)	99		False	True	0.0
T0015	OpenAI	openai/gpt-5.6-sol	Profile A	CENTER_FIRST	3		Qwen3-14B (Prompt)	99		False	True	0.0
T0016	OpenAI	openai/gpt-5.6-sol	Profile A	CENTER_FIRST	4		Qwen3-14B (Prompt)	99		False	True	0.0
T0017	OpenAI	openai/gpt-5.6-sol	Profile A	CENTER_FIRST	5		Qwen3-14B (Prompt)	99		False	True	0.0
T0018	OpenAI	openai/gpt-5.6-sol	Profile A	CENTER_FIRST	6		Qwen3-14B (Prompt)	99		False	True	0.0
T0019	OpenAI	openai/gpt-5.6-sol	Profile A	CENTER_FIRST	7		Qwen3-14B (Prompt)	99		False	True	0.0
T0020	OpenAI	openai/gpt-5.6-sol	Profile A	CENTER_FIRST	8		Qwen3-14B (Prompt)	99		False	True	0.0
T0021	OpenAI	openai/gpt-5.6-sol	Profile A	CENTER_FIRST	9		Qwen3-14B (Prompt)	99		False	True	0.0
T0022	OpenAI	openai/gpt-5.6-sol	Profile A	CENTER_FIRST	10		Qwen3-14B (Prompt)	98		False	True	0.0
T0023	OpenAI	openai/gpt-5.6-sol	Profile A	CENTER_FIRST	11		Qwen3-14B (Prompt)	99		False	True	0.0
T0024	OpenAI	openai/gpt-5.6-sol	Profile A	CENTER_FIRST	12		Qwen3-14B (Prompt)	98		False	True	0.0
T0025	OpenAI	openai/gpt-5.6-sol	Profile A	MRI_ORDER	1	Qwen3-14B (Prompt)	Qwen3-14B (Prompt)	98	False	False	True	0.0
T0026	OpenAI	openai/gpt-5.6-sol	Profile A	MRI_ORDER	2	Qwen3-14B (Prompt)	Qwen3-14B (Prompt)	97	False	False	True	0.0
T0027	OpenAI	openai/gpt-5.6-sol	Profile A	MRI_ORDER	3	Qwen3-14B (Prompt)	Qwen3-14B (Prompt)	97	False	False	True	0.0
T0028	OpenAI	openai/gpt-5.6-sol	Profile A	MRI_ORDER	4	Qwen3-14B (Prompt)	Qwen3-14B (Prompt)	97	False	False	True	0.0
T0029	OpenAI	openai/gpt-5.6-sol	Profile A	MRI_ORDER	5	Qwen3-14B (Prompt)	Qwen3-14B (Prompt)	98	False	False	True	0.0
T0030	OpenAI	openai/gpt-5.6-sol	Profile A	MRI_ORDER	6	Qwen3-14B (Prompt)	Qwen3-14B (Prompt)	97	False	False	True	0.0
T0031	OpenAI	openai/gpt-5.6-sol	Profile A	MRI_ORDER	7	Qwen3-14B (Prompt)	Qwen3-14B (Prompt)	97	False	False	True	0.0
T0032	OpenAI	openai/gpt-5.6-sol	Profile A	MRI_ORDER	8	Qwen3-14B (Prompt)	Qwen3-14B (Prompt)	98	False	False	True	0.0
T0033	OpenAI	openai/gpt-5.6-sol	Profile A	MRI_ORDER	9	Qwen3-14B (Prompt)	Qwen3-14B (Prompt)	98	False	False	True	0.0
T0034	OpenAI	openai/gpt-5.6-sol	Profile A	MRI_ORDER	10	Qwen3-14B (Prompt)	Qwen3-14B (Prompt)	98	False	False	True	0.0
T0035	OpenAI	openai/gpt-5.6-sol	Profile A	MRI_ORDER	11	Qwen3-14B (Prompt)	Qwen3-14B (Prompt)	98	False	False	True	0.0
T0036	OpenAI	openai/gpt-5.6-sol	Profile A	MRI_ORDER	12	Qwen3-14B (Prompt)	Qwen3-14B (Prompt)	98	False	False	True	0.0
T0037	OpenAI	openai/gpt-5.6-sol	Profile C	EVIDENCE_FIRST	1		Claude-Opus-4-5-20251101 (FC)	84		True	True	0.0
T0038	OpenAI	openai/gpt-5.6-sol	Profile C	EVIDENCE_FIRST	2		Claude-Opus-4-5-20251101 (FC)	96		True	True	0.0
T0039	OpenAI	openai/gpt-5.6-sol	Profile C	EVIDENCE_FIRST	3		Claude-Opus-4-5-20251101 (FC)	94		True	True	0.0
T0040	OpenAI	openai/gpt-5.6-sol	Profile C	EVIDENCE_FIRST	4		Claude-Opus-4-5-20251101 (FC)	93		True	True	0.0
T0041	OpenAI	openai/gpt-5.6-sol	Profile C	EVIDENCE_FIRST	5		Claude-Opus-4-5-20251101 (FC)	89		True	True	0.0
T0042	OpenAI	openai/gpt-5.6-sol	Profile C	EVIDENCE_FIRST	6		Claude-Opus-4-5-20251101 (FC)	86		True	True	0.0
T0043	OpenAI	openai/gpt-5.6-sol	Profile C	EVIDENCE_FIRST	7		Claude-Opus-4-5-20251101 (FC)	87		True	True	0.0
T0044	OpenAI	openai/gpt-5.6-sol	Profile C	EVIDENCE_FIRST	8		Claude-Opus-4-5-20251101 (FC)	91		True	True	0.0
T0045	OpenAI	openai/gpt-5.6-sol	Profile C	EVIDENCE_FIRST	9		Claude-Opus-4-5-20251101 (FC)	92		True	True	0.0
T0046	OpenAI	openai/gpt-5.6-sol	Profile C	EVIDENCE_FIRST	10		Claude-Opus-4-5-20251101 (FC)	91		True	True	0.0
T0047	OpenAI	openai/gpt-5.6-sol	Profile C	EVIDENCE_FIRST	11		Claude-Opus-4-5-20251101 (FC)	91		True	True	0.0
T0048	OpenAI	openai/gpt-5.6-sol	Profile C	EVIDENCE_FIRST	12		Claude-Opus-4-5-20251101 (FC)	92		True	True	0.0
T0049	OpenAI	openai/gpt-5.6-sol	Profile C	CENTER_FIRST	1		Claude-Opus-4-5-20251101 (FC)	88		True	True	0.0
T0050	OpenAI	openai/gpt-5.6-sol	Profile C	CENTER_FIRST	2		Claude-Opus-4-5-20251101 (FC)	93		True	True	0.0
T0051	OpenAI	openai/gpt-5.6-sol	Profile C	CENTER_FIRST	3		Claude-Opus-4-5-20251101 (FC)	91		True	True	0.0
T0052	OpenAI	openai/gpt-5.6-sol	Profile C	CENTER_FIRST	4		Claude-Opus-4-5-20251101 (FC)	94		True	True	0.0
T0053	OpenAI	openai/gpt-5.6-sol	Profile C	CENTER_FIRST	5		Claude-Opus-4-5-20251101 (FC)	93		True	True	0.0
T0054	OpenAI	openai/gpt-5.6-sol	Profile C	CENTER_FIRST	6		Claude-Opus-4-5-20251101 (FC)	94		True	True	0.0
T0055	OpenAI	openai/gpt-5.6-sol	Profile C	CENTER_FIRST	7		Claude-Opus-4-5-20251101 (FC)	88		True	True	0.0
T0056	OpenAI	openai/gpt-5.6-sol	Profile C	CENTER_FIRST	8		Claude-Opus-4-5-20251101 (FC)	91		True	True	0.0
T0057	OpenAI	openai/gpt-5.6-sol	Profile C	CENTER_FIRST	9		Claude-Opus-4-5-20251101 (FC)	93		True	True	0.0
T0058	OpenAI	openai/gpt-5.6-sol	Profile C	CENTER_FIRST	10		Claude-Opus-4-5-20251101 (FC)	92		True	True	0.0
T0059	OpenAI	openai/gpt-5.6-sol	Profile C	CENTER_FIRST	11		Claude-Opus-4-5-20251101 (FC)	92		True	True	0.0
T0060	OpenAI	openai/gpt-5.6-sol	Profile C	CENTER_FIRST	12		Claude-Opus-4-5-20251101 (FC)	94		True	True	0.0
T0061	OpenAI	openai/gpt-5.6-sol	Profile C	MRI_ORDER	1	Claude-Opus-4-5-20251101 (FC)	Claude-Opus-4-5-20251101 (FC)	94	False	True	True	0.0
T0062	OpenAI	openai/gpt-5.6-sol	Profile C	MRI_ORDER	2	Claude-Opus-4-5-20251101 (FC)	Claude-Opus-4-5-20251101 (FC)	94	False	True	True	0.0
T0063	OpenAI	openai/gpt-5.6-sol	Profile C	MRI_ORDER	3	Claude-Opus-4-5-20251101 (FC)	Claude-Opus-4-5-20251101 (FC)	95	False	True	True	0.0
T0064	OpenAI	openai/gpt-5.6-sol	Profile C	MRI_ORDER	4	Claude-Opus-4-5-20251101 (FC)	Claude-Opus-4-5-20251101 (FC)	97	False	True	True	0.0
T0065	OpenAI	openai/gpt-5.6-sol	Profile C	MRI_ORDER	5	Claude-Opus-4-5-20251101 (FC)	Claude-Opus-4-5-20251101 (FC)	92	False	True	True	0.0
T0066	OpenAI	openai/gpt-5.6-sol	Profile C	MRI_ORDER	6	Claude-Opus-4-5-20251101 (FC)	Claude-Opus-4-5-20251101 (FC)	93	False	True	True	0.0
T0067	OpenAI	openai/gpt-5.6-sol	Profile C	MRI_ORDER	7	Claude-Opus-4-5-20251101 (FC)	Claude-Opus-4-5-20251101 (FC)	94	False	True	True	0.0
T0068	OpenAI	openai/gpt-5.6-sol	Profile C	MRI_ORDER	8	Claude-Opus-4-5-20251101 (FC)	Claude-Opus-4-5-20251101 (FC)	96	False	True	True	0.0
T0069	OpenAI	openai/gpt-5.6-sol	Profile C	MRI_ORDER	9	Claude-Opus-4-5-20251101 (FC)	Claude-Opus-4-5-20251101 (FC)	94	False	True	True	0.0
T0070	OpenAI	openai/gpt-5.6-sol	Profile C	MRI_ORDER	10	Claude-Opus-4-5-20251101 (FC)	Claude-Opus-4-5-20251101 (FC)	95	False	True	True	0.0
T0071	OpenAI	openai/gpt-5.6-sol	Profile C	MRI_ORDER	11	Claude-Opus-4-5-20251101 (FC)	Claude-Opus-4-5-20251101 (FC)	93	False	True	True	0.0
T0072	OpenAI	openai/gpt-5.6-sol	Profile C	MRI_ORDER	12	Claude-Opus-4-5-20251101 (FC)	Claude-Opus-4-5-20251101 (FC)	94	False	True	True	0.0
T0073	Anthropic	anthropic/claude-opus-5	Profile A	EVIDENCE_FIRST	1					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1110 (char 1109)
T0074	Anthropic	anthropic/claude-opus-5	Profile A	EVIDENCE_FIRST	2					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1145 (char 1144)
T0075	Anthropic	anthropic/claude-opus-5	Profile A	EVIDENCE_FIRST	3		Qwen3-14B (Prompt)	97		False	True	0.0
T0076	Anthropic	anthropic/claude-opus-5	Profile A	EVIDENCE_FIRST	4		Qwen3-14B (Prompt)	97		False	True	0.0
T0077	Anthropic	anthropic/claude-opus-5	Profile A	EVIDENCE_FIRST	5		Qwen3-14B (Prompt)	97		False	True	0.0
T0078	Anthropic	anthropic/claude-opus-5	Profile A	EVIDENCE_FIRST	6		Qwen3-14B (Prompt)	97		False	True	0.0
T0079	Anthropic	anthropic/claude-opus-5	Profile A	EVIDENCE_FIRST	7					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1121 (char 1120)
T0080	Anthropic	anthropic/claude-opus-5	Profile A	EVIDENCE_FIRST	8		Qwen3-14B (Prompt)	97		False	True	0.0
T0081	Anthropic	anthropic/claude-opus-5	Profile A	EVIDENCE_FIRST	9		Qwen3-14B (Prompt)	97		False	True	0.0
T0082	Anthropic	anthropic/claude-opus-5	Profile A	EVIDENCE_FIRST	10		Qwen3-14B (Prompt)	97		False	True	0.0
T0083	Anthropic	anthropic/claude-opus-5	Profile A	EVIDENCE_FIRST	11		Qwen3-14B (Prompt)	97		False	True	0.0
T0084	Anthropic	anthropic/claude-opus-5	Profile A	EVIDENCE_FIRST	12		Qwen3-14B (Prompt)	97		False	True	0.0
T0085	Anthropic	anthropic/claude-opus-5	Profile A	CENTER_FIRST	1					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 970 (char 969)
T0086	Anthropic	anthropic/claude-opus-5	Profile A	CENTER_FIRST	2					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 928 (char 927)
T0087	Anthropic	anthropic/claude-opus-5	Profile A	CENTER_FIRST	3					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1096 (char 1095)
T0088	Anthropic	anthropic/claude-opus-5	Profile A	CENTER_FIRST	4					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 888 (char 887)
T0089	Anthropic	anthropic/claude-opus-5	Profile A	CENTER_FIRST	5					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1043 (char 1042)
T0090	Anthropic	anthropic/claude-opus-5	Profile A	CENTER_FIRST	6					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1020 (char 1019)
T0091	Anthropic	anthropic/claude-opus-5	Profile A	CENTER_FIRST	7					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1225 (char 1224)
T0092	Anthropic	anthropic/claude-opus-5	Profile A	CENTER_FIRST	8					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 975 (char 974)
T0093	Anthropic	anthropic/claude-opus-5	Profile A	CENTER_FIRST	9					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 901 (char 900)
T0094	Anthropic	anthropic/claude-opus-5	Profile A	CENTER_FIRST	10					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1017 (char 1016)
T0095	Anthropic	anthropic/claude-opus-5	Profile A	CENTER_FIRST	11		Qwen3-14B (Prompt)	93		False	True	0.0
T0096	Anthropic	anthropic/claude-opus-5	Profile A	CENTER_FIRST	12					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1101 (char 1100)
T0097	Anthropic	anthropic/claude-opus-5	Profile A	MRI_ORDER	1					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1011 (char 1010)
T0098	Anthropic	anthropic/claude-opus-5	Profile A	MRI_ORDER	2					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 869 (char 868)
T0099	Anthropic	anthropic/claude-opus-5	Profile A	MRI_ORDER	3					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1107 (char 1106)
T0100	Anthropic	anthropic/claude-opus-5	Profile A	MRI_ORDER	4					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1071 (char 1070)
T0101	Anthropic	anthropic/claude-opus-5	Profile A	MRI_ORDER	5	Qwen3-14B (Prompt)				False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1077 (char 1076)
T0102	Anthropic	anthropic/claude-opus-5	Profile A	MRI_ORDER	6	Qwen3-14B (Prompt)				False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1098 (char 1097)
T0103	Anthropic	anthropic/claude-opus-5	Profile A	MRI_ORDER	7					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1085 (char 1084)
T0104	Anthropic	anthropic/claude-opus-5	Profile A	MRI_ORDER	8					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1071 (char 1070)
T0105	Anthropic	anthropic/claude-opus-5	Profile A	MRI_ORDER	9	Qwen3-14B (Prompt)				False	False		JSONDecodeError: Unterminated string starting at: line 1 column 890 (char 889)
T0106	Anthropic	anthropic/claude-opus-5	Profile A	MRI_ORDER	10	Qwen3-14B (Prompt)				False	False		JSONDecodeError: Unterminated string starting at: line 1 column 982 (char 981)
T0107	Anthropic	anthropic/claude-opus-5	Profile A	MRI_ORDER	11					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1064 (char 1063)
T0108	Anthropic	anthropic/claude-opus-5	Profile A	MRI_ORDER	12					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1027 (char 1026)
T0109	Anthropic	anthropic/claude-opus-5	Profile C	EVIDENCE_FIRST	1					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1129 (char 1128)
T0110	Anthropic	anthropic/claude-opus-5	Profile C	EVIDENCE_FIRST	2					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 74 (char 73)
T0111	Anthropic	anthropic/claude-opus-5	Profile C	EVIDENCE_FIRST	3					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 74 (char 73)
T0112	Anthropic	anthropic/claude-opus-5	Profile C	EVIDENCE_FIRST	4					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1160 (char 1159)
T0113	Anthropic	anthropic/claude-opus-5	Profile C	EVIDENCE_FIRST	5					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1085 (char 1084)
T0114	Anthropic	anthropic/claude-opus-5	Profile C	EVIDENCE_FIRST	6					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1157 (char 1156)
T0115	Anthropic	anthropic/claude-opus-5	Profile C	EVIDENCE_FIRST	7					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1155 (char 1154)
T0116	Anthropic	anthropic/claude-opus-5	Profile C	EVIDENCE_FIRST	8					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 74 (char 73)
T0117	Anthropic	anthropic/claude-opus-5	Profile C	EVIDENCE_FIRST	9					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 74 (char 73)
T0118	Anthropic	anthropic/claude-opus-5	Profile C	EVIDENCE_FIRST	10					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1123 (char 1122)
T0119	Anthropic	anthropic/claude-opus-5	Profile C	EVIDENCE_FIRST	11					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1080 (char 1079)
T0120	Anthropic	anthropic/claude-opus-5	Profile C	EVIDENCE_FIRST	12					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1067 (char 1066)
T0121	Anthropic	anthropic/claude-opus-5	Profile C	CENTER_FIRST	1					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1141 (char 1140)
T0122	Anthropic	anthropic/claude-opus-5	Profile C	CENTER_FIRST	2					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1174 (char 1173)
T0123	Anthropic	anthropic/claude-opus-5	Profile C	CENTER_FIRST	3					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1232 (char 1231)
T0124	Anthropic	anthropic/claude-opus-5	Profile C	CENTER_FIRST	4					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1017 (char 1016)
T0125	Anthropic	anthropic/claude-opus-5	Profile C	CENTER_FIRST	5					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 899 (char 898)
T0126	Anthropic	anthropic/claude-opus-5	Profile C	CENTER_FIRST	6					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 74 (char 73)
T0127	Anthropic	anthropic/claude-opus-5	Profile C	CENTER_FIRST	7					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 74 (char 73)
T0128	Anthropic	anthropic/claude-opus-5	Profile C	CENTER_FIRST	8					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1184 (char 1183)
T0129	Anthropic	anthropic/claude-opus-5	Profile C	CENTER_FIRST	9					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 74 (char 73)
T0130	Anthropic	anthropic/claude-opus-5	Profile C	CENTER_FIRST	10					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 74 (char 73)
T0131	Anthropic	anthropic/claude-opus-5	Profile C	CENTER_FIRST	11					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 74 (char 73)
T0132	Anthropic	anthropic/claude-opus-5	Profile C	CENTER_FIRST	12					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1131 (char 1130)
T0133	Anthropic	anthropic/claude-opus-5	Profile C	MRI_ORDER	1					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 924 (char 923)
T0134	Anthropic	anthropic/claude-opus-5	Profile C	MRI_ORDER	2					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1198 (char 1197)
T0135	Anthropic	anthropic/claude-opus-5	Profile C	MRI_ORDER	3					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1193 (char 1192)
T0136	Anthropic	anthropic/claude-opus-5	Profile C	MRI_ORDER	4					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1118 (char 1117)
T0137	Anthropic	anthropic/claude-opus-5	Profile C	MRI_ORDER	5					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1146 (char 1145)
T0138	Anthropic	anthropic/claude-opus-5	Profile C	MRI_ORDER	6					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1164 (char 1163)
T0139	Anthropic	anthropic/claude-opus-5	Profile C	MRI_ORDER	7					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1078 (char 1077)
T0140	Anthropic	anthropic/claude-opus-5	Profile C	MRI_ORDER	8					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 80 (char 79)
T0141	Anthropic	anthropic/claude-opus-5	Profile C	MRI_ORDER	9					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 80 (char 79)
T0142	Anthropic	anthropic/claude-opus-5	Profile C	MRI_ORDER	10					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1188 (char 1187)
T0143	Anthropic	anthropic/claude-opus-5	Profile C	MRI_ORDER	11					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1205 (char 1204)
T0144	Anthropic	anthropic/claude-opus-5	Profile C	MRI_ORDER	12					False	False		JSONDecodeError: Unterminated string starting at: line 1 column 1108 (char 1107)
T0145	Google	google/gemini-2.5-pro	Profile A	EVIDENCE_FIRST	1					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 33 (char 34)
T0146	Google	google/gemini-2.5-pro	Profile A	EVIDENCE_FIRST	2					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 25 (char 26)
T0147	Google	google/gemini-2.5-pro	Profile A	EVIDENCE_FIRST	3					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 23 (char 24)
T0148	Google	google/gemini-2.5-pro	Profile A	EVIDENCE_FIRST	4					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 25 (char 26)
T0149	Google	google/gemini-2.5-pro	Profile A	EVIDENCE_FIRST	5					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 25 (char 26)
T0150	Google	google/gemini-2.5-pro	Profile A	EVIDENCE_FIRST	6					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 35 (char 36)
T0151	Google	google/gemini-2.5-pro	Profile A	EVIDENCE_FIRST	7					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 25 (char 26)
T0152	Google	google/gemini-2.5-pro	Profile A	EVIDENCE_FIRST	8					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 35 (char 36)
T0153	Google	google/gemini-2.5-pro	Profile A	EVIDENCE_FIRST	9					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 25 (char 26)
T0154	Google	google/gemini-2.5-pro	Profile A	EVIDENCE_FIRST	10					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 35 (char 36)
T0155	Google	google/gemini-2.5-pro	Profile A	EVIDENCE_FIRST	11					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 25 (char 26)
T0156	Google	google/gemini-2.5-pro	Profile A	EVIDENCE_FIRST	12					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 35 (char 36)
T0157	Google	google/gemini-2.5-pro	Profile A	CENTER_FIRST	1					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 25 (char 26)
T0158	Google	google/gemini-2.5-pro	Profile A	CENTER_FIRST	2					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 25 (char 26)
T0159	Google	google/gemini-2.5-pro	Profile A	CENTER_FIRST	3					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 25 (char 26)
T0160	Google	google/gemini-2.5-pro	Profile A	CENTER_FIRST	4					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 35 (char 36)
T0161	Google	google/gemini-2.5-pro	Profile A	CENTER_FIRST	5					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 35 (char 36)
T0162	Google	google/gemini-2.5-pro	Profile A	CENTER_FIRST	6					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 35 (char 36)
T0163	Google	google/gemini-2.5-pro	Profile A	CENTER_FIRST	7					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 35 (char 36)
T0164	Google	google/gemini-2.5-pro	Profile A	CENTER_FIRST	8					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 33 (char 34)
T0165	Google	google/gemini-2.5-pro	Profile A	CENTER_FIRST	9					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 25 (char 26)
T0166	Google	google/gemini-2.5-pro	Profile A	CENTER_FIRST	10					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 35 (char 36)
T0167	Google	google/gemini-2.5-pro	Profile A	CENTER_FIRST	11					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 33 (char 34)
T0168	Google	google/gemini-2.5-pro	Profile A	CENTER_FIRST	12					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 35 (char 36)
T0169	Google	google/gemini-2.5-pro	Profile A	MRI_ORDER	1					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 31 (char 32)
T0170	Google	google/gemini-2.5-pro	Profile A	MRI_ORDER	2					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 29 (char 30)
T0171	Google	google/gemini-2.5-pro	Profile A	MRI_ORDER	3					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 41 (char 42)
T0172	Google	google/gemini-2.5-pro	Profile A	MRI_ORDER	4					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 41 (char 42)
T0173	Google	google/gemini-2.5-pro	Profile A	MRI_ORDER	5					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 29 (char 30)
T0174	Google	google/gemini-2.5-pro	Profile A	MRI_ORDER	6					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 31 (char 32)
T0175	Google	google/gemini-2.5-pro	Profile A	MRI_ORDER	7					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 31 (char 32)
T0176	Google	google/gemini-2.5-pro	Profile A	MRI_ORDER	8					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 29 (char 30)
T0177	Google	google/gemini-2.5-pro	Profile A	MRI_ORDER	9					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 29 (char 30)
T0178	Google	google/gemini-2.5-pro	Profile A	MRI_ORDER	10					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 41 (char 42)
T0179	Google	google/gemini-2.5-pro	Profile A	MRI_ORDER	11					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 39 (char 40)
T0180	Google	google/gemini-2.5-pro	Profile A	MRI_ORDER	12					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 31 (char 32)
T0181	Google	google/gemini-2.5-pro	Profile C	EVIDENCE_FIRST	1					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 25 (char 26)
T0182	Google	google/gemini-2.5-pro	Profile C	EVIDENCE_FIRST	2					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 25 (char 26)
T0183	Google	google/gemini-2.5-pro	Profile C	EVIDENCE_FIRST	3					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 25 (char 26)
T0184	Google	google/gemini-2.5-pro	Profile C	EVIDENCE_FIRST	4					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 35 (char 36)
T0185	Google	google/gemini-2.5-pro	Profile C	EVIDENCE_FIRST	5					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 35 (char 36)
T0186	Google	google/gemini-2.5-pro	Profile C	EVIDENCE_FIRST	6					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 35 (char 36)
T0187	Google	google/gemini-2.5-pro	Profile C	EVIDENCE_FIRST	7					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 25 (char 26)
T0188	Google	google/gemini-2.5-pro	Profile C	EVIDENCE_FIRST	8					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 25 (char 26)
T0189	Google	google/gemini-2.5-pro	Profile C	EVIDENCE_FIRST	9					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 35 (char 36)
T0190	Google	google/gemini-2.5-pro	Profile C	EVIDENCE_FIRST	10					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 25 (char 26)
T0191	Google	google/gemini-2.5-pro	Profile C	EVIDENCE_FIRST	11					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 25 (char 26)
T0192	Google	google/gemini-2.5-pro	Profile C	EVIDENCE_FIRST	12					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 25 (char 26)
T0193	Google	google/gemini-2.5-pro	Profile C	CENTER_FIRST	1					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 35 (char 36)
T0194	Google	google/gemini-2.5-pro	Profile C	CENTER_FIRST	2					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 35 (char 36)
T0195	Google	google/gemini-2.5-pro	Profile C	CENTER_FIRST	3					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 25 (char 26)
T0196	Google	google/gemini-2.5-pro	Profile C	CENTER_FIRST	4					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 25 (char 26)
T0197	Google	google/gemini-2.5-pro	Profile C	CENTER_FIRST	5					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 35 (char 36)
T0198	Google	google/gemini-2.5-pro	Profile C	CENTER_FIRST	6					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 25 (char 26)
T0199	Google	google/gemini-2.5-pro	Profile C	CENTER_FIRST	7					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 25 (char 26)
T0200	Google	google/gemini-2.5-pro	Profile C	CENTER_FIRST	8					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 25 (char 26)
T0201	Google	google/gemini-2.5-pro	Profile C	CENTER_FIRST	9					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 25 (char 26)
T0202	Google	google/gemini-2.5-pro	Profile C	CENTER_FIRST	10					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 25 (char 26)
T0203	Google	google/gemini-2.5-pro	Profile C	CENTER_FIRST	11					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 35 (char 36)
T0204	Google	google/gemini-2.5-pro	Profile C	CENTER_FIRST	12					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 33 (char 34)
T0205	Google	google/gemini-2.5-pro	Profile C	MRI_ORDER	1					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 31 (char 32)
T0206	Google	google/gemini-2.5-pro	Profile C	MRI_ORDER	2					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 29 (char 30)
T0207	Google	google/gemini-2.5-pro	Profile C	MRI_ORDER	3					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 31 (char 32)
T0208	Google	google/gemini-2.5-pro	Profile C	MRI_ORDER	4					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 31 (char 32)
T0209	Google	google/gemini-2.5-pro	Profile C	MRI_ORDER	5					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 31 (char 32)
T0210	Google	google/gemini-2.5-pro	Profile C	MRI_ORDER	6					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 31 (char 32)
T0211	Google	google/gemini-2.5-pro	Profile C	MRI_ORDER	7					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 29 (char 30)
T0212	Google	google/gemini-2.5-pro	Profile C	MRI_ORDER	8					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 41 (char 42)
T0213	Google	google/gemini-2.5-pro	Profile C	MRI_ORDER	9					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 31 (char 32)
T0214	Google	google/gemini-2.5-pro	Profile C	MRI_ORDER	10					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 31 (char 32)
T0215	Google	google/gemini-2.5-pro	Profile C	MRI_ORDER	11					False	False		JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 41 (char 42)
T0216	Google	google/gemini-2.5-pro	Profile C	MRI_ORDER	12					False	False		JSONDecodeError: Unterminated string starting at: line 2 column 31 (char 32)
T0217	xAI	x-ai/grok-4.5	Profile A	EVIDENCE_FIRST	1		Qwen3-14B (Prompt)	95		False	True	0.0
T0218	xAI	x-ai/grok-4.5	Profile A	EVIDENCE_FIRST	2		Qwen3-14B (Prompt)	92		False	True	0.0
T0219	xAI	x-ai/grok-4.5	Profile A	EVIDENCE_FIRST	3		Qwen3-14B (Prompt)	95		False	True	0.0
T0220	xAI	x-ai/grok-4.5	Profile A	EVIDENCE_FIRST	4		Qwen3-14B (Prompt)	95		False	True	0.0
T0221	xAI	x-ai/grok-4.5	Profile A	EVIDENCE_FIRST	5		Qwen3-14B (Prompt)	95		False	True	0.0
T0222	xAI	x-ai/grok-4.5	Profile A	EVIDENCE_FIRST	6		Qwen3-14B (Prompt)	95		False	True	0.0
T0223	xAI	x-ai/grok-4.5	Profile A	EVIDENCE_FIRST	7		Qwen3-14B (Prompt)	95		False	True	0.0
T0224	xAI	x-ai/grok-4.5	Profile A	EVIDENCE_FIRST	8		Qwen3-14B (Prompt)	95		False	True	0.0
T0225	xAI	x-ai/grok-4.5	Profile A	EVIDENCE_FIRST	9		Qwen3-14B (Prompt)	95		False	True	0.0
T0226	xAI	x-ai/grok-4.5	Profile A	EVIDENCE_FIRST	10		Qwen3-14B (Prompt)	95		False	True	0.0
T0227	xAI	x-ai/grok-4.5	Profile A	EVIDENCE_FIRST	11		Qwen3-14B (Prompt)	95		False	True	0.0
T0228	xAI	x-ai/grok-4.5	Profile A	EVIDENCE_FIRST	12		Qwen3-14B (Prompt)	95		False	True	0.0
T0229	xAI	x-ai/grok-4.5	Profile A	CENTER_FIRST	1		Qwen3-14B (Prompt)	92		False	True	0.0
T0230	xAI	x-ai/grok-4.5	Profile A	CENTER_FIRST	2		Qwen3-14B (Prompt)	92		False	True	0.0
T0231	xAI	x-ai/grok-4.5	Profile A	CENTER_FIRST	3		Qwen3-14B (Prompt)	88		False	True	0.0
T0232	xAI	x-ai/grok-4.5	Profile A	CENTER_FIRST	4		Qwen3-14B (Prompt)	88		False	True	0.0
T0233	xAI	x-ai/grok-4.5	Profile A	CENTER_FIRST	5		Qwen3-14B (Prompt)	92		False	True	0.0
T0234	xAI	x-ai/grok-4.5	Profile A	CENTER_FIRST	6		Qwen3-14B (Prompt)	92		False	True	0.0
T0235	xAI	x-ai/grok-4.5	Profile A	CENTER_FIRST	7		Qwen3-14B (Prompt)	92		False	True	0.0
T0236	xAI	x-ai/grok-4.5	Profile A	CENTER_FIRST	8		Qwen3-14B (Prompt)	92		False	True	0.0
T0237	xAI	x-ai/grok-4.5	Profile A	CENTER_FIRST	9		Qwen3-14B (Prompt)	92		False	True	0.0
T0238	xAI	x-ai/grok-4.5	Profile A	CENTER_FIRST	10		Qwen3-14B (Prompt)	92		False	True	0.0
T0239	xAI	x-ai/grok-4.5	Profile A	CENTER_FIRST	11		Qwen3-14B (Prompt)	92		False	True	0.0
T0240	xAI	x-ai/grok-4.5	Profile A	CENTER_FIRST	12		Qwen3-14B (Prompt)	92		False	True	0.0
T0241	xAI	x-ai/grok-4.5	Profile A	MRI_ORDER	1	Qwen3-14B (Prompt)				False	False		JSONDecodeError: Expecting ':' delimiter: line 1 column 73 (char 72)
T0242	xAI	x-ai/grok-4.5	Profile A	MRI_ORDER	2	Qwen3-14B (Prompt)	Qwen3-14B (Prompt)	93	False	False	True	0.0
T0243	xAI	x-ai/grok-4.5	Profile A	MRI_ORDER	3	Qwen3-14B (Prompt)				False	False		JSONDecodeError: Expecting ':' delimiter: line 1 column 67 (char 66)
T0244	xAI	x-ai/grok-4.5	Profile A	MRI_ORDER	4	Qwen3-14B (Prompt)	Qwen3-14B (Prompt)	93	False	False	True	0.0
T0245	xAI	x-ai/grok-4.5	Profile A	MRI_ORDER	5	Qwen3-14B (Prompt)	Qwen3-14B (Prompt)	95	False	False	True	0.0
T0246	xAI	x-ai/grok-4.5	Profile A	MRI_ORDER	6	Qwen3-14B (Prompt)	Qwen3-14B (Prompt)	95	False	False	True	0.0
T0247	xAI	x-ai/grok-4.5	Profile A	MRI_ORDER	7	Qwen3-14B (Prompt)	Qwen3-14B (Prompt)	93	False	False	True	0.0
T0248	xAI	x-ai/grok-4.5	Profile A	MRI_ORDER	8	Qwen3-14B (Prompt)	Qwen3-14B (Prompt)	93	False	False	True	0.0
T0249	xAI	x-ai/grok-4.5	Profile A	MRI_ORDER	9	Qwen3-14B (Prompt)	Qwen3-14B (Prompt)	93	False	False	True	0.0
T0250	xAI	x-ai/grok-4.5	Profile A	MRI_ORDER	10	Qwen3-14B (Prompt)	Qwen3-14B (Prompt)	93	False	False	True	0.0
T0251	xAI	x-ai/grok-4.5	Profile A	MRI_ORDER	11	Qwen3-14B (Prompt)	Qwen3-14B (Prompt)	93	False	False	True	0.0
T0252	xAI	x-ai/grok-4.5	Profile A	MRI_ORDER	12	Qwen3-14B (Prompt)				False	False		JSONDecodeError: Expecting ':' delimiter: line 1 column 73 (char 72)
T0253	xAI	x-ai/grok-4.5	Profile C	EVIDENCE_FIRST	1		Claude-Opus-4-5-20251101 (FC)	72		True	True	0.0
T0254	xAI	x-ai/grok-4.5	Profile C	EVIDENCE_FIRST	2		Claude-Opus-4-5-20251101 (FC)	78		True	True	0.0
T0255	xAI	x-ai/grok-4.5	Profile C	EVIDENCE_FIRST	3		Claude-Opus-4-5-20251101 (FC)	72		True	True	0.0
T0256	xAI	x-ai/grok-4.5	Profile C	EVIDENCE_FIRST	4		Claude-Opus-4-5-20251101 (FC)	78		True	True	0.0
T0257	xAI	x-ai/grok-4.5	Profile C	EVIDENCE_FIRST	5		Claude-Opus-4-5-20251101 (FC)	72		True	True	0.0
T0258	xAI	x-ai/grok-4.5	Profile C	EVIDENCE_FIRST	6		GLM-4.6 (FC thinking)	72		False	False	3.8251670000000075
T0259	xAI	x-ai/grok-4.5	Profile C	EVIDENCE_FIRST	7		Claude-Opus-4-5-20251101 (FC)	78		True	True	0.0
T0260	xAI	x-ai/grok-4.5	Profile C	EVIDENCE_FIRST	8		Claude-Opus-4-5-20251101 (FC)	78		True	True	0.0
T0261	xAI	x-ai/grok-4.5	Profile C	EVIDENCE_FIRST	9		Claude-Opus-4-5-20251101 (FC)	78		True	True	0.0
T0262	xAI	x-ai/grok-4.5	Profile C	EVIDENCE_FIRST	10		Claude-Opus-4-5-20251101 (FC)	78		True	True	0.0
T0263	xAI	x-ai/grok-4.5	Profile C	EVIDENCE_FIRST	11		Claude-Opus-4-5-20251101 (FC)	78		True	True	0.0
T0264	xAI	x-ai/grok-4.5	Profile C	EVIDENCE_FIRST	12		Claude-Opus-4-5-20251101 (FC)	78		True	True	0.0
T0265	xAI	x-ai/grok-4.5	Profile C	CENTER_FIRST	1		Claude-Opus-4-5-20251101 (FC)	78		True	True	0.0
T0266	xAI	x-ai/grok-4.5	Profile C	CENTER_FIRST	2		Claude-Opus-4-5-20251101 (FC)	78		True	True	0.0
T0267	xAI	x-ai/grok-4.5	Profile C	CENTER_FIRST	3		Claude-Opus-4-5-20251101 (FC)	82		True	True	0.0
T0268	xAI	x-ai/grok-4.5	Profile C	CENTER_FIRST	4		Claude-Opus-4-5-20251101 (FC)	78		True	True	0.0
T0269	xAI	x-ai/grok-4.5	Profile C	CENTER_FIRST	5		Claude-Opus-4-5-20251101 (FC)	78		True	True	0.0
T0270	xAI	x-ai/grok-4.5	Profile C	CENTER_FIRST	6		Claude-Opus-4-5-20251101 (FC)	78		True	True	0.0
T0271	xAI	x-ai/grok-4.5	Profile C	CENTER_FIRST	7		Claude-Opus-4-5-20251101 (FC)	78		True	True	0.0
T0272	xAI	x-ai/grok-4.5	Profile C	CENTER_FIRST	8		Claude-Opus-4-5-20251101 (FC)	78		True	True	0.0
T0273	xAI	x-ai/grok-4.5	Profile C	CENTER_FIRST	9		Claude-Opus-4-5-20251101 (FC)	78		True	True	0.0
T0274	xAI	x-ai/grok-4.5	Profile C	CENTER_FIRST	10		Claude-Opus-4-5-20251101 (FC)	78		True	True	0.0
T0275	xAI	x-ai/grok-4.5	Profile C	CENTER_FIRST	11		Claude-Opus-4-5-20251101 (FC)	78		True	True	0.0
T0276	xAI	x-ai/grok-4.5	Profile C	CENTER_FIRST	12		Claude-Opus-4-5-20251101 (FC)	78		True	True	0.0
T0277	xAI	x-ai/grok-4.5	Profile C	MRI_ORDER	1	Claude-Opus-4-5-20251101 (FC)	Claude-Opus-4-5-20251101 (FC)	85	False	True	True	0.0
T0278	xAI	x-ai/grok-4.5	Profile C	MRI_ORDER	2	Claude-Opus-4-5-20251101 (FC)	Claude-Opus-4-5-20251101 (FC)	82	False	True	True	0.0
T0279	xAI	x-ai/grok-4.5	Profile C	MRI_ORDER	3	Claude-Opus-4-5-20251101 (FC)	Claude-Opus-4-5-20251101 (FC)	85	False	True	True	0.0
T0280	xAI	x-ai/grok-4.5	Profile C	MRI_ORDER	4	Claude-Opus-4-5-20251101 (FC)	Claude-Opus-4-5-20251101 (FC)	85	False	True	True	0.0
T0281	xAI	x-ai/grok-4.5	Profile C	MRI_ORDER	5	Claude-Opus-4-5-20251101 (FC)				False	False		JSONDecodeError: Expecting ':' delimiter: line 1 column 67 (char 66)
T0282	xAI	x-ai/grok-4.5	Profile C	MRI_ORDER	6	Claude-Opus-4-5-20251101 (FC)	Claude-Opus-4-5-20251101 (FC)	85	False	True	True	0.0
T0283	xAI	x-ai/grok-4.5	Profile C	MRI_ORDER	7	Claude-Opus-4-5-20251101 (FC)	Claude-Opus-4-5-20251101 (FC)	88	False	True	True	0.0
T0284	xAI	x-ai/grok-4.5	Profile C	MRI_ORDER	8	Claude-Opus-4-5-20251101 (FC)	Claude-Opus-4-5-20251101 (FC)	85	False	True	True	0.0
T0285	xAI	x-ai/grok-4.5	Profile C	MRI_ORDER	9	Claude-Opus-4-5-20251101 (FC)	Claude-Opus-4-5-20251101 (FC)	80	False	True	True	0.0
T0286	xAI	x-ai/grok-4.5	Profile C	MRI_ORDER	10	Claude-Opus-4-5-20251101 (FC)	Claude-Opus-4-5-20251101 (FC)	85	False	True	True	0.0
T0287	xAI	x-ai/grok-4.5	Profile C	MRI_ORDER	11	Claude-Opus-4-5-20251101 (FC)	Claude-Opus-4-5-20251101 (FC)	88	False	True	True	0.0
T0288	xAI	x-ai/grok-4.5	Profile C	MRI_ORDER	12	Claude-Opus-4-5-20251101 (FC)				False	False		JSONDecodeError: Expecting ':' delimiter: line 1 column 73 (char 72)

AI-assisted research disclosure

MRI-001 used AI-assisted research, coding and analysis.

AI models, Python and Codex were used to help collect and organize evidence, implement calculations, process the model field, run sensitivity tests, preserve outputs, prepare tables and audit the analysis.

The research question, conceptual boundaries, test structure, interpretation, publication decisions and limits on what could be claimed remained human decisions.

Outputs that weakened the preferred interpretation were retained. This includes malformed responses, alternative winners under sensitivity tests and the failed center-pull pilot.

The evidence package is published so the distinction between the research judgment and the mechanical work can be inspected rather than taken on trust.

Provenance & Reproducibility

The manifests, reports, scripts, and ZIP below preserve the provenance trail for the frozen evidence package.

Show evidence package manifest

artifact_path	original_path	file_type	purpose	sha256	source_unchanged	notes
mri/001/evidence/code/CODE_MAP.md	generated_for_evidence_package	markdown	Maps copied code files to original paths and generated outputs.	a6587389f02e0a72417c3bec0e80b408f3be71421578c51d86b7228d0e04a338	N/A	generated package documentation; no source data modified
mri/001/evidence/data/bfcl_relevant_baseline_rows.csv	frame_audit_001_bfcl/data/processed/bfcl_v4_baseline.csv	csv_extract	Exact baseline rows for public demonstration models, preserving every source column.	4ea2d3638fe82d0f757ecd05e6a9a73174a15bae572a8d34e0f231d06494c440	TRUE	values copied exactly from source rows; column order preserved
mri/001/evidence/data/profile_a_relevant_results.csv	frame_audit_001_bfcl/data/processed/bfcl_profile_results.csv	csv_extract	Exact Profile A rows for top five models, published winner, and leave-one-factor selected models where present in profile results.	814f2cf598cd97c137934618c7ccd46d5050a06ea7a7809d4e2a3b52baafa39c	TRUE	values copied exactly from source rows; top-five membership determined from stored profile_rank
mri/001/evidence/data/profile_a_sensitivity_results.csv	frame_audit_001_bfcl/data/processed/bfcl_sensitivity_results.csv	csv_extract	Exact Profile A rows available in bfcl_sensitivity_results.csv.	281989e32907f8dccf2dd484559d65c2a106b6b4e8942a99af19910c181946c6	TRUE	source CSV contains Profile A buyer_profile_weighting rows only; leave-one-factor diagnostics are recorded in BFCL_CALCULATION_AND_MATERIALITY_AUDIT.md, not in this CSV
mri/001/evidence/pilot/PILOT_FILE_MAP.md	generated_for_evidence_package	markdown	Maps copied pilot files to originals and records pilot run counts/formulas/verdict location.	3afcb9fd84a1c95c1283f573f0f1b0f1382aab369474e6be018c058af95660f7	N/A	generated package documentation; no source data modified
mri/001/evidence/provenance/PROVENANCE_SUMMARY.md	generated_for_evidence_package	markdown	Summarizes frozen BFCL provenance and baseline reproducibility metadata.	d0217b4d6029f890b33edbd4b79be8d7e68b0c295c311ec6a9c162233afe633c	N/A	generated package documentation; no source data modified
mri/001/evidence/README.md	generated_for_evidence_package	markdown	Top-level statement of what the evidence package supports and does not support.	1a0f0282b107980c928cfdbc89bc4a98e7345016c8cfc5a5b76f493e86f20a93	N/A	generated package documentation; no source data modified
mri/001/evidence/code/run_bfcl_v4_sensitivity.py	code/run_bfcl_v4_sensitivity.py	python_source	Calculates buyer-profile scores, eligibility, disqualifications, profile rankings, cost-efficiency normalization, latency scoring, and registered sensitivity outputs.	6812d4352ae1e76bbb45d92b7b3bf1e8656cda986ddd176091449440f844e15d	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/code/audit_bfcl_v4_sensitivity.py	code/audit_bfcl_v4_sensitivity.py	python_source	Verifies Profile A-D calculations, runs leave-one-factor-out diagnostics, decomposes Profile A/D scores, and writes the calculation/materiality audit.	48eafa44157041edb8399a5af64ec3babc1edf6cc492caade8bad08f7fa5e2b4	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/code/freeze_bfcl_v4_snapshot.py	code/freeze_bfcl_v4_snapshot.py	python_source	Downloads and freezes BFCL public files, records source hashes and retrieval metadata, reconstructs the published aggregate, and writes baseline outputs.	d614d22ba31cef4362a4aeb959cefb02610897cd56a41f22e0d4c59dec8866ca	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/pilot/run_mri_rank_before_work_pilot.py	pilot/run_mri_rank_before_work_pilot.py	python_source	Authority-pull pilot source containing CENTER FIRST, EVIDENCE FIRST, MRI ORDER, parsing, metrics, and verdict logic.	7db5205b0c7139fa74501210e24c426b11e23d0fd0dda1dc34a5d8e354e304f3	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/pilot/00_PREREGISTRATION.md	mri_rank_before_work/00_PREREGISTRATION.md	markdown	Pilot preregistration, conditions, scenarios, measures, and pass/mixed/fail rules.	c201a0c4ec90a4c9465d11928a22ce3705e38eb8e0e14d36a811b74fb003b750	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/pilot/01_EXPERIMENTAL_CARDS.md	mri_rank_before_work/01_EXPERIMENTAL_CARDS.md	markdown	Experimental cards and prompt/condition wording.	6f8409fa395c51561fac1e19f0f87b57490d0d8ee9e7f5c81399d707db0c4ada	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/pilot/02_RAW_RESPONSES.md	mri_rank_before_work/02_RAW_RESPONSES.md	markdown	Raw judge responses preserved from the pilot.	aa9e8f5dbf982c7e62d0d81422c022a71793345911c2d653885d646ca93c8a3b	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/pilot/03_RESULTS.csv	mri_rank_before_work/03_RESULTS.csv	csv	Row-level pilot decisions including scenario, condition, selected model, preferred model, regret, citations, and errors.	c6b09ad15cf08adf8632773617a11e8a6b68ab9b2122010a29204236e79e7810	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/pilot/04_MRI_METRICS.csv	mri_rank_before_work/04_MRI_METRICS.csv	csv	Pilot metric calculations including center pull, task-fit accuracy, regret, suppression, and correction capacity.	241cb93e935b4d193facd9053dc4008ddf555bc284ac492d80431a55de07acbf	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/pilot/05_TECHNICAL_REPORT.md	mri_rank_before_work/05_TECHNICAL_REPORT.md	markdown	Technical pilot report and final PILOT FAIL verdict.	cc14f3b813b8c35879947b92ee853e5c5400738136c1df58322519e89b7682dc	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/pilot/06_PLAIN_LANGUAGE_RESULT.md	mri_rank_before_work/06_PLAIN_LANGUAGE_RESULT.md	markdown	Plain-language pilot result and final verdict.	f01015c71dda4973e3d349e10ccf14f2d4b9c43e6ac0b9a20793f06a645d1ad6	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/provenance/BFCL_SNAPSHOT_MANIFEST.csv	frame_audit_001_bfcl/data/raw/BFCL_SNAPSHOT_MANIFEST.csv	csv	Frozen manifest with source URLs, retrieval timestamps, byte sizes, SHA-256 hashes, row counts, column names, and update metadata.	878a5709b4593a5e75885f203fb090001a4101eed84340472d38aa36b098ab4d	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/provenance/bfcl_download_http_metadata.json	frame_audit_001_bfcl/data/raw/metadata/bfcl_download_http_metadata.json	json	HTTP metadata from the public BFCL file retrieval.	6a5d7838671e854188f667a078136480ebe02f6776a584cd5ca47e4899aa0424	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/provenance/bfcl_snapshot_metadata.json	frame_audit_001_bfcl/data/raw/metadata/bfcl_snapshot_metadata.json	json	Frozen BFCL package/commit/archive/update metadata.	8f82e568ffc69c3d00e07ef2a5203663e16d42da9acb45f1f4924ce81bd1a0ab	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/provenance/bfcl_result_archive_index_2025-12-16.csv	frame_audit_001_bfcl/data/raw/metadata/bfcl_result_archive_index_2025-12-16.csv	csv	Public result archive index snapshot.	1818d09c44854b9a390009e2b258a67d30d732e3cb02f2dc70fd5e06187be1cb	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/provenance/bfcl_result_archive_result_index_2025-12-16.json	frame_audit_001_bfcl/data/raw/metadata/bfcl_result_archive_result_index_2025-12-16.json	json	Public result archive result index snapshot.	18837f30b7944f3b491d2e921f9b7c5502ff539680e03be42d56aa33823532ba	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/provenance/bfcl_result_archive_score_index_2025-12-16.json	frame_audit_001_bfcl/data/raw/metadata/bfcl_result_archive_score_index_2025-12-16.json	json	Public result archive score index snapshot.	5f7e763e7faef873d6f6e3f696fe13157c3aec5f5f0c4963febdcd826d33b2f7	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/provenance/bfcl_result_archive_root_index.json	frame_audit_001_bfcl/data/raw/metadata/bfcl_result_archive_root_index.json	json	Public result archive root index snapshot.	8ceee9c838bfcb3817ba4dff109c016926e5fbedfcffb601dde014b80b984637	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/provenance/bfcl_readme.md	frame_audit_001_bfcl/data/raw/metadata/bfcl_readme.md	markdown	BFCL public README copied from the frozen metadata set.	c6c95683cce9653788c8b3f3f5002d2f5c27b95a1288544fe82c2a2faf5e12d0	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/provenance/bfcl_changelog.md	frame_audit_001_bfcl/data/raw/metadata/bfcl_changelog.md	markdown	BFCL public changelog copied from the frozen metadata set.	27cbabdb95424b5b03b6b9b34eebdf99a5c5b64fc5d2964726ef42ecbac104db	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/provenance/bfcl_v4_score_composition.html	frame_audit_001_bfcl/data/raw/metadata/bfcl_v4_score_composition.html	html	BFCL V4 score composition documentation snapshot.	c6967f17bd54a2151940384075abb2af2a3adfc26ddec9f7afae411519977c62	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/provenance/BFCL_AUDIT_PROTOCOL.md	frame_audit_001_bfcl/BFCL_AUDIT_PROTOCOL.md	markdown	Frozen audit protocol.	891059021f8ec16e5bdfe45cd5afccab1b8e210b07511c8dcc87986190090956	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/provenance/BFCL_DATA_INVENTORY.md	frame_audit_001_bfcl/BFCL_DATA_INVENTORY.md	markdown	Frozen data inventory.	414fd2be8b47aac98ea178e70ced9ae95127e271e271ff2cd2be274e91f1fda3	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/provenance/BFCL_DECISION_PROFILES.md	frame_audit_001_bfcl/BFCL_DECISION_PROFILES.md	markdown	Frozen Profile A-D definitions.	82151cc02b4b9855e21657c9ac64688777f67119866332867ebe4d9b2a7f51ff	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/provenance/BFCL_PASS_FAIL_RULES.md	frame_audit_001_bfcl/BFCL_PASS_FAIL_RULES.md	markdown	Frozen pass/fail rules.	cba30090055e554a96c23ac4152f23061923a13b7018b7d645ec84c603267e2c	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/provenance/BFCL_BASELINE_RECONSTRUCTION_REPORT.md	frame_audit_001_bfcl/outputs/BFCL_BASELINE_RECONSTRUCTION_REPORT.md	markdown	Baseline reconstruction result and maximum discrepancy.	8d4279db2b6414fa9a6b73d53b3f30c19735ca2d34a46030eaa25d1ab4582236	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/provenance/BFCL_DECISION_FLIP_SUMMARY.md	frame_audit_001_bfcl/outputs/BFCL_DECISION_FLIP_SUMMARY.md	markdown	Existing decision-flip summary.	61787bed11a954c4e0a9c1ceed2f9251f9620809e817f2bc3f46a50012229329	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved
mri/001/evidence/provenance/BFCL_CALCULATION_AND_MATERIALITY_AUDIT.md	frame_audit_001_bfcl/outputs/BFCL_CALCULATION_AND_MATERIALITY_AUDIT.md	markdown	Existing verified calculation/materiality audit, including leave-one-factor-out diagnostics.	29106929037310f15e253931f2d5d1ab1b2648402f21fb61e485943ccf9d7a3b	TRUE	source values copied; publication paths sanitized where present; evidentiary values preserved

Show BFCL snapshot manifest

local_path	file_name	file_kind	source_url	retrieval_datetime_utc	http_last_modified	http_etag	byte_size	sha256	row_count	column_names	latest_model_update_represented	leaderboard_last_updated	bfcl_eval_package	bfcl_eval_commit	missing_or_malformed_fields
frame_audit_001_bfcl/data/raw/data_overall.csv	data_overall.csv	summary_csv	https://gorilla.cs.berkeley.edu/data_overall.csv	2026-07-30T22:19:25Z	Mon, 13 Apr 2026 03:20:44 GMT	"69dc610c-8116"	33046	ede5081a24bc5ac7f9b3b09a52af6bcfcd35aa0f1709239fed979db77a8159a6	109	Rank; Overall Acc; Model; Model Link; Total Cost ($); Latency Mean (s); Latency Standard Deviation (s); Latency 95th Percentile (s); Non-Live AST Acc; Non-Live Simple AST; Non-Live Multiple AST; Non-Live Parallel AST; Non-Live Parallel Multiple AST; Live Acc; Live Simple AST; Live Multiple AST; Live Parallel AST; Live Parallel Multiple AST; Multi Turn Acc; Multi Turn Base; Multi Turn Miss Func; Multi Turn Miss Param; Multi Turn Long Context; Web Search Acc; Web Search Base; Web Search No Snippet; Memory Acc; Memory KV; Memory Vector; Memory Recursive Summarization; Relevance Detection; Irrelevance Detection; Format Sensitivity Max Delta; Format Sensitivity Standard Deviation; Organization; License	2025-12-11	2026-04-12	bfcl-eval==2025.12.17	f7cf735	missing: Format Sensitivity Max Delta=69; Format Sensitivity Standard Deviation=69; malformed: none
frame_audit_001_bfcl/data/raw/data_agentic.csv	data_agentic.csv	summary_csv	https://gorilla.cs.berkeley.edu/data_agentic.csv	2026-07-30T22:19:25Z	Mon, 13 Apr 2026 03:20:44 GMT	"69dc610c-22ff"	8959	92cc312ff1fa7935ff20687dd1075540243dc1366f52efd71e2192265919bb0d	109	Rank; Model; Agentic Overall Acc; Web Search Summary; Web Search Base; Web Search No Snippet; Memory Summary; Memory KV; Memory Vector; Memory Recursive Summarization	2025-12-11	2026-04-12	bfcl-eval==2025.12.17	f7cf735	missing: none; malformed: none
frame_audit_001_bfcl/data/raw/data_format_sensitivity.csv	data_format_sensitivity.csv	summary_csv	https://gorilla.cs.berkeley.edu/data_format_sensitivity.csv	2026-07-30T22:19:25Z	Mon, 13 Apr 2026 03:20:44 GMT	"69dc610c-51e6"	20966	eef15e10a428aeb2a13beea423db9c8267af913bff67bd260cd698c9e026eab6	109	Rank; Model; Format Sensitivity Max Delta; Format Sensitivity Standard Deviation; Config ret_fmt=python&tool_call_tag=True&func_doc_fmt=python&prompt_fmt=plaintext&style=classic; Config ret_fmt=python&tool_call_tag=True&func_doc_fmt=xml&prompt_fmt=plaintext&style=classic; Config ret_fmt=python&tool_call_tag=True&func_doc_fmt=json&prompt_fmt=plaintext&style=classic; Config ret_fmt=python&tool_call_tag=False&func_doc_fmt=python&prompt_fmt=plaintext&style=classic; Config ret_fmt=python&tool_call_tag=False&func_doc_fmt=xml&prompt_fmt=plaintext&style=classic; Config ret_fmt=python&tool_call_tag=False&func_doc_fmt=json&prompt_fmt=plaintext&style=classic; Config ret_fmt=json&tool_call_tag=True&func_doc_fmt=python&prompt_fmt=plaintext&style=classic; Config ret_fmt=json&tool_call_tag=True&func_doc_fmt=xml&prompt_fmt=plaintext&style=classic; Config ret_fmt=json&tool_call_tag=True&func_doc_fmt=json&prompt_fmt=plaintext&style=classic; Config ret_fmt=json&tool_call_tag=False&func_doc_fmt=python&prompt_fmt=plaintext&style=classic; Config ret_fmt=json&tool_call_tag=False&func_doc_fmt=xml&prompt_fmt=plaintext&style=classic; Config ret_fmt=json&tool_call_tag=False&func_doc_fmt=json&prompt_fmt=plaintext&style=classic; Config ret_fmt=verbose_xml&tool_call_tag=True&func_doc_fmt=python&prompt_fmt=plaintext&style=classic; Config ret_fmt=verbose_xml&tool_call_tag=True&func_doc_fmt=xml&prompt_fmt=plaintext&style=classic; Config ret_fmt=verbose_xml&tool_call_tag=True&func_doc_fmt=json&prompt_fmt=plaintext&style=classic; Config ret_fmt=verbose_xml&tool_call_tag=False&func_doc_fmt=python&prompt_fmt=plaintext&style=classic; Config ret_fmt=verbose_xml&tool_call_tag=False&func_doc_fmt=xml&prompt_fmt=plaintext&style=classic; Config ret_fmt=verbose_xml&tool_call_tag=False&func_doc_fmt=json&prompt_fmt=plaintext&style=classic; Config ret_fmt=concise_xml&tool_call_tag=True&func_doc_fmt=python&prompt_fmt=plaintext&style=classic; Config ret_fmt=concise_xml&tool_call_tag=True&func_doc_fmt=xml&prompt_fmt=plaintext&style=classic; Config ret_fmt=concise_xml&tool_call_tag=True&func_doc_fmt=json&prompt_fmt=plaintext&style=classic; Config ret_fmt=concise_xml&tool_call_tag=False&func_doc_fmt=python&prompt_fmt=plaintext&style=classic; Config ret_fmt=concise_xml&tool_call_tag=False&func_doc_fmt=xml&prompt_fmt=plaintext&style=classic; Config ret_fmt=concise_xml&tool_call_tag=False&func_doc_fmt=json&prompt_fmt=plaintext&style=classic; Config ret_fmt=python&tool_call_tag=False&func_doc_fmt=json&prompt_fmt=markdown&style=classic; Config ret_fmt=python&tool_call_tag=False&func_doc_fmt=json&prompt_fmt=plaintext&style=experimental	2025-12-11	2026-04-12	bfcl-eval==2025.12.17	f7cf735	missing: Format Sensitivity Max Delta=69; Format Sensitivity Standard Deviation=69; Config ret_fmt=python&tool_call_tag=True&func_doc_fmt=python&prompt_fmt=plaintext&style=classic=69; Config ret_fmt=python&tool_call_tag=True&func_doc_fmt=xml&prompt_fmt=plaintext&style=classic=69; Config ret_fmt=python&tool_call_tag=True&func_doc_fmt=json&prompt_fmt=plaintext&style=classic=69; Config ret_fmt=python&tool_call_tag=False&func_doc_fmt=python&prompt_fmt=plaintext&style=classic=69; Config ret_fmt=python&tool_call_tag=False&func_doc_fmt=xml&prompt_fmt=plaintext&style=classic=69; Config ret_fmt=python&tool_call_tag=False&func_doc_fmt=json&prompt_fmt=plaintext&style=classic=69; Config ret_fmt=json&tool_call_tag=True&func_doc_fmt=python&prompt_fmt=plaintext&style=classic=69; Config ret_fmt=json&tool_call_tag=True&func_doc_fmt=xml&prompt_fmt=plaintext&style=classic=69; Config ret_fmt=json&tool_call_tag=True&func_doc_fmt=json&prompt_fmt=plaintext&style=classic=69; Config ret_fmt=json&tool_call_tag=False&func_doc_fmt=python&prompt_fmt=plaintext&style=classic=69; Config ret_fmt=json&tool_call_tag=False&func_doc_fmt=xml&prompt_fmt=plaintext&style=classic=69; Config ret_fmt=json&tool_call_tag=False&func_doc_fmt=json&prompt_fmt=plaintext&style=classic=69; Config ret_fmt=verbose_xml&tool_call_tag=True&func_doc_fmt=python&prompt_fmt=plaintext&style=classic=69; Config ret_fmt=verbose_xml&tool_call_tag=True&func_doc_fmt=xml&prompt_fmt=plaintext&style=classic=69; Config ret_fmt=verbose_xml&tool_call_tag=True&func_doc_fmt=json&prompt_fmt=plaintext&style=classic=69; Config ret_fmt=verbose_xml&tool_call_tag=False&func_doc_fmt=python&prompt_fmt=plaintext&style=classic=69; Config ret_fmt=verbose_xml&tool_call_tag=False&func_doc_fmt=xml&prompt_fmt=plaintext&style=classic=69; Config ret_fmt=verbose_xml&tool_call_tag=False&func_doc_fmt=json&prompt_fmt=plaintext&style=classic=69; Config ret_fmt=concise_xml&tool_call_tag=True&func_doc_fmt=python&prompt_fmt=plaintext&style=classic=69; Config ret_fmt=concise_xml&tool_call_tag=True&func_doc_fmt=xml&prompt_fmt=plaintext&style=classic=69; Config ret_fmt=concise_xml&tool_call_tag=True&func_doc_fmt=json&prompt_fmt=plaintext&style=classic=69; Config ret_fmt=concise_xml&tool_call_tag=False&func_doc_fmt=python&prompt_fmt=plaintext&style=classic=69; Config ret_fmt=concise_xml&tool_call_tag=False&func_doc_fmt=xml&prompt_fmt=plaintext&style=classic=69; Config ret_fmt=concise_xml&tool_call_tag=False&func_doc_fmt=json&prompt_fmt=plaintext&style=classic=69; Config ret_fmt=python&tool_call_tag=False&func_doc_fmt=json&prompt_fmt=markdown&style=classic=69; Config ret_fmt=python&tool_call_tag=False&func_doc_fmt=json&prompt_fmt=plaintext&style=experimental=69; malformed: none
frame_audit_001_bfcl/data/raw/data_live.csv	data_live.csv	summary_csv	https://gorilla.cs.berkeley.edu/data_live.csv	2026-07-30T22:19:25Z	Mon, 13 Apr 2026 03:20:44 GMT	"69dc610c-2476"	9334	8c37538602fb45a10ee15c64478af6384271b31e90961d8daa17b28472b1c970	109	Rank; Model; Live Overall Acc; AST Summary; Python Simple AST; Python Multiple AST; Python Parallel AST; Python Parallel Multiple AST; Irrelevance Detection; Relevance Detection	2025-12-11	2026-04-12	bfcl-eval==2025.12.17	f7cf735	missing: none; malformed: none
frame_audit_001_bfcl/data/raw/data_multi_turn.csv	data_multi_turn.csv	summary_csv	https://gorilla.cs.berkeley.edu/data_multi_turn.csv	2026-07-30T22:19:25Z	Mon, 13 Apr 2026 03:20:44 GMT	"69dc610c-1a69"	6761	504d79bfaf9d67dda30f519ed3f77c335b117bd4131722a1605694a26b9e00ad	109	Rank; Model; Multi Turn Overall Acc; Base; Miss Func; Miss Param; Long Context	2025-12-11	2026-04-12	bfcl-eval==2025.12.17	f7cf735	missing: none; malformed: none
frame_audit_001_bfcl/data/raw/data_non_live.csv	data_non_live.csv	summary_csv	https://gorilla.cs.berkeley.edu/data_non_live.csv	2026-07-30T22:19:25Z	Mon, 13 Apr 2026 03:20:43 GMT	"69dc610b-2a6b"	10859	19eec28695654ece206b343252db836127a96f18552456c1c3cf496b2dc306d0	109	Rank; Model; Non-Live Overall Acc; AST Summary; Simple AST; Python Simple AST; Java Simple AST; JavaScript Simple AST; Multiple AST; Parallel AST; Parallel Multiple AST; Irrelevance Detection	2025-12-11	2026-04-12	bfcl-eval==2025.12.17	f7cf735	missing: none; malformed: none
frame_audit_001_bfcl/data/raw/metadata/bfcl_leaderboard.html	bfcl_leaderboard.html	metadata	https://gorilla.cs.berkeley.edu/leaderboard	2026-07-30T22:19:25Z	Mon, 13 Apr 2026 03:20:43 GMT	"69dc610b-3f4f"	16207	ef72a8621b6ea1b8aa1d1b7691f73cc3079fd38e86f599c9a38cb626c911fc29	N/A		2025-12-11	2026-04-12	bfcl-eval==2025.12.17	f7cf735	missing: not applicable; non-tabular file with 350 lines; malformed: not applicable
frame_audit_001_bfcl/data/raw/metadata/bfcl_v4_score_composition.html	bfcl_v4_score_composition.html	metadata	https://gorilla.cs.berkeley.edu/blogs/15_bfcl_v4_web_search.html	2026-07-30T22:19:25Z	Mon, 13 Apr 2026 03:20:44 GMT	"69dc610c-124f1"	74993	c6967f17bd54a2151940384075abb2af2a3adfc26ddec9f7afae411519977c62	N/A		2025-12-11	2026-04-12	bfcl-eval==2025.12.17	f7cf735	missing: not applicable; non-tabular file with 1479 lines; malformed: not applicable
frame_audit_001_bfcl/data/raw/metadata/bfcl_readme.md	bfcl_readme.md	metadata	https://raw.githubusercontent.com/ShishirPatil/gorilla/main/berkeley-function-call-leaderboard/README.md	2026-07-30T22:19:25Z		"b8736777e8f2efe31c60ef0ba6fd81754ad938236bee5d0cbac3574d751970fd"	17311	c6c95683cce9653788c8b3f3f5002d2f5c27b95a1288544fe82c2a2faf5e12d0	N/A		2025-12-11	2026-04-12	bfcl-eval==2025.12.17	f7cf735	missing: not applicable; non-tabular file with 351 lines; malformed: not applicable
frame_audit_001_bfcl/data/raw/metadata/bfcl_changelog.md	bfcl_changelog.md	metadata	https://raw.githubusercontent.com/ShishirPatil/gorilla/main/berkeley-function-call-leaderboard/CHANGELOG.md	2026-07-30T22:19:25Z		"c07c2489a1238fbb1bfba1900e45ed6cccae38ca679affdd9869ee362259aa5d"	44359	27cbabdb95424b5b03b6b9b34eebdf99a5c5b64fc5d2964726ef42ecbac104db	N/A		2025-12-11	2026-04-12	bfcl-eval==2025.12.17	f7cf735	missing: not applicable; non-tabular file with 474 lines; malformed: not applicable
frame_audit_001_bfcl/data/raw/metadata/bfcl_result_archive_root_index.json	bfcl_result_archive_root_index.json	archive_index	https://api.github.com/repos/HuanzhiMao/BFCL-Result/contents?per_page=100	2026-07-30T22:19:26Z	Thu, 30 Jul 2026 02:03:12 GMT	"b1ff27ae1610770b567de112f02a9cd9a06f0f36"	17689	8ceee9c838bfcb3817ba4dff109c016926e5fbedfcffb601dde014b80b984637	25	_links; download_url; git_url; html_url; name; path; sha; size; type; url	2025-12-11	2026-04-12	bfcl-eval==2025.12.17	f7cf735	missing: not evaluated for JSON index; malformed: none
frame_audit_001_bfcl/data/raw/metadata/bfcl_result_archive_score_index_2025-12-16.json	bfcl_result_archive_score_index_2025-12-16.json	archive_index	https://api.github.com/repos/HuanzhiMao/BFCL-Result/contents/2025-12-16/score?per_page=100	2026-07-30T22:19:26Z	Thu, 30 Jul 2026 02:03:12 GMT	"b1ff27ae1610770b567de112f02a9cd9a06f0f36"	99708	5f7e763e7faef873d6f6e3f696fe13157c3aec5f5f0c4963febdcd826d33b2f7	115	_links; download_url; git_url; html_url; name; path; sha; size; type; url	2025-12-11	2026-04-12	bfcl-eval==2025.12.17	f7cf735	missing: not evaluated for JSON index; malformed: none
frame_audit_001_bfcl/data/raw/metadata/bfcl_result_archive_result_index_2025-12-16.json	bfcl_result_archive_result_index_2025-12-16.json	archive_index	https://api.github.com/repos/HuanzhiMao/BFCL-Result/contents/2025-12-16/result?per_page=100	2026-07-30T22:19:26Z	Thu, 30 Jul 2026 02:03:12 GMT	"b1ff27ae1610770b567de112f02a9cd9a06f0f36"	94676	18837f30b7944f3b491d2e921f9b7c5502ff539680e03be42d56aa33823532ba	109	_links; download_url; git_url; html_url; name; path; sha; size; type; url	2025-12-11	2026-04-12	bfcl-eval==2025.12.17	f7cf735	missing: not evaluated for JSON index; malformed: none