# BFCL V4 Data Inventory

## Public Methodology Sources

- Live leaderboard: `https://gorilla.cs.berkeley.edu/leaderboard`
- BFCL code and docs: `https://github.com/ShishirPatil/gorilla/tree/main/berkeley-function-call-leaderboard`
- BFCL result archive: `https://github.com/HuanzhiMao/BFCL-Result`
- BFCL V4 score-composition blog: `https://gorilla.cs.berkeley.edu/blogs/15_bfcl_v4_web_search.html`
- BFCL dataset directory: `https://github.com/ShishirPatil/gorilla/tree/main/berkeley-function-call-leaderboard/bfcl_eval/data`
- Public result archive snapshot inspected: `https://github.com/HuanzhiMao/BFCL-Result/tree/main/2025-12-16`

## Snapshot Metadata Inspected

- Leaderboard page displayed last updated: `2026-04-12`.
- Direct CSV HTTP last-modified: `Mon, 13 Apr 2026 03:20:44 GMT`.
- Direct CSV row count inspected: 109 model variants in each of the six summary CSVs.
- Public result archive top-level snapshot used for file-pattern inspection: `2025-12-16`.
- Live leaderboard advertised evaluation commit: `f7cf735`.
- Live leaderboard advertised package: `bfcl-eval==2025.12.17`.

## Direct Public Summary CSVs

| File | URL | Rows | Columns | Bytes | SHA-256 |
| --- | --- | ---: | ---: | ---: | --- |
| `data_overall.csv` | `https://gorilla.cs.berkeley.edu/data_overall.csv` | 109 | 36 | 33046 | `ede5081a24bc5ac7f9b3b09a52af6bcfcd35aa0f1709239fed979db77a8159a6` |
| `data_agentic.csv` | `https://gorilla.cs.berkeley.edu/data_agentic.csv` | 109 | 10 | 8959 | `92cc312ff1fa7935ff20687dd1075540243dc1366f52efd71e2192265919bb0d` |
| `data_format_sensitivity.csv` | `https://gorilla.cs.berkeley.edu/data_format_sensitivity.csv` | 109 | 30 | 20966 | `eef15e10a428aeb2a13beea423db9c8267af913bff67bd260cd698c9e026eab6` |
| `data_live.csv` | `https://gorilla.cs.berkeley.edu/data_live.csv` | 109 | 10 | 9334 | `8c37538602fb45a10ee15c64478af6384271b31e90961d8daa17b28472b1c970` |
| `data_multi_turn.csv` | `https://gorilla.cs.berkeley.edu/data_multi_turn.csv` | 109 | 7 | 6761 | `504d79bfaf9d67dda30f519ed3f77c335b117bd4131722a1605694a26b9e00ad` |
| `data_non_live.csv` | `https://gorilla.cs.berkeley.edu/data_non_live.csv` | 109 | 12 | 10859 | `19eec28695654ece206b343252db836127a96f18552456c1c3cf496b2dc306d0` |

Archive fallback URLs use the pattern:

`https://raw.githubusercontent.com/HuanzhiMao/BFCL-Result/main/2025-12-16/score/<CSV_FILE>`

## Score Components In `data_overall.csv`

- `Rank`
- `Overall Acc`
- `Model`
- `Model Link`
- `Total Cost ($)`
- `Latency Mean (s)`
- `Latency Standard Deviation (s)`
- `Latency 95th Percentile (s)`
- `Non-Live AST Acc`
- `Non-Live Simple AST`
- `Non-Live Multiple AST`
- `Non-Live Parallel AST`
- `Non-Live Parallel Multiple AST`
- `Live Acc`
- `Live Simple AST`
- `Live Multiple AST`
- `Live Parallel AST`
- `Live Parallel Multiple AST`
- `Multi Turn Acc`
- `Multi Turn Base`
- `Multi Turn Miss Func`
- `Multi Turn Miss Param`
- `Multi Turn Long Context`
- `Web Search Acc`
- `Web Search Base`
- `Web Search No Snippet`
- `Memory Acc`
- `Memory KV`
- `Memory Vector`
- `Memory Recursive Summarization`
- `Relevance Detection`
- `Irrelevance Detection`
- `Format Sensitivity Max Delta`
- `Format Sensitivity Standard Deviation`
- `Organization`
- `License`

## Components In `data_agentic.csv`

- `Rank`
- `Model`
- `Agentic Overall Acc`
- `Web Search Summary`
- `Web Search Base`
- `Web Search No Snippet`
- `Memory Summary`
- `Memory KV`
- `Memory Vector`
- `Memory Recursive Summarization`

## Components In `data_live.csv`

- `Rank`
- `Model`
- `Live Overall Acc`
- `AST Summary`
- `Python Simple AST`
- `Python Multiple AST`
- `Python Parallel AST`
- `Python Parallel Multiple AST`
- `Irrelevance Detection`
- `Relevance Detection`

## Components In `data_multi_turn.csv`

- `Rank`
- `Model`
- `Multi Turn Overall Acc`
- `Base`
- `Miss Func`
- `Miss Param`
- `Long Context`

## Components In `data_non_live.csv`

- `Rank`
- `Model`
- `Non-Live Overall Acc`
- `AST Summary`
- `Simple AST`
- `Python Simple AST`
- `Java Simple AST`
- `JavaScript Simple AST`
- `Multiple AST`
- `Parallel AST`
- `Parallel Multiple AST`
- `Irrelevance Detection`

## Components In `data_format_sensitivity.csv`

- `Rank`
- `Model`
- `Format Sensitivity Max Delta`
- `Format Sensitivity Standard Deviation`
- `Config ret_fmt=python&tool_call_tag=True&func_doc_fmt=python&prompt_fmt=plaintext&style=classic`
- `Config ret_fmt=python&tool_call_tag=True&func_doc_fmt=xml&prompt_fmt=plaintext&style=classic`
- `Config ret_fmt=python&tool_call_tag=True&func_doc_fmt=json&prompt_fmt=plaintext&style=classic`
- `Config ret_fmt=python&tool_call_tag=False&func_doc_fmt=python&prompt_fmt=plaintext&style=classic`
- `Config ret_fmt=python&tool_call_tag=False&func_doc_fmt=xml&prompt_fmt=plaintext&style=classic`
- `Config ret_fmt=python&tool_call_tag=False&func_doc_fmt=json&prompt_fmt=plaintext&style=classic`
- `Config ret_fmt=json&tool_call_tag=True&func_doc_fmt=python&prompt_fmt=plaintext&style=classic`
- `Config ret_fmt=json&tool_call_tag=True&func_doc_fmt=xml&prompt_fmt=plaintext&style=classic`
- `Config ret_fmt=json&tool_call_tag=True&func_doc_fmt=json&prompt_fmt=plaintext&style=classic`
- `Config ret_fmt=json&tool_call_tag=False&func_doc_fmt=python&prompt_fmt=plaintext&style=classic`
- `Config ret_fmt=json&tool_call_tag=False&func_doc_fmt=xml&prompt_fmt=plaintext&style=classic`
- `Config ret_fmt=json&tool_call_tag=False&func_doc_fmt=json&prompt_fmt=plaintext&style=classic`
- `Config ret_fmt=verbose_xml&tool_call_tag=True&func_doc_fmt=python&prompt_fmt=plaintext&style=classic`
- `Config ret_fmt=verbose_xml&tool_call_tag=True&func_doc_fmt=xml&prompt_fmt=plaintext&style=classic`
- `Config ret_fmt=verbose_xml&tool_call_tag=True&func_doc_fmt=json&prompt_fmt=plaintext&style=classic`
- `Config ret_fmt=verbose_xml&tool_call_tag=False&func_doc_fmt=python&prompt_fmt=plaintext&style=classic`
- `Config ret_fmt=verbose_xml&tool_call_tag=False&func_doc_fmt=xml&prompt_fmt=plaintext&style=classic`
- `Config ret_fmt=verbose_xml&tool_call_tag=False&func_doc_fmt=json&prompt_fmt=plaintext&style=classic`
- `Config ret_fmt=concise_xml&tool_call_tag=True&func_doc_fmt=python&prompt_fmt=plaintext&style=classic`
- `Config ret_fmt=concise_xml&tool_call_tag=True&func_doc_fmt=xml&prompt_fmt=plaintext&style=classic`
- `Config ret_fmt=concise_xml&tool_call_tag=True&func_doc_fmt=json&prompt_fmt=plaintext&style=classic`
- `Config ret_fmt=concise_xml&tool_call_tag=False&func_doc_fmt=python&prompt_fmt=plaintext&style=classic`
- `Config ret_fmt=concise_xml&tool_call_tag=False&func_doc_fmt=xml&prompt_fmt=plaintext&style=classic`
- `Config ret_fmt=concise_xml&tool_call_tag=False&func_doc_fmt=json&prompt_fmt=plaintext&style=classic`
- `Config ret_fmt=python&tool_call_tag=False&func_doc_fmt=json&prompt_fmt=markdown&style=classic`
- `Config ret_fmt=python&tool_call_tag=False&func_doc_fmt=json&prompt_fmt=plaintext&style=experimental`

## Public BFCL V4 Dataset Files

Dataset files are available in the BFCL code repository under:

`https://github.com/ShishirPatil/gorilla/tree/main/berkeley-function-call-leaderboard/bfcl_eval/data`

V4 category files visible there:

- `BFCL_v4_format_sensitivity.json`
- `BFCL_v4_irrelevance.json`
- `BFCL_v4_live_irrelevance.json`
- `BFCL_v4_live_multiple.json`
- `BFCL_v4_live_parallel.json`
- `BFCL_v4_live_parallel_multiple.json`
- `BFCL_v4_live_relevance.json`
- `BFCL_v4_live_simple.json`
- `BFCL_v4_memory.json`
- `BFCL_v4_multi_turn_base.json`
- `BFCL_v4_multi_turn_long_context.json`
- `BFCL_v4_multi_turn_miss_func.json`
- `BFCL_v4_multi_turn_miss_param.json`
- `BFCL_v4_multiple.json`
- `BFCL_v4_parallel.json`
- `BFCL_v4_parallel_multiple.json`
- `BFCL_v4_simple_java.json`
- `BFCL_v4_simple_javascript.json`
- `BFCL_v4_simple_python.json`
- `BFCL_v4_web_search.json`

Related public directories:

- `memory_prereq_conversation/`
- `multi_turn_func_doc/`
- `possible_answer/`
- `unused_datasets/`

## Downloadable Result And Score File Patterns

The public result archive states that it hosts model responses used to generate the BFCL leaderboard together with evaluation scores. REST category API keys are redacted in the archive.

The current inspected archive layout has 109 model-variant directories under each of:

- `https://github.com/HuanzhiMao/BFCL-Result/tree/main/2025-12-16/result`
- `https://github.com/HuanzhiMao/BFCL-Result/tree/main/2025-12-16/score`

Per-model raw-result files use these patterns:

- `result/<MODEL>/agentic/BFCL_v4_web_search_base_result.json`
- `result/<MODEL>/agentic/BFCL_v4_web_search_no_snippet_result.json`
- `result/<MODEL>/agentic/memory/kv/BFCL_v4_memory_kv_prereq_result.json`
- `result/<MODEL>/agentic/memory/kv/BFCL_v4_memory_kv_result.json`
- `result/<MODEL>/agentic/memory/vector/BFCL_v4_memory_vector_prereq_result.json`
- `result/<MODEL>/agentic/memory/vector/BFCL_v4_memory_vector_result.json`
- `result/<MODEL>/agentic/memory/rec_sum/BFCL_v4_memory_rec_sum_prereq_result.json`
- `result/<MODEL>/agentic/memory/rec_sum/BFCL_v4_memory_rec_sum_result.json`
- `result/<MODEL>/agentic/memory/<BACKEND>/memory_snapshot/customer_final.json`
- `result/<MODEL>/agentic/memory/<BACKEND>/memory_snapshot/finance_final.json`
- `result/<MODEL>/agentic/memory/<BACKEND>/memory_snapshot/healthcare_final.json`
- `result/<MODEL>/agentic/memory/<BACKEND>/memory_snapshot/notetaker_final.json`
- `result/<MODEL>/agentic/memory/<BACKEND>/memory_snapshot/student_final.json`
- `result/<MODEL>/agentic/memory/<BACKEND>/memory_snapshot/prereq_checkpoints/*.json`
- `result/<MODEL>/live/BFCL_v4_live_simple_result.json`
- `result/<MODEL>/live/BFCL_v4_live_multiple_result.json`
- `result/<MODEL>/live/BFCL_v4_live_parallel_result.json`
- `result/<MODEL>/live/BFCL_v4_live_parallel_multiple_result.json`
- `result/<MODEL>/live/BFCL_v4_live_irrelevance_result.json`
- `result/<MODEL>/live/BFCL_v4_live_relevance_result.json`
- `result/<MODEL>/multi_turn/BFCL_v4_multi_turn_base_result.json`
- `result/<MODEL>/multi_turn/BFCL_v4_multi_turn_miss_func_result.json`
- `result/<MODEL>/multi_turn/BFCL_v4_multi_turn_miss_param_result.json`
- `result/<MODEL>/multi_turn/BFCL_v4_multi_turn_long_context_result.json`
- `result/<MODEL>/non_live/BFCL_v4_simple_python_result.json`
- `result/<MODEL>/non_live/BFCL_v4_simple_java_result.json`
- `result/<MODEL>/non_live/BFCL_v4_simple_javascript_result.json`
- `result/<MODEL>/non_live/BFCL_v4_multiple_result.json`
- `result/<MODEL>/non_live/BFCL_v4_parallel_result.json`
- `result/<MODEL>/non_live/BFCL_v4_parallel_multiple_result.json`
- `result/<MODEL>/non_live/BFCL_v4_irrelevance_result.json`
- `result/<MODEL>/format_sensitivity/BFCL_v4_format_sensitivity_result.json` for prompt/non-FC variants where present.

Per-model score files use parallel patterns:

- `score/<MODEL>/agentic/BFCL_v4_web_search_base_score.json`
- `score/<MODEL>/agentic/BFCL_v4_web_search_no_snippet_score.json`
- `score/<MODEL>/agentic/memory/kv/BFCL_v4_memory_kv_score.json`
- `score/<MODEL>/agentic/memory/vector/BFCL_v4_memory_vector_score.json`
- `score/<MODEL>/agentic/memory/rec_sum/BFCL_v4_memory_rec_sum_score.json`
- `score/<MODEL>/live/BFCL_v4_live_simple_score.json`
- `score/<MODEL>/live/BFCL_v4_live_multiple_score.json`
- `score/<MODEL>/live/BFCL_v4_live_parallel_score.json`
- `score/<MODEL>/live/BFCL_v4_live_parallel_multiple_score.json`
- `score/<MODEL>/live/BFCL_v4_live_irrelevance_score.json`
- `score/<MODEL>/live/BFCL_v4_live_relevance_score.json`
- `score/<MODEL>/multi_turn/BFCL_v4_multi_turn_base_score.json`
- `score/<MODEL>/multi_turn/BFCL_v4_multi_turn_miss_func_score.json`
- `score/<MODEL>/multi_turn/BFCL_v4_multi_turn_miss_param_score.json`
- `score/<MODEL>/multi_turn/BFCL_v4_multi_turn_long_context_score.json`
- `score/<MODEL>/non_live/BFCL_v4_simple_python_score.json`
- `score/<MODEL>/non_live/BFCL_v4_simple_java_score.json`
- `score/<MODEL>/non_live/BFCL_v4_simple_javascript_score.json`
- `score/<MODEL>/non_live/BFCL_v4_multiple_score.json`
- `score/<MODEL>/non_live/BFCL_v4_parallel_score.json`
- `score/<MODEL>/non_live/BFCL_v4_parallel_multiple_score.json`
- `score/<MODEL>/non_live/BFCL_v4_irrelevance_score.json`
- `score/<MODEL>/format_sensitivity/BFCL_v4_format_sensitivity_score.json` for prompt/non-FC variants where present.

## Model Variant Directories Inspected

The following 109 model-variant directories were visible in both `score/` and `result/` for the inspected public archive snapshot:

`BitAgent_BitAgent-Bounty-8B`, `DeepSeek-V3.2-Exp-FC`, `DeepSeek-V3.2-Exp-thinking`, `MadeAgents_Hammer2.1-0.5b`, `MadeAgents_Hammer2.1-1.5b`, `MadeAgents_Hammer2.1-3b`, `MadeAgents_Hammer2.1-7b`, `Nanbeige3.5-Pro-Thinking-FC`, `Nanbeige_Nanbeige4-3B-Thinking-2511`, `Qwen_Qwen3-32B-FC`, `Qwen_Qwen3-32B`, `Qwen_Qwen3-4B-Instruct-2507-FC`, `Qwen_Qwen3-4B-Instruct-2507`, `Qwen_Qwen3-8B-FC`, `Qwen_Qwen3-8B`, `Salesforce_Llama-xLAM-2-70b-fc-r`, `Salesforce_Llama-xLAM-2-8b-fc-r`, `Salesforce_xLAM-2-1b-fc-r`, `Salesforce_xLAM-2-32b-fc-r`, `Salesforce_xLAM-2-3b-fc-r`, `Team-ACE_ToolACE-2-8B`, `claude-haiku-4-5-20251001-FC`, `claude-haiku-4-5-20251001`, `claude-opus-4-5-20251101-FC`, `claude-opus-4-5-20251101`, `claude-sonnet-4-5-20250929-FC`, `claude-sonnet-4-5-20250929`, `command-a-03-2025-FC`, `command-a-reasoning-08-2025-FC`, `command-r7b-12-2024-FC`, `gemini-2.5-flash-FC`, `gemini-2.5-flash-lite-FC`, `gemini-2.5-flash-lite`, `gemini-2.5-flash`, `gemini-3-pro-preview-FC`, `gemini-3-pro-preview`, `glm-4.6-FC`, `google_gemma-3-12b-it`, `google_gemma-3-1b-it`, `google_gemma-3-27b-it`, `google_gemma-3-4b-it`, `gpt-4.1-2025-04-14-FC`, `gpt-4.1-2025-04-14`, `gpt-4.1-mini-2025-04-14-FC`, `gpt-4.1-mini-2025-04-14`, `gpt-4.1-nano-2025-04-14-FC`, `gpt-4.1-nano-2025-04-14`, `gpt-5-mini-2025-08-07-FC`, `gpt-5-mini-2025-08-07`, `gpt-5-nano-2025-08-07-FC`, `gpt-5-nano-2025-08-07`, `gpt-5.2-2025-12-11-FC`, `gpt-5.2-2025-12-11`, `grok-4-0709-FC`, `grok-4-0709`, `grok-4-1-fast-non-reasoning-FC`, `grok-4-1-fast-reasoning-FC`, `ibm-granite_granite-20b-functioncalling`, `ibm-granite_granite-3.1-8b-instruct`, `ibm-granite_granite-3.2-8b-instruct`, `ibm-granite_granite-4.0-350m`, `katanemo_Arch-Agent-1.5B`, `katanemo_Arch-Agent-32B`, `katanemo_Arch-Agent-3B`, `kimi-k2-0905-preview-FC`, `meta-llama_Llama-3.1-8B-Instruct`, `meta-llama_Llama-3.2-1B-Instruct-FC`, `meta-llama_Llama-3.2-3B-Instruct-FC`, `meta-llama_Llama-3.3-70B-Instruct-FC`, `meta-llama_Llama-4-Maverick-17B-128E-Instruct-FP8-FC`, `meta-llama_Llama-4-Scout-17B-16E-Instruct-FC`, `microsoft_phi-4`, `mistral-large-2411-FC`, `mistral-large-2411`, `mistral-medium-2505-FC`, `mistral-medium-2505`, `mistral-small-2506-FC`, `mistral-small-2506`, `mistralai_Ministral-8B-Instruct-2410`, `nova-2-lite-v1.0`, `nova-micro-v1.0`, `nova-pro-v1.0`, `nvidia_llama-3.1-nemotron-ultra-253b-v1`, `o3-2025-04-16-FC`, `o3-2025-04-16`, `o4-mini-2025-04-16-FC`, `o4-mini-2025-04-16`, `open-mistral-nemo-2407-FC`, `open-mistral-nemo-2407`, `openbmb_MiniCPM3-4B-FC`, `openbmb_MiniCPM3-4B`, `palmyra-x-004`, `phronetic-ai_RZN-T`, `qwen3-0.6b-FC`, `qwen3-0.6b`, `qwen3-1.7b-FC`, `qwen3-14b-FC`, `qwen3-14b`, `qwen3-235b-a22b-instruct-2507-FC`, `qwen3-235b-a22b-instruct-2507`, `qwen3-30b-a3b-instruct-2507-FC`, `qwen3-30b-a3b-instruct-2507`, `speakleash_Bielik-11B-v2.3-Instruct`, `tiiuae_Falcon3-10B-Instruct-FC`, `tiiuae_Falcon3-1B-Instruct-FC`, `tiiuae_Falcon3-3B-Instruct-FC`, `tiiuae_Falcon3-7B-Instruct-FC`, `uiuc-convai_CoALM-70B`, `uiuc-convai_CoALM-8B`.

## Aggregation And Missing-Data Notes

- The live leaderboard states that cost is the estimated cost for the entire benchmark and latency is measured in seconds.
- The live leaderboard states that format sensitivity test cases are only supported for prompt/non-FC models.
- The BFCL README states that unevaluated categories appear as `N/A`, and summary columns treat unevaluated categories as 0.
- The V4 blog and changelog describe a revised score composition favoring agentic and multi-turn tasks.
- The live page wording also says overall accuracy is an unweighted average of subcategories. This should be reconciled against the V4 blog, changelog, and code before interpreting any rank reversal.

## Likely Data Gaps

- Exact code path used to compute the live `Overall Acc` must be verified against the public code.
- The live CSVs and result archive should be checked for exact byte/hash equality at audit start.
- Some model variants may lack `format_sensitivity` per-model JSON files.
- Some rows may include `N/A` for categories that are not supported or not evaluated.
- Cost is benchmark-level estimated cost, not buyer production cost.
- Latency is benchmark-measured latency, not necessarily buyer deployment latency.