# BFCL Calculation And Materiality Audit

Status: internal calculation audit only. No source data, buyer-profile definitions, or thresholds were changed. No API calls were made.

## 1. Sensitivity Summary Correction

The previous profile report contained contradictory sensitivity lines such as `no eligible model; eligible 109; disqualified 0`. The underlying `bfcl_sensitivity_results.csv` had rank values in a fallback `rank` column but an empty `sensitivity_rank` column, so the report lookup missed eligible winners.

- Filled `sensitivity_rank` cells from the existing `rank` column: 446.
- Rewrote the `## Sensitivity Tests` section of `BFCL_PROFILE_RESULTS_REPORT.md` using the corrected rank field.
- No frozen source data or buyer-profile definitions were changed.

## 2. Calculation Verification

Each Profile A-D score, eligibility decision, disqualification reason, and rank was recomputed from `bfcl_v4_baseline.csv` using the frozen profile definitions.

- Profile counts: A: 52 eligible/57 disqualified; B: 0 eligible/109 disqualified; C: 7 eligible/102 disqualified; D: 64 eligible/45 disqualified.
- Eligibility mismatches: 0.
- Disqualification-reason mismatches: 0.
- Rank mismatches: 0.
- Maximum profile score difference versus stored CSV: 0.000000499.

## 3. Documented BFCL Data

- Published aggregate comparator: `Claude-Opus-4-5-20251101 (FC)` at official rank 1.
- Model variants in processed baseline: 109.
- Used documented BFCL fields already frozen in `bfcl_v4_baseline.csv`: overall score, component scores, cost, latency, organization, license, model link, leaderboard update, package, commit, and archive snapshot.

## 4. Pre-Registered Buyer Assumptions

- Profile A emphasizes single-turn tool-call accuracy, relevance/irrelevance behavior, cost efficiency, and latency.
- Profile B applies strict regulated-workflow thresholds for hallucination/relevance, multi-turn missing-function behavior, and latency.
- Profile C emphasizes multi-turn, memory, web search, reliability, latency, and cost.
- Profile D emphasizes general task success, cost per expected success, latency-adjusted success, relevance/irrelevance behavior, and optional format robustness.
- Optional format sensitivity was only used where BFCL provided fields for that model variant; absent optional format terms were omitted and remaining registered weights were renormalized.

## 5. Profile A And D Score Decomposition

| Profile | Role | Model | Published Rank | Profile Score | Benchmark Capability | Cost | Latency | Reliability/Workflow | Format | Raw Terms |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| A | profile_selected_model | Qwen3-14B (Prompt) | 47 | 90.93 | 34.14 | 24.94 | 15.00 | 16.84 | 0 | single_turn=85.361; relevance_behavior=84.215; cost_efficiency=99.768; latency=100.000 |
| A | published_bfcl_winner | Claude-Opus-4-5-20251101 (FC) | 1 | 81.70 | 34.07 | 17.91 | 15.00 | 14.72 | 0 | single_turn=85.177; relevance_behavior=73.610; cost_efficiency=71.631; latency=100.000 |
| A | second_ranked_profile_model | Qwen3-235B-A22B-Instruct-2507 (Prompt) | 23 | 90.76 | 33.71 | 24.79 | 15.00 | 17.26 | 0 | single_turn=84.267; relevance_behavior=86.320; cost_efficiency=99.172; latency=100.000 |
| D | profile_selected_model | GLM-4.6 (FC thinking) | 4 | 85.58 | 28.45 | 36.52 | 12.19 | 8.42 | 0 | task_success=77.210; cost_efficiency=99.127; latency_adjusted_success=77.210; relevance_behavior=79.980 |
| D | published_bfcl_winner | Claude-Opus-4-5-20251101 (FC) | 1 | 78.68 | 28.94 | 29.58 | 12.40 | 7.75 | 0 | task_success=78.555; cost_efficiency=80.301; latency_adjusted_success=78.555; relevance_behavior=73.610 |
| D | second_ranked_profile_model | xLAM-2-32b-fc-r (FC) | 18 | 82.93 | 26.64 | 36.37 | 11.42 | 8.50 | 0 | task_success=72.315; cost_efficiency=98.714; latency_adjusted_success=72.315; relevance_behavior=80.740 |

## 6. Leave-One-Factor-Out Checks

These are diagnostic checks only. They remove one registered factor at a time and renormalize the remaining registered factors; they do not redefine or replace the frozen buyer profiles.

| Profile | Check | Selected Model | Published Rank | Selected Score | Published Winner Score | Flip Survives |
| --- | --- | --- | --- | --- | --- | --- |
| A | full_registered_profile | Qwen3-14B (Prompt) | 47 | 90.93 | 81.70 | Yes |
| A | remove_cost | GPT-4.1-2025-04-14 (Prompt) | 45 | 89.83 | 85.06 | Yes |
| A | remove_latency | Qwen3-14B (Prompt) | 47 | 89.33 | 78.47 | Yes |
| A | remove_single_turn_capability | Mistral-small-2506 (FC) | 51 | 95.17 | 79.38 | Yes |
| A | remove_relevance_reliability | Qwen3-14B (Prompt) | 47 | 92.61 | 83.72 | Yes |
| A | capability_only_weighting | GPT-4.1-2025-04-14 (Prompt) | 45 | 87.29 | 81.32 | Yes |
| D | full_registered_profile | GLM-4.6 (FC thinking) | 4 | 85.58 | 78.68 | Yes |
| D | remove_cost | Claude-Opus-4-5-20251101 (FC) | 1 | 77.73 | 77.73 | No |
| D | remove_latency | GLM-4.6 (FC thinking) | 4 | 87.14 | 78.70 | Yes |
| D | remove_task_success_capability | GLM-4.6 (FC thinking) | 4 | 90.46 | 78.75 | Yes |
| D | remove_relevance_reliability | GLM-4.6 (FC thinking) | 4 | 86.23 | 79.27 | Yes |
| D | remove_optional_format_sensitivity | GLM-4.6 (FC thinking) | 4 | 85.58 | 78.68 | Yes |
| D | capability_only_weighting | Gemini-3-Pro-Preview (Prompt) | 3 | 79.14 | 77.46 | Yes |

Profile A: the decision flip survives without cost, without latency, and under capability-only weighting. The exact selected model changes under some diagnostics, but the published aggregate winner is not selected once the workflow frame is narrowed to low-risk high-volume single-turn automation.

Profile D: the full-profile selected model does not survive removal of cost; removing cost returns the published aggregate winner. The D flip is therefore materially cost-dependent.

## 7. BFCL Cost Comparability Limitations

- Prompted versus native function-calling variants may not carry equivalent tool-schema serialization, parser behavior, or token accounting.
- Reasoning and non-reasoning variants may differ in hidden reasoning effort, output length, latency, and provider billing behavior; the frozen baseline does not expose reasoning-token controls.
- Provider cost fields are benchmark-level estimates, not buyer production prices; provider discounts, caching, routing, regional deployment, and rate-limit behavior are not represented.
- Open-weight models and hosted proprietary models may reflect different deployment economics; BFCL `Total Cost ($)` is useful for benchmark comparison but not a complete total-cost-of-ownership model.
- Model versions and provider pricing can change after the leaderboard snapshot; the frozen metadata records leaderboard update and archive snapshot but does not verify live current availability.
- These limitations especially affect Profile D, where the selected model is cost-dependent.

## 8. Profile B Threshold Audit

Individual failure counts across 109 models:
- Irrelevance Detection below 90%: 94
- Relevance Detection below 85%: 69
- Multi Turn Miss Func below 70%: 106
- Latency 95th Percentile above 60 seconds: 25
- missing required safety-related fields: 0

Failure combinations:
- 44: Irrelevance Detection below 90% + Multi Turn Miss Func below 70% + Relevance Detection below 85%
- 27: Irrelevance Detection below 90% + Multi Turn Miss Func below 70%
- 12: Irrelevance Detection below 90% + Latency 95th Percentile above 60 seconds + Multi Turn Miss Func below 70%
- 12: Multi Turn Miss Func below 70% + Relevance Detection below 85%
- 8: Irrelevance Detection below 90% + Latency 95th Percentile above 60 seconds + Multi Turn Miss Func below 70% + Relevance Detection below 85%
- 3: Latency 95th Percentile above 60 seconds + Multi Turn Miss Func below 70% + Relevance Detection below 85%
- 1: Irrelevance Detection below 90% + Relevance Detection below 85%
- 1: Irrelevance Detection below 90% + Latency 95th Percentile above 60 seconds + Relevance Detection below 85%
- 1: Irrelevance Detection below 90% + Latency 95th Percentile above 60 seconds

Profile B's zero-eligible result is primarily driven by the `Multi Turn Miss Func below 70%` threshold, which individually fails 106 of 109 models. The remaining models that pass that threshold still fail one or more hallucination/relevance or latency thresholds, so the zero-eligible result is caused by the conjunction, with one dominant bottleneck.

## 9. Procurement Plausibility From Frozen Metadata

| Profile | Model | Organization | License | Model Link | Model Date | Leaderboard Updated | BFCL Package | Commit | Archive | Plausibility |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| A | Qwen3-14B (Prompt) | Qwen | apache-2.0 | yes | not encoded in model identifier | 2026-04-12 | bfcl-eval==2025.12.17 | f7cf735 | 2025-12-16 | plausible from frozen BFCL metadata; live availability not rechecked |
| C | Claude-Opus-4-5-20251101 (FC) | Anthropic | Proprietary | yes | 2025-11-01 | 2026-04-12 | bfcl-eval==2025.12.17 | f7cf735 | 2025-12-16 | plausible from frozen BFCL metadata; live availability not rechecked |
| D | GLM-4.6 (FC thinking) | Zhipu AI | MIT | yes | not encoded in model identifier | 2026-04-12 | bfcl-eval==2025.12.17 | f7cf735 | 2025-12-16 | plausible from frozen BFCL metadata; live availability not rechecked |

The profile-selected models remain plausible candidates only in the limited sense that they are present in the frozen BFCL snapshot with model links, organization/license fields, and reproducibility metadata. No live availability check was performed.

## 10. Decision-Flip Materiality Table

| Profile | Selected Model | Survives Without Cost | Survives Without Latency | Capability-Only Flip | Operational Materiality | Questionable Input Dependence |
| --- | --- | --- | --- | --- | --- | --- |
| A | Qwen3-14B (Prompt) | Yes | Yes | Yes | Yes: profile gap vs published winner is about 9.23 points and capability-only still rejects the aggregate winner. | No for the aggregate-vs-profile flip; exact selected model changes when cost is removed. |
| B | none | N/A | N/A | N/A | No eligible model; threshold audit only. | Conjunction is very strict; dominated by Multi Turn Miss Func threshold. |
| C | Claude-Opus-4-5-20251101 (FC) | N/A | N/A | N/A | No flip from published aggregate winner. | No decision flip to validate. |
| D | GLM-4.6 (FC thinking) | No | Yes | Yes | Conditional: profile gap is about 6.90 points, but the exact full-profile winner depends on cost. | Yes: the full-profile winner is cost-dependent and BFCL cost comparability is limited. |

## 11. Interpretation

Profile A provides the strongest verified frame-audit evidence: a defensible workflow frame selects a different model than BFCL's published aggregate winner, and that difference survives removal of cost and latency. Profile D adds supporting but weaker evidence because its full-profile winner is cost-dependent and BFCL cost comparability is limited. Profile B is too strict to produce an eligible model, and Profile C does not flip.

VERIFIED FRAME AUDIT PASS