# BFCL V4 Decision Profiles

These profiles are pre-registered before analysis. They are intended to reflect plausible procurement or deployment decisions, not to manufacture a rank reversal.

All profiles use BFCL's published aggregate as the baseline comparison. Each profile excludes model variants with missing required categories unless stated otherwise.

## A. High-Volume Low-Risk Automation

Buyer example: product team deploying routine CRM, ticket triage, or back-office tool calls where individual failures are reviewable and low consequence, but volume is high.

Required categories:

- Non-live AST: simple, multiple, parallel, parallel multiple.
- Live AST: simple, multiple, parallel, parallel multiple.
- Relevance and irrelevance detection.
- Cost and latency fields.

Category weights:

- Single-turn tool-call accuracy: 40%.
- Relevance/irrelevance behavior: 20%.
- Cost efficiency: 25%.
- Latency: 15%.

Hard disqualification constraints:

- Required single-turn categories cannot be `N/A`.
- `Irrelevance Detection` below 70% disqualifies the model.
- `Latency 95th Percentile (s)` above 20 seconds disqualifies the model.
- Missing `Total Cost ($)` disqualifies the model.

Cost treatment:

- Compute cost per expected successful call from `Total Cost ($)` and selected success rate.
- Cost is a major decision factor because high volume magnifies small per-call differences.

Latency treatment:

- Use `Latency 95th Percentile (s)` as the primary latency risk.
- Apply a penalty above a 10-second target and a hard disqualification above 20 seconds.

Rationale:

In low-risk automation, the buyer can tolerate occasional errors with review, but cannot tolerate runaway inference cost or sluggish workflow throughput.

## B. Regulated Or High-Consequence Workflow

Buyer example: healthcare administration, financial operations, legal operations, or compliance workflow where an incorrect tool call can create regulatory, financial, or safety exposure.

Required categories:

- Relevance Detection.
- Irrelevance Detection.
- Multi-turn base, missing-function, missing-parameter, and long-context categories.
- Live and non-live AST summary categories.
- Format sensitivity max delta and standard deviation if present for the variant.

Category weights:

- Hallucination / abstention behavior: 30%.
- Multi-turn correctness: 25%.
- Agentic web-search and memory behavior: 15%.
- Single-turn tool-call accuracy: 15%.
- Format robustness: 10%.
- Cost and latency: 5%.

Hard disqualification constraints:

- `Irrelevance Detection` below 90% disqualifies the model.
- `Relevance Detection` below 85% disqualifies the model.
- `Multi Turn Miss Func` below 70% disqualifies the model.
- Missing required safety-related categories disqualifies the model.

Cost treatment:

- Cost is secondary unless two models are operationally equivalent.
- Excess cost is acceptable if it materially lowers false tool calls or missed abstentions.

Latency treatment:

- Use latency as a service-level risk, not the primary selector.
- Disqualify only if `Latency 95th Percentile (s)` is above 60 seconds.

Rationale:

The dominant financial consequence is not token spend. It is failed workflow cost, human escalation burden, compliance exposure, and procurement risk from preventable tool misuse.

## C. Multi-Turn Enterprise Agent

Buyer example: enterprise agent platform handling multi-step workflows such as account changes, procurement requests, technical support, workflow orchestration, or knowledge-work automation.

Required categories:

- Multi-turn base.
- Multi-turn missing function.
- Multi-turn missing parameter.
- Multi-turn long context.
- Agentic memory: key-value, vector, recursive summarization.
- Web search base and no-snippet.
- Relevance and irrelevance detection.

Category weights:

- Multi-turn performance: 35%.
- Agentic memory performance: 25%.
- Web-search performance: 15%.
- Relevance/irrelevance behavior: 10%.
- Latency: 10%.
- Cost: 5%.

Hard disqualification constraints:

- `Multi Turn Acc` below 60% disqualifies the model.
- Any required multi-turn subcategory missing disqualifies the model.
- Any required memory backend missing disqualifies the model.
- `Latency 95th Percentile (s)` above 60 seconds disqualifies the model.

Cost treatment:

- Cost is included but secondary to successful completion of multi-step workflows.
- Report cost per expected completed workflow.

Latency treatment:

- Penalize slow tail latency because multi-turn workflows compound delay across steps.
- Use a 30-second target and 60-second hard limit for the 95th percentile.

Rationale:

For enterprise agents, the model that wins single-turn function calling may not be the model that survives long-context, missing-parameter, missing-function, and persistent-memory workflows.

## D. Cost-Constrained General Tool User

Buyer example: startup, internal-tools team, or small business using a general tool-calling model across heterogeneous but non-regulated workflows.

Required categories:

- Overall accuracy.
- Non-live and live AST summaries.
- Multi-turn overall.
- Relevance and irrelevance detection.
- Cost and latency fields.

Category weights:

- Task success across general BFCL components: 35%.
- Cost per expected success: 35%.
- Latency-adjusted success: 15%.
- Relevance/irrelevance behavior: 10%.
- Format robustness: 5%.

Hard disqualification constraints:

- Missing cost field disqualifies the model.
- Missing `Overall Acc` disqualifies the model.
- `Irrelevance Detection` below 70% disqualifies the model.
- `Latency 95th Percentile (s)` above 45 seconds disqualifies the model.

Cost treatment:

- Normalize total benchmark cost to a lower-is-better index.
- Use cost per expected success as the main cost metric, not raw cost alone.

Latency treatment:

- Apply a moderate penalty for high tail latency.
- Low latency can break ties among similarly cheap models.

Rationale:

This buyer is not seeking the absolute best model. The economic question is whether a cheaper model is good enough to reduce spend without creating a review burden that erases savings.

## Shared Rules

- These weights are locked before analysis.
- If a buyer profile produces a different winner, the dossier must explain the operational constraint that caused the change.
- If a profile's winner changes only because of arbitrary weight movement, it does not count as a pass.
- If all profiles select the published BFCL winner, the audit can still be valuable as a negative result.