AIHubMix Fidelitas Standard v2.0

AIHubMix Fidelitas measures the delivery of the same model by different model providers, evaluating task outcome correctness, capability availability, response performance, and first-request completion separately. The overall score and the four dimension scores are presented together to compare the service performance of the same model under specified test conditions. This page uses AIHubMix Fidelitas Standard v2.0, with project version AF v2.0. The primary score uses the functionality support and cache measurement results configured according to provider documentation; interface compatibility and cache results under the default configuration are reported separately as diagnostic metrics.

Scope and how to read the scores

Score item	Symbol	Weight	What is measured
Answer quality	Q (Quality)	35%	Correctness of task outcomes for tool calling, knowledge and reasoning, and long-document comprehension
Functionality	F (Functionality)	25%	Availability of streaming, tools, structured output, parameters, context, error responses, and usage metering
Response speed	P (Performance)	20%	Time to first answer content, generation rate, concurrent valid throughput, and effective cache hits
Service reliability	R (Reliability)	20%	Proportion of a fixed request set that completes validly on first execution

The unit of scoring is the model provider × model version, that is, the service of a given model version provided by a given model provider. Each object forms one set of four dimension scores and an overall score under the specified evaluation batch and configuration. Thinking mode, request parameters, account service tier, test region, and observation period are measurement conditions and are recorded with the results. Results under different conditions are labeled separately and are not merged directly. When the model version, region, or service tier cannot be verified, the corresponding unverified status is noted, not presumed confirmed. Evaluations are executed through the AIHubMix API, and results describe the actual observed performance under that access method. The official reference service is also measured through the AIHubMix API and is not reported as a measurement made through an independent direct connection to the official API. All members of a comparison group use the same task set, rounds, timeout, request budget, failure scoring, and statistical method. Required request fields, streaming structure, and usage fields specified by provider documentation are handled by the access adapter layer (adapter), keeping the measurement target consistent. This method is not intended to rank intelligence across models, and model provider identity does not confer a scoring advantage.

Score states

Status	Meaning
Final	Test registration, sampling, grading, and evidence verification have been completed under the applicable standard
Reference result (PROVISIONAL)	A result can be computed, but sampling, grading, adaptation registration, or evidence is subject to applicable limitations
Unavailable (UNAVAILABLE)	Required grading, denominator, or reference is missing; no numeric value is assigned
Not comparable	Model configuration, observation conditions, or result views are inconsistent, so the result is excluded from like-for-like comparison

Missing values and scores of 0 are handled separately. When a service explicitly fails or does not provide a required capability, it is scored as a failure under the corresponding rule; when a measurement is missing, the denominator is unknown, or evidence is insufficient, the corresponding status is retained. If any dimension is unavailable, the overall score is also marked unavailable, and its weight is not redistributed to other dimensions.

Interpreting results

Raw metrics are converted to absolute performance A by the scoring formula and, after composition with the relative reference and evidence, produce the dimension composite score D. The overall score uses D and does not directly weight raw correctness rates, functionality pass rates, or service success rates.
Scores reflect performance under a specific task set, load, region, configuration, and observation window; applicable conditions are recorded with the results.
Display precision does not indicate statistical discriminative power. Answer quality differences are interpreted together with paired statistical intervals; decimal differences alone cannot establish superiority.
The overall score is a summary index; it is not a certification and has no confidence interval. Statistical conclusions rest on the evidence for each specific dimension.
Service reliability reports the number of first-attempt successes, the number of planned opportunities, and a two-sided 95% interval, which cannot be interpreted directly as a long-term SLA.

If review confirms a model identity mismatch, silent input truncation, sensitive content leakage, severe protocol corruption, or a billing anomaly, the result is separately marked “Failed admission,” and reproduction evidence and usage limitations are retained. Dimension scores and the overall score do not offset this limitation.

Overall score and dimension composition

Overall score formula

The overall score is calculated by weighting the four dimension composite scores. D_Q, D_F, D_P, and D_R denote the composite scores for answer quality, functionality, response speed, and service reliability, respectively.

Overall score

Overall = 0.35 × D_Q + 0.25 × D_F + 0.20 × D_P + 0.20 × D_R

All four dimension composite scores use a 0–100 scale. The overall score summarizes the overall result, and the four dimension scores identify specific differences.

Three-layer composition

Each dimension composite score D is computed by weighting absolute performance A, performance relative to the official reference B, and evidence completeness C.

Dimension score

D = 0.65 × A + 0.20 × B + 0.15 × C

Input layer	Weight	Definition
A: Absolute performance	65%	Actual measurement results obtained under the specified test and statistical methods
B: Relative to the official reference	20%	Difference in absolute performance for the same dimension relative to the official reference service
C: Evidence completeness	15%	Degree to which sample coverage, observation consistency, and metering evidence support this result

A, B, and C are computation-layer symbols and do not denote access grades. Absolute performance uses the specified formula; the fastest or highest-scoring service in the batch is not set to 100, and no renormalization is performed by batch. The prefix before the underscore denotes the computation layer, and the suffix denotes the scoring dimension. For example, A_F is the functionality score at the absolute performance layer, and D_F is the functionality score after three-layer composition. The primary score and diagnostic values also use the following configuration suffixes:

Symbol	Full English name	Meaning and use
`F_native`	Native Functionality Score	Functionality score obtained after configuration and parsing according to provider documentation, used as `A_F` in the primary score
`F_strict`	Strict Interface Compatibility Score	Strict compatibility score obtained by checking directly against the AIHubMix interface contract, reported as a separate diagnostic
`K`	Cache Hit Score	Effective cache hit subscore, included in the absolute performance of response speed
`K_native`	Native Cache Hit Score	Cache subscore measured with the configuration required by provider documentation, used in the primary score
`K_default`	Default-Configuration Cache Hit Score	Cache subscore measured uniformly without session affinity configuration, reported as a separate diagnostic

native means configured according to provider documentation, strict means strict interface checking, and default means the default configuration defined in this page. K is the cache symbol in this method and does not denote Quality. The primary score consistently uses F_native and K_native, and diagnostic values are not mixed with the primary score.

Relative to the official reference B

A_official denotes the absolute performance score of the official reference service for the current dimension; its relative reference score B_official is fixed at 50.

Relative reference layer

B_official = 50
B = 50 + 50 × clip((A − A_official) / δ, −1, 1)

clip(x, −1, 1) is a clipping function that constrains x to the range −1 to 1. The difference scale parameter δ is defined by dimension:

Dimension	δ
Answer quality Q	2
Functionality F	15
Response speed P	10
Service reliability R	2

Answer quality uses a 5,000-iteration paired-task cluster bootstrap to estimate the difference interval relative to the official reference. When the two-sided 95% interval includes 0, B_Q = 50; when the interval excludes 0, the mapping above applies. The official reference service independently undergoes absolute performance measurement; its own B is fixed at 50, with no full score assumed. When configurations, timeouts, periods, or result views are not comparable, no presumed value is entered in the relative layer.

Evidence completeness C

C consists of the following five items, each worth up to 20 points, for a total of 100. Each item is assigned 20, 10, or 0 points according to the degree of coverage and evidence in the records:

Item	What is verified
Sample coverage	Degree to which actual valid records match the planned sample size
Window coverage	Completeness of the required periods or answer quality collection segments
Usage completeness	Coverage and verifiability of the raw `usage`
Denominator evidence	Reviewability of the denominators for question grading, functionality determination, and cache metering
Protocol consistency	Consistency of request configuration, record attribution, and observation protocol

C measures measurement evidence and does not directly measure model capability. Each assignment is linked to evidence records, and result status is retained independently; the evidence-layer score does not replace required data validity verification.

Answer quality

Answer quality Q measures the proportion of all valid task instances that are correct, based on reference answers and the specified grading protocol.

Task sets and grading

Track	What is measured	Questions × rounds	Planned instances	Grading method
BFCL	Function selection, argument filling, parallel calls, irrelevant-call suppression, and a three-stage tool chain	500 × 3	1,500	Python-only JSON matching adaptation based on the official AST checker of the specified version
HLE	High-difficulty text knowledge and reasoning questions stratified by subject	250 × 10	2,500	Answer equivalence judge; the judge model, prompt, and version are recorded
AA-LCR	Multi-document synthesis QA of about 74,000 to 114,000 tokens	25 × 10	250	Designated documents and reference answers, graded by answer equivalence
Total	775 distinct questions	Executed at the rounds specified for each track	4,250	Aggregated over all valid instances

Absolute performance for answer quality

A_Q = 100 × correct instances / valid scored instances

Full planned denominator = 1,500 + 2,500 + 250 = 4,250

The three tracks are aggregated by instance count, not by an equal-weight average of per-track correctness rates. Multi-round tool tasks may correspond to multiple API requests; task instances and requests are counted separately.

This task set is a derived task set of AF Suite. It is not an exact subset of the official Artificial Analysis set and does not reproduce its leaderboard results. The 50 BFCL multi-round tasks are constructed from real questions and are defined separately from the BFCL official stateful multi-round execution environment. Sources and versions are listed in Dataset sources.

Comparison conditions and missing-data handling

Task correctness is determined against reference answers; official reference responses are not used as ground truth.
All parties use the same controllable budget, sampling count, and semantically equivalent parameters. When thinking mode, tool templates, or effort cannot be aligned, the result is marked “Configuration not comparable.”
Service failures, timeouts, refusals, and budget exhaustion are counted as task failures in the denominator.
Records with invalid judging or without a valid determination are labeled separately, excluded from the computable denominator, and retain the reference result status.
Question filtering rules are specified in test registration, and reports list planned counts, filtered counts, and actual scored counts.
Repeated rounds belong to the same question cluster and are not treated as additional independent questions.

Relative differences use a per-track paired-task cluster bootstrap with 5,000 repetitions. All parties use the same question sampling index, and the two-sided 95% interval of the difference is reported. When the interval includes 0, the result is marked “No difference established,” and the relative reference layer takes 50.

Functionality

Functionality F measures whether interface capabilities are available under the specified configuration and does not double-count task answer correctness.

Functional groups and weights

Functional group	Weight	What is checked
Streaming and message protocol	20%	Streaming deltas, event order, terminal chunk, `finish_reason`, and `usage` placement
Tool interface	20%	Tool names, schema, argument parsing, parallel calls, tool result return, and error handling
Output constraints and parameters	20%	Actual behavior of `temperature`, `max_tokens`, `stop`, structured output, and thinking configuration
Context and state	15%	Window boundaries, over-limit handling, message history, and long-input integrity
Errors and recovery	15%	Consistency of error codes, error bodies, rate limiting, and retry semantics with documented conventions
Metering and cache protocol	10%	`usage` completeness, total consistency, and cache metering

f[组] (Functional Group Pass Score) denotes the pass rate of valid applicable instances for a single functional group. In the formula, f流式, f工具, f约束, f上下文, f错误, and f计量 correspond to the six groups in the table.

Absolute performance for functionality

f[组] = 100 × passing instances / valid applicable instances
A_F = 0.20 × f流式 + 0.20 × f工具 + 0.20 × f约束
    + 0.15 × f上下文 + 0.15 × f错误 + 0.10 × f计量

Each group has 10 test cases, each executed 1 time in each of 3 windows, for 60 test cases and 180 determinations. Repeated requests specified within a case are recorded separately and do not change the case grouping or weights.

Determination states

State	Meaning	Aggregation
PASS	All applicable checks pass	Counted in passing and valid applicable instances
FAIL	Missing capability, contract mismatch, value error, or unexpected service failure	Counted in valid applicable instances, not in passing
NA	The model does not natively support the capability, with sufficient evidence and prior registration	Excluded from the applicable denominator
UNTESTED	Test not executed	Retains a restricted or unavailable status when required test records are insufficient
INVALID	Tester, fixture, or configuration evidence is invalid and cannot be attributed to the service under test	Not treated as a functionality failure; valid denominator and evidence limitations are recorded
UNOBSERVED	A preceding step was interrupted and no subsequent observation was formed	Not counted as passing; the missing-observation status is retained

The native capability score is computed from valid applicable records. When INVALID or UNOBSERVED records exist, coverage and denominator limitations are disclosed together; when required checks lack evidence, the strict compatibility score is unavailable. A capability that the model natively supports but the service does not provide is recorded as FAIL and cannot be classified as NA.

Native capability and interface compatibility

Metric	Full English name and meaning	Basis for checking	Use
`F_native`	Native Functionality Score; functionality support score in the native configuration	Interpret fields, streaming structure, and required request configuration according to provider documentation, and verify that capabilities exist with correct semantics	Used as `A_F` in the primary score
`F_strict`	Strict Interface Compatibility Score; interface compatibility score under strict checking	Check fields, event structure, and response behavior directly against the AIHubMix interface contract	Reported as a separate interface compatibility diagnostic

When a field exists and its placement or naming conforms to provider documentation, the native capability check parses it according to its actual structure. When a field is entirely missing, has an incorrect value, or is contradictory, the determination follows the actual check result. The adapter layer retains the original fields and mapping basis and does not presume an unknown value to be a passing capability. F_native and F_strict are reported separately and cannot be used interchangeably in the primary score of the same comparison table. The effective cache hit ratio is included in Response speed, and billing accuracy is verified separately.

Response speed

Response speed P consists of response performance for answer content and effective cache hits. The primary score uses K_native.

Load and sampling

Load profile I (Non-thinking Profile) and load profile T (Thinking Profile) are measured and reported separately. I and T are load profile identifiers; profile I requires the model to natively support non-thinking mode, and profile T records effort, thinking budget, and output budget. C1 (Concurrency 1, single concurrency) and C10 (Concurrency 10, 10-way concurrency) denote the number of requests executed concurrently. Here C denotes Concurrency and is used separately from the evidence completeness layer C.

Configuration item	Measurement convention
Input	10,000 ± 2% tokens, measured with `o200k_base`; a unique prefix is added at the input start to control unexpected cache hits
Output	1,000 to 1,200 answer-content tokens, including fixed verification points
Timeout	300 seconds, consistent within a comparison group
Single concurrency C1	90 requests, 3 windows × 30
10-way concurrency C10	90 requests, 3 windows × 30
Cache subtest	30 requests, 3 windows × 8k/32k two levels × 5
Isolation	Answer quality and response speed are executed separately; response speed uses an independent measurement period

Concurrency testing uses a closed-loop replenishment mechanism: one request is issued whenever a request completes. C10 represents the observed result under this load and does not constitute a complete capacity curve.

Timing and valid output

Metric	Full English name	Definition and use
Measurement start	Measurement Start Time	The moment the formal request begins to be written after the pre-established connection is ready; connection cold-start time is recorded separately
TTFT	Time to First Token	Wait time until the first payload of actual reasoning, answer content, or tool arguments; empty-role and heartbeat events are not counted; reported as a raw metric
TTFA	Time to First Answer	Wait time until the first non-empty final answer-content chunk, including prior thinking time; used for latency scoring
TPOT	Time per Output Token	Mean time per token during answer generation, computed from the first and last answer-content chunk times and the corresponding token counts
goodput	Valid-Output Throughput	Rate of valid answer-content tokens that meet correctness and completeness requirements over the full C10 wall-clock time

When answer content is returned in a single block, TPOT is recorded as invalid and is not used to construct a per-token rate. The scoring statistic uses P50 (50th Percentile, the median) including failure handling; empirical P95 (95th Percentile) and between-window variation are reported separately, and 90 samples per level do not support conclusions about P99 (99th Percentile). Here P denotes Percentile and is used separately from the response speed dimension P.

Scoring formulas

The function names and subscore symbols in this section are computational notation for this method, with the following meanings:

Symbol	Full English name	Meaning
`S(x; τ)`	Latency Score Function	Converts wait time x into a latency subscore; τ is the latency scale
`L(x; t, k)`	Lower-is-Better Score Function	Scoring function where lower values are better; used for TPOT in this section, with t as the full-score target and k as the range multiplier
`H(x; t)`	Higher-is-Better Score Function	Scoring function where higher values are better; used for valid throughput in this section, with t as the full-score target
`l_ttfa`	Answer Latency Subscore	Time-to-first-answer subscore, taking the smaller of the C1 and C10 scoring results
`l_tpot`	Output Token Time Subscore	Time-per-answer-token subscore, taking the smaller of the C1 and C10 scoring results
`P_response`	Response Performance Score	Response performance subscore composed from latency, time per token, and valid throughput, excluding the cache subscore
`P50_TTFA_C1`, `P50_TTFA_C10`	Median Time to First Answer at Concurrency 1 / 10	Median TTFA including failure handling at single or 10-way concurrency
`P50_TPOT_C1`, `P50_TPOT_C10`	Median Time per Output Token at Concurrency 1 / 10	Median TPOT including failure handling at single or 10-way concurrency

A_P is the absolute response speed score, including response performance and the cache subscore; D_P is the response speed dimension score after composition with the relative reference and evidence, which participates in the overall score. Below, clip constrains a value to the range 0 to 1, ln is the natural logarithm, min takes the smaller value, and Σ denotes summation. The parameters τ, b, c, and k in the formulas are defined in the parameter table immediately following:

Absolute performance for response speed

S(x; τ) = 100 × τ / (τ + x)
L(x; t, k) = 100 × clip[ln(k·t/x) / ln(k)]
H(x; t) = 100 × clip[ln(x / (0.05·t)) / ln(20)]

l_ttfa = min(S(P50_TTFA_C1; τ), S(P50_TTFA_C10; τ))
l_tpot = min(L(P50_TPOT_C1; b, 10), L(P50_TPOT_C10; b, 10))

goodput = Σ valid requests min(answer-content tokens, 1000) / Σ C10 window wall-clock duration

P_response = 0.35 × l_ttfa + 0.35 × l_tpot + 0.30 × H(goodput; c)
A_P = 0.80 × P_response + 0.20 × K_native

Parameter	Full English name	Value	Meaning
τ	Latency Scale	2 seconds	TTFA at which the latency subscore reaches 50
b	Target Time per Output Token	5 ms/token	TPOT full-score target; 100 when at or below this value
c	Target Valid-Output Throughput	1,200 token/s	C10 valid throughput full-score target; 100 when at or above this value
k	TPOT Range Multiplier	10	Range multiplier for the TPOT logarithmic mapping

These targets apply to the load specified in this standard and do not represent universal thresholds for all models and all workloads.

TTFA	0 seconds	1 second	2 seconds	4 seconds	10 seconds
Latency subscore	100	66.7	50	33.3	16.7

Invalid output and measurement isolation

Outputs with incorrect length, a wrong verification point, degenerate looping, in-stream errors, abnormal termination, or truncation are judged invalid and still count toward load and elapsed time. Before computing P50, the following handling applies:

TTFA is counted as positive infinity, with a corresponding latency score of 0.
TPOT is counted as 10b, that is, 50 ms/token.
The valid-output contribution to goodput is 0, and the time remains in the corresponding C10 window.
The raw measured time is retained separately and is not overwritten by the scoring-processed value.

The number of valid requests, the planned number, and the valid output rate are also reported. Performance results include the effect of service failures on delivery and cannot be interpreted solely from the latency of successful requests.

Cache hits

The cache subscore K (Cache Hit Score) measures the proportion of tokens in a prefix expected to be reusable that are actually reported as cache reads. For each length, the read count and expected reusable count of hot requests are aggregated first, and the two lengths, 8k and 32k, are then combined with equal weights.

Symbol	Full English name	Meaning
`K_length`	Length-Specific Cache Hit Score	Effective cache hit subscore for one prefix length level
`K_8k`	8k-Prefix Cache Hit Score	Effective cache hit subscore for the approximately 8k-token prefix level
`K_32k`	32k-Prefix Cache Hit Score	Effective cache hit subscore for the approximately 32k-token prefix level
`cache_read_native_tokens`	Cache-Read Native Token Count	Number of native tokens actually reported as cache reads for hot requests, excluding cache writes
`expected_reusable_native_tokens`	Expected Reusable Native Token Count	Number of native prefix tokens expected to be reusable under the rules, used as the hit-rate denominator

Here native_tokens denotes tokens measured with the designated native tokenizer and is interpreted separately from native in K_native, which denotes configuration according to provider documentation.

Cache hit subscore

K_length = 100 × Σ hot requests cache_read_native_tokens
                 / Σ hot requests expected_reusable_native_tokens

K = (K_8k + K_32k) / 2

The token-weighted hit rate after combining lengths may serve as a diagnostic metric and does not replace the equal-weight K in the primary score.

Test sequence and denominator

Each group uses a fixed long prefix and changing new questions, for 5 requests in total:

The first request writes the cache, and a unique random identifier is added at the prefix start to isolate different test groups.
Requests 2 through 4 keep the prefix byte-identical and use different short questions at the end, issued 5 seconds, 30 seconds, and 120 seconds after the previous request completes, respectively.
The 5th request replaces the prefix with a completely independent new prefix as a cold control.

Each of the three windows covers the approximately 8k and 32k levels, for 30 requests in total. Only requests 2 through 4 enter hot-hit scoring, giving each length 9 hot-request opportunities. The first write and the cold control are verified separately. The denominator uses the registered number of native prefix tokens expected to be reusable, based on the designated tokenizer, template, and cache rules. New questions and non-reusable portions are identified separately; the denominator is not replaced by all input tokens, the actual hit count, or an o200k_base count. Read counts reported in blocks retain their original values and block size. The primary score uses K_native: the configuration required to enable caching according to provider documentation, with necessary session affinity fields treated as part of access adaptation. K_default records the result obtained uniformly without session affinity configuration, serving as a default-configuration diagnostic. All providers follow the same configuration determination principle, and reports retain the specific configuration.

Anomalies and unobservable cases

Case	Handling
Reusable denominator unknown	K is unavailable, not recorded as 0
Successful request missing the cache-read field	Recorded as unobservable by default; parsed as 0 only when the documentation explicitly states that the field is omitted on zero hits, with registration and consistent evidence
Read count exceeds the denominator or fields conflict	The anomaly and original values are recorded; not truncated to 100%
Cold control succeeded with 0 reads	May serve as cold-control evidence; controls that could not be completed retain a restricted status
Hot request failed and the denominator is determinable	Reads are included as 0, and the failure is recorded separately
Model natively supports caching but the service does not provide it	K = 0
Model does not natively support caching and this was defined in advance	Listed as a separate No-cache level, with `A_P = P_response`, and reported separately from the cache-enabled level
Hit metering not supported	K and P are marked unavailable; cache weight is not transferred

usage can only confirm the cache read count reported by the interface. Actual cost may be marked verified only after reconciliation; a cache hit does not mean the request is free and does not guarantee lower total cost.

Service reliability

Service reliability R measures the proportion of a fixed request set that completes validly on first execution; scoring does not use automatic retries or fallback across model providers:

Absolute performance for service reliability

A_R = 100 × first-attempt valid completions / valid planned opportunities

Full planned opportunities = 300

Test set and success conditions

The test consists of 300 opportunities distributed across 3 windows, with 100 in each window. Each set of 100 uses the same task mix: 40 short-text, 30 long-input of about 10k, 20 tool-protocol calls, and 10 single calls with multi-round history. Tool tasks use a pre-built valid history to keep the scoring denominator stable. The default is single concurrency, and the request rate stays within verified RPM and TPM quotas. This suite is executed independently and does not mix in answer quality questions, functionality negative test cases, diagnostic retries, warm-up, or stress-load requests. A first-attempt success satisfies all of the following:

HTTP status is as expected, and the response structure and event sequence are valid.
The answer content or tool payload is non-empty and ends normally.
There are no in-stream errors, unexpected truncation, or missing required fields.
The specified usage fields are complete and parse according to the corresponding interface structure.
The request completes within the 300-second deadline.

HTTP 200 alone is not sufficient for a success determination; task answer errors and service failures are counted separately.

Failures, invalid records, and intervals

429 responses within quota, server errors, connection failures, timeouts, empty responses, stream interruptions, and abnormal truncation are counted as failures. When a model provider failure prevents a planned request from being sent, it is counted as a failure and the reason is recorded. Test environment or tester anomalies are marked INVALID and are not attributed to the service under test, while the number of valid opportunities and coverage limitations are retained. Diagnostic requests are recorded independently and do not replace first-attempt results. The success rate is reported with a two-sided 95% Clopper–Pearson interval. An observed success rate of 300/300 is 100%, and the lower bound of the interval is approximately 98.78%, which still does not prove a long-term service success rate of 100%. Failure distribution by window and task category, and the longest consecutive failure streak, are recorded.

Test scale and comparability

Sample size

The following are the planned scales of AF Full within AF Suite:

Item	Planned scale per provider	Composition
Answer quality	4,250 instances	BFCL 1,500, HLE 2,500, AA-LCR 250
Functionality	180 determinations	60 test cases × 3 windows
Response speed	180 requests	90 each for C1/C10, distributed across 3 windows
Cache	30 requests	3 windows × two prefix lengths × 5
Service reliability	300 opportunities	3 windows × 100

Task instances, API requests, and judge requests are counted separately; multi-stage tool tasks and in-case repetitions may correspond to additional requests.

Execution conditions

Within a comparison group, the task set, rounds, timeout, temperature, output budget, failure handling, and scoring formula are the same.
The same client, network region, and connection strategy are used, and account service tier, actual start and end times, and execution order are recorded.
A full test uses 3 windows at least 60 minutes apart; answer quality and response speed are executed separately.
Model ID, authentication, required fields, streaming format, usage mapping, cache semantics, and tokenizer basis form an adaptation record.
Access adaptation is registered and verified before sampling, and the same rules apply to all eligible model providers.
The primary scoring view consistently uses F_native and K_native; compatibility and default-configuration diagnostics are reported separately.
First-execution results are retained, and planned counts, actual counts, filtering, failures, invalid records, and unobserved records are reconciled separately.

Tests are executed within approved quotas, permissions, and cost limits. When required conditions are not met, results retain a restricted, incomplete, or unavailable status, and remaining items are not renormalized.

Dataset sources and result records

Dataset sources

Answer quality uses a versioned derived task set, retaining source, sampling records, and integrity verification information. The AF Full answer quality task set contains 775 distinct questions, with the following source snapshots:

Track	Source snapshot	Version record	License and usage boundaries
BFCL	Berkeley Function Calling Leaderboard data and checker	`6ea57973c7a6`	Source repository Apache-2.0; this evaluation uses a derived task set and checker adaptation
HLE	Text mirror snapshot; official dataset	Mirror `e3e9cb5a550e`	Recorded as MIT at the source; byte-for-byte equivalence with the current official version has not been verified
AA-LCR	Artificial Analysis AA-LCR data and documentation snapshot	`9a77ef56b717`	Dataset repository Apache-2.0; third-party rights in referenced documents remain with their respective owners

BFCL is sampled by category quota; HLE and AA-LCR are sampled by stratified proportion with minimum coverage guaranteed for each stratum. Sampling uses a designated seed and SHA-256 ordering, and the task set version, filter list, and grading protocol are recorded with the results. The links above serve to verify original sources and are not equivalent to a complete evaluation package that can directly reproduce AF scores. Restricted data is not redistributed in this page, and specific licenses follow the respective sources.

Result records and verification

Provider reports are collectively referred to as AF Profile and record the following evaluation information:

Record category	Required content
Evaluation identity	Standard and task set versions, model provider, exact model version, and the official reference service
Measurement conditions	Thinking configuration, account tier, region, period, timeout, concurrency, and performance parameters
Adaptation basis	Documentation review date, required fields, streaming structure, usage parsing, and cache configuration
Sample coverage	Planned count, actual count, scoring denominator, filtering, failures, invalid, and unobserved items
Grading basis	Reference answers, checker or judge configuration, grading status, and bias assessment
Raw measurements	Redacted requests and responses, event times, status, termination reasons, and raw `usage`
Aggregated results	A/B/C layer inputs, the four D scores, overall score, dimension intervals, and unrounded values
Diagnostic metrics	`F_strict`, `K_default`, valid rate, error categories, and corresponding applicability limits

Service success, task correctness, and output validity are labeled separately. Negative test cases are not mixed into the service reliability score, cache reads and writes are metered separately, and the verification status of estimated and actual costs is recorded separately. Keys, user identifiers, and sensitive content are removed before external presentation; values carry units, and missing values retain their status.

Reference data and calculation examples

The following uses the results of five providers’ services for DeepSeek V4.1 Flash 1017 to illustrate the calculation method. The primary score uses F_native and K_native. Values are displayed to two decimal places, and calculations use the intermediate values saved in the result files.

This section presents reference results (PROVISIONAL). Prior registration evidence for collection and adaptation is incomplete, so these cannot serve as official AF Full results for AIHubMix Fidelitas Standard. Answer quality counts 4,240 instances per provider, with 10 additional filtered instances. The HLE/AA-LCR judges belong to the same DeepSeek family as the models under test, and independent calibration evidence is insufficient. Results are limited to the recorded configuration and observation scope.

Absolute performance and cache subscores

In the table below, Q, F, P, and R are all absolute performance A and are not the four dimension display scores in the overall score. K is the cache subscore within P.

Model provider	A_Q	F_native (A_F)	A_P	K_native	A_R
ByteDance	42.05	74.98	84.37	93.83	100.00
Baidu	38.00	88.39	37.77	68.85	100.00
Alibaba Cloud	41.75	89.78	88.58	99.37	100.00
DeepSeek Official Reference	42.45	94.00	86.14	98.38	100.00
Fireworks	41.42	71.78	79.51	60.68	100.00

All five providers’ service reliability suites are 300/300; service failures in other suites still count toward their own dimensions and cannot be replaced by the R suite result.
Fireworks’ K_native uses the session affinity configuration specified in its documentation, and the other services are measured with the configuration required by their respective documentation.
Baidu’s K uses the sample rule that parses an omitted field as 0 on zero hits, and the related prior registration evidence is incomplete; the 8k cold control has an incomplete record, so K retains the reference status.
DeepSeek’s F_strict is unavailable because of 27 INVALID records; F_native is computed from valid applicable records.

Functionality dimension composition

Taking Alibaba Cloud as an example, the absolute functionality score is A_F = 89.777778, the official reference’s absolute score for the same item is 94, the difference scale is δ_F = 15, and evidence completeness is C_F = 100. The relative reference score B_F is calculated first, and the dimension score D_F is then composed:

Relative reference and functionality dimension score

B_F = 50 + 50 × clip((89.777778 − 94) / 15, −1, 1)
    ≈ 35.925926

D_F = 0.65 × 89.777778 + 0.20 × 35.925926 + 0.15 × 100
    ≈ 80.540741

89.78 is the native functionality absolute performance, and 80.54 is the functionality dimension display score used in the overall score calculation.

Service reliability score of the official reference

The official reference’s R suite is 300/300, its absolute service reliability score is A_R = 100, its relative reference score is B_R = 50, and its evidence completeness is C_R = 100:

Service reliability dimension composition

D_R = 0.65 × 100 + 0.20 × 50 + 0.15 × 100
    = 90

100% is the observed success rate of the service reliability suite in this round, and 90 is the dimension display score after three-layer composition.

Dimension display scores and overall score

The table below uses D values and corresponds to the overall score formula. The table follows the provider display order and is not sorted by score.

Model provider	Answer quality D_Q	Functionality D_F	Response speed D_P	Service reliability D_R	Overall score
ByteDance	50.83	62.24	76.57	88.50	66.37
Baidu	36.70	75.71	36.55	87.00	56.48
Alibaba Cloud	52.13	80.54	85.01	90.00	73.38
DeepSeek Official Reference	52.59	84.60	80.99	90.00	73.76
Fireworks	50.42	60.16	68.55	88.50	64.10

Taking the official reference as an example:

Overall score calculation

Overall = 0.35 × 52.594340 + 0.25 × 84.600000
        + 0.20 × 80.990116 + 0.20 × 90.000000
        ≈ 73.76

Independent diagnostic metrics

Model provider	F_strict	K_default
ByteDance	56.83	93.83
Baidu	66.17	68.85
Alibaba Cloud	89.78	99.37
DeepSeek Official Reference	Unavailable	98.38
Fireworks	71.78	11.06

F_strict expresses interface compatibility, and K_default expresses the cache result obtained uniformly without session affinity configuration. Neither column replaces a primary score input. In the difference intervals for answer quality relative to the official reference, the 95% intervals for ByteDance, Alibaba Cloud, and Fireworks all include 0, so B_Q = 50 for them. Baidu’s difference is about −4.46 percentage points, with a 95% interval of approximately −5.78 to −3.16 percentage points. This conclusion is limited to this task set and observation conditions and is not extended to a statistical significance conclusion about the overall score.

Standard and applicability conditions

Standard identifier: AIHubMix Fidelitas Standard v2.0. Results also record AF Suite, adaptation configuration, grading protocol, and measurement conditions. Results under different configurations or statistical definitions are interpreted separately. The four scores on model pages are D values; correctness rates, functionality pass rates, latency, cache ratios, and first-attempt success rates in reports are the corresponding raw metrics or A values. The overall score is used for summary presentation, and specific integration decisions are based on dimension results, functional requirements, and evidence conditions. Related: Choose a model · Model details and comparison Document date: 2026-11-05

​Scope and how to read the scores

​Score states

​Interpreting results

​Overall score and dimension composition

​Overall score formula

​Three-layer composition

​Relative to the official reference B

​Evidence completeness C

​Answer quality

​Task sets and grading

​Comparison conditions and missing-data handling

​Functionality

​Functional groups and weights

​Determination states

​Native capability and interface compatibility

​Response speed

​Load and sampling

​Timing and valid output

​Scoring formulas

​Invalid output and measurement isolation

​Cache hits

​Test sequence and denominator

​Anomalies and unobservable cases

​Service reliability

​Test set and success conditions

​Failures, invalid records, and intervals

​Test scale and comparability

​Sample size

​Execution conditions

​Dataset sources and result records

​Dataset sources

​Result records and verification

​Reference data and calculation examples

​Absolute performance and cache subscores

​Functionality dimension composition

​Service reliability score of the official reference

​Dimension display scores and overall score

​Independent diagnostic metrics

​Standard and applicability conditions

Scope and how to read the scores

Score states

Interpreting results

Overall score and dimension composition

Overall score formula

Three-layer composition

Relative to the official reference B

Evidence completeness C

Answer quality

Task sets and grading

Comparison conditions and missing-data handling

Functionality

Functional groups and weights

Determination states

Native capability and interface compatibility

Response speed

Load and sampling

Timing and valid output

Scoring formulas

Invalid output and measurement isolation

Cache hits

Test sequence and denominator

Anomalies and unobservable cases

Service reliability

Test set and success conditions

Failures, invalid records, and intervals

Test scale and comparability

Sample size

Execution conditions

Dataset sources and result records

Dataset sources

Result records and verification

Reference data and calculation examples

Absolute performance and cache subscores

Functionality dimension composition

Service reliability score of the official reference

Dimension display scores and overall score

Independent diagnostic metrics

Standard and applicability conditions