> ## Documentation Index > Fetch the complete documentation index at: https://docs.aihubmix.com/llms.txt > Use this file to discover all available pages before exploring further. # AIHubMix Fidelitas Standard v2.0 > Learn about the four scoring dimensions, overall score formula, native capability and interface compatibility determinations, response speed and cache measurement, and task set sources and statistical boundaries of AIHubMix Fidelitas Standard. AIHubMix Fidelitas measures the delivery of the same model by different model providers, evaluating task outcome correctness, capability availability, response performance, and first-request completion separately. The overall score and the four dimension scores are presented together to compare the service performance of the same model under specified test conditions. This page uses **AIHubMix Fidelitas Standard v2.0**, with project version **AF v2.0**. The primary score uses the functionality support and cache measurement results configured according to provider documentation; interface compatibility and cache results under the default configuration are reported separately as diagnostic metrics. ## Scope and how to read the scores | Score item | Symbol | Weight | What is measured | | - | - | - | - | | Answer quality | Q (Quality) | 35% | Correctness of task outcomes for tool calling, knowledge and reasoning, and long-document comprehension | | Functionality | F (Functionality) | 25% | Availability of streaming, tools, structured output, parameters, context, error responses, and usage metering | | Response speed | P (Performance) | 20% | Time to first answer content, generation rate, concurrent valid throughput, and effective cache hits | | Service reliability | R (Reliability) | 20% | Proportion of a fixed request set that completes validly on first execution | **The unit of scoring is the model provider × model version**, that is, the service of a given model version provided by a given model provider. Each object forms one set of four dimension scores and an overall score under the specified evaluation batch and configuration. Thinking mode, request parameters, account service tier, test region, and observation period are measurement conditions and are recorded with the results. Results under different conditions are labeled separately and are not merged directly. When the model version, region, or service tier cannot be verified, the corresponding unverified status is noted, not presumed confirmed. Evaluations are executed through the AIHubMix API, and results describe the actual observed performance under that access method. The official reference service is also measured through the AIHubMix API and is not reported as a measurement made through an independent direct connection to the official API. All members of a comparison group use the same task set, rounds, timeout, request budget, failure scoring, and statistical method. Required request fields, streaming structure, and usage fields specified by provider documentation are handled by the access adapter layer (adapter), keeping the measurement target consistent. This method is not intended to rank intelligence across models, and model provider identity does not confer a scoring advantage. ### Score states | Status | Meaning | | - | - | | Final | Test registration, sampling, grading, and evidence verification have been completed under the applicable standard | | Reference result (PROVISIONAL) | A result can be computed, but sampling, grading, adaptation registration, or evidence is subject to applicable limitations | | Unavailable (UNAVAILABLE) | Required grading, denominator, or reference is missing; no numeric value is assigned | | Not comparable | Model configuration, observation conditions, or result views are inconsistent, so the result is excluded from like-for-like comparison | Missing values and scores of 0 are handled separately. When a service explicitly fails or does not provide a required capability, it is scored as a failure under the corresponding rule; when a measurement is missing, the denominator is unknown, or evidence is insufficient, the corresponding status is retained. If any dimension is unavailable, the overall score is also marked unavailable, and its weight is not redistributed to other dimensions. ### Interpreting results * Raw metrics are converted to absolute performance A by the scoring formula and, after composition with the relative reference and evidence, produce the dimension composite score D. The overall score uses D and does not directly weight raw correctness rates, functionality pass rates, or service success rates. * Scores reflect performance under a specific task set, load, region, configuration, and observation window; applicable conditions are recorded with the results. * Display precision does not indicate statistical discriminative power. Answer quality differences are interpreted together with paired statistical intervals; decimal differences alone cannot establish superiority. * The overall score is a summary index; it is not a certification and has no confidence interval. Statistical conclusions rest on the evidence for each specific dimension. * Service reliability reports the number of first-attempt successes, the number of planned opportunities, and a two-sided 95% interval, which cannot be interpreted directly as a long-term SLA. If review confirms a model identity mismatch, silent input truncation, sensitive content leakage, severe protocol corruption, or a billing anomaly, the result is separately marked “Failed admission,” and reproduction evidence and usage limitations are retained. Dimension scores and the overall score do not offset this limitation. ## Overall score and dimension composition ### Overall score formula The overall score is calculated by weighting the four dimension composite scores. `D_Q`, `D_F`, `D_P`, and `D_R` denote the composite scores for answer quality, functionality, response speed, and service reliability, respectively. ```text Overall score theme={null} Overall = 0.35 × D_Q + 0.25 × D_F + 0.20 × D_P + 0.20 × D_R ``` All four dimension composite scores use a 0–100 scale. The overall score summarizes the overall result, and the four dimension scores identify specific differences. ### Three-layer composition Each dimension composite score D is computed by weighting absolute performance A, performance relative to the official reference B, and evidence completeness C. ```text Dimension score theme={null} D = 0.65 × A + 0.20 × B + 0.15 × C ``` | Input layer | Weight | Definition | | - | - | - | | A: Absolute performance | 65% | Actual measurement results obtained under the specified test and statistical methods | | B: Relative to the official reference | 20% | Difference in absolute performance for the same dimension relative to the official reference service | | C: Evidence completeness | 15% | Degree to which sample coverage, observation consistency, and metering evidence support this result | A, B, and C are computation-layer symbols and do not denote access grades. Absolute performance uses the specified formula; the fastest or highest-scoring service in the batch is not set to 100, and no renormalization is performed by batch. The prefix before the underscore denotes the computation layer, and the suffix denotes the scoring dimension. For example, `A_F` is the functionality score at the absolute performance layer, and `D_F` is the functionality score after three-layer composition. The primary score and diagnostic values also use the following configuration suffixes: | Symbol | Full English name | Meaning and use | | - | - | - | | `F_native` | Native Functionality Score | Functionality score obtained after configuration and parsing according to provider documentation, used as `A_F` in the primary score | | `F_strict` | Strict Interface Compatibility Score | Strict compatibility score obtained by checking directly against the AIHubMix interface contract, reported as a separate diagnostic | | `K` | Cache Hit Score | Effective cache hit subscore, included in the absolute performance of response speed | | `K_native` | Native Cache Hit Score | Cache subscore measured with the configuration required by provider documentation, used in the primary score | | `K_default` | Default-Configuration Cache Hit Score | Cache subscore measured uniformly without session affinity configuration, reported as a separate diagnostic | `native` means configured according to provider documentation, `strict` means strict interface checking, and `default` means the default configuration defined in this page. `K` is the cache symbol in this method and does not denote Quality. The primary score consistently uses `F_native` and `K_native`, and diagnostic values are not mixed with the primary score. ### Relative to the official reference B `A_official` denotes the absolute performance score of the official reference service for the current dimension; its relative reference score `B_official` is fixed at 50. ```text Relative reference layer theme={null} B_official = 50 B = 50 + 50 × clip((A − A_official) / δ, −1, 1) ``` `clip(x, −1, 1)` is a clipping function that constrains x to the range −1 to 1. The difference scale parameter δ is defined by dimension: | Dimension | δ | | - | - | | Answer quality Q | 2 | | Functionality F | 15 | | Response speed P | 10 | | Service reliability R | 2 | Answer quality uses a 5,000-iteration paired-task cluster bootstrap to estimate the difference interval relative to the official reference. **When the two-sided 95% interval includes 0, `B_Q = 50`**; when the interval excludes 0, the mapping above applies. The official reference service independently undergoes absolute performance measurement; its own B is fixed at 50, with no full score assumed. When configurations, timeouts, periods, or result views are not comparable, no presumed value is entered in the relative layer. ### Evidence completeness C C consists of the following five items, each worth up to 20 points, for a total of 100. Each item is assigned 20, 10, or 0 points according to the degree of coverage and evidence in the records: | Item | What is verified | | - | - | | Sample coverage | Degree to which actual valid records match the planned sample size | | Window coverage | Completeness of the required periods or answer quality collection segments | | Usage completeness | Coverage and verifiability of the raw `usage` | | Denominator evidence | Reviewability of the denominators for question grading, functionality determination, and cache metering | | Protocol consistency | Consistency of request configuration, record attribution, and observation protocol | C measures measurement evidence and does not directly measure model capability. Each assignment is linked to evidence records, and result status is retained independently; the evidence-layer score does not replace required data validity verification. ## Answer quality Answer quality Q measures the proportion of all valid task instances that are correct, based on reference answers and the specified grading protocol. ### Task sets and grading | Track | What is measured | Questions × rounds | Planned instances | Grading method | | - | - | - | - | - | | BFCL | Function selection, argument filling, parallel calls, irrelevant-call suppression, and a three-stage tool chain | 500 × 3 | 1,500 | Python-only JSON matching adaptation based on the official AST checker of the specified version | | HLE | High-difficulty text knowledge and reasoning questions stratified by subject | 250 × 10 | 2,500 | Answer equivalence judge; the judge model, prompt, and version are recorded | | AA-LCR | Multi-document synthesis QA of about 74,000 to 114,000 tokens | 25 × 10 | 250 | Designated documents and reference answers, graded by answer equivalence | | Total | 775 distinct questions | Executed at the rounds specified for each track | 4,250 | Aggregated over all valid instances | ```text Absolute performance for answer quality theme={null} A_Q = 100 × correct instances / valid scored instances Full planned denominator = 1,500 + 2,500 + 250 = 4,250 ``` The three tracks are aggregated by instance count, not by an equal-weight average of per-track correctness rates. Multi-round tool tasks may correspond to multiple API requests; task instances and requests are counted separately. > [!NOTE] > > This task set is a derived task set of AF Suite. It is not an exact subset of the official Artificial Analysis set and does not reproduce its leaderboard results. The 50 BFCL multi-round tasks are constructed from real questions and are defined separately from the BFCL official stateful multi-round execution environment. Sources and versions are listed in [Dataset sources](#dataset-sources). ### Comparison conditions and missing-data handling * Task correctness is determined against reference answers; official reference responses are not used as ground truth. * All parties use the same controllable budget, sampling count, and semantically equivalent parameters. When thinking mode, tool templates, or `effort` cannot be aligned, the result is marked “Configuration not comparable.” * Service failures, timeouts, refusals, and budget exhaustion are counted as task failures in the denominator. * Records with invalid judging or without a valid determination are labeled separately, excluded from the computable denominator, and retain the reference result status. * Question filtering rules are specified in test registration, and reports list planned counts, filtered counts, and actual scored counts. * Repeated rounds belong to the same question cluster and are not treated as additional independent questions. Relative differences use a per-track paired-task cluster bootstrap with 5,000 repetitions. All parties use the same question sampling index, and the two-sided 95% interval of the difference is reported. When the interval includes 0, the result is marked “No difference established,” and the relative reference layer takes 50. ## Functionality Functionality F measures whether interface capabilities are available under the specified configuration and does not double-count task answer correctness. ### Functional groups and weights | Functional group | Weight | What is checked | | - | - | - | | Streaming and message protocol | 20% | Streaming deltas, event order, terminal chunk, `finish_reason`, and `usage` placement | | Tool interface | 20% | Tool names, schema, argument parsing, parallel calls, tool result return, and error handling | | Output constraints and parameters | 20% | Actual behavior of `temperature`, `max_tokens`, `stop`, structured output, and thinking configuration | | Context and state | 15% | Window boundaries, over-limit handling, message history, and long-input integrity | | Errors and recovery | 15% | Consistency of error codes, error bodies, rate limiting, and retry semantics with documented conventions | | Metering and cache protocol | 10% | `usage` completeness, total consistency, and cache metering | `f[组]` (Functional Group Pass Score) denotes the pass rate of valid applicable instances for a single functional group. In the formula, `f流式`, `f工具`, `f约束`, `f上下文`, `f错误`, and `f计量` correspond to the six groups in the table. ```text Absolute performance for functionality theme={null} f[组] = 100 × passing instances / valid applicable instances A_F = 0.20 × f流式 + 0.20 × f工具 + 0.20 × f约束 + 0.15 × f上下文 + 0.15 × f错误 + 0.10 × f计量 ``` Each group has 10 test cases, each executed 1 time in each of 3 windows, for 60 test cases and 180 determinations. Repeated requests specified within a case are recorded separately and do not change the case grouping or weights. ### Determination states | State | Meaning | Aggregation | | - | - | - | | PASS | All applicable checks pass | Counted in passing and valid applicable instances | | FAIL | Missing capability, contract mismatch, value error, or unexpected service failure | Counted in valid applicable instances, not in passing | | NA | The model does not natively support the capability, with sufficient evidence and prior registration | Excluded from the applicable denominator | | UNTESTED | Test not executed | Retains a restricted or unavailable status when required test records are insufficient | | INVALID | Tester, fixture, or configuration evidence is invalid and cannot be attributed to the service under test | Not treated as a functionality failure; valid denominator and evidence limitations are recorded | | UNOBSERVED | A preceding step was interrupted and no subsequent observation was formed | Not counted as passing; the missing-observation status is retained | The native capability score is computed from valid applicable records. When INVALID or UNOBSERVED records exist, coverage and denominator limitations are disclosed together; when required checks lack evidence, the strict compatibility score is unavailable. A capability that the model natively supports but the service does not provide is recorded as FAIL and cannot be classified as NA. ### Native capability and interface compatibility | Metric | Full English name and meaning | Basis for checking | Use | | - | - | - | - | | `F_native` | Native Functionality Score; functionality support score in the native configuration | Interpret fields, streaming structure, and required request configuration according to provider documentation, and verify that capabilities exist with correct semantics | Used as `A_F` in the primary score | | `F_strict` | Strict Interface Compatibility Score; interface compatibility score under strict checking | Check fields, event structure, and response behavior directly against the AIHubMix interface contract | Reported as a separate interface compatibility diagnostic | When a field exists and its placement or naming conforms to provider documentation, the native capability check parses it according to its actual structure. When a field is entirely missing, has an incorrect value, or is contradictory, the determination follows the actual check result. The adapter layer retains the original fields and mapping basis and does not presume an unknown value to be a passing capability. `F_native` and `F_strict` are reported separately and cannot be used interchangeably in the primary score of the same comparison table. The effective cache hit ratio is included in [Response speed](#speed), and billing accuracy is verified separately. ## Response speed Response speed P consists of response performance for answer content and effective cache hits. The primary score uses `K_native`. ### Load and sampling Load profile I (Non-thinking Profile) and load profile T (Thinking Profile) are measured and reported separately. I and T are load profile identifiers; profile I requires the model to natively support non-thinking mode, and profile T records `effort`, thinking budget, and output budget. C1 (Concurrency 1, single concurrency) and C10 (Concurrency 10, 10-way concurrency) denote the number of requests executed concurrently. Here C denotes Concurrency and is used separately from the evidence completeness layer C. | Configuration item | Measurement convention | | - | - | | Input | 10,000 ± 2% tokens, measured with `o200k_base`; a unique prefix is added at the input start to control unexpected cache hits | | Output | 1,000 to 1,200 answer-content tokens, including fixed verification points | | Timeout | 300 seconds, consistent within a comparison group | | Single concurrency C1 | 90 requests, 3 windows × 30 | | 10-way concurrency C10 | 90 requests, 3 windows × 30 | | Cache subtest | 30 requests, 3 windows × 8k/32k two levels × 5 | | Isolation | Answer quality and response speed are executed separately; response speed uses an independent measurement period | Concurrency testing uses a closed-loop replenishment mechanism: one request is issued whenever a request completes. C10 represents the observed result under this load and does not constitute a complete capacity curve. ### Timing and valid output | Metric | Full English name | Definition and use | | - | - | - | | Measurement start | Measurement Start Time | The moment the formal request begins to be written after the pre-established connection is ready; connection cold-start time is recorded separately | | TTFT | Time to First Token | Wait time until the first payload of actual reasoning, answer content, or tool arguments; empty-role and heartbeat events are not counted; reported as a raw metric | | TTFA | Time to First Answer | Wait time until the first non-empty final answer-content chunk, including prior thinking time; used for latency scoring | | TPOT | Time per Output Token | Mean time per token during answer generation, computed from the first and last answer-content chunk times and the corresponding token counts | | goodput | Valid-Output Throughput | Rate of valid answer-content tokens that meet correctness and completeness requirements over the full C10 wall-clock time | When answer content is returned in a single block, TPOT is recorded as invalid and is not used to construct a per-token rate. The scoring statistic uses P50 (50th Percentile, the median) including failure handling; empirical P95 (95th Percentile) and between-window variation are reported separately, and 90 samples per level do not support conclusions about P99 (99th Percentile). Here P denotes Percentile and is used separately from the response speed dimension P. ### Scoring formulas The function names and subscore symbols in this section are computational notation for this method, with the following meanings: | Symbol | Full English name | Meaning | | - | - | - | | `S(x; τ)` | Latency Score Function | Converts wait time x into a latency subscore; τ is the latency scale | | `L(x; t, k)` | Lower-is-Better Score Function | Scoring function where lower values are better; used for TPOT in this section, with t as the full-score target and k as the range multiplier | | `H(x; t)` | Higher-is-Better Score Function | Scoring function where higher values are better; used for valid throughput in this section, with t as the full-score target | | `l_ttfa` | Answer Latency Subscore | Time-to-first-answer subscore, taking the smaller of the C1 and C10 scoring results | | `l_tpot` | Output Token Time Subscore | Time-per-answer-token subscore, taking the smaller of the C1 and C10 scoring results | | `P_response` | Response Performance Score | Response performance subscore composed from latency, time per token, and valid throughput, excluding the cache subscore | | `P50_TTFA_C1`, `P50_TTFA_C10` | Median Time to First Answer at Concurrency 1 / 10 | Median TTFA including failure handling at single or 10-way concurrency | | `P50_TPOT_C1`, `P50_TPOT_C10` | Median Time per Output Token at Concurrency 1 / 10 | Median TPOT including failure handling at single or 10-way concurrency | `A_P` is the absolute response speed score, including response performance and the cache subscore; `D_P` is the response speed dimension score after composition with the relative reference and evidence, which participates in the overall score. Below, `clip` constrains a value to the range 0 to 1, `ln` is the natural logarithm, `min` takes the smaller value, and Σ denotes summation. The parameters τ, b, c, and k in the formulas are defined in the parameter table immediately following: ```text Absolute performance for response speed theme={null} S(x; τ) = 100 × τ / (τ + x) L(x; t, k) = 100 × clip[ln(k·t/x) / ln(k)] H(x; t) = 100 × clip[ln(x / (0.05·t)) / ln(20)] l_ttfa = min(S(P50_TTFA_C1; τ), S(P50_TTFA_C10; τ)) l_tpot = min(L(P50_TPOT_C1; b, 10), L(P50_TPOT_C10; b, 10)) goodput = Σ valid requests min(answer-content tokens, 1000) / Σ C10 window wall-clock duration P_response = 0.35 × l_ttfa + 0.35 × l_tpot + 0.30 × H(goodput; c) A_P = 0.80 × P_response + 0.20 × K_native ``` | Parameter | Full English name | Value | Meaning | | - | - | - | - | | τ | Latency Scale | 2 seconds | TTFA at which the latency subscore reaches 50 | | b | Target Time per Output Token | 5 ms/token | TPOT full-score target; 100 when at or below this value | | c | Target Valid-Output Throughput | 1,200 token/s | C10 valid throughput full-score target; 100 when at or above this value | | k | TPOT Range Multiplier | 10 | Range multiplier for the TPOT logarithmic mapping | These targets apply to the load specified in this standard and do not represent universal thresholds for all models and all workloads. | TTFA | 0 seconds | 1 second | 2 seconds | 4 seconds | 10 seconds | | - | - | - | - | - | - | | Latency subscore | 100 | 66.7 | 50 | 33.3 | 16.7 | ### Invalid output and measurement isolation Outputs with incorrect length, a wrong verification point, degenerate looping, in-stream errors, abnormal termination, or truncation are judged invalid and still count toward load and elapsed time. Before computing P50, the following handling applies: * TTFA is counted as positive infinity, with a corresponding latency score of 0. * TPOT is counted as `10b`, that is, 50 ms/token. * The valid-output contribution to goodput is 0, and the time remains in the corresponding C10 window. * The raw measured time is retained separately and is not overwritten by the scoring-processed value. The number of valid requests, the planned number, and the valid output rate are also reported. Performance results include the effect of service failures on delivery and cannot be interpreted solely from the latency of successful requests. ## Cache hits The cache subscore K (Cache Hit Score) measures the proportion of tokens in a prefix expected to be reusable that are actually reported as cache reads. For each length, the read count and expected reusable count of hot requests are aggregated first, and the two lengths, 8k and 32k, are then combined with equal weights. | Symbol | Full English name | Meaning | | - | - | - | | `K_length` | Length-Specific Cache Hit Score | Effective cache hit subscore for one prefix length level | | `K_8k` | 8k-Prefix Cache Hit Score | Effective cache hit subscore for the approximately 8k-token prefix level | | `K_32k` | 32k-Prefix Cache Hit Score | Effective cache hit subscore for the approximately 32k-token prefix level | | `cache_read_native_tokens` | Cache-Read Native Token Count | Number of native tokens actually reported as cache reads for hot requests, excluding cache writes | | `expected_reusable_native_tokens` | Expected Reusable Native Token Count | Number of native prefix tokens expected to be reusable under the rules, used as the hit-rate denominator | Here `native_tokens` denotes tokens measured with the designated native tokenizer and is interpreted separately from `native` in `K_native`, which denotes configuration according to provider documentation. ```text Cache hit subscore theme={null} K_length = 100 × Σ hot requests cache_read_native_tokens / Σ hot requests expected_reusable_native_tokens K = (K_8k + K_32k) / 2 ``` The token-weighted hit rate after combining lengths may serve as a diagnostic metric and does not replace the equal-weight K in the primary score. ### Test sequence and denominator Each group uses a fixed long prefix and changing new questions, for 5 requests in total: 1. The first request writes the cache, and a unique random identifier is added at the prefix start to isolate different test groups. 2. Requests 2 through 4 keep the prefix byte-identical and use different short questions at the end, issued 5 seconds, 30 seconds, and 120 seconds after the previous request completes, respectively. 3. The 5th request replaces the prefix with a completely independent new prefix as a cold control. Each of the three windows covers the approximately 8k and 32k levels, for 30 requests in total. Only requests 2 through 4 enter hot-hit scoring, giving each length 9 hot-request opportunities. The first write and the cold control are verified separately. The denominator uses the registered number of native prefix tokens expected to be reusable, based on the designated tokenizer, template, and cache rules. New questions and non-reusable portions are identified separately; the denominator is not replaced by all input tokens, the actual hit count, or an `o200k_base` count. Read counts reported in blocks retain their original values and block size. The primary score uses **`K_native`**: the configuration required to enable caching according to provider documentation, with necessary session affinity fields treated as part of access adaptation. **`K_default`** records the result obtained uniformly without session affinity configuration, serving as a default-configuration diagnostic. All providers follow the same configuration determination principle, and reports retain the specific configuration. ### Anomalies and unobservable cases | Case | Handling | | - | - | | Reusable denominator unknown | K is unavailable, not recorded as 0 | | Successful request missing the cache-read field | Recorded as unobservable by default; parsed as 0 only when the documentation explicitly states that the field is omitted on zero hits, with registration and consistent evidence | | Read count exceeds the denominator or fields conflict | The anomaly and original values are recorded; not truncated to 100% | | Cold control succeeded with 0 reads | May serve as cold-control evidence; controls that could not be completed retain a restricted status | | Hot request failed and the denominator is determinable | Reads are included as 0, and the failure is recorded separately | | Model natively supports caching but the service does not provide it | K = 0 | | Model does not natively support caching and this was defined in advance | Listed as a separate No-cache level, with `A_P = P_response`, and reported separately from the cache-enabled level | | Hit metering not supported | K and P are marked unavailable; cache weight is not transferred | `usage` can only confirm the cache read count reported by the interface. Actual cost may be marked verified only after reconciliation; a cache hit does not mean the request is free and does not guarantee lower total cost. ## Service reliability Service reliability R measures the proportion of a fixed request set that completes validly on first execution; scoring does not use automatic retries or fallback across model providers: ```text Absolute performance for service reliability theme={null} A_R = 100 × first-attempt valid completions / valid planned opportunities Full planned opportunities = 300 ``` ### Test set and success conditions The test consists of 300 opportunities distributed across 3 windows, with 100 in each window. Each set of 100 uses the same task mix: 40 short-text, 30 long-input of about 10k, 20 tool-protocol calls, and 10 single calls with multi-round history. Tool tasks use a pre-built valid history to keep the scoring denominator stable. The default is single concurrency, and the request rate stays within verified RPM and TPM quotas. This suite is executed independently and does not mix in answer quality questions, functionality negative test cases, diagnostic retries, warm-up, or stress-load requests. A first-attempt success satisfies all of the following: * HTTP status is as expected, and the response structure and event sequence are valid. * The answer content or tool payload is non-empty and ends normally. * There are no in-stream errors, unexpected truncation, or missing required fields. * The specified `usage` fields are complete and parse according to the corresponding interface structure. * The request completes within the 300-second deadline. HTTP 200 alone is not sufficient for a success determination; task answer errors and service failures are counted separately. ### Failures, invalid records, and intervals 429 responses within quota, server errors, connection failures, timeouts, empty responses, stream interruptions, and abnormal truncation are counted as failures. When a model provider failure prevents a planned request from being sent, it is counted as a failure and the reason is recorded. Test environment or tester anomalies are marked INVALID and are not attributed to the service under test, while the number of valid opportunities and coverage limitations are retained. Diagnostic requests are recorded independently and do not replace first-attempt results. The success rate is reported with a two-sided 95% Clopper–Pearson interval. An observed success rate of 300/300 is 100%, and the lower bound of the interval is approximately 98.78%, which still does not prove a long-term service success rate of 100%. Failure distribution by window and task category, and the longest consecutive failure streak, are recorded. ## Test scale and comparability ### Sample size The following are the planned scales of AF Full within AF Suite: | Item | Planned scale per provider | Composition | | - | - | - | | Answer quality | 4,250 instances | BFCL 1,500, HLE 2,500, AA-LCR 250 | | Functionality | 180 determinations | 60 test cases × 3 windows | | Response speed | 180 requests | 90 each for C1/C10, distributed across 3 windows | | Cache | 30 requests | 3 windows × two prefix lengths × 5 | | Service reliability | 300 opportunities | 3 windows × 100 | Task instances, API requests, and judge requests are counted separately; multi-stage tool tasks and in-case repetitions may correspond to additional requests. ### Execution conditions * Within a comparison group, the task set, rounds, timeout, temperature, output budget, failure handling, and scoring formula are the same. * The same client, network region, and connection strategy are used, and account service tier, actual start and end times, and execution order are recorded. * A full test uses 3 windows at least 60 minutes apart; answer quality and response speed are executed separately. * Model ID, authentication, required fields, streaming format, usage mapping, cache semantics, and tokenizer basis form an adaptation record. * Access adaptation is registered and verified before sampling, and the same rules apply to all eligible model providers. * The primary scoring view consistently uses `F_native` and `K_native`; compatibility and default-configuration diagnostics are reported separately. * First-execution results are retained, and planned counts, actual counts, filtering, failures, invalid records, and unobserved records are reconciled separately. Tests are executed within approved quotas, permissions, and cost limits. When required conditions are not met, results retain a restricted, incomplete, or unavailable status, and remaining items are not renormalized. ## Dataset sources and result records ### Dataset sources Answer quality uses a versioned derived task set, retaining source, sampling records, and integrity verification information. The AF Full answer quality task set contains 775 distinct questions, with the following source snapshots: | Track | Source snapshot | Version record | License and usage boundaries | | - | - | - | - | | BFCL | [Berkeley Function Calling Leaderboard data and checker](https://github.com/ShishirPatil/gorilla/tree/6ea57973c7a6097fd7c5915698c54c17c5b1b6c8/berkeley-function-call-leaderboard) | `6ea57973c7a6` | Source repository Apache-2.0; this evaluation uses a derived task set and checker adaptation | | HLE | [Text mirror snapshot](https://huggingface.co/datasets/macabdul9/hle_text_only/tree/e3e9cb5a550ec7de645fac553b0e2768d30a03ee); [official dataset](https://huggingface.co/datasets/cais/hle) | Mirror `e3e9cb5a550e` | Recorded as MIT at the source; byte-for-byte equivalence with the current official version has not been verified | | AA-LCR | [Artificial Analysis AA-LCR data and documentation snapshot](https://huggingface.co/datasets/ArtificialAnalysis/AA-LCR/tree/9a77ef56b717057ade24ceab4d273712a0b4f19e) | `9a77ef56b717` | Dataset repository Apache-2.0; third-party rights in referenced documents remain with their respective owners | BFCL is sampled by category quota; HLE and AA-LCR are sampled by stratified proportion with minimum coverage guaranteed for each stratum. Sampling uses a designated seed and SHA-256 ordering, and the task set version, filter list, and grading protocol are recorded with the results. The links above serve to verify original sources and are not equivalent to a complete evaluation package that can directly reproduce AF scores. Restricted data is not redistributed in this page, and specific licenses follow the respective sources. ### Result records and verification Provider reports are collectively referred to as AF Profile and record the following evaluation information: | Record category | Required content | | - | - | | Evaluation identity | Standard and task set versions, model provider, exact model version, and the official reference service | | Measurement conditions | Thinking configuration, account tier, region, period, timeout, concurrency, and performance parameters | | Adaptation basis | Documentation review date, required fields, streaming structure, usage parsing, and cache configuration | | Sample coverage | Planned count, actual count, scoring denominator, filtering, failures, invalid, and unobserved items | | Grading basis | Reference answers, checker or judge configuration, grading status, and bias assessment | | Raw measurements | Redacted requests and responses, event times, status, termination reasons, and raw `usage` | | Aggregated results | A/B/C layer inputs, the four D scores, overall score, dimension intervals, and unrounded values | | Diagnostic metrics | `F_strict`, `K_default`, valid rate, error categories, and corresponding applicability limits | Service success, task correctness, and output validity are labeled separately. Negative test cases are not mixed into the service reliability score, cache reads and writes are metered separately, and the verification status of estimated and actual costs is recorded separately. Keys, user identifiers, and sensitive content are removed before external presentation; values carry units, and missing values retain their status. ## Reference data and calculation examples The following uses the results of five providers' services for DeepSeek V4.1 Flash 1017 to illustrate the calculation method. The primary score uses `F_native` and `K_native`. Values are displayed to two decimal places, and calculations use the intermediate values saved in the result files. > [!NOTE] > > This section presents reference results (PROVISIONAL). Prior registration evidence for collection and adaptation is incomplete, so these cannot serve as official AF Full results for AIHubMix Fidelitas Standard. Answer quality counts 4,240 instances per provider, with 10 additional filtered instances. The HLE/AA-LCR judges belong to the same DeepSeek family as the models under test, and independent calibration evidence is insufficient. Results are limited to the recorded configuration and observation scope. ### Absolute performance and cache subscores In the table below, Q, F, P, and R are all absolute performance A and **are not the four dimension display scores in the overall score**. K is the cache subscore within P. | Model provider | A\_Q | F\_native (A\_F) | A\_P | K\_native | A\_R | | - | - | - | - | - | - | | ByteDance | 42.05 | 74.98 | 84.37 | 93.83 | 100.00 | | Baidu | 38.00 | 88.39 | 37.77 | 68.85 | 100.00 | | Alibaba Cloud | 41.75 | 89.78 | 88.58 | 99.37 | 100.00 | | DeepSeek Official Reference | 42.45 | 94.00 | 86.14 | 98.38 | 100.00 | | Fireworks | 41.42 | 71.78 | 79.51 | 60.68 | 100.00 | * All five providers' service reliability suites are 300/300; service failures in other suites still count toward their own dimensions and cannot be replaced by the R suite result. * Fireworks' `K_native` uses the session affinity configuration specified in its documentation, and the other services are measured with the configuration required by their respective documentation. * Baidu's K uses the sample rule that parses an omitted field as 0 on zero hits, and the related prior registration evidence is incomplete; the 8k cold control has an incomplete record, so K retains the reference status. * DeepSeek's `F_strict` is unavailable because of 27 INVALID records; `F_native` is computed from valid applicable records. ### Functionality dimension composition Taking Alibaba Cloud as an example, the absolute functionality score is `A_F = 89.777778`, the official reference's absolute score for the same item is 94, the difference scale is `δ_F = 15`, and evidence completeness is `C_F = 100`. The relative reference score `B_F` is calculated first, and the dimension score `D_F` is then composed: ```text Relative reference and functionality dimension score theme={null} B_F = 50 + 50 × clip((89.777778 − 94) / 15, −1, 1) ≈ 35.925926 D_F = 0.65 × 89.777778 + 0.20 × 35.925926 + 0.15 × 100 ≈ 80.540741 ``` 89.78 is the native functionality absolute performance, and 80.54 is the functionality dimension display score used in the overall score calculation. ### Service reliability score of the official reference The official reference's R suite is 300/300, its absolute service reliability score is `A_R = 100`, its relative reference score is `B_R = 50`, and its evidence completeness is `C_R = 100`: ```text Service reliability dimension composition theme={null} D_R = 0.65 × 100 + 0.20 × 50 + 0.15 × 100 = 90 ``` 100% is the observed success rate of the service reliability suite in this round, and 90 is the dimension display score after three-layer composition. ### Dimension display scores and overall score The table below uses D values and corresponds to the overall score formula. The table follows the provider display order and is not sorted by score. | Model provider | Answer quality D\_Q | Functionality D\_F | Response speed D\_P | Service reliability D\_R | Overall score | | - | - | - | - | - | - | | ByteDance | 50.83 | 62.24 | 76.57 | 88.50 | **66.37** | | Baidu | 36.70 | 75.71 | 36.55 | 87.00 | **56.48** | | Alibaba Cloud | 52.13 | 80.54 | 85.01 | 90.00 | **73.38** | | DeepSeek Official Reference | 52.59 | 84.60 | 80.99 | 90.00 | **73.76** | | Fireworks | 50.42 | 60.16 | 68.55 | 88.50 | **64.10** | Taking the official reference as an example: ```text Overall score calculation theme={null} Overall = 0.35 × 52.594340 + 0.25 × 84.600000 + 0.20 × 80.990116 + 0.20 × 90.000000 ≈ 73.76 ``` ### Independent diagnostic metrics | Model provider | F\_strict | K\_default | | - | - | - | | ByteDance | 56.83 | 93.83 | | Baidu | 66.17 | 68.85 | | Alibaba Cloud | 89.78 | 99.37 | | DeepSeek Official Reference | Unavailable | 98.38 | | Fireworks | 71.78 | 11.06 | `F_strict` expresses interface compatibility, and `K_default` expresses the cache result obtained uniformly without session affinity configuration. Neither column replaces a primary score input. In the difference intervals for answer quality relative to the official reference, the 95% intervals for ByteDance, Alibaba Cloud, and Fireworks all include 0, so `B_Q = 50` for them. Baidu's difference is about −4.46 percentage points, with a 95% interval of approximately −5.78 to −3.16 percentage points. This conclusion is limited to this task set and observation conditions and is not extended to a statistical significance conclusion about the overall score. ## Standard and applicability conditions Standard identifier: **AIHubMix Fidelitas Standard v2.0**. Results also record AF Suite, adaptation configuration, grading protocol, and measurement conditions. Results under different configurations or statistical definitions are interpreted separately. The four scores on model pages are D values; correctness rates, functionality pass rates, latency, cache ratios, and first-attempt success rates in reports are the corresponding raw metrics or A values. The overall score is used for summary presentation, and specific integration decisions are based on dimension results, functional requirements, and evidence conditions. Related: [Choose a model](https://docs.aihubmix.com/en/guides/choose-models.md) · [Model details and comparison](https://docs.aihubmix.com/en/guides/model-details.md) Document date: 2026-11-05 This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.