eval_framework.metrics.loglikelihood package

Submodules

eval_framework.metrics.loglikelihood.accuracy_loglikelihood module

class eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyBayesianLoglikelihood[source]

Bases: BaseMetric[Loglikelihood]

Accuracy after adjusting the loglikelihoods for the byte-length bias of the completion. See https://arxiv.org/html/2607.12767v1 for more details.

NAME: str = 'Accuracy Bayesian Loglikelihood'
calculate(response)[source]
Return type:

list[MetricResult]

Parameters:

response (Loglikelihood)

prepare(responses)[source]

Estimating the length decay factor. See Equation (24) in https://arxiv.org/html/2607.12767v1

Return type:

None

Parameters:

responses (list[Loglikelihood])

class eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood[source]

Bases: BaseMetric[Loglikelihood]

NAME: str = 'Accuracy Loglikelihood'
calculate(response)[source]
Return type:

list[MetricResult]

Parameters:

response (Loglikelihood)

class eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood[source]

Bases: BaseMetric[Loglikelihood]

NAME: str = 'Accuracy Normalized Loglikelihood'
calculate(response)[source]
Return type:

list[MetricResult]

Parameters:

response (Loglikelihood)

class eval_framework.metrics.loglikelihood.accuracy_loglikelihood.PartialEvalAccuracy[source]

Bases: BaseMetric[Loglikelihood]

An accuracy metric for partial evaluation tasks, e.g. WinograndeCloze.

Here, for each item, we generate a pair of two samples, one for each option. We then calculate the accuracy of the model’s completion for each option, and then use the accuracy of the correct option to calculate the overall accuracy.

NOTE: The current implementation relies on the assumption that it comes in pairs of samples, which can be identified by having consecutive ids (odd and even). This is how it is implemented in the WinograndeCloze tasks, but if other tasks use this metric, it might not be the case and require a more general implementation (e.g. storing item_id in the Sample.context).

NAME: str = 'Partial Evaluation Accuracy'
calculate(response)[source]
Return type:

list[MetricResult]

Parameters:

response (Loglikelihood)

eval_framework.metrics.loglikelihood.base module

class eval_framework.metrics.loglikelihood.base.BaseLoglikelihoodMetric(*, len_normalised=True)[source]

Bases: BaseMetric[Loglikelihood]

Base class for metrics that operate on loglikelihood responses.

Parameters:

len_normalised (bool)

eval_framework.metrics.loglikelihood.bits_per_byte module

class eval_framework.metrics.loglikelihood.bits_per_byte.BitsPerByteLoglikelihood[source]

Bases: BaseMetric[Loglikelihood]

Bits-per-byte metric for loglikelihood responses.

This follows the Paloma definition: the negative log-likelihood of the answer divided by the number of UTF-8 bytes in the answer string.

NAME: str = 'BitsPerByte'
calculate(response)[source]
Return type:

list[MetricResult]

Parameters:

response (Loglikelihood)

eval_framework.metrics.loglikelihood.bpb_variants module

Extended bits-per-byte for loglikelihood responses.

Emits the standard BitsPerByte ratio, corpus-aggregation fields, and prefix BPB when per-token logprobs exist.

class eval_framework.metrics.loglikelihood.bpb_variants.BitsPerByteVariantsLoglikelihood[source]

Bases: BaseMetric[Loglikelihood]

Standard BPB plus prefix, prior, and corpus companion fields.

NAME: str = 'BitsPerByteVariants'
calculate(response)[source]
Return type:

list[MetricResult]

Parameters:

response (Loglikelihood)

eval_framework.metrics.loglikelihood.bpb_variants_common module

Shared logic for BitsPerByteVariantsLoglikelihood.

eval_framework.metrics.loglikelihood.bpb_variants_common.aggregate_prior_bpb_metrics(responses)[source]

Prior BPB from per-token logprobs (geometric prior, MU_PRIOR default 111).

Also emits Prior BPB prior_mass_in_support: the share of prior weight on byte positions k where at least one ground truth has length >= k (see prior_bpb in bpb_estimators.py). It is near 1 on tasks with long ground truths and lower on tasks with short ground truths under a large mu, such as single-letter labels.

Return type:

dict[str, float | None]

Parameters:

responses (list[Loglikelihood])

eval_framework.metrics.loglikelihood.bpb_variants_common.build_prefix_item(response, ground_truth)[source]
Return type:

PrefixItem | None

Parameters:

  • response (Loglikelihood)

  • ground_truth (str)

eval_framework.metrics.loglikelihood.bpb_variants_common.collect_prefix_items(responses)[source]
Return type:

list[PrefixItem]

Parameters:

responses (list[Loglikelihood])

eval_framework.metrics.loglikelihood.bpb_variants_common.compute_all_bpb_results(response)[source]
Return type:

list[MetricResult]

Parameters:

response (Loglikelihood)

eval_framework.metrics.loglikelihood.bpb_variants_common.compute_prefix_bpb_results(response, ground_truth)[source]
Return type:

list[MetricResult]

Parameters:

  • response (Loglikelihood)

  • ground_truth (str)

eval_framework.metrics.loglikelihood.bpb_variants_common.compute_prior_bpb_for_items(items, mu=None)[source]
Return type:

dict[str, float | None]

Parameters:
eval_framework.metrics.loglikelihood.bpb_variants_common.compute_standard_bpb_results(response, answer_text, candidates)[source]
Return type:

list[MetricResult]

Parameters:

  • response (Loglikelihood)

  • answer_text (str)

  • candidates (list[str])

eval_framework.metrics.loglikelihood.bpb_variants_common.list_ground_truth_candidates(response)[source]
Return type:

list[str]

Parameters:

response (Loglikelihood)

eval_framework.metrics.loglikelihood.bpb_variants_common.select_ground_truth(response, candidates=None)[source]
Return type:

str | None

Parameters:

  • response (Loglikelihood)

  • candidates (list[str] | None)

eval_framework.metrics.loglikelihood.bpb_variants_common.standard_bpb_error(message, response)[source]
Return type:

list[MetricResult]

Parameters:

  • message (str)

  • response (Loglikelihood)

eval_framework.metrics.loglikelihood.bpb_variants_estimators module

Corpus-level BPB estimators.

Pure functions over parallel (bits, nbytes[, tokens]) arrays. Not per-item BaseMetrics: each collapses a batch of (bits, nbytes) pairs to one scalar.

class eval_framework.metrics.loglikelihood.bpb_variants_estimators.AffineFit(intercept, slope, r2, method)[source]

Bases: object

Parameters:

  • intercept (float)

  • slope (float)

  • r2 (float)

  • method (str)

intercept: float
method: str
r2: float
slope: float
class eval_framework.metrics.loglikelihood.bpb_variants_estimators.PrefixItem(bits, byte_lens, offset=0)[source]

Bases: object

Byte-indexed costs for one item (prefix / Prior BPB).

bits[j]: -log2 prob of token j; byte_lens[j]: UTF-8 length of token j; offset: content bytes to skip at the start (0 or 1 for a leading space).

Parameters:

  • bits (ndarray)

  • byte_lens (ndarray)

  • offset (int)

bits: ndarray
byte_lens: ndarray
property content_bytes: int
offset: int = 0
property total_bytes: int
eval_framework.metrics.loglikelihood.bpb_variants_estimators.bin_mass(nbytes, bins)[source]

Fraction of nbytes falling in each bin. Sums to 1.

Return type:

ndarray

Parameters:

  • nbytes (ndarray)

  • bins (list[tuple[int, int]])

eval_framework.metrics.loglikelihood.bpb_variants_estimators.bits_per_answer(bits)[source]
Return type:

float

Parameters:

bits (ndarray)

eval_framework.metrics.loglikelihood.bpb_variants_estimators.bpb_at_nstar(bits, nbytes, n_star, method='ols')[source]
Return type:

dict[str, Any]

Parameters:

  • bits (ndarray)

  • nbytes (ndarray)

  • n_star (float)

  • method (str)

eval_framework.metrics.loglikelihood.bpb_variants_estimators.continuation_rate(items, k1, k2)[source]

R(k1,k2): mean per-byte bits from content byte k1 to k2, over items with support.

Return type:

float

Parameters:
eval_framework.metrics.loglikelihood.bpb_variants_estimators.corpus_bpb(bits, nbytes)[source]
Return type:

float

Parameters:

  • bits (ndarray)

  • nbytes (ndarray)

eval_framework.metrics.loglikelihood.bpb_variants_estimators.cumulative_cost(bits, byte_lens, k_abs)[source]

Bits for the first k_abs bytes. Linear split inside a token that straddles k.

Return type:

float

Parameters:

  • bits (ndarray)

  • byte_lens (ndarray)

  • k_abs (float)

eval_framework.metrics.loglikelihood.bpb_variants_estimators.effective_length_bpb(bits, nbytes)[source]

mean L_i / (n_i + alpha/beta). Undefined when beta <= 0.

Return type:

dict[str, Any]

Parameters:

  • bits (ndarray)

  • nbytes (ndarray)

eval_framework.metrics.loglikelihood.bpb_variants_estimators.entry_cost(items, k0)[source]

A(k0): mean bits to cover the first k0 content bytes, over items with support.

Return type:

float

Parameters:
eval_framework.metrics.loglikelihood.bpb_variants_estimators.geometric_survival(mu, kmax)[source]

s(k) = (1 - 1/mu)^(k-1), k=1..kmax. Mean-mu geometric length prior.

Return type:

ndarray

Parameters:

  • mu (float)

  • kmax (int)

eval_framework.metrics.loglikelihood.bpb_variants_estimators.huber_affine(nbytes, bits, c=1.345, max_iter=50)[source]

IRLS Huber regression of bits on nbytes with intercept.

Return type:

AffineFit

Parameters:

  • nbytes (ndarray)

  • bits (ndarray)

  • c (float)

  • max_iter (int)

eval_framework.metrics.loglikelihood.bpb_variants_estimators.ls_bpb(bits, nbytes, q_nbytes=None, bins=None, n_quintiles=5, q_mass=None)[source]

Length-standardized BPB with fixed nonnegative stratum weights.

q_nbytes: lengths that define q (default: nbytes). bins=None uses quintiles of q_nbytes. q_mass: explicit stratum weights instead of a q_nbytes histogram (e.g. equal task weight).

Return type:

dict[str, Any]

Parameters:

  • bits (ndarray)

  • nbytes (ndarray)

  • q_nbytes (ndarray | None)

  • bins (list[tuple[int, int]] | None)

  • n_quintiles (int)

  • q_mass (ndarray | None)

eval_framework.metrics.loglikelihood.bpb_variants_estimators.make_prefix_items(per_token, choices=None)[source]

Build PrefixItems from parallel (bits, byte_lens) pairs.

If choices is given, a leading space sets offset=1 (excluded from content bytes).

Return type:

list[PrefixItem]

Parameters:

  • per_token (list[tuple[list[float] | ndarray, list[float] | ndarray]])

  • choices (list[str] | None)

eval_framework.metrics.loglikelihood.bpb_variants_estimators.marginal_rate_curve(items, kmax=None)[source]

(k, R(k), coverage(k)) for k=1..kmax, using every byte of every item.

R(k) is the mean marginal cost at content byte k over all items that reach it.

Return type:

tuple[ndarray, ndarray, ndarray]

Parameters:
eval_framework.metrics.loglikelihood.bpb_variants_estimators.mean_cost_curve(items, kmax=None)[source]

Mean cumulative cost A(k) = sum_{j<=k} R(j).

Return type:

tuple[ndarray, ndarray]

Parameters:
eval_framework.metrics.loglikelihood.bpb_variants_estimators.mean_of_ratios(bits, nbytes)[source]
Return type:

float

Parameters:

  • bits (ndarray)

  • nbytes (ndarray)

eval_framework.metrics.loglikelihood.bpb_variants_estimators.ols_affine(nbytes, bits)[source]
Return type:

AffineFit

Parameters:

  • nbytes (ndarray)

  • bits (ndarray)

eval_framework.metrics.loglikelihood.bpb_variants_estimators.ols_leverage_weights(nbytes, n_star)[source]

h_i such that predicted L(n*) = sum h_i L_i.

Return type:

ndarray

Parameters:

  • nbytes (ndarray)

  • n_star (float)

eval_framework.metrics.loglikelihood.bpb_variants_estimators.pbpb(items, k)[source]

Prefix BPB at budget k: mean_i C_i(k)/k over items with content_bytes >= k.

Return type:

float

Parameters:
eval_framework.metrics.loglikelihood.bpb_variants_estimators.per_byte_cost(item)[source]

Bits per content byte. Same linear split as cumulative_cost; zero-byte tokens attach to the previous byte; offset bytes are skipped.

Return type:

ndarray

Parameters:

item (PrefixItem)

eval_framework.metrics.loglikelihood.bpb_variants_estimators.prefix_curve(items, ks)[source]

PBPB and coverage across a grid of byte budgets ks.

Return type:

list[dict[str, Any]]

Parameters:

  • items (list[PrefixItem])

  • ks (list[float] | ndarray)

eval_framework.metrics.loglikelihood.bpb_variants_estimators.prefix_decompose(items, k0, k)[source]

Exact split PBPB(k) = [A(k0) + R(k0,k)*(k-k0)] / k on a common support (>= k).

Return type:

dict[str, Any]

Parameters:
eval_framework.metrics.loglikelihood.bpb_variants_estimators.prior_bpb(items, prior='geometric', mu=None, ref_lengths=None, kmax=None)[source]

Prior BPB: weighted average of R(k) under a fixed length prior.

prior=”geometric”: mean mu. prior=”reference”: ref_lengths (task lengths give corpus BPB).

Return type:

dict[str, Any]

Parameters:

  • items (list[PrefixItem])

  • prior (str)

  • mu (float | None)

  • ref_lengths (ndarray | None)

  • kmax (int | None)

eval_framework.metrics.loglikelihood.bpb_variants_estimators.reference_survival(ref_lengths, kmax)[source]

s(k) = P(N >= k) for a fixed reference length sample, k=1..kmax.

Return type:

ndarray

Parameters:

  • ref_lengths (ndarray)

  • kmax (int)

eval_framework.metrics.loglikelihood.bpb_variants_estimators.resolve_nstar(nbytes, spec)[source]
Return type:

float

Parameters:

  • nbytes (ndarray)

  • spec (float | int | str)

eval_framework.metrics.loglikelihood.bpb_variants_estimators.space_stripped_corpus_bpb(bits, nbytes, leading_space)[source]

Corpus BPB after removing one leading-space byte where the gold had one.

Return type:

float

Parameters:

  • bits (ndarray)

  • nbytes (ndarray)

  • leading_space (ndarray)

eval_framework.metrics.loglikelihood.bpb_variants_estimators.summarize_all(bits, nbytes, tokens=None, q_nbytes=None, leading_space=None)[source]
Return type:

dict[str, Any]

Parameters:

  • bits (ndarray)

  • nbytes (ndarray)

  • tokens (ndarray | None)

  • q_nbytes (ndarray | None)

  • leading_space (ndarray | None)

eval_framework.metrics.loglikelihood.bpb_variants_instrumented module

Deprecated BitsPerByteVariantsLoglikelihood wrapper.

class eval_framework.metrics.loglikelihood.bpb_variants_instrumented.InstrumentedBitsPerByte[source]

Bases: BaseMetric[Loglikelihood]

Use BitsPerByteVariantsLoglikelihood instead.

NAME: str = 'BitsPerByteVariants'
calculate(response)[source]
Return type:

list[MetricResult]

Parameters:

response (Loglikelihood)

eval_framework.metrics.loglikelihood.bpb_variants_instrumented.compute_standard_bpb_results(response, answer_text, candidates)[source]
Return type:

list[MetricResult]

Parameters:

  • response (Loglikelihood)

  • answer_text (str)

  • candidates (list[str])

eval_framework.metrics.loglikelihood.bpb_variants_instrumented.list_ground_truth_candidates(response)[source]
Return type:

list[str]

Parameters:

response (Loglikelihood)

eval_framework.metrics.loglikelihood.bpb_variants_instrumented.select_ground_truth(response, candidates=None)[source]
Return type:

str | None

Parameters:

  • response (Loglikelihood)

  • candidates (list[str] | None)

eval_framework.metrics.loglikelihood.bpb_variants_instrumented.standard_bpb_error(message, response)[source]
Return type:

list[MetricResult]

Parameters:

  • message (str)

  • response (Loglikelihood)

eval_framework.metrics.loglikelihood.bpb_variants_prefix module

Deprecated wrapper for prefix BPB. Use BitsPerByteVariantsLoglikelihood.

class eval_framework.metrics.loglikelihood.bpb_variants_prefix.PrefixBitsPerByte[source]

Bases: BaseMetric[Loglikelihood]

Use BitsPerByteVariantsLoglikelihood instead.

NAME: str = 'PrefixBPB'
calculate(response)[source]
Return type:

list[MetricResult]

Parameters:

response (Loglikelihood)

eval_framework.metrics.loglikelihood.bpb_variants_prefix.compute_prefix_bpb_results(response, ground_truth)[source]
Return type:

list[MetricResult]

Parameters:

  • response (Loglikelihood)

  • ground_truth (str)

eval_framework.metrics.loglikelihood.confidence_weighted_accuracy module

class eval_framework.metrics.loglikelihood.confidence_weighted_accuracy.ConfidenceWeightedAccuracy(*, len_normalised=True)[source]

Bases: BaseLoglikelihoodMetric

Parameters:

len_normalised (bool)

NAME: str = 'Confidence-weighted Accuracy'
calculate(response)[source]
Return type:

list[MetricResult]

Parameters:

response (Loglikelihood)

eval_framework.metrics.loglikelihood.dcs module

class eval_framework.metrics.loglikelihood.dcs.DistributionalCorrectnessScore(*, lc=1.0, lw=1.0, len_normalised=True)[source]

Bases: BaseLoglikelihoodMetric

Based on Burns (2025) Measuring Language Model Hallucinations Through Distributional Correctness.

Parameters:

  • lc (float)

  • lw (float)

  • len_normalised (bool)

NAME: str = 'Distributional Correctness Score'
calculate(response)[source]
Return type:

list[MetricResult]

Parameters:

response (Loglikelihood)

eval_framework.metrics.loglikelihood.probability_mass module

class eval_framework.metrics.loglikelihood.probability_mass.ProbabilityMass[source]

Bases: BaseMetric[Loglikelihood]

NAME: str = 'Probability Mass'
calculate(response)[source]
Return type:

list[MetricResult]

Parameters:

response (Loglikelihood)

class eval_framework.metrics.loglikelihood.probability_mass.ProbabilityMassNorm[source]

Bases: BaseMetric[Loglikelihood]

NAME: str = 'Probability Mass Normalized'
calculate(response)[source]
Return type:

list[MetricResult]

Parameters:

response (Loglikelihood)

eval_framework.metrics.loglikelihood.ternary module

class eval_framework.metrics.loglikelihood.ternary.TernaryScore(*, lc=1.0, lw=1.0, len_normalised=True)[source]

Bases: BaseLoglikelihoodMetric

Based on Kalai et al. (2025) Why language models hallucinate. arXiv:2509.04664

Parameters:

  • lc (float)

  • lw (float)

  • len_normalised (bool)

NAME: str = 'Ternary Score'
calculate(response)[source]
Return type:

list[MetricResult]

Parameters:

response (Loglikelihood)

Module contents