eval_framework.metrics package¶
Subpackages¶
- eval_framework.metrics.aggregators package
- eval_framework.metrics.completion package
- Submodules
- eval_framework.metrics.completion.accuracy_completion module
- eval_framework.metrics.completion.code_assertion module
- eval_framework.metrics.completion.code_execution_pass_at_one module
- eval_framework.metrics.completion.concordance_index module
- eval_framework.metrics.completion.csv_format module
- eval_framework.metrics.completion.drop_completion module
- eval_framework.metrics.completion.exponential_similarity module
- eval_framework.metrics.completion.f1 module
- eval_framework.metrics.completion.format_checker module
- eval_framework.metrics.completion.grid_difference module
- eval_framework.metrics.completion.ifeval module
- eval_framework.metrics.completion.json_format module
- eval_framework.metrics.completion.language_checker module
- eval_framework.metrics.completion.length_control module
- eval_framework.metrics.completion.math_minerva_completion module
- eval_framework.metrics.completion.math_reasoning_completion module
- eval_framework.metrics.completion.minerva_math_utils module
- eval_framework.metrics.completion.multipl_e_assertion module
- eval_framework.metrics.completion.placeholder_checker module
- eval_framework.metrics.completion.repetition module
- eval_framework.metrics.completion.rouge_1 module
- eval_framework.metrics.completion.rouge_2 module
- eval_framework.metrics.completion.rouge_geometric_mean module
- eval_framework.metrics.completion.rouge_l module
- eval_framework.metrics.completion.text_counter module
- Module contents
- eval_framework.metrics.efficiency package
- Submodules
- eval_framework.metrics.efficiency.bytes_per_sequence_position module
- eval_framework.metrics.efficiency.finish_reason module
- eval_framework.metrics.efficiency.token_counters module
- Module contents
- eval_framework.metrics.llm package
- eval_framework.metrics.loglikelihood package
- Submodules
- eval_framework.metrics.loglikelihood.accuracy_loglikelihood module
- eval_framework.metrics.loglikelihood.base module
- eval_framework.metrics.loglikelihood.bits_per_byte module
- eval_framework.metrics.loglikelihood.bpb_variants module
- eval_framework.metrics.loglikelihood.bpb_variants_common module
- eval_framework.metrics.loglikelihood.bpb_variants_estimators module
AffineFitPrefixItembin_mass()bits_per_answer()bpb_at_nstar()continuation_rate()corpus_bpb()cumulative_cost()effective_length_bpb()entry_cost()geometric_survival()huber_affine()ls_bpb()make_prefix_items()marginal_rate_curve()mean_cost_curve()mean_of_ratios()ols_affine()ols_leverage_weights()pbpb()per_byte_cost()prefix_curve()prefix_decompose()prior_bpb()reference_survival()resolve_nstar()space_stripped_corpus_bpb()summarize_all()
- eval_framework.metrics.loglikelihood.bpb_variants_instrumented module
- eval_framework.metrics.loglikelihood.bpb_variants_prefix module
- eval_framework.metrics.loglikelihood.confidence_weighted_accuracy module
- eval_framework.metrics.loglikelihood.dcs module
- eval_framework.metrics.loglikelihood.probability_mass module
- eval_framework.metrics.loglikelihood.ternary module
- Module contents
Submodules¶
eval_framework.metrics.base module¶
- class eval_framework.metrics.base.BaseMetric[source]¶
Bases:
ABC,Generic-
AGGREGATORS:
list[Aggregator] = []¶
-
KEYS:
list[str] |None= None¶
-
MAX_WORKERS:
int= 1¶
-
NAME:
str¶
- NAMES¶
- abstractmethod calculate(response)[source]¶
- Return type:
list[MetricResult]- Parameters:
response (Response)
-
AGGREGATORS:
- class eval_framework.metrics.base.MetricResult(**data)[source]¶
Bases:
BaseModel- Parameters:
metric_name (str)
value (float | None)
higher_is_better (bool)
llm_judge_prompt (str | None)
llm_judge_response (str | None)
code_execution_trace (str | None)
error (Error | None)
-
code_execution_trace:
str|None¶
-
error:
Error|None¶
-
higher_is_better:
bool¶
-
llm_judge_prompt:
str|None¶
-
llm_judge_response:
str|None¶
-
metric_name:
str¶
- model_config: ClassVar[ConfigDict] = {'extra': 'forbid'}¶
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
-
value:
float|None¶