eval_framework.metrics package

Subpackages

• eval_framework.metrics.completion package
  • Submodules
  • eval_framework.metrics.completion.accuracy_completion module
    • AccuracyCompletion
      • AccuracyCompletion.NAME
      • AccuracyCompletion.calculate()
  • eval_framework.metrics.completion.aidanbench module
    • AidanBenchMetric
      • AidanBenchMetric.NAME
      • AidanBenchMetric.calculate()
  • eval_framework.metrics.completion.bleu module
    • BLEU
      • BLEU.NAME
      • BLEU.calculate()
    • LINEWISE_BLEU
      • LINEWISE_BLEU.NAME
      • LINEWISE_BLEU.calculate()
    • ResponseToOriginalBLEU
      • ResponseToOriginalBLEU.NAME
      • ResponseToOriginalBLEU.calculate()
  • eval_framework.metrics.completion.chrf module
    • CHRF
      • CHRF.NAME
      • CHRF.calculate()
    • LINEWISE_CHRF
      • LINEWISE_CHRF.NAME
      • LINEWISE_CHRF.calculate()
  • eval_framework.metrics.completion.code_assertion module
    • CodeCompletionAssertion
      • CodeCompletionAssertion.NAME
      • CodeCompletionAssertion.calculate()
  • eval_framework.metrics.completion.code_execution_pass_at_one module
    • CodeExecutionBaseContext
      • CodeExecutionBaseContext.benchmark_timeout
      • CodeExecutionBaseContext.code_prompt
      • CodeExecutionBaseContext.model_config
      • CodeExecutionBaseContext.package_downloads
      • CodeExecutionBaseContext.run_env
      • CodeExecutionBaseContext.test_code
    • CodeExecutionPassAtOne
      • CodeExecutionPassAtOne.NAME
      • CodeExecutionPassAtOne.calculate()
    • CodeExecutionPassAtOneContext
      • CodeExecutionPassAtOneContext.model_config
      • CodeExecutionPassAtOneContext.output_parse_fn
      • CodeExecutionPassAtOneContext.snippet_merge_fn
    • RealtimeCodeExectionContext
      • RealtimeCodeExectionContext.from_context()
      • RealtimeCodeExectionContext.model_config
      • RealtimeCodeExectionContext.output_parse_fn
      • RealtimeCodeExectionContext.snippet_merge_fn
    • estimate_pass_at_k()
  • eval_framework.metrics.completion.comet module
    • COMET
      • COMET.NAME
      • COMET.calculate()
  • eval_framework.metrics.completion.concordance_index module
    • ConcordanceIndex
      • ConcordanceIndex.NAME
      • ConcordanceIndex.calculate()
    • calculate_concordance_index()
  • eval_framework.metrics.completion.csv_format module
    • CSVFormat
      • CSVFormat.KEYS
      • CSVFormat.NAME
      • CSVFormat.calculate()
    • CSVFormatEvaluation
      • CSVFormatEvaluation.has_csv
      • CSVFormatEvaluation.implicit
      • CSVFormatEvaluation.is_column_count_respected
      • CSVFormatEvaluation.is_separator_respected
      • CSVFormatEvaluation.model_config
    • evaluate_csv_format()
    • extract_csv_from_text()
  • eval_framework.metrics.completion.cwe_accuracy module
    • CWEAccuracy
      • CWEAccuracy.NAME
      • CWEAccuracy.calculate()
  • eval_framework.metrics.completion.exponential_similarity module
    • ExponentialSimilarity
      • ExponentialSimilarity.NAME
      • ExponentialSimilarity.calculate()
    • calculate_exponential_similarity()
  • eval_framework.metrics.completion.f1 module
    • F1
      • F1.NAME
      • F1.calculate()
    • calculate_f1()
  • eval_framework.metrics.completion.format_checker module
    • CheckJsonFormat
      • CheckJsonFormat.NAME
      • CheckJsonFormat.calculate()
    • CheckPostScriptFormat
      • CheckPostScriptFormat.NAME
      • CheckPostScriptFormat.calculate()
  • eval_framework.metrics.completion.grid_difference module
    • GridDifference
      • GridDifference.NAME
      • GridDifference.calculate()
      • GridDifference.calculate_score()
      • GridDifference.count_differences()
      • GridDifference.extract_grid_from_prompt()
  • eval_framework.metrics.completion.ifeval module
    • IFEvalMetric
      • IFEvalMetric.NAME
      • IFEvalMetric.calculate()
    • IFEvalMetricContext
      • IFEvalMetricContext.additional_kwargs
      • IFEvalMetricContext.instruction_id_list
      • IFEvalMetricContext.key
      • IFEvalMetricContext.model_config
      • IFEvalMetricContext.prompt
  • eval_framework.metrics.completion.json_format module
    • JsonFormat
      • JsonFormat.NAME
      • JsonFormat.calculate()
    • JsonFormatEvaluation
      • JsonFormatEvaluation.exact_match
      • JsonFormatEvaluation.fulfills_schema
      • JsonFormatEvaluation.is_just_json
      • JsonFormatEvaluation.is_valid_json
      • JsonFormatEvaluation.json_parsing_error
      • JsonFormatEvaluation.model_config
      • JsonFormatEvaluation.schema_validation_error
    • get_json_object()
    • remove_comments()
  • eval_framework.metrics.completion.language_checker module
    • GermanCompletionChecker
      • GermanCompletionChecker.NAME
      • GermanCompletionChecker.calculate()
    • LanguageChecker
      • LanguageChecker.NAME
      • LanguageChecker.calculate()
    • LanguageConsistencyChecker
      • LanguageConsistencyChecker.NAME
      • LanguageConsistencyChecker.calculate()
    • LanguageRawConsistencyChecker
      • LanguageRawConsistencyChecker.NAME
      • LanguageRawConsistencyChecker.calculate()
  • eval_framework.metrics.completion.length_control module
    • LengthControl
      • LengthControl.NAME
      • LengthControl.calculate()
    • LengthRequirementType
      • LengthRequirementType.MAX
      • LengthRequirementType.MIN
      • LengthRequirementType.TARGET
    • LengthRequirementUnit
      • LengthRequirementUnit.PARAGRAPHS
      • LengthRequirementUnit.SENTENCES
      • LengthRequirementUnit.WORDS
  • eval_framework.metrics.completion.math_reasoning_completion module
    • MathReasoningCompletion
      • MathReasoningCompletion.NAME
      • MathReasoningCompletion.REMOVED_EXPRESSIONS_FORMAT
      • MathReasoningCompletion.REMOVED_EXPRESSIONS_UNITS
      • MathReasoningCompletion.SUBSTITUTIONS
      • MathReasoningCompletion.calculate()
      • MathReasoningCompletion.check_for_equation()
      • MathReasoningCompletion.normalize_expression()
    • timeout_handler()
  • eval_framework.metrics.completion.niah_accuracy module
    • NIAHAccuracy
      • NIAHAccuracy.NAME
      • NIAHAccuracy.calculate()
    • clean_text()
  • eval_framework.metrics.completion.placeholder_checker module
    • PlaceholderChecker
      • PlaceholderChecker.NAME
      • PlaceholderChecker.calculate()
    • PlaceholderCheckerMetricContext
      • PlaceholderCheckerMetricContext.model_config
      • PlaceholderCheckerMetricContext.num_placeholders
  • eval_framework.metrics.completion.repetition module
    • WordRepetition
      • WordRepetition.HIGHER_IS_BETTER
      • WordRepetition.NAME
      • WordRepetition.calculate()
  • eval_framework.metrics.completion.rouge_1 module
    • ROUGE_1
      • ROUGE_1.NAME
      • ROUGE_1.calculate()
  • eval_framework.metrics.completion.rouge_2 module
    • ROUGE_2
      • ROUGE_2.NAME
      • ROUGE_2.calculate()
  • eval_framework.metrics.completion.rouge_geometric_mean module
    • ROUGE_GEOMETRIC_MEAN
      • ROUGE_GEOMETRIC_MEAN.NAME
      • ROUGE_GEOMETRIC_MEAN.calculate()
  • eval_framework.metrics.completion.rouge_l module
    • ROUGE_L
      • ROUGE_L.NAME
      • ROUGE_L.calculate()
  • eval_framework.metrics.completion.struct_eval_metrics module
    • RenderableStructMetric
      • RenderableStructMetric.NAME
      • RenderableStructMetric.calculate()
    • RenderableStructMetricContext
      • RenderableStructMetricContext.keywords
      • RenderableStructMetricContext.model_config
      • RenderableStructMetricContext.output_type
    • StructMetric
      • StructMetric.NAME
      • StructMetric.calculate()
    • StructMetricContext
      • StructMetricContext.model_config
      • StructMetricContext.output_type
      • StructMetricContext.paths
    • is_valid_html()
    • path_exists()
    • tokenize_path()
  • eval_framework.metrics.completion.ter module
    • LINEWISE_TER
      • LINEWISE_TER.NAME
      • LINEWISE_TER.calculate()
    • TER
      • TER.NAME
      • TER.calculate()
  • eval_framework.metrics.completion.text_counter module
    • ParagraphCounter
      • ParagraphCounter.NAME
      • ParagraphCounter.calculate()
    • ParagraphCounterMetricContext
      • ParagraphCounterMetricContext.comparison
      • ParagraphCounterMetricContext.model_config
      • ParagraphCounterMetricContext.paragraph_count
    • ResponseToOriginalLengthRatio
      • ResponseToOriginalLengthRatio.NAME
      • ResponseToOriginalLengthRatio.calculate()
    • SentenceCounter
      • SentenceCounter.NAME
      • SentenceCounter.calculate()
    • SentenceCounterMetricContext
      • SentenceCounterMetricContext.comparison
      • SentenceCounterMetricContext.model_config
      • SentenceCounterMetricContext.sentence_count
    • WordCounter
      • WordCounter.NAME
      • WordCounter.calculate()
    • WordCounterMetricContext
      • WordCounterMetricContext.comparison
      • WordCounterMetricContext.model_config
      • WordCounterMetricContext.word_count
  • Module contents
• eval_framework.metrics.efficiency package
  • Submodules
  • eval_framework.metrics.efficiency.bytes_per_sequence_position module
    • BytesCompletion
      • BytesCompletion.NAME
      • BytesCompletion.calculate()
    • BytesLoglikelihood
      • BytesLoglikelihood.NAME
      • BytesLoglikelihood.calculate()
    • SequencePositionsCompletion
      • SequencePositionsCompletion.NAME
      • SequencePositionsCompletion.calculate()
    • SequencePositionsLoglikelihood
      • SequencePositionsLoglikelihood.NAME
      • SequencePositionsLoglikelihood.calculate()
  • Module contents
• eval_framework.metrics.llm package
  • Submodules
  • eval_framework.metrics.llm.base module
    • BaseLLMJudgeMetric
  • eval_framework.metrics.llm.llm_judge_chatbot_style module
    • LLMJudgeChatbotStyle
      • LLMJudgeChatbotStyle.NAME
      • LLMJudgeChatbotStyle.calculate()
  • eval_framework.metrics.llm.llm_judge_coherence module
    • LLMJudgeCoherence
      • LLMJudgeCoherence.KEYS
      • LLMJudgeCoherence.NAME
      • LLMJudgeCoherence.calculate()
  • eval_framework.metrics.llm.llm_judge_completion_accuracy module
    • LLMJudgeCompletionAccuracy
      • LLMJudgeCompletionAccuracy.NAME
      • LLMJudgeCompletionAccuracy.calculate()
  • eval_framework.metrics.llm.llm_judge_conciseness module
    • LLMJudgeConciseness
      • LLMJudgeConciseness.NAME
      • LLMJudgeConciseness.calculate()
  • eval_framework.metrics.llm.llm_judge_contains_names module
    • LLMJudgeAvoidsNames
      • LLMJudgeAvoidsNames.NAME
      • LLMJudgeAvoidsNames.calculate()
  • eval_framework.metrics.llm.llm_judge_format_correctness module
    • LLMJudgeFormatCorrectness
      • LLMJudgeFormatCorrectness.NAME
      • LLMJudgeFormatCorrectness.calculate()
    • LLMJudgeFormatCorrectnessContext
      • LLMJudgeFormatCorrectnessContext.language
      • LLMJudgeFormatCorrectnessContext.model_config
  • eval_framework.metrics.llm.llm_judge_instruction module
    • LLMJudgeInstruction
      • LLMJudgeInstruction.KEYS
      • LLMJudgeInstruction.NAME
      • LLMJudgeInstruction.calculate()
  • eval_framework.metrics.llm.llm_judge_mtbench_pair module
    • MTBenchJudgePair
      • MTBenchJudgePair.NAME
      • MTBenchJudgePair.calculate()
    • MTBenchJudgePairMetricContext
      • MTBenchJudgePairMetricContext.answer
      • MTBenchJudgePairMetricContext.category
      • MTBenchJudgePairMetricContext.model_config
      • MTBenchJudgePairMetricContext.reference
    • PromptToJudge
      • PromptToJudge.candidate_is_a
      • PromptToJudge.comparison_type
      • PromptToJudge.model_config
      • PromptToJudge.prompt_text
    • generate_pair_judge_prompts()
  • eval_framework.metrics.llm.llm_judge_mtbench_single module
    • MTBenchJudgeSingle
      • MTBenchJudgeSingle.NAME
      • MTBenchJudgeSingle.calculate()
    • MTBenchJudgeSingleMetricContext
      • MTBenchJudgeSingleMetricContext.category
      • MTBenchJudgeSingleMetricContext.model_config
      • MTBenchJudgeSingleMetricContext.reference
    • PromptToJudge
      • PromptToJudge.comparison_type
      • PromptToJudge.model_config
      • PromptToJudge.prompt_text
    • generate_single_judge_prompts()
  • eval_framework.metrics.llm.llm_judge_refusal module
    • LLMJudgeRefusal
      • LLMJudgeRefusal.NAME
      • LLMJudgeRefusal.calculate()
  • eval_framework.metrics.llm.llm_judge_sql module
    • LLMJudgeSql
      • LLMJudgeSql.NAME
      • LLMJudgeSql.calculate()
      • LLMJudgeSql.connect_to_mysql()
      • LLMJudgeSql.connect_to_postgres()
      • LLMJudgeSql.validate_query()
      • LLMJudgeSql.validate_query_mysql()
      • LLMJudgeSql.validate_query_postgres()
      • LLMJudgeSql.validate_query_sqlite()
    • LLMJudgeSqlMetricContext
      • LLMJudgeSqlMetricContext.db_schema
      • LLMJudgeSqlMetricContext.dialect
      • LLMJudgeSqlMetricContext.model_config
    • SqlDialects
      • SqlDialects.mysql
      • SqlDialects.postgres
      • SqlDialects.sqlite
      • SqlDialects.standard_sql
    • SqlOutputComparison
      • SqlOutputComparison.matches_column_count
      • SqlOutputComparison.matches_results_count
      • SqlOutputComparison.model_config
      • SqlOutputComparison.results_equal
    • SqlValidationResult
      • SqlValidationResult.model_config
      • SqlValidationResult.query_error
      • SqlValidationResult.results
      • SqlValidationResult.schema_error
      • SqlValidationResult.success
    • count_result_columns()
    • extract_query_from_completions()
    • is_create_table_statement()
    • separate_statements()
  • eval_framework.metrics.llm.llm_judge_world_knowledge module
    • LLMJudgeWorldKnowledge
      • LLMJudgeWorldKnowledge.NAME
      • LLMJudgeWorldKnowledge.calculate()
  • eval_framework.metrics.llm.utils module
    • order_answers_for_comparison()
  • Module contents
• eval_framework.metrics.loglikelihood package
  • Submodules
  • eval_framework.metrics.loglikelihood.accuracy_loglikelihood module
    • AccuracyLoglikelihood
      • AccuracyLoglikelihood.NAME
      • AccuracyLoglikelihood.calculate()
    • AccuracyNormLoglikelihood
      • AccuracyNormLoglikelihood.NAME
      • AccuracyNormLoglikelihood.calculate()
  • eval_framework.metrics.loglikelihood.base module
    • BaseLoglikelihoodMetric
  • eval_framework.metrics.loglikelihood.confidence_weighted_accuracy module
    • ConfidenceWeightedAccuracy
      • ConfidenceWeightedAccuracy.NAME
      • ConfidenceWeightedAccuracy.calculate()
  • eval_framework.metrics.loglikelihood.dcs module
    • DistributionalCorrectnessScore
      • DistributionalCorrectnessScore.NAME
      • DistributionalCorrectnessScore.calculate()
  • eval_framework.metrics.loglikelihood.probability_mass module
    • ProbabilityMass
      • ProbabilityMass.NAME
      • ProbabilityMass.calculate()
    • ProbabilityMassNorm
      • ProbabilityMassNorm.NAME
      • ProbabilityMassNorm.calculate()
  • eval_framework.metrics.loglikelihood.ternary module
    • TernaryScore
      • TernaryScore.NAME
      • TernaryScore.calculate()
  • Module contents

Submodules

eval_framework.metrics.base module

class eval_framework.metrics.base.BaseMetric

Bases: ABC, Generic

KEYS: list[str] | None = None

NAME: str

NAMES

abstractmethod calculate(response)

Return type:

list[MetricResult]

Parameters:

response (Response)

class eval_framework.metrics.base.MetricResult(**data)

Bases: BaseModel

Parameters:

• metric_name (str)

• value (float | None)

• higher_is_better (bool)

• llm_judge_prompt (str | None)

• llm_judge_response (str | None)

• code_execution_trace (str | None)

• error (Error | None)

code_execution_trace: str | None

error: Error | None

higher_is_better: bool

llm_judge_prompt: str | None

llm_judge_response: str | None

metric_name: str

model_config: ClassVar[ConfigDict] = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

value: float | None

class eval_framework.metrics.base.classproperty(method)

Bases: object

Parameters:

method (Any)

Module contents