eval_framework.tasks.benchmarks package

Submodules

eval_framework.tasks.benchmarks.arc module

class eval_framework.tasks.benchmarks.arc.ARC(num_fewshot=0)

Bases: BaseTask[str]

ARC dataset: https://huggingface.co/datasets/allenai/ai2_arc

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'allenai/ai2_arc'

FEWSHOT_SPLIT: str = 'train'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.bits_per_byte.BitsPerByteLoglikelihood'>]

NAME: str = 'ARC'

RESPONSE_TYPE = 'loglikelihoods'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'test'

SUBJECTS: list[SubjectType] = ['ARC-Easy', 'ARC-Challenge']

class eval_framework.tasks.benchmarks.arc.ARC_IDK(num_fewshot=0)

Bases: ARC

Parameters:

num_fewshot (int)

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.confidence_weighted_accuracy.ConfidenceWeightedAccuracy'>, <class 'eval_framework.metrics.loglikelihood.dcs.DistributionalCorrectnessScore'>, <class 'eval_framework.metrics.loglikelihood.ternary.TernaryScore'>]

NAME: str = 'ARC_IDK'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.arc.ARC_OLMES(num_fewshot=0)

Bases: ARC

ARC with OLMES-style prompt: options shown with space-prefixed labels (” A.”, “ B.”, …); loglikelihood over “ A”/” B”/ etc.

Parameters:

num_fewshot (int)

NAME: str = 'ARC_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

eval_framework.tasks.benchmarks.arc_de module

class eval_framework.tasks.benchmarks.arc_de.ARC_DE(num_fewshot=0)

Bases: BaseTask[str]

ARC-DE dataset: https://huggingface.co/datasets/LeoLM/ArcChallenge_de

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'LeoLM/ArcChallenge_de'

FEWSHOT_SPLIT: str = 'validation'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'German'

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.bits_per_byte.BitsPerByteLoglikelihood'>]

NAME: str = 'ARC German'

RESPONSE_TYPE = 'loglikelihoods'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'test'

SUBJECTS: list[SubjectType] = ['no_subject']

eval_framework.tasks.benchmarks.arc_ellamind module

German ARC (EllaMind) tasks.

https://huggingface.co/datasets/ellamind/arc-multilingual

class eval_framework.tasks.benchmarks.arc_ellamind.ARC_ELLAMIND_BPB_DE(num_fewshot=0)

Bases: _ARC_ELLAMIND_DE_Base

German ARC - BPB format.

Parameters:

num_fewshot (int)

NAME: str = 'ARC_ELLAMIND_BPB_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.BPBStyle object>

class eval_framework.tasks.benchmarks.arc_ellamind.ARC_ELLAMIND_CLOZE_DE(num_fewshot=0)

Bases: _ARC_ELLAMIND_DE_Base

German ARC - Cloze format.

Parameters:

num_fewshot (int)

NAME: str = 'ARC_ELLAMIND_CLOZE_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.ClozeStyle object>

class eval_framework.tasks.benchmarks.arc_ellamind.ARC_ELLAMIND_MC_DE(num_fewshot=0)

Bases: _ARC_ELLAMIND_DE_Base

German ARC - Multiple-choice format.

Parameters:

num_fewshot (int)

NAME: str = 'ARC_ELLAMIND_MC_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.MCStyle object>

eval_framework.tasks.benchmarks.bigcodebench module

class eval_framework.tasks.benchmarks.bigcodebench.BigCodeBench(num_fewshot=0)

Bases: BaseTask[str]

BigCodeBench dataset: https://huggingface.co/datasets/bigcode/bigcodebench

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'bigcode/bigcodebench'

FEWSHOT_SPLIT: str = 'v0.1.4'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.completion.code_execution_pass_at_one.CodeExecutionPassAtOne'>]

NAME: str = 'BigCodeBench'

RESPONSE_TYPE = 'completion'

SAMPLE_SPLIT: str = 'v0.1.4'

SUBJECTS: list[SubjectType] = ['original', 'calibrated']

post_process_generated_completion(completion_text, sample=None)

Return type:

str

Parameters:

• completion_text (str)

• sample (Sample | None)

class eval_framework.tasks.benchmarks.bigcodebench.BigCodeBenchHard(num_fewshot=0)

Bases: BigCodeBench

BigCodeBench dataset: https://huggingface.co/datasets/bigcode/bigcodebench-hard

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'bigcode/bigcodebench-hard'

NAME: str = 'BigCodeBenchHard'

class eval_framework.tasks.benchmarks.bigcodebench.BigCodeBenchHardInstruct(num_fewshot=0)

Bases: BigCodeBenchHard

BigCodeBench dataset: https://huggingface.co/datasets/bigcode/bigcodebench-hard

Parameters:

num_fewshot (int)

NAME: str = 'BigCodeBenchHardInstruct'

class eval_framework.tasks.benchmarks.bigcodebench.BigCodeBenchInstruct(num_fewshot=0)

Bases: BigCodeBench

BigCodeBench dataset: https://huggingface.co/datasets/bigcode/bigcodebench

Parameters:

num_fewshot (int)

NAME: str = 'BigCodeBenchInstruct'

class eval_framework.tasks.benchmarks.bigcodebench.BigCodeBench_OLMES(num_fewshot=3)

Bases: BigCodeBench

BigCodeBench variant matching oe_eval bigcodebench:3shot::olmo3:v2.

Recommended run settings for parity with oe_eval: temperature=0.6, top_p=0.6, repeats=5 (n=5), then compute pass@1 over the 5 samples per problem (post-process if needed).

Parameters:

num_fewshot (int)

FEWSHOT_SPLIT: str = 'v0.1.2'

METRICS = [<class 'eval_framework.metrics.completion.code_execution_pass_at_one.CodeExecutionPassAtOneWithCodebench'>]

NAME: str = 'BigCodeBench_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'v0.1.2'

post_process_generated_completion(completion_text, sample=None)

Return type:

str

Parameters:

• completion_text (str)

• sample (Sample | None)

eval_framework.tasks.benchmarks.bigcodebench.extract_executable_code(llm_response)

Return type:

str

Parameters:

llm_response (str)

eval_framework.tasks.benchmarks.copa module

class eval_framework.tasks.benchmarks.copa.COPA(num_fewshot=0)

Bases: COPAEvalHarness

Unlike the original COPA task, this version uses the test split for evaluation and the validation split for few-shot examples. Previously, the test split labels were unavailable in the original dataset, but they are now accessible, allowing this configuration.

Parameters:

num_fewshot (int)

FEWSHOT_SPLIT: str = 'validation'

NAME: str = 'COPA'

SAMPLE_SPLIT: str = 'test'

class eval_framework.tasks.benchmarks.copa.COPAEvalHarness(num_fewshot=0)

Bases: BaseTask[str]

COPA dataset: https://huggingface.co/datasets/aps/super_glue This version uses samples from the validation split as evaluation examples (same as lm-eval-harness).

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'aps/super_glue'

FEWSHOT_SPLIT: str = 'test'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>]

NAME: str = 'COPAEvalHarness'

RESPONSE_TYPE = 'loglikelihoods'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'validation'

SUBJECTS: list[SubjectType] = ['copa']

convert_choice(choice)

Return type:

str

Parameters:

choice (str)

class eval_framework.tasks.benchmarks.copa.COPA_IDK(num_fewshot=0)

Bases: COPA_IDKEvalHarness

Parameters:

num_fewshot (int)

FEWSHOT_SPLIT: str = 'validation'

NAME: str = 'COPA_IDK'

SAMPLE_SPLIT: str = 'test'

class eval_framework.tasks.benchmarks.copa.COPA_IDKEvalHarness(num_fewshot=0)

Bases: COPAEvalHarness

Parameters:

num_fewshot (int)

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.confidence_weighted_accuracy.ConfidenceWeightedAccuracy'>, <class 'eval_framework.metrics.loglikelihood.dcs.DistributionalCorrectnessScore'>, <class 'eval_framework.metrics.loglikelihood.ternary.TernaryScore'>]

NAME: str = 'COPA_IDKEvalHarness'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.copa.COPA_OLMES(num_fewshot=0)

Bases: COPAEvalHarness

COPA multiple choice (OLMES/oe_eval style): prompt shows premise + connector and options with space-prefixed labels (” A.”, “ B.”); loglikelihood over “ A”/” B”.

Parameters:

num_fewshot (int)

NAME: str = 'COPA_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

eval_framework.tasks.benchmarks.csqa module

class eval_framework.tasks.benchmarks.csqa.CommonsenseQACloze(num_fewshot=0)

Bases: BaseTask[str]

CommonsenseQA dataset: https://huggingface.co/datasets/tau/commonsense_qa

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'tau/commonsense_qa'

FEWSHOT_SPLIT: str = 'validation'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.bits_per_byte.BitsPerByteLoglikelihood'>]

NAME: str = 'CommonsenseQACloze'

RESPONSE_TYPE = 'loglikelihoods'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'validation'

SUBJECTS: list[SubjectType] = ['no_subject']

class eval_framework.tasks.benchmarks.csqa.CommonsenseQAFullTextCloze(num_fewshot=0)

Bases: CommonsenseQACloze

CommonsenseQA cloze with full answer text as ground truth (not just the letter). Scores loglikelihood over the full correct choice text; includes bits-per-byte.

Parameters:

num_fewshot (int)

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.bits_per_byte.BitsPerByteLoglikelihood'>]

NAME: str = 'CommonsenseQAFullTextCloze'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.csqa.CommonsenseQAMC(num_fewshot=0)

Bases: CommonsenseQACloze

Multiple-choice variant of CommonsenseQA where the model selects a letter (A-E).

Parameters:

num_fewshot (int)

NAME: str = 'CommonsenseQAMC'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.csqa.CommonsenseQAMC_OLMES(num_fewshot=0)

Bases: CommonsenseQAMC

CommonsenseQA MC with OLMES-style prompt: space before each label in the prompt (” A.”, “ B.”, …).

Parameters:

num_fewshot (int)

FEWSHOT_SPLIT: str = 'train'

NAME: str = 'CommonsenseQAMC_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'train'

eval_framework.tasks.benchmarks.csqa_ellamind module

German CommonsenseQA (EllaMind) tasks.

https://huggingface.co/datasets/ellamind/csqa-multilingual

CSQA supplies separate easy and hard distractors. Each base class uses a _DISTRACTOR_LEVEL class attribute ("easy" or "hard") that the registered subclass overrides.

class eval_framework.tasks.benchmarks.csqa_ellamind.CSQA_ELLAMIND_BPB_DE(num_fewshot=0)

Bases: CSQA_ELLAMIND_CLOZE_EASY_DE

German CSQA - BPB format (easy distractors; distractor set is irrelevant for BPB).

Parameters:

num_fewshot (int)

NAME: str = 'CSQA_ELLAMIND_BPB_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.BPBStyle object>

class eval_framework.tasks.benchmarks.csqa_ellamind.CSQA_ELLAMIND_CLOZE_EASY_DE(num_fewshot=0)

Bases: _CSQA_ELLAMIND_DE_Base

German CSQA - Cloze format with easy distractors.

Parameters:

num_fewshot (int)

NAME: str = 'CSQA_ELLAMIND_CLOZE_EASY_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.ClozeStyle object>

class eval_framework.tasks.benchmarks.csqa_ellamind.CSQA_ELLAMIND_CLOZE_HARD_DE(num_fewshot=0)

Bases: _CSQA_ELLAMIND_DE_Base

German CSQA - Cloze format with hard distractors.

Parameters:

num_fewshot (int)

NAME: str = 'CSQA_ELLAMIND_CLOZE_HARD_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.ClozeStyle object>

class eval_framework.tasks.benchmarks.csqa_ellamind.CSQA_ELLAMIND_MC_EASY_DE(num_fewshot=0)

Bases: _CSQA_ELLAMIND_DE_Base

German CSQA - MC format with easy distractors.

Parameters:

num_fewshot (int)

NAME: str = 'CSQA_ELLAMIND_MC_EASY_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.MCStyle object>

class eval_framework.tasks.benchmarks.csqa_ellamind.CSQA_ELLAMIND_MC_HARD_DE(num_fewshot=0)

Bases: _CSQA_ELLAMIND_DE_Base

German CSQA - MC format with hard distractors.

Parameters:

num_fewshot (int)

NAME: str = 'CSQA_ELLAMIND_MC_HARD_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.MCStyle object>

eval_framework.tasks.benchmarks.drop module

class eval_framework.tasks.benchmarks.drop.DropCloze(num_fewshot=0)

Bases: BaseTask[str]

Cloze variant: loglikelihood ranking over full choice texts (allenai/drop-gen2mc).

Same dataset as DropMC; options not shown in prompt; model scores full text of each choice. Includes BitsPerByte on the correct choice.

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'allenai/drop-gen2mc'

FEWSHOT_SPLIT: str = 'validation'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.bits_per_byte.BitsPerByteLoglikelihood'>]

NAME: str = 'DropCloze'

RESPONSE_TYPE = 'loglikelihoods'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'validation'

SUBJECTS: list[SubjectType] = ['no_subject']

class eval_framework.tasks.benchmarks.drop.DropCompletion(num_fewshot=0)

Bases: BaseTask[str]

DROP completion benchmark (EleutherAI/drop): passage, question, model generates answer.

Uses DROP F1 and exact match. Stop at new paragraph or repeated prefixes.

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'EleutherAI/drop'

FEWSHOT_SPLIT: str = 'validation'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.completion.drop_completion.DropF1ExactMatch'>]

NAME: str = 'DropCompletion'

RESPONSE_TYPE = 'completion'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'validation'

SUBJECTS: list[SubjectType] = ['no_subject']

class eval_framework.tasks.benchmarks.drop.DropCompletion_OLMES(num_fewshot=0)

Bases: DropCompletion

DropCompletion matching OLMES, using train split for fewshot and max tokens 100.

Parameters:

num_fewshot (int)

FEWSHOT_SPLIT: str = 'train'

NAME: str = 'DropCompletion_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.drop.DropMC(num_fewshot=0)

Bases: BaseTask[str]

Multiple-choice variant using allenai/drop-gen2mc (passage_original, question_original, choices, answerKey).

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'allenai/drop-gen2mc'

FEWSHOT_SPLIT: str = 'validation'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.bits_per_byte.BitsPerByteLoglikelihood'>]

NAME: str = 'DropMC'

RESPONSE_TYPE = 'loglikelihoods'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'validation'

SUBJECTS: list[SubjectType] = ['no_subject']

class eval_framework.tasks.benchmarks.drop.DropMC_OLMES(num_fewshot=0)

Bases: DropMC

DropMC with OLMES-style prompt: space before each label in the prompt (” A.”, “ B.”, …).

Parameters:

num_fewshot (int)

NAME: str = 'DropMC_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

eval_framework.tasks.benchmarks.global_mmlu module

class eval_framework.tasks.benchmarks.global_mmlu.GlobalMMLU(num_fewshot=0)

Bases: BaseTask[tuple[str, str]]

MMLU dataset: https://huggingface.co/datasets/CohereLabs/Global-MMLU

Currently, we only support prompting in French, German, Spanish, Italian, Portugese, and Arabic.

TO-DO: Suggest we adjust prompting for languages individually, e.g., South-East Asian languages available here: https://github.com/aisingapore/SEA-HELM/blob/main/seahelm_tasks/knowledge/global_mmlu/abstract_algebra/config.yaml

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'CohereLabs/Global-MMLU'

FEWSHOT_SPLIT: str = 'dev'

LANGUAGE: Language | dict[str, Language] | None = {"('ar', 'abstract_algebra')": Language.ARB, "('ar', 'anatomy')": Language.ARB, "('ar', 'astronomy')": Language.ARB, "('ar', 'business_ethics')": Language.ARB, "('ar', 'clinical_knowledge')": Language.ARB, "('ar', 'college_biology')": Language.ARB, "('ar', 'college_chemistry')": Language.ARB, "('ar', 'college_computer_science')": Language.ARB, "('ar', 'college_mathematics')": Language.ARB, "('ar', 'college_medicine')": Language.ARB, "('ar', 'college_physics')": Language.ARB, "('ar', 'computer_security')": Language.ARB, "('ar', 'conceptual_physics')": Language.ARB, "('ar', 'econometrics')": Language.ARB, "('ar', 'electrical_engineering')": Language.ARB, "('ar', 'elementary_mathematics')": Language.ARB, "('ar', 'formal_logic')": Language.ARB, "('ar', 'global_facts')": Language.ARB, "('ar', 'high_school_biology')": Language.ARB, "('ar', 'high_school_chemistry')": Language.ARB, "('ar', 'high_school_computer_science')": Language.ARB, "('ar', 'high_school_european_history')": Language.ARB, "('ar', 'high_school_geography')": Language.ARB, "('ar', 'high_school_government_and_politics')": Language.ARB, "('ar', 'high_school_macroeconomics')": Language.ARB, "('ar', 'high_school_mathematics')": Language.ARB, "('ar', 'high_school_microeconomics')": Language.ARB, "('ar', 'high_school_physics')": Language.ARB, "('ar', 'high_school_psychology')": Language.ARB, "('ar', 'high_school_statistics')": Language.ARB, "('ar', 'high_school_us_history')": Language.ARB, "('ar', 'high_school_world_history')": Language.ARB, "('ar', 'human_aging')": Language.ARB, "('ar', 'human_sexuality')": Language.ARB, "('ar', 'international_law')": Language.ARB, "('ar', 'jurisprudence')": Language.ARB, "('ar', 'logical_fallacies')": Language.ARB, "('ar', 'machine_learning')": Language.ARB, "('ar', 'management')": Language.ARB, "('ar', 'marketing')": Language.ARB, "('ar', 'medical_genetics')": Language.ARB, "('ar', 'miscellaneous')": Language.ARB, "('ar', 'moral_disputes')": Language.ARB, "('ar', 'moral_scenarios')": Language.ARB, "('ar', 'nutrition')": Language.ARB, "('ar', 'philosophy')": Language.ARB, "('ar', 'prehistory')": Language.ARB, "('ar', 'professional_accounting')": Language.ARB, "('ar', 'professional_law')": Language.ARB, "('ar', 'professional_medicine')": Language.ARB, "('ar', 'professional_psychology')": Language.ARB, "('ar', 'public_relations')": Language.ARB, "('ar', 'security_studies')": Language.ARB, "('ar', 'sociology')": Language.ARB, "('ar', 'us_foreign_policy')": Language.ARB, "('ar', 'virology')": Language.ARB, "('ar', 'world_religions')": Language.ARB, "('de', 'abstract_algebra')": Language.DEU, "('de', 'anatomy')": Language.DEU, "('de', 'astronomy')": Language.DEU, "('de', 'business_ethics')": Language.DEU, "('de', 'clinical_knowledge')": Language.DEU, "('de', 'college_biology')": Language.DEU, "('de', 'college_chemistry')": Language.DEU, "('de', 'college_computer_science')": Language.DEU, "('de', 'college_mathematics')": Language.DEU, "('de', 'college_medicine')": Language.DEU, "('de', 'college_physics')": Language.DEU, "('de', 'computer_security')": Language.DEU, "('de', 'conceptual_physics')": Language.DEU, "('de', 'econometrics')": Language.DEU, "('de', 'electrical_engineering')": Language.DEU, "('de', 'elementary_mathematics')": Language.DEU, "('de', 'formal_logic')": Language.DEU, "('de', 'global_facts')": Language.DEU, "('de', 'high_school_biology')": Language.DEU, "('de', 'high_school_chemistry')": Language.DEU, "('de', 'high_school_computer_science')": Language.DEU, "('de', 'high_school_european_history')": Language.DEU, "('de', 'high_school_geography')": Language.DEU, "('de', 'high_school_government_and_politics')": Language.DEU, "('de', 'high_school_macroeconomics')": Language.DEU, "('de', 'high_school_mathematics')": Language.DEU, "('de', 'high_school_microeconomics')": Language.DEU, "('de', 'high_school_physics')": Language.DEU, "('de', 'high_school_psychology')": Language.DEU, "('de', 'high_school_statistics')": Language.DEU, "('de', 'high_school_us_history')": Language.DEU, "('de', 'high_school_world_history')": Language.DEU, "('de', 'human_aging')": Language.DEU, "('de', 'human_sexuality')": Language.DEU, "('de', 'international_law')": Language.DEU, "('de', 'jurisprudence')": Language.DEU, "('de', 'logical_fallacies')": Language.DEU, "('de', 'machine_learning')": Language.DEU, "('de', 'management')": Language.DEU, "('de', 'marketing')": Language.DEU, "('de', 'medical_genetics')": Language.DEU, "('de', 'miscellaneous')": Language.DEU, "('de', 'moral_disputes')": Language.DEU, "('de', 'moral_scenarios')": Language.DEU, "('de', 'nutrition')": Language.DEU, "('de', 'philosophy')": Language.DEU, "('de', 'prehistory')": Language.DEU, "('de', 'professional_accounting')": Language.DEU, "('de', 'professional_law')": Language.DEU, "('de', 'professional_medicine')": Language.DEU, "('de', 'professional_psychology')": Language.DEU, "('de', 'public_relations')": Language.DEU, "('de', 'security_studies')": Language.DEU, "('de', 'sociology')": Language.DEU, "('de', 'us_foreign_policy')": Language.DEU, "('de', 'virology')": Language.DEU, "('de', 'world_religions')": Language.DEU, "('es', 'abstract_algebra')": Language.SPA, "('es', 'anatomy')": Language.SPA, "('es', 'astronomy')": Language.SPA, "('es', 'business_ethics')": Language.SPA, "('es', 'clinical_knowledge')": Language.SPA, "('es', 'college_biology')": Language.SPA, "('es', 'college_chemistry')": Language.SPA, "('es', 'college_computer_science')": Language.SPA, "('es', 'college_mathematics')": Language.SPA, "('es', 'college_medicine')": Language.SPA, "('es', 'college_physics')": Language.SPA, "('es', 'computer_security')": Language.SPA, "('es', 'conceptual_physics')": Language.SPA, "('es', 'econometrics')": Language.SPA, "('es', 'electrical_engineering')": Language.SPA, "('es', 'elementary_mathematics')": Language.SPA, "('es', 'formal_logic')": Language.SPA, "('es', 'global_facts')": Language.SPA, "('es', 'high_school_biology')": Language.SPA, "('es', 'high_school_chemistry')": Language.SPA, "('es', 'high_school_computer_science')": Language.SPA, "('es', 'high_school_european_history')": Language.SPA, "('es', 'high_school_geography')": Language.SPA, "('es', 'high_school_government_and_politics')": Language.SPA, "('es', 'high_school_macroeconomics')": Language.SPA, "('es', 'high_school_mathematics')": Language.SPA, "('es', 'high_school_microeconomics')": Language.SPA, "('es', 'high_school_physics')": Language.SPA, "('es', 'high_school_psychology')": Language.SPA, "('es', 'high_school_statistics')": Language.SPA, "('es', 'high_school_us_history')": Language.SPA, "('es', 'high_school_world_history')": Language.SPA, "('es', 'human_aging')": Language.SPA, "('es', 'human_sexuality')": Language.SPA, "('es', 'international_law')": Language.SPA, "('es', 'jurisprudence')": Language.SPA, "('es', 'logical_fallacies')": Language.SPA, "('es', 'machine_learning')": Language.SPA, "('es', 'management')": Language.SPA, "('es', 'marketing')": Language.SPA, "('es', 'medical_genetics')": Language.SPA, "('es', 'miscellaneous')": Language.SPA, "('es', 'moral_disputes')": Language.SPA, "('es', 'moral_scenarios')": Language.SPA, "('es', 'nutrition')": Language.SPA, "('es', 'philosophy')": Language.SPA, "('es', 'prehistory')": Language.SPA, "('es', 'professional_accounting')": Language.SPA, "('es', 'professional_law')": Language.SPA, "('es', 'professional_medicine')": Language.SPA, "('es', 'professional_psychology')": Language.SPA, "('es', 'public_relations')": Language.SPA, "('es', 'security_studies')": Language.SPA, "('es', 'sociology')": Language.SPA, "('es', 'us_foreign_policy')": Language.SPA, "('es', 'virology')": Language.SPA, "('es', 'world_religions')": Language.SPA, "('fr', 'abstract_algebra')": Language.FRA, "('fr', 'anatomy')": Language.FRA, "('fr', 'astronomy')": Language.FRA, "('fr', 'business_ethics')": Language.FRA, "('fr', 'clinical_knowledge')": Language.FRA, "('fr', 'college_biology')": Language.FRA, "('fr', 'college_chemistry')": Language.FRA, "('fr', 'college_computer_science')": Language.FRA, "('fr', 'college_mathematics')": Language.FRA, "('fr', 'college_medicine')": Language.FRA, "('fr', 'college_physics')": Language.FRA, "('fr', 'computer_security')": Language.FRA, "('fr', 'conceptual_physics')": Language.FRA, "('fr', 'econometrics')": Language.FRA, "('fr', 'electrical_engineering')": Language.FRA, "('fr', 'elementary_mathematics')": Language.FRA, "('fr', 'formal_logic')": Language.FRA, "('fr', 'global_facts')": Language.FRA, "('fr', 'high_school_biology')": Language.FRA, "('fr', 'high_school_chemistry')": Language.FRA, "('fr', 'high_school_computer_science')": Language.FRA, "('fr', 'high_school_european_history')": Language.FRA, "('fr', 'high_school_geography')": Language.FRA, "('fr', 'high_school_government_and_politics')": Language.FRA, "('fr', 'high_school_macroeconomics')": Language.FRA, "('fr', 'high_school_mathematics')": Language.FRA, "('fr', 'high_school_microeconomics')": Language.FRA, "('fr', 'high_school_physics')": Language.FRA, "('fr', 'high_school_psychology')": Language.FRA, "('fr', 'high_school_statistics')": Language.FRA, "('fr', 'high_school_us_history')": Language.FRA, "('fr', 'high_school_world_history')": Language.FRA, "('fr', 'human_aging')": Language.FRA, "('fr', 'human_sexuality')": Language.FRA, "('fr', 'international_law')": Language.FRA, "('fr', 'jurisprudence')": Language.FRA, "('fr', 'logical_fallacies')": Language.FRA, "('fr', 'machine_learning')": Language.FRA, "('fr', 'management')": Language.FRA, "('fr', 'marketing')": Language.FRA, "('fr', 'medical_genetics')": Language.FRA, "('fr', 'miscellaneous')": Language.FRA, "('fr', 'moral_disputes')": Language.FRA, "('fr', 'moral_scenarios')": Language.FRA, "('fr', 'nutrition')": Language.FRA, "('fr', 'philosophy')": Language.FRA, "('fr', 'prehistory')": Language.FRA, "('fr', 'professional_accounting')": Language.FRA, "('fr', 'professional_law')": Language.FRA, "('fr', 'professional_medicine')": Language.FRA, "('fr', 'professional_psychology')": Language.FRA, "('fr', 'public_relations')": Language.FRA, "('fr', 'security_studies')": Language.FRA, "('fr', 'sociology')": Language.FRA, "('fr', 'us_foreign_policy')": Language.FRA, "('fr', 'virology')": Language.FRA, "('fr', 'world_religions')": Language.FRA, "('it', 'abstract_algebra')": Language.ITA, "('it', 'anatomy')": Language.ITA, "('it', 'astronomy')": Language.ITA, "('it', 'business_ethics')": Language.ITA, "('it', 'clinical_knowledge')": Language.ITA, "('it', 'college_biology')": Language.ITA, "('it', 'college_chemistry')": Language.ITA, "('it', 'college_computer_science')": Language.ITA, "('it', 'college_mathematics')": Language.ITA, "('it', 'college_medicine')": Language.ITA, "('it', 'college_physics')": Language.ITA, "('it', 'computer_security')": Language.ITA, "('it', 'conceptual_physics')": Language.ITA, "('it', 'econometrics')": Language.ITA, "('it', 'electrical_engineering')": Language.ITA, "('it', 'elementary_mathematics')": Language.ITA, "('it', 'formal_logic')": Language.ITA, "('it', 'global_facts')": Language.ITA, "('it', 'high_school_biology')": Language.ITA, "('it', 'high_school_chemistry')": Language.ITA, "('it', 'high_school_computer_science')": Language.ITA, "('it', 'high_school_european_history')": Language.ITA, "('it', 'high_school_geography')": Language.ITA, "('it', 'high_school_government_and_politics')": Language.ITA, "('it', 'high_school_macroeconomics')": Language.ITA, "('it', 'high_school_mathematics')": Language.ITA, "('it', 'high_school_microeconomics')": Language.ITA, "('it', 'high_school_physics')": Language.ITA, "('it', 'high_school_psychology')": Language.ITA, "('it', 'high_school_statistics')": Language.ITA, "('it', 'high_school_us_history')": Language.ITA, "('it', 'high_school_world_history')": Language.ITA, "('it', 'human_aging')": Language.ITA, "('it', 'human_sexuality')": Language.ITA, "('it', 'international_law')": Language.ITA, "('it', 'jurisprudence')": Language.ITA, "('it', 'logical_fallacies')": Language.ITA, "('it', 'machine_learning')": Language.ITA, "('it', 'management')": Language.ITA, "('it', 'marketing')": Language.ITA, "('it', 'medical_genetics')": Language.ITA, "('it', 'miscellaneous')": Language.ITA, "('it', 'moral_disputes')": Language.ITA, "('it', 'moral_scenarios')": Language.ITA, "('it', 'nutrition')": Language.ITA, "('it', 'philosophy')": Language.ITA, "('it', 'prehistory')": Language.ITA, "('it', 'professional_accounting')": Language.ITA, "('it', 'professional_law')": Language.ITA, "('it', 'professional_medicine')": Language.ITA, "('it', 'professional_psychology')": Language.ITA, "('it', 'public_relations')": Language.ITA, "('it', 'security_studies')": Language.ITA, "('it', 'sociology')": Language.ITA, "('it', 'us_foreign_policy')": Language.ITA, "('it', 'virology')": Language.ITA, "('it', 'world_religions')": Language.ITA, "('pt', 'abstract_algebra')": Language.POR, "('pt', 'anatomy')": Language.POR, "('pt', 'astronomy')": Language.POR, "('pt', 'business_ethics')": Language.POR, "('pt', 'clinical_knowledge')": Language.POR, "('pt', 'college_biology')": Language.POR, "('pt', 'college_chemistry')": Language.POR, "('pt', 'college_computer_science')": Language.POR, "('pt', 'college_mathematics')": Language.POR, "('pt', 'college_medicine')": Language.POR, "('pt', 'college_physics')": Language.POR, "('pt', 'computer_security')": Language.POR, "('pt', 'conceptual_physics')": Language.POR, "('pt', 'econometrics')": Language.POR, "('pt', 'electrical_engineering')": Language.POR, "('pt', 'elementary_mathematics')": Language.POR, "('pt', 'formal_logic')": Language.POR, "('pt', 'global_facts')": Language.POR, "('pt', 'high_school_biology')": Language.POR, "('pt', 'high_school_chemistry')": Language.POR, "('pt', 'high_school_computer_science')": Language.POR, "('pt', 'high_school_european_history')": Language.POR, "('pt', 'high_school_geography')": Language.POR, "('pt', 'high_school_government_and_politics')": Language.POR, "('pt', 'high_school_macroeconomics')": Language.POR, "('pt', 'high_school_mathematics')": Language.POR, "('pt', 'high_school_microeconomics')": Language.POR, "('pt', 'high_school_physics')": Language.POR, "('pt', 'high_school_psychology')": Language.POR, "('pt', 'high_school_statistics')": Language.POR, "('pt', 'high_school_us_history')": Language.POR, "('pt', 'high_school_world_history')": Language.POR, "('pt', 'human_aging')": Language.POR, "('pt', 'human_sexuality')": Language.POR, "('pt', 'international_law')": Language.POR, "('pt', 'jurisprudence')": Language.POR, "('pt', 'logical_fallacies')": Language.POR, "('pt', 'machine_learning')": Language.POR, "('pt', 'management')": Language.POR, "('pt', 'marketing')": Language.POR, "('pt', 'medical_genetics')": Language.POR, "('pt', 'miscellaneous')": Language.POR, "('pt', 'moral_disputes')": Language.POR, "('pt', 'moral_scenarios')": Language.POR, "('pt', 'nutrition')": Language.POR, "('pt', 'philosophy')": Language.POR, "('pt', 'prehistory')": Language.POR, "('pt', 'professional_accounting')": Language.POR, "('pt', 'professional_law')": Language.POR, "('pt', 'professional_medicine')": Language.POR, "('pt', 'professional_psychology')": Language.POR, "('pt', 'public_relations')": Language.POR, "('pt', 'security_studies')": Language.POR, "('pt', 'sociology')": Language.POR, "('pt', 'us_foreign_policy')": Language.POR, "('pt', 'virology')": Language.POR, "('pt', 'world_religions')": Language.POR}

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.bits_per_byte.BitsPerByteLoglikelihood'>]

NAME: str = 'GlobalMMLU'

OPTION_KEYS = {'A': 'option_a', 'B': 'option_b', 'C': 'option_c', 'D': 'option_d'}

RESPONSE_TYPE = 'loglikelihoods'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'test'

SUBJECTS: list[SubjectType] = [('fr', 'abstract_algebra'), ('fr', 'anatomy'), ('fr', 'astronomy'), ('fr', 'business_ethics'), ('fr', 'clinical_knowledge'), ('fr', 'college_biology'), ('fr', 'college_chemistry'), ('fr', 'college_computer_science'), ('fr', 'college_mathematics'), ('fr', 'college_medicine'), ('fr', 'college_physics'), ('fr', 'computer_security'), ('fr', 'conceptual_physics'), ('fr', 'econometrics'), ('fr', 'electrical_engineering'), ('fr', 'elementary_mathematics'), ('fr', 'formal_logic'), ('fr', 'global_facts'), ('fr', 'high_school_biology'), ('fr', 'high_school_chemistry'), ('fr', 'high_school_computer_science'), ('fr', 'high_school_european_history'), ('fr', 'high_school_geography'), ('fr', 'high_school_government_and_politics'), ('fr', 'high_school_macroeconomics'), ('fr', 'high_school_mathematics'), ('fr', 'high_school_microeconomics'), ('fr', 'high_school_physics'), ('fr', 'high_school_psychology'), ('fr', 'high_school_statistics'), ('fr', 'high_school_us_history'), ('fr', 'high_school_world_history'), ('fr', 'human_aging'), ('fr', 'human_sexuality'), ('fr', 'international_law'), ('fr', 'jurisprudence'), ('fr', 'logical_fallacies'), ('fr', 'machine_learning'), ('fr', 'management'), ('fr', 'marketing'), ('fr', 'medical_genetics'), ('fr', 'miscellaneous'), ('fr', 'moral_disputes'), ('fr', 'moral_scenarios'), ('fr', 'nutrition'), ('fr', 'philosophy'), ('fr', 'prehistory'), ('fr', 'professional_accounting'), ('fr', 'professional_law'), ('fr', 'professional_medicine'), ('fr', 'professional_psychology'), ('fr', 'public_relations'), ('fr', 'security_studies'), ('fr', 'sociology'), ('fr', 'us_foreign_policy'), ('fr', 'virology'), ('fr', 'world_religions'), ('de', 'abstract_algebra'), ('de', 'anatomy'), ('de', 'astronomy'), ('de', 'business_ethics'), ('de', 'clinical_knowledge'), ('de', 'college_biology'), ('de', 'college_chemistry'), ('de', 'college_computer_science'), ('de', 'college_mathematics'), ('de', 'college_medicine'), ('de', 'college_physics'), ('de', 'computer_security'), ('de', 'conceptual_physics'), ('de', 'econometrics'), ('de', 'electrical_engineering'), ('de', 'elementary_mathematics'), ('de', 'formal_logic'), ('de', 'global_facts'), ('de', 'high_school_biology'), ('de', 'high_school_chemistry'), ('de', 'high_school_computer_science'), ('de', 'high_school_european_history'), ('de', 'high_school_geography'), ('de', 'high_school_government_and_politics'), ('de', 'high_school_macroeconomics'), ('de', 'high_school_mathematics'), ('de', 'high_school_microeconomics'), ('de', 'high_school_physics'), ('de', 'high_school_psychology'), ('de', 'high_school_statistics'), ('de', 'high_school_us_history'), ('de', 'high_school_world_history'), ('de', 'human_aging'), ('de', 'human_sexuality'), ('de', 'international_law'), ('de', 'jurisprudence'), ('de', 'logical_fallacies'), ('de', 'machine_learning'), ('de', 'management'), ('de', 'marketing'), ('de', 'medical_genetics'), ('de', 'miscellaneous'), ('de', 'moral_disputes'), ('de', 'moral_scenarios'), ('de', 'nutrition'), ('de', 'philosophy'), ('de', 'prehistory'), ('de', 'professional_accounting'), ('de', 'professional_law'), ('de', 'professional_medicine'), ('de', 'professional_psychology'), ('de', 'public_relations'), ('de', 'security_studies'), ('de', 'sociology'), ('de', 'us_foreign_policy'), ('de', 'virology'), ('de', 'world_religions'), ('es', 'abstract_algebra'), ('es', 'anatomy'), ('es', 'astronomy'), ('es', 'business_ethics'), ('es', 'clinical_knowledge'), ('es', 'college_biology'), ('es', 'college_chemistry'), ('es', 'college_computer_science'), ('es', 'college_mathematics'), ('es', 'college_medicine'), ('es', 'college_physics'), ('es', 'computer_security'), ('es', 'conceptual_physics'), ('es', 'econometrics'), ('es', 'electrical_engineering'), ('es', 'elementary_mathematics'), ('es', 'formal_logic'), ('es', 'global_facts'), ('es', 'high_school_biology'), ('es', 'high_school_chemistry'), ('es', 'high_school_computer_science'), ('es', 'high_school_european_history'), ('es', 'high_school_geography'), ('es', 'high_school_government_and_politics'), ('es', 'high_school_macroeconomics'), ('es', 'high_school_mathematics'), ('es', 'high_school_microeconomics'), ('es', 'high_school_physics'), ('es', 'high_school_psychology'), ('es', 'high_school_statistics'), ('es', 'high_school_us_history'), ('es', 'high_school_world_history'), ('es', 'human_aging'), ('es', 'human_sexuality'), ('es', 'international_law'), ('es', 'jurisprudence'), ('es', 'logical_fallacies'), ('es', 'machine_learning'), ('es', 'management'), ('es', 'marketing'), ('es', 'medical_genetics'), ('es', 'miscellaneous'), ('es', 'moral_disputes'), ('es', 'moral_scenarios'), ('es', 'nutrition'), ('es', 'philosophy'), ('es', 'prehistory'), ('es', 'professional_accounting'), ('es', 'professional_law'), ('es', 'professional_medicine'), ('es', 'professional_psychology'), ('es', 'public_relations'), ('es', 'security_studies'), ('es', 'sociology'), ('es', 'us_foreign_policy'), ('es', 'virology'), ('es', 'world_religions'), ('it', 'abstract_algebra'), ('it', 'anatomy'), ('it', 'astronomy'), ('it', 'business_ethics'), ('it', 'clinical_knowledge'), ('it', 'college_biology'), ('it', 'college_chemistry'), ('it', 'college_computer_science'), ('it', 'college_mathematics'), ('it', 'college_medicine'), ('it', 'college_physics'), ('it', 'computer_security'), ('it', 'conceptual_physics'), ('it', 'econometrics'), ('it', 'electrical_engineering'), ('it', 'elementary_mathematics'), ('it', 'formal_logic'), ('it', 'global_facts'), ('it', 'high_school_biology'), ('it', 'high_school_chemistry'), ('it', 'high_school_computer_science'), ('it', 'high_school_european_history'), ('it', 'high_school_geography'), ('it', 'high_school_government_and_politics'), ('it', 'high_school_macroeconomics'), ('it', 'high_school_mathematics'), ('it', 'high_school_microeconomics'), ('it', 'high_school_physics'), ('it', 'high_school_psychology'), ('it', 'high_school_statistics'), ('it', 'high_school_us_history'), ('it', 'high_school_world_history'), ('it', 'human_aging'), ('it', 'human_sexuality'), ('it', 'international_law'), ('it', 'jurisprudence'), ('it', 'logical_fallacies'), ('it', 'machine_learning'), ('it', 'management'), ('it', 'marketing'), ('it', 'medical_genetics'), ('it', 'miscellaneous'), ('it', 'moral_disputes'), ('it', 'moral_scenarios'), ('it', 'nutrition'), ('it', 'philosophy'), ('it', 'prehistory'), ('it', 'professional_accounting'), ('it', 'professional_law'), ('it', 'professional_medicine'), ('it', 'professional_psychology'), ('it', 'public_relations'), ('it', 'security_studies'), ('it', 'sociology'), ('it', 'us_foreign_policy'), ('it', 'virology'), ('it', 'world_religions'), ('pt', 'abstract_algebra'), ('pt', 'anatomy'), ('pt', 'astronomy'), ('pt', 'business_ethics'), ('pt', 'clinical_knowledge'), ('pt', 'college_biology'), ('pt', 'college_chemistry'), ('pt', 'college_computer_science'), ('pt', 'college_mathematics'), ('pt', 'college_medicine'), ('pt', 'college_physics'), ('pt', 'computer_security'), ('pt', 'conceptual_physics'), ('pt', 'econometrics'), ('pt', 'electrical_engineering'), ('pt', 'elementary_mathematics'), ('pt', 'formal_logic'), ('pt', 'global_facts'), ('pt', 'high_school_biology'), ('pt', 'high_school_chemistry'), ('pt', 'high_school_computer_science'), ('pt', 'high_school_european_history'), ('pt', 'high_school_geography'), ('pt', 'high_school_government_and_politics'), ('pt', 'high_school_macroeconomics'), ('pt', 'high_school_mathematics'), ('pt', 'high_school_microeconomics'), ('pt', 'high_school_physics'), ('pt', 'high_school_psychology'), ('pt', 'high_school_statistics'), ('pt', 'high_school_us_history'), ('pt', 'high_school_world_history'), ('pt', 'human_aging'), ('pt', 'human_sexuality'), ('pt', 'international_law'), ('pt', 'jurisprudence'), ('pt', 'logical_fallacies'), ('pt', 'machine_learning'), ('pt', 'management'), ('pt', 'marketing'), ('pt', 'medical_genetics'), ('pt', 'miscellaneous'), ('pt', 'moral_disputes'), ('pt', 'moral_scenarios'), ('pt', 'nutrition'), ('pt', 'philosophy'), ('pt', 'prehistory'), ('pt', 'professional_accounting'), ('pt', 'professional_law'), ('pt', 'professional_medicine'), ('pt', 'professional_psychology'), ('pt', 'public_relations'), ('pt', 'security_studies'), ('pt', 'sociology'), ('pt', 'us_foreign_policy'), ('pt', 'virology'), ('pt', 'world_religions'), ('ar', 'abstract_algebra'), ('ar', 'anatomy'), ('ar', 'astronomy'), ('ar', 'business_ethics'), ('ar', 'clinical_knowledge'), ('ar', 'college_biology'), ('ar', 'college_chemistry'), ('ar', 'college_computer_science'), ('ar', 'college_mathematics'), ('ar', 'college_medicine'), ('ar', 'college_physics'), ('ar', 'computer_security'), ('ar', 'conceptual_physics'), ('ar', 'econometrics'), ('ar', 'electrical_engineering'), ('ar', 'elementary_mathematics'), ('ar', 'formal_logic'), ('ar', 'global_facts'), ('ar', 'high_school_biology'), ('ar', 'high_school_chemistry'), ('ar', 'high_school_computer_science'), ('ar', 'high_school_european_history'), ('ar', 'high_school_geography'), ('ar', 'high_school_government_and_politics'), ('ar', 'high_school_macroeconomics'), ('ar', 'high_school_mathematics'), ('ar', 'high_school_microeconomics'), ('ar', 'high_school_physics'), ('ar', 'high_school_psychology'), ('ar', 'high_school_statistics'), ('ar', 'high_school_us_history'), ('ar', 'high_school_world_history'), ('ar', 'human_aging'), ('ar', 'human_sexuality'), ('ar', 'international_law'), ('ar', 'jurisprudence'), ('ar', 'logical_fallacies'), ('ar', 'machine_learning'), ('ar', 'management'), ('ar', 'marketing'), ('ar', 'medical_genetics'), ('ar', 'miscellaneous'), ('ar', 'moral_disputes'), ('ar', 'moral_scenarios'), ('ar', 'nutrition'), ('ar', 'philosophy'), ('ar', 'prehistory'), ('ar', 'professional_accounting'), ('ar', 'professional_law'), ('ar', 'professional_medicine'), ('ar', 'professional_psychology'), ('ar', 'public_relations'), ('ar', 'security_studies'), ('ar', 'sociology'), ('ar', 'us_foreign_policy'), ('ar', 'virology'), ('ar', 'world_religions')]

class eval_framework.tasks.benchmarks.global_mmlu.GlobalMMLU_German(num_fewshot=0)

Bases: GlobalMMLU

Parameters:

num_fewshot (int)

LANGUAGE: Language | dict[str, Language] | None = 'German'

NAME: str = 'GlobalMMLU_German'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SUBJECTS: list[SubjectType] = [('de', 'abstract_algebra'), ('de', 'anatomy'), ('de', 'astronomy'), ('de', 'business_ethics'), ('de', 'clinical_knowledge'), ('de', 'college_biology'), ('de', 'college_chemistry'), ('de', 'college_computer_science'), ('de', 'college_mathematics'), ('de', 'college_medicine'), ('de', 'college_physics'), ('de', 'computer_security'), ('de', 'conceptual_physics'), ('de', 'econometrics'), ('de', 'electrical_engineering'), ('de', 'elementary_mathematics'), ('de', 'formal_logic'), ('de', 'global_facts'), ('de', 'high_school_biology'), ('de', 'high_school_chemistry'), ('de', 'high_school_computer_science'), ('de', 'high_school_european_history'), ('de', 'high_school_geography'), ('de', 'high_school_government_and_politics'), ('de', 'high_school_macroeconomics'), ('de', 'high_school_mathematics'), ('de', 'high_school_microeconomics'), ('de', 'high_school_physics'), ('de', 'high_school_psychology'), ('de', 'high_school_statistics'), ('de', 'high_school_us_history'), ('de', 'high_school_world_history'), ('de', 'human_aging'), ('de', 'human_sexuality'), ('de', 'international_law'), ('de', 'jurisprudence'), ('de', 'logical_fallacies'), ('de', 'machine_learning'), ('de', 'management'), ('de', 'marketing'), ('de', 'medical_genetics'), ('de', 'miscellaneous'), ('de', 'moral_disputes'), ('de', 'moral_scenarios'), ('de', 'nutrition'), ('de', 'philosophy'), ('de', 'prehistory'), ('de', 'professional_accounting'), ('de', 'professional_law'), ('de', 'professional_medicine'), ('de', 'professional_psychology'), ('de', 'public_relations'), ('de', 'security_studies'), ('de', 'sociology'), ('de', 'us_foreign_policy'), ('de', 'virology'), ('de', 'world_religions')]

eval_framework.tasks.benchmarks.goldenswag module

class eval_framework.tasks.benchmarks.goldenswag.GOLDENSWAG(num_fewshot=0)

Bases: HELLASWAG

GoldenSwag dataset: https://huggingface.co/datasets/PleIAs/GoldenSwag available data set sections: validation

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'PleIAs/GoldenSwag'

FEWSHOT_SPLIT: str = 'validation'

NAME: str = 'GoldenSwag'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'validation'

class eval_framework.tasks.benchmarks.goldenswag.GOLDENSWAG_IDK(num_fewshot=0)

Bases: GOLDENSWAG

Parameters:

num_fewshot (int)

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.confidence_weighted_accuracy.ConfidenceWeightedAccuracy'>, <class 'eval_framework.metrics.loglikelihood.dcs.DistributionalCorrectnessScore'>, <class 'eval_framework.metrics.loglikelihood.ternary.TernaryScore'>]

NAME: str = 'GoldenSwag_IDK'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

eval_framework.tasks.benchmarks.gpqa module

class eval_framework.tasks.benchmarks.gpqa.GPQA(num_fewshot=0)

Bases: BaseTask[str]

GPQA dataset: https://huggingface.co/datasets/Idavidrein/gpqa

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'Idavidrein/gpqa'

FEWSHOT_SPLIT: str = 'train'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>]

NAME: str = 'GPQA'

RESPONSE_TYPE = 'loglikelihoods'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'train'

SUBJECTS: list[SubjectType] = ['gpqa_extended']

class eval_framework.tasks.benchmarks.gpqa.GPQA_COT(num_fewshot=0)

Bases: GPQA

Parameters:

num_fewshot (int)

ANS_RE = re.compile('Therefore, the answer is \\(([ABCDEFGHIJ])\\)')

METRICS = [<class 'eval_framework.metrics.completion.accuracy_completion.AccuracyCompletion'>]

NAME: str = 'GPQA_COT'

RESPONSE_TYPE = 'completion'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

post_process_generated_completion(completion_text, sample=None)

Return type:

str

Parameters:

• completion_text (str)

• sample (Sample | None)

stop_sequences: list[str]

class eval_framework.tasks.benchmarks.gpqa.GPQA_DIAMOND_COT(num_fewshot=0)

Bases: GPQA_COT

Parameters:

num_fewshot (int)

NAME: str = 'GPQA_DIAMOND_COT'

SUBJECTS: list[SubjectType] = ['gpqa_diamond']

class eval_framework.tasks.benchmarks.gpqa.GPQA_IDK(num_fewshot=0)

Bases: GPQA

Parameters:

num_fewshot (int)

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.confidence_weighted_accuracy.ConfidenceWeightedAccuracy'>, <class 'eval_framework.metrics.loglikelihood.dcs.DistributionalCorrectnessScore'>, <class 'eval_framework.metrics.loglikelihood.ternary.TernaryScore'>]

NAME: str = 'GPQA_IDK'

class eval_framework.tasks.benchmarks.gpqa.GPQA_OLMES(num_fewshot=0)

Bases: GPQA

GPQA multiple choice (OLMES/oe_eval style): prompt shows options with space-prefixed labels (” A.”, “ B.”, “ C.”, “ D.”); loglikelihood over “ A”/” B”/” C”/” D”.

Parameters:

num_fewshot (int)

NAME: str = 'GPQA_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

eval_framework.tasks.benchmarks.gpqa_ellamind module

German GPQA (Graduate-level Professional QA, EllaMind) tasks.

https://huggingface.co/datasets/ellamind/gpqa-multilingual

GPQA uses a single distractor set (incorrect_answers). Its diamond subset (the 198 hardest questions) is exposed via _DIAMOND_ONLY = True on the subclass.

class eval_framework.tasks.benchmarks.gpqa_ellamind.GPQA_ELLAMIND_BPB_DE(num_fewshot=0)

Bases: _GPQA_ELLAMIND_DE_Base

German GPQA - BPB format (all 448 items).

Parameters:

num_fewshot (int)

NAME: str = 'GPQA_ELLAMIND_BPB_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.BPBStyle object>

class eval_framework.tasks.benchmarks.gpqa_ellamind.GPQA_ELLAMIND_CLOZE_DE(num_fewshot=0)

Bases: _GPQA_ELLAMIND_DE_Base

German GPQA - Cloze format (all 448 items).

Parameters:

num_fewshot (int)

NAME: str = 'GPQA_ELLAMIND_CLOZE_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.ClozeStyle object>

class eval_framework.tasks.benchmarks.gpqa_ellamind.GPQA_ELLAMIND_DIAMOND_BPB_DE(num_fewshot=0)

Bases: _GPQA_ELLAMIND_DE_Base

German GPQA - BPB format, diamond subset (198 hardest items).

Parameters:

num_fewshot (int)

NAME: str = 'GPQA_ELLAMIND_DIAMOND_BPB_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.BPBStyle object>

class eval_framework.tasks.benchmarks.gpqa_ellamind.GPQA_ELLAMIND_DIAMOND_CLOZE_DE(num_fewshot=0)

Bases: _GPQA_ELLAMIND_DE_Base

German GPQA - Cloze format, diamond subset (198 hardest items).

Parameters:

num_fewshot (int)

NAME: str = 'GPQA_ELLAMIND_DIAMOND_CLOZE_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.ClozeStyle object>

class eval_framework.tasks.benchmarks.gpqa_ellamind.GPQA_ELLAMIND_DIAMOND_MC_DE(num_fewshot=0)

Bases: _GPQA_ELLAMIND_DE_Base

German GPQA - MC format, diamond subset (198 hardest items).

Parameters:

num_fewshot (int)

NAME: str = 'GPQA_ELLAMIND_DIAMOND_MC_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.MCStyle object>

class eval_framework.tasks.benchmarks.gpqa_ellamind.GPQA_ELLAMIND_MC_DE(num_fewshot=0)

Bases: _GPQA_ELLAMIND_DE_Base

German GPQA - MC format (all 448 items).

Parameters:

num_fewshot (int)

NAME: str = 'GPQA_ELLAMIND_MC_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.MCStyle object>

eval_framework.tasks.benchmarks.gsm8k module

class eval_framework.tasks.benchmarks.gsm8k.GSM8K(num_fewshot=0)

Bases: GSM8KEvalHarness

Parameters:

num_fewshot (int)

FEWSHOT_SPLIT: str = ''

NAME: str = 'GSM8K'

class eval_framework.tasks.benchmarks.gsm8k.GSM8KBPB(num_fewshot=8)

Bases: GSM8K_OLMES

Parameters:

num_fewshot (int)

NAME: str = 'GSM8KBPB'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.BPBStyle object>

class eval_framework.tasks.benchmarks.gsm8k.GSM8KEvalHarness(num_fewshot=0)

Bases: BaseTask[str]

GSM8K dataset: https://huggingface.co/datasets/openai/gsm8k This version uses samples from the train split as fewshot examples.

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'openai/gsm8k'

FEWSHOT_SPLIT: str = 'train'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.completion.accuracy_completion.AccuracyCompletion'>]

NAME: str = 'GSM8KEvalHarness'

RESPONSE_TYPE = 'completion'

SAMPLE_SPLIT: str = 'test'

SUBJECTS: list[SubjectType] = ['main']

post_process_generated_completion(completion_text, sample=None)

Return type:

str

Parameters:

• completion_text (str)

• sample (Sample | None)

stop_sequences: list[str]

class eval_framework.tasks.benchmarks.gsm8k.GSM8K_OLMES(num_fewshot=8)

Bases: GSM8K

Parameters:

num_fewshot (int)

METRICS = [<class 'eval_framework.metrics.completion.accuracy_completion.AccuracyCompletionOLMES'>]

NAME: str = 'GSM8K_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

add_spaces_around_operators_no_regex(_str)

Add spacing around special operators if it does not exist

Return type:

str

Parameters:

_str (str)

normalize_answer_str(item)

Rephrases the answer to a more natural-appearing string to improve bpb calculation (according to Olmes)

TODO: Verify the necessity of this operation.

Return type:

str

Parameters:

item (dict[str, Any])

post_process_generated_completion(completion_text, sample=None)

Return type:

str

Parameters:

• completion_text (str)

• sample (Sample | None)

eval_framework.tasks.benchmarks.gsm8k_ellamind module

German GSM8K (EllaMind) tasks.

https://huggingface.co/datasets/ellamind/gsm8k-platinum-multilingual

class eval_framework.tasks.benchmarks.gsm8k_ellamind.GSM8K_Ellamind_DE_BPB_Platinum(num_fewshot=0)

Bases: BaseTask

German GSM8K (Platinum) BPB format.

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'ellamind/gsm8k-platinum-multilingual'

FEWSHOT_SPLIT: str = 'test'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'German'

NAME: str = 'GSM8K_Ellamind_DE_BPB_Platinum'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'test'

SUBJECTS: list[SubjectType] = ['deu']

TASK_STYLER = <eval_framework.tasks.task_style.BPBStyle object>

class eval_framework.tasks.benchmarks.gsm8k_ellamind.GSM8K_Ellamind_DE_Platinum(num_fewshot=0)

Bases: GSM8KEvalHarness

German GSM8K (Platinum) generative, mirroring the English GSM8K.

Dataset: https://huggingface.co/datasets/ellamind/gsm8k-platinum-multilingual

German adaptations vs. the English GSM8K:

• prompt markers translated: Frage: / Antwort:.

• few-shot targets end with a German final-answer line (Daher ist die Antwort N.). The EllaMind final_answer column is the ground truth directly, so no #### N extraction is needed.

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'ellamind/gsm8k-platinum-multilingual'

FEWSHOT_SPLIT: str = 'test'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'German'

NAME: str = 'GSM8K_Ellamind_DE_Platinum'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'test'

SUBJECTS: list[SubjectType] = ['deu']

post_process_generated_completion(completion_text, sample=None)

Extract the last number from the completion text.

Return type:

str

Parameters:

• completion_text (str)

• sample (Sample | None)

eval_framework.tasks.benchmarks.hellaswag module

class eval_framework.tasks.benchmarks.hellaswag.HELLASWAG(num_fewshot=0)

Bases: BaseTask[str]

Hellaswag dataset: https://huggingface.co/datasets/Rowan/hellaswag available data set sections: train, validation, test

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'Rowan/hellaswag'

FEWSHOT_SPLIT: str = 'train'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.bits_per_byte.BitsPerByteLoglikelihood'>]

NAME: str = 'HellaSwag'

RESPONSE_TYPE = 'loglikelihoods'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'validation'

SUBJECTS: list[SubjectType] = ['no_subject']

class eval_framework.tasks.benchmarks.hellaswag.HELLASWAG_IDK(num_fewshot=0)

Bases: HELLASWAG

Parameters:

num_fewshot (int)

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.confidence_weighted_accuracy.ConfidenceWeightedAccuracy'>, <class 'eval_framework.metrics.loglikelihood.dcs.DistributionalCorrectnessScore'>, <class 'eval_framework.metrics.loglikelihood.ternary.TernaryScore'>]

NAME: str = 'HellaSwag_IDK'

class eval_framework.tasks.benchmarks.hellaswag.HELLASWAG_OLMES(num_fewshot=0)

Bases: HELLASWAG

Parameters:

num_fewshot (int)

NAME: str = 'HellaSwag_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'train'

eval_framework.tasks.benchmarks.hellaswag_ellamind module

German HellaSwag (EllaMind) tasks.

https://huggingface.co/datasets/ellamind/hellaswag-multilingual

HellaSwag supplies separate easy and hard distractors. Each task class uses a _DISTRACTOR_LEVEL class attribute ("easy" or "hard").

class eval_framework.tasks.benchmarks.hellaswag_ellamind.HELLASWAG_ELLAMIND_BPB_DE(num_fewshot=0)

Bases: HELLASWAG_ELLAMIND_EASY_DE

German HellaSwag - BPB format.

Parameters:

num_fewshot (int)

NAME: str = 'HELLASWAG_ELLAMIND_BPB_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.BPBStyle object>

class eval_framework.tasks.benchmarks.hellaswag_ellamind.HELLASWAG_ELLAMIND_EASY_DE(num_fewshot=0)

Bases: BaseTask[str]

German HellaSwag - Cloze (sentence-completion) format with easy distractors.

Dataset: https://huggingface.co/datasets/ellamind/hellaswag-multilingual

HellaSwag is a sentence-completion task: the prompt is a partial sentence ("{activity}: {context}") and the model scores full sentence endings. There is no natural MC variant for this task (would be possible, but not natural).

Set _DISTRACTOR_LEVEL = "easy" or "hard" on the task class.

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'ellamind/hellaswag-multilingual'

FEWSHOT_SPLIT: str = 'validation'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'German'

NAME: str = 'HELLASWAG_ELLAMIND_EASY_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'validation'

SUBJECTS: list[SubjectType] = ['deu']

TASK_STYLER = <eval_framework.tasks.task_style.ClozeStyle object>

class eval_framework.tasks.benchmarks.hellaswag_ellamind.HELLASWAG_ELLAMIND_HARD_DE(num_fewshot=0)

Bases: HELLASWAG_ELLAMIND_EASY_DE

German HellaSwag - Cloze (sentence-completion) format with hard distractors.

Parameters:

num_fewshot (int)

NAME: str = 'HELLASWAG_ELLAMIND_HARD_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

eval_framework.tasks.benchmarks.hendrycks_math_ellamind module

German Hendrycks Math (EllaMind) tasks.

https://huggingface.co/datasets/ellamind/hendrycks-math-multilingual

class eval_framework.tasks.benchmarks.hendrycks_math_ellamind.MATHMinervaDE_BPB_OLMES(num_fewshot=4)

Bases: MATHMinervaDE_OLMES

German HENDRYCKS Math - BPB format.

Dataset: https://huggingface.co/datasets/ellamind/hendrycks-math-multilingual

Parameters:

num_fewshot (int)

NAME: str = 'MATHMinervaDE_BPB_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.BPBStyle object>

class eval_framework.tasks.benchmarks.hendrycks_math_ellamind.MATHMinervaDE_OLMES(num_fewshot=4)

Bases: MATHMinerva

German Hendrycks Math - MATHMinerva-style reasoning generation.

Dataset: https://huggingface.co/datasets/ellamind/hendrycks-math-multilingual

Matches the English MATHMinerva_OLMES task with these German adaptations:

• prompt markers translated: Aufgabe: / Lösung:

• stop sequences adjusted accordingly

• few-shot examples drawn from the dataset (not the hardcoded English _OLMES_FEWSHOTS list the parent uses)

• few-shot targets end with a German Minerva-style final-answer line.

The EllaMind dataset is test-only; few-shot sampling uses the same oversample-and-drop guard as BaseTask._sample_fewshot_examples while filtering out multi-paragraph solutions. Solutions preserve the original \boxed{...} markers, so the parent’s ground-truth extraction and the Minerva metric’s boxed-answer fallback both keep working unchanged.

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'ellamind/hendrycks-math-multilingual'

FEWSHOT_SPLIT: str = 'test'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'German'

METRICS = [<class 'eval_framework.metrics.completion.math_minerva_completion.MathMinervaCompletionDE'>, <class 'eval_framework.metrics.completion.math_minerva_completion.MathMinervaCompletionRelaxedDE'>]

NAME: str = 'MATHMinervaDE_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'test'

SUBJECTS: list[SubjectType] = ['deu']

post_process_generated_completion(completion_text, sample=None)

Primary answer for storage; uses relaxed final-answer extraction.

Return type:

str

Parameters:

• completion_text (str)

• sample (Sample | None)

class eval_framework.tasks.benchmarks.hendrycks_math_ellamind.MATHMinervaDE_OLMES_NONL(num_fewshot=4)

Bases: MATHMinervaDE_OLMES

Same as MATHMinervaDE_OLMES, but without nn as stop sequence.

Parameters:

num_fewshot (int)

NAME: str = 'MATHMinervaDE_OLMES_NONL'

eval_framework.tasks.benchmarks.hle_ellamind module

German HLE (Humanity’s Last Exam, EllaMind) tasks.

https://huggingface.co/datasets/ellamind/hle-multilingual

HLE uses a single distractor set (incorrect_answers). The natively multiple-choice subset is exposed via _NATIVE_MC_ONLY = True on the subclass.

class eval_framework.tasks.benchmarks.hle_ellamind.HLE_ELLAMIND_BPB_DE(num_fewshot=0)

Bases: _HLE_ELLAMIND_DE_Base

German HLE - BPB format (all 800 items).

Parameters:

num_fewshot (int)

NAME: str = 'HLE_ELLAMIND_BPB_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.BPBStyle object>

class eval_framework.tasks.benchmarks.hle_ellamind.HLE_ELLAMIND_CLOZE_DE(num_fewshot=0)

Bases: _HLE_ELLAMIND_DE_Base

German HLE - Cloze format (all 800 items).

Parameters:

num_fewshot (int)

NAME: str = 'HLE_ELLAMIND_CLOZE_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.ClozeStyle object>

class eval_framework.tasks.benchmarks.hle_ellamind.HLE_ELLAMIND_CLOZE_NATIVE_DE(num_fewshot=0)

Bases: _HLE_ELLAMIND_DE_Base

German HLE - Cloze format, native multiple-choice items only.

Parameters:

num_fewshot (int)

NAME: str = 'HLE_ELLAMIND_CLOZE_NATIVE_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.ClozeStyle object>

class eval_framework.tasks.benchmarks.hle_ellamind.HLE_ELLAMIND_MC_DE(num_fewshot=0)

Bases: _HLE_ELLAMIND_DE_Base

German HLE - MC format (all 800 items).

Parameters:

num_fewshot (int)

NAME: str = 'HLE_ELLAMIND_MC_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.MCStyle object>

class eval_framework.tasks.benchmarks.hle_ellamind.HLE_ELLAMIND_MC_NATIVE_DE(num_fewshot=0)

Bases: _HLE_ELLAMIND_DE_Base

German HLE - MC format, native multiple-choice items only.

Parameters:

num_fewshot (int)

NAME: str = 'HLE_ELLAMIND_MC_NATIVE_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.MCStyle object>

eval_framework.tasks.benchmarks.humaneval module

class eval_framework.tasks.benchmarks.humaneval.HumanEval(num_fewshot=0)

Bases: BaseTask[str]

HumanEval dataset: https://huggingface.co/datasets/openai/openai_humaneval/

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'openai/openai_humaneval'

FEWSHOT_SPLIT: str = 'test'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.completion.code_assertion.CodeCompletionAssertion'>]

NAME: str = 'Human Eval'

RESPONSE_TYPE = 'completion'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'test'

SUBJECTS: list[SubjectType] = ['no_subject']

post_process_generated_completion(completion_text, sample=None)

Return type:

str

Parameters:

• completion_text (str)

• sample (Sample | None)

class eval_framework.tasks.benchmarks.humaneval.HumanEvalBPB(num_fewshot=0)

Bases: HumanEval

HumanEval variant that scores loglikelihood of the gold canonical solution. Reports bits-per-byte on the reference completion.

Parameters:

num_fewshot (int)

METRICS = [<class 'eval_framework.metrics.loglikelihood.bits_per_byte.BitsPerByteLoglikelihood'>]

NAME: str = 'Human Eval BPB'

RESPONSE_TYPE = 'loglikelihoods'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.humaneval.HumanEvalBPB_V2(num_fewshot=0)

Bases: HumanEvalBPB

HumanEvalBPB variant that wraps the prompt and canonical solution in markdown code fences instead of prefixing the solution with a leading space, so the loglikelihood boundary starts on a new line.

Parameters:

num_fewshot (int)

NAME: str = 'Human Eval BPB V2'

class eval_framework.tasks.benchmarks.humaneval.HumanEvalInstruct(num_fewshot=0)

Bases: HumanEval

Parameters:

num_fewshot (int)

CUE_PREFIX = 'Here is the completed function:\n```python\n'

NAME: str = 'Human Eval Instruct'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.humaneval.HumanEvalMetricContext(**data)

Bases: BaseMetricContext

Parameters:

• test (str)

• entry_point (str)

• prompt (str)

• extra_data (Any)

entry_point: str

model_config: ClassVar[ConfigDict] = {'extra': 'allow'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

prompt: str

test: str

class eval_framework.tasks.benchmarks.humaneval.HumanEval_OLMES(num_fewshot=3)

Bases: HumanEval

HumanEval OLMES variant replicating codex_humaneval:3shot::olmo3:n32:v2 from oe_eval.

Recommended EvalConfig settings for full replication:

repeats: 32 llm_args: {sampling_params: {temperature: 0.6, top_p: 0.6}}

Parameters:

num_fewshot (int)

NAME: str = 'Human Eval OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.humaneval.HumanEval_OLMES_V2(num_fewshot=3)

Bases: HumanEval_OLMES

Parameters:

num_fewshot (int)

NAME: str = 'Human Eval Olmes v2'

eval_framework.tasks.benchmarks.humaneval_ellamind module

German HumanEval (EllaMind) tasks.

https://huggingface.co/datasets/ellamind/humaneval-multilingual

class eval_framework.tasks.benchmarks.humaneval_ellamind.HumanEvalDE_BPB_OLMES(num_fewshot=0)

Bases: BaseTask[str]

German HumanEval - BPB format (loglikelihood of the canonical solution).

Dataset: https://huggingface.co/datasets/ellamind/humaneval-multilingual

We use the following prompt format: {prompt} {canonical_solution}

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'ellamind/humaneval-multilingual'

FEWSHOT_SPLIT: str = 'test'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'German'

NAME: str = 'HumanEvalDE_BPB_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'test'

SUBJECTS: list[SubjectType] = ['deu']

TASK_STYLER = <eval_framework.tasks.task_style.BPBStyle object>

class eval_framework.tasks.benchmarks.humaneval_ellamind.HumanEvalDE_BPB_OLMES_V2(num_fewshot=0)

Bases: HumanEvalDE_BPB_OLMES

HumanEvalDE_BPB_OLMES variant that wraps the prompt and canonical solution in markdown code fences, mirroring the HumanEvalDE_OLMES completion prompt exactly.

Parameters:

num_fewshot (int)

NAME: str = 'HumanEvalDE_BPB_OLMES V2'

class eval_framework.tasks.benchmarks.humaneval_ellamind.HumanEvalDE_OLMES(num_fewshot=3)

Bases: HumanEval_OLMES

German HumanEval - code generation format (pass@1 via test execution).

Dataset: https://huggingface.co/datasets/ellamind/humaneval-multilingual

The EllaMind dataset mirrors the original HumanEval dataset, so only the dataset path and language metadata are overridden.

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'ellamind/humaneval-multilingual'

FEWSHOT_SPLIT: str = 'test'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'German'

NAME: str = 'HumanEvalDE_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'test'

SUBJECTS: list[SubjectType] = ['deu']

class eval_framework.tasks.benchmarks.humaneval_ellamind.HumanEvalDE_OLMES_V2(num_fewshot=3)

Bases: HumanEval_OLMES_V2

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'ellamind/humaneval-multilingual'

FEWSHOT_SPLIT: str = 'test'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'German'

NAME: str = 'Human Eval DE Newline OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'test'

SUBJECTS: list[SubjectType] = ['deu']

eval_framework.tasks.benchmarks.ifeval module

class eval_framework.tasks.benchmarks.ifeval.IFEval(num_fewshot=0)

Bases: BaseTask[str]

IFEval: Instruction Following Eval (https://arxiv.org/pdf/2311.07911).

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'google/IFEval'

FEWSHOT_SPLIT: str = 'train'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = {'no_subject': Language.ENG}

METRICS = [<class 'eval_framework.metrics.completion.ifeval.IFEvalMetric'>]

NAME: str = 'IFEval'

RESPONSE_TYPE = 'completion'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'train'

SUBJECTS: list[SubjectType] = ['no_subject']

class eval_framework.tasks.benchmarks.ifeval.IFEvalDe(num_fewshot=0)

Bases: IFEval

German version of the Instruction Following Evaluation (IFEval) benchmark.

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'jzhang86/de_ifeval'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = {'no_subject': Language.DEU}

METRICS = [<class 'eval_framework.metrics.completion.ifeval.IFEvalMetric'>, <class 'eval_framework.metrics.completion.language_checker.LanguageRawConsistencyChecker'>]

NAME: str = 'IFEval German'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SUBJECTS: list[SubjectType] = ['no_subject']

class eval_framework.tasks.benchmarks.ifeval.IFEvalFiSv(num_fewshot=0)

Bases: IFEval

Machine translated versions of the Instruction Following Evaluation (IFEval) benchmark.

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'LumiOpen/ifeval_mt'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = {'fi': Language.FIN, 'sv': Language.SWE}

NAME: str = 'IFEval Finnish & Swedish'

SUBJECTS: list[SubjectType] = ['fi', 'sv']

eval_framework.tasks.benchmarks.math_reasoning module

class eval_framework.tasks.benchmarks.math_reasoning.AIME2024(num_fewshot=0)

Bases: MATHReasoning

AIME 2024 dataset: https://huggingface.co/datasets/HuggingFaceH4/aime_2024

This dataset contains a single train split of 30 questions. Data contains

ID | Problem | Solution | Answer

pass@1 evaluation

Parameters:

num_fewshot (int)

ANSWER_PATTERN = 'Therefore, the final answer is:(.*?). I hope it is correct.'

DATASET_PATH: str = 'HuggingFaceH4/aime_2024'

FEWSHOT_SPLIT: str = 'train'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.completion.math_reasoning_completion.MathReasoningCompletion'>, <class 'eval_framework.metrics.completion.language_checker.LanguageRawConsistencyChecker'>]

NAME: str = 'AIME2024'

QUERY_TEMPLATE = 'Solve the following math problem efficiently and clearly:\n\n    - For simple problems (2 steps or fewer):\n    Provide a concise solution with minimal explanation.\n\n    - For complex problems (3 steps or more):\n    Use this step-by-step format:\n\n    ## Step 1: [Concise description]\n    [Brief explanation and calculations]\n\n    ## Step 2: [Concise description]\n    [Brief explanation and calculations]\n\n    ...\n\n    Regardless of the approach, always conclude with:\n\n    Therefore, the final answer is: $\\boxed{{answer}}$. I hope it is correct.\n\n    Where [answer] is just the final number or expression that solves the problem.\n\n    Problem: {Question}'

RESPONSE_TYPE = 'completion'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'train'

SUBJECTS: list[SubjectType] = ['no_subject']

class eval_framework.tasks.benchmarks.math_reasoning.AIME2025(num_fewshot=0)

Bases: AIME2024

AIME 2025 dataset: https://huggingface.co/datasets/math-ai/aime25

This dataset contains a single test split of 30 questions. Data contains problem | answer | id

pass@1 evaluation

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'math-ai/aime25'

FEWSHOT_SPLIT: str = 'test'

NAME: str = 'AIME2025'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'test'

class eval_framework.tasks.benchmarks.math_reasoning.AIME2026(num_fewshot=0)

Bases: AIME2024

AIME 2026 dataset: https://huggingface.co/datasets/math-ai/aime26

This dataset contains a single test split of 30 questions. Data contains problem | answer | id

pass@1 evaluation

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'math-ai/aime26'

FEWSHOT_SPLIT: str = 'test'

NAME: str = 'AIME2026'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'test'

class eval_framework.tasks.benchmarks.math_reasoning.GSM8KReasoning(num_fewshot=0)

Bases: MATHReasoning

GSM8K dataset with reasoning prompt: https://huggingface.co/datasets/openai/gsm8k

Zero-shot reasoning version that expects answers in boxed format.

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'openai/gsm8k'

FEWSHOT_SPLIT: str = 'train'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.completion.accuracy_completion.AccuracyCompletion'>, <class 'eval_framework.metrics.completion.language_checker.LanguageRawConsistencyChecker'>]

NAME: str = 'GSM8KReasoning'

QUERY_TEMPLATE = 'Solve the following math problem step by step. Think through the problem carefully and show your reasoning.\n\nPlease provide your answer in the format: $\\boxed{{answer}}$ where answer is the final numerical result.\n\nQuestion: {question}\n\nAnswer:'

RESPONSE_TYPE = 'completion'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'test'

SUBJECTS: list[SubjectType] = ['main']

post_process_generated_completion(completion_text, sample=None)

Return type:

str

Parameters:

• completion_text (str)

• sample (Sample | None)

stop_sequences: list[str]

class eval_framework.tasks.benchmarks.math_reasoning.MATH(num_fewshot=0)

Bases: MATHReasoning

MATH dataset: https://huggingface.co/datasets/EleutherAI/hendrycks_math

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'EleutherAI/hendrycks_math'

FEWSHOT_SPLIT: str = 'train'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.completion.math_reasoning_completion.MathReasoningCompletion'>, <class 'eval_framework.metrics.completion.language_checker.LanguageRawConsistencyChecker'>]

NAME: str = 'Math'

QUERY_TEMPLATE = 'Solve the following math problem step by step. The last line of your response should be of the form Answer: $ANSWER (without quotes) where $ANSWER is the answer to the problem.\n\n    {Question}\n\n    Remember to put your answer in $\\boxed{{answer}}$\n\n    where [answer] is just the final number or expression that solves the problem.'

RESPONSE_TYPE = 'completion'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'test'

SUBJECTS: list[SubjectType] = ['algebra', 'counting_and_probability', 'geometry', 'intermediate_algebra', 'number_theory', 'prealgebra', 'precalculus']

extract_last_two_dollar_text(s)

extract_last_two_dollar_text finds text between the last two dollar signs in a string :type s: str :param s: the string to extract text from

Return type:

str

Returns:

the extracted text

Parameters:

s (str)

post_process_generated_completion(completion_text, sample=None)

post_process_generated_completion extracts via flex extraction/matching. if there is a boxed answer, then this gets used first if there is no boxed answer, and latex math symbols (“$”) then this will be extracted and used if there is an answer text (“Answer:”) then this will be used last

Return type:

str

Parameters:

• completion_text (str)

• sample (Sample | None)

class eval_framework.tasks.benchmarks.math_reasoning.MATH500(num_fewshot=0)

Bases: MATHReasoning

MATH500 dataset: https://huggingface.co/datasets/HuggingFaceH4/MATH-500

This dataset contains a single test split of 500 questions. Data contains

ID | Problem | Solution | Answer

pass@1 evaluation

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'HuggingFaceH4/MATH-500'

FEWSHOT_SPLIT: str = 'test'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.completion.math_reasoning_completion.MathReasoningCompletion'>, <class 'eval_framework.metrics.completion.language_checker.LanguageRawConsistencyChecker'>]

NAME: str = 'MATH500'

QUERY_TEMPLATE = 'Solve the following math problem step by step. The last line of your response should be of the form Answer: $ANSWER (without quotes) where $ANSWER is the answer to the problem.\n\n    {Question}\n\n    Remember to put your answer in $\\boxed{{answer}}$\n\n    where [answer] is just the final number or expression that solves the problem.'

RESPONSE_TYPE = 'completion'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'test'

SUBJECTS: list[SubjectType] = ['no_subject']

post_process_generated_completion(completion_text, sample=None)

Return type:

str

Parameters:

• completion_text (str)

• sample (Sample | None)

class eval_framework.tasks.benchmarks.math_reasoning.MATH500Minerva(num_fewshot=0)

Bases: MATHMinerva

MATH-500 with Minerva-style prompt and scoring (OLMES minerva_math_500 parity). Uses HuggingFaceH4/MATH-500 which has a single ‘default’ config (no subject splits).

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'HuggingFaceH4/MATH-500'

FEWSHOT_SPLIT: str = 'test'

NAME: str = 'MATH500Minerva'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'test'

SUBJECTS: list[SubjectType] = ['no_subject']

class eval_framework.tasks.benchmarks.math_reasoning.MATHLvl5(num_fewshot=0)

Bases: MATH

Parameters:

num_fewshot (int)

NAME: str = 'Math Lvl 5'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.math_reasoning.MATHMinerva(num_fewshot=0)

Bases: MATHMinervaEvalHarness

MATH with Minerva-style prompt and relaxed final-answer string matching. Same as MATHMinervaEvalHarness but allows flexible whitespace and case for variations of “(The )Final Answer: The (final )answer is …( I hope it is correct.)”, where parentheses are optional.

Parameters:

num_fewshot (int)

METRICS = [<class 'eval_framework.metrics.completion.math_minerva_completion.MathMinervaCompletionRelaxed'>]

NAME: str = 'MATHMinerva'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

post_process_generated_completion(completion_text, sample=None)

Primary answer for storage; uses relaxed final-answer extraction.

Return type:

str

Parameters:

• completion_text (str)

• sample (Sample | None)

class eval_framework.tasks.benchmarks.math_reasoning.MATHMinervaBPB(num_fewshot=4)

Bases: MATHMinerva_OLMES

Parameters:

num_fewshot (int)

NAME: str = 'MATHMinervaBPB'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.BPBStyle object>

class eval_framework.tasks.benchmarks.math_reasoning.MATHMinervaEvalHarness(num_fewshot=0)

Bases: MATHReasoning

MATH with Minerva-style prompt and scoring (lm-evaluation-harness / oe_eval parity). Uses strict final-answer string matching: “Final Answer: The final answer is … I hope it is correct.” Prompt: “Problem:n” + problem + “nn” + “Solution:” Gold: normalized_gold_from_solution(solution) Metrics: Exact Match, Exact Match (Flex) via MathMinervaCompletion.

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'EleutherAI/hendrycks_math'

FEWSHOT_SPLIT: str = 'train'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.completion.math_minerva_completion.MathMinervaCompletion'>]

NAME: str = 'MATHMinervaEvalHarness'

RESPONSE_TYPE = 'completion'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'test'

SUBJECTS: list[SubjectType] = ['algebra', 'counting_and_probability', 'geometry', 'intermediate_algebra', 'number_theory', 'prealgebra', 'precalculus']

post_process_generated_completion(completion_text, sample=None)

Primary answer for storage; metric uses raw_completion for exact_match_flex (strict matching).

Return type:

str

Parameters:

• completion_text (str)

• sample (Sample | None)

class eval_framework.tasks.benchmarks.math_reasoning.MATHMinerva_OLMES(num_fewshot=4)

Bases: MATHMinerva

Parameters:

num_fewshot (int)

METRICS = [<class 'eval_framework.metrics.completion.math_minerva_completion.MathMinervaCompletion'>, <class 'eval_framework.metrics.completion.math_minerva_completion.MathMinervaCompletionRelaxed'>]

NAME: str = 'MATHMinerva_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.math_reasoning.MATHMinerva_OLMES_NONL(num_fewshot=4)

Bases: MATHMinerva_OLMES

Parameters:

num_fewshot (int)

NAME: str = 'MATHMinerva_OLMES_NONL'

class eval_framework.tasks.benchmarks.math_reasoning.MATHReasoning(num_fewshot=0)

Bases: BaseTask[str]

AIME 2024 dataset: https://huggingface.co/datasets/HuggingFaceH4/aime_2024

This dataset contains a single train split of 30 questions. Data contains

ID | Problem | Solution | Answer

pass@1 evaluation

Parameters:

num_fewshot (int)

ANSWER_PATTERN = '(?i)Answer\\s*:\\s*(.*)'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.completion.math_reasoning_completion.MathReasoningCompletion'>]

RESPONSE_TYPE = 'completion'

SUBJECTS: list[SubjectType] = ['no_subject']

post_process_generated_completion(completion_text, sample=None)

Return type:

str

Parameters:

• completion_text (str)

• sample (Sample | None)

eval_framework.tasks.benchmarks.mbpp module

class eval_framework.tasks.benchmarks.mbpp.MBPP(num_fewshot=0)

Bases: BaseTask[str]

MBPP provides both the problem statement and the test cases upfront. It says, “Here’s the problem and here are the tests; write code that passes them.”. Note that LLMs can cheat and only write code that passes the tests without solving the given problem.

MBPP_PROMPT_WITHOUT_TESTS, on the other hand, only gives you the problem statement and function signature initially. It says, “Here’s the problem and function signature; write code, then we’ll run tests later.”

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'google-research-datasets/mbpp'

FEWSHOT_SPLIT: str = 'train'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.completion.code_assertion.CodeCompletionAssertion'>]

NAME: str = 'MBPP'

RESPONSE_TYPE = 'completion'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'test'

SUBJECTS: list[SubjectType] = ['full']

post_process_generated_completion(completion_text, sample)

Return type:

str

Parameters:

• completion_text (str)

• sample (Sample)

class eval_framework.tasks.benchmarks.mbpp.MBPPBPB(num_fewshot=0)

Bases: MBPP

MBPP variant that scores loglikelihood of the gold reference code. Reports bits-per-byte on the reference solution.

Parameters:

num_fewshot (int)

METRICS = [<class 'eval_framework.metrics.loglikelihood.bits_per_byte.BitsPerByteLoglikelihood'>]

NAME: str = 'MBPP BPB'

RESPONSE_TYPE = 'loglikelihoods'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.mbpp.MBPPMetricContext(**data)

Bases: BaseMetricContext

Parameters:

• tests_code (str)

• extra_data (Any)

model_config: ClassVar[ConfigDict] = {'extra': 'allow'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

tests_code: str

class eval_framework.tasks.benchmarks.mbpp.MBPP_BPB_EvalPlus(num_fewshot=3)

Bases: BaseTask[str]

BPB (loglikelihood) counterpart of MBPP_EvalPlus.

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'google-research-datasets/mbpp'

FEWSHOT_SPLIT: str = 'test'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

NAME: str = 'MBPP_BPB_EvalPlus'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'test'

SUBJECTS: list[SubjectType] = ['full']

TASK_STYLER = <eval_framework.tasks.task_style.BPBStyle object>

class eval_framework.tasks.benchmarks.mbpp.MBPP_EvalPlus(num_fewshot=3)

Bases: MBPP

Version that faithfully follows the EvalPlus prompt format, which OLMES only approximated.

Parameters:

num_fewshot (int)

FEWSHOT_SPLIT: str = 'test'

NAME: str = 'MBPP_EvalPlus'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

post_process_generated_completion(completion_text, sample)

Return type:

str

Parameters:

• completion_text (str)

• sample (Sample)

class eval_framework.tasks.benchmarks.mbpp.MBPP_OLMES(num_fewshot=3)

Bases: MBPP

MBPP OLMES variant replicating oe_eval’s mbpp:3shot::olmo3:n32:v2.

Uses the EvalPlus prompt format with 3 hardcoded fewshot examples from the original MBPP “prompt” split (matching oe_eval’s ordering). Each prompt shows one test case (the first) instead of all.

Recommended EvalConfig settings for full replication:

split: test
num_fewshot: 3 (hardcoded, prompt split)
metric: pass_at_1
temperature: 0.6
top_p: 0.6
repeats: 32

Parameters:

num_fewshot (int)

FEWSHOT_SPLIT: str = 'test'

NAME: str = 'MBPP_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

post_process_generated_completion(completion_text, sample)

Return type:

str

Parameters:

• completion_text (str)

• sample (Sample)

class eval_framework.tasks.benchmarks.mbpp.MBPP_PROMPT_WITHOUT_TESTS(num_fewshot=0)

Bases: MBPP

MBPP provides both the problem statement and the test cases upfront. It says, “Here’s the problem and here are the tests; write code that passes them.”. Note that LLMs can cheat and only write code that passes the tests without solving the given problem.

MBPP_PROMPT_WITHOUT_TESTS, on the other hand, only gives you the problem statement and function signature initially. It says, “Here’s the problem and function signature; write code, then we’ll run tests later.”

Parameters:

num_fewshot (int)

NAME: str = 'MBPP_PROMPT_WITHOUT_TESTS'

post_process_generated_completion(completion_text, sample)

Return type:

str

Parameters:

• completion_text (str)

• sample (Sample)

class eval_framework.tasks.benchmarks.mbpp.MBPP_PROMPT_WITHOUT_TESTS_SANITIZED(num_fewshot=0)

Bases: MBPP_PROMPT_WITHOUT_TESTS

Parameters:

num_fewshot (int)

NAME: str = 'MBPP_PROMPT_WITHOUT_TESTS_SANITIZED'

SUBJECTS: list[SubjectType] = ['sanitized']

class eval_framework.tasks.benchmarks.mbpp.MBPP_SANITIZED(num_fewshot=0)

Bases: MBPP

Parameters:

num_fewshot (int)

NAME: str = 'MBPP_SANITZED'

SUBJECTS: list[SubjectType] = ['sanitized']

eval_framework.tasks.benchmarks.mbpp_ellamind module

German MBPP (EllaMind) tasks.

https://huggingface.co/datasets/ellamind/mbpp-multilingual

class eval_framework.tasks.benchmarks.mbpp_ellamind.MBPPDE_BPB_EvalPlus(num_fewshot=3)

Bases: MBPP_BPB_EvalPlus

German MBPP - EvalPlus BPB format (loglikelihood of the reference Python solution).

German pendant of MBPP_BPB_EvalPlus: same BPBStyle mechanism and EvalPlus prompt, with the German instruction wrapper, German response preamble, and same-split (German) few-shot sampling.

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'ellamind/mbpp-multilingual'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'German'

NAME: str = 'MBPPDE_BPB_EvalPlus'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SUBJECTS: list[SubjectType] = ['deu']

TASK_STYLER = <eval_framework.tasks.task_style.BPBStyle object>

class eval_framework.tasks.benchmarks.mbpp_ellamind.MBPPDE_BPB_OLMES(num_fewshot=0)

Bases: BaseTask[str]

German MBPP - BPB format (loglikelihood of the reference Python solution).

Dataset: https://huggingface.co/datasets/ellamind/mbpp-multilingual

We use the following prompt format: {text} `python {code} `

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'ellamind/mbpp-multilingual'

FEWSHOT_SPLIT: str = 'test'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'German'

NAME: str = 'MBPPDE_BPB_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'test'

SUBJECTS: list[SubjectType] = ['deu']

TASK_STYLER = <eval_framework.tasks.task_style.BPBStyle object>

class eval_framework.tasks.benchmarks.mbpp_ellamind.MBPPDE_EvalPlus(num_fewshot=3)

Bases: MBPP_EvalPlus

German MBPP - EvalPlus code-generation format (pass@1 via test execution).

German pendant of MBPP_EvalPlus: same prompt structure and post-processing, with the German instruction wrapper, German response preamble, and same-split (German) few-shot sampling.

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'ellamind/mbpp-multilingual'

FEWSHOT_SPLIT: str = 'test'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'German'

NAME: str = 'MBPPDE_EvalPlus'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'test'

SUBJECTS: list[SubjectType] = ['deu']

class eval_framework.tasks.benchmarks.mbpp_ellamind.MBPPDE_OLMES(num_fewshot=3)

Bases: MBPP_OLMES

German MBPP - code generation format (pass@1 via test execution).

Dataset: https://huggingface.co/datasets/ellamind/mbpp-multilingual

The EllaMind dataset mirrors the google-research-datasets/mbpp schema (text / code / test_list), so the upstream MBPP stop sequence, cue text, metric context, and post-processing (which expands the generated code with the ground-truth asserts and prints True/False) are reused unchanged. Only the dataset path, language metadata, and the German instruction wrapper around item["text"] are overridden.

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'ellamind/mbpp-multilingual'

FEWSHOT_SPLIT: str = 'test'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'German'

NAME: str = 'MBPPDE_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'test'

SUBJECTS: list[SubjectType] = ['deu']

eval_framework.tasks.benchmarks.medqa module

MedQA (English): Open-domain medical question answering from medical exams.

class eval_framework.tasks.benchmarks.medqa.MedQACloze(num_fewshot=0)

Bases: BaseTask[str]

MedQA cloze (loglikelihood over choice text).

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'davidheineman/medqa-en'

FEWSHOT_SPLIT: str = 'dev'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.bits_per_byte.BitsPerByteLoglikelihood'>]

NAME: str = 'MedQACloze'

RESPONSE_TYPE = 'loglikelihoods'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'test'

SUBJECTS: list[SubjectType] = ['no_subject']

class eval_framework.tasks.benchmarks.medqa.MedQAMC(num_fewshot=0)

Bases: MedQACloze

MedQA multiple choice (loglikelihood over A/B/C/D/…).

Parameters:

num_fewshot (int)

NAME: str = 'MedQAMC'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.medqa.MedQAMC_OLMES(num_fewshot=0)

Bases: MedQAMC

MedQA multiple choice with OLMES-style prompt: space before each label (” A.”, “ B.”, …).

Parameters:

num_fewshot (int)

FEWSHOT_SPLIT: str = 'train'

NAME: str = 'MedQAMC_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

eval_framework.tasks.benchmarks.mmlu module

class eval_framework.tasks.benchmarks.mmlu.FullTextMMLU(num_fewshot=0)

Bases: MMLU

MMLU dataset but where the model is expected to replicate choice text, rather than just the key.

Parameters:

num_fewshot (int)

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.bits_per_byte.BitsPerByteLoglikelihood'>]

NAME: str = 'Full Text MMLU'

class eval_framework.tasks.benchmarks.mmlu.MMLU(num_fewshot=0)

Bases: BaseTask[str]

MMLU dataset: https://huggingface.co/datasets/cais/mmlu

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'cais/mmlu'

FEWSHOT_SPLIT: str = 'dev'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.bits_per_byte.BitsPerByteLoglikelihood'>]

NAME: str = 'MMLU'

RESPONSE_TYPE = 'loglikelihoods'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'test'

SUBJECTS: list[SubjectType] = ['abstract_algebra', 'anatomy', 'astronomy', 'business_ethics', 'clinical_knowledge', 'college_biology', 'college_chemistry', 'college_computer_science', 'college_mathematics', 'college_medicine', 'college_physics', 'computer_security', 'conceptual_physics', 'econometrics', 'electrical_engineering', 'elementary_mathematics', 'formal_logic', 'global_facts', 'high_school_biology', 'high_school_chemistry', 'high_school_computer_science', 'high_school_european_history', 'high_school_geography', 'high_school_government_and_politics', 'high_school_macroeconomics', 'high_school_mathematics', 'high_school_microeconomics', 'high_school_physics', 'high_school_psychology', 'high_school_statistics', 'high_school_us_history', 'high_school_world_history', 'human_aging', 'human_sexuality', 'international_law', 'jurisprudence', 'logical_fallacies', 'machine_learning', 'management', 'marketing', 'medical_genetics', 'miscellaneous', 'moral_disputes', 'moral_scenarios', 'nutrition', 'philosophy', 'prehistory', 'professional_accounting', 'professional_law', 'professional_medicine', 'professional_psychology', 'public_relations', 'security_studies', 'sociology', 'us_foreign_policy', 'virology', 'world_religions']

class eval_framework.tasks.benchmarks.mmlu.MMLU_COT(num_fewshot=0)

Bases: MMLU

MMLU dataset with instruction to summarize reasoning and conclude with answer. Inspired by https://arxiv.org/pdf/2411.15124 (Table 44)

Parameters:

num_fewshot (int)

ANS_RE = re.compile('Therefore, the answer is: ([ABCD])')

METRICS = [<class 'eval_framework.metrics.completion.accuracy_completion.AccuracyCompletion'>]

NAME: str = 'MMLU_COT'

RESPONSE_TYPE = 'completion'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

post_process_generated_completion(completion_text, sample=None)

Return type:

str

Parameters:

• completion_text (str)

• sample (Sample | None)

stop_sequences: list[str]

class eval_framework.tasks.benchmarks.mmlu.MMLU_IDK(num_fewshot=0)

Bases: MMLU

Parameters:

num_fewshot (int)

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.confidence_weighted_accuracy.ConfidenceWeightedAccuracy'>, <class 'eval_framework.metrics.loglikelihood.dcs.DistributionalCorrectnessScore'>, <class 'eval_framework.metrics.loglikelihood.ternary.TernaryScore'>]

NAME: str = 'MMLU_IDK'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.mmlu.MMLU_OLMES(num_fewshot=0)

Bases: MMLU

MMLU with OLMES-style prompt: space before each label in the prompt (” A.”, “ B.”, …).

Parameters:

num_fewshot (int)

NAME: str = 'MMLU_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

eval_framework.tasks.benchmarks.mmlu_pro module

class eval_framework.tasks.benchmarks.mmlu_pro.MMLU_PRO(num_fewshot=0)

Bases: BaseTask[str]

MMLU_PRO dataset: https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'TIGER-Lab/MMLU-Pro'

FEWSHOT_SPLIT: str = 'test'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>]

NAME: str = 'MMLU Pro'

RESPONSE_TYPE = 'loglikelihoods'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'test'

SUBJECTS: list[SubjectType] = ['engineering', 'physics', 'psychology', 'chemistry', 'biology', 'law', 'philosophy', 'computer science', 'other', 'economics', 'business', 'history', 'math', 'health']

class eval_framework.tasks.benchmarks.mmlu_pro.MMLU_PRO_COT(num_fewshot=0)

Bases: MMLU_PRO

Parameters:

num_fewshot (int)

ANS_RE = re.compile('Therefore, the answer is \\(([ABCDEFGHIJ])\\)')

METRICS = [<class 'eval_framework.metrics.completion.accuracy_completion.AccuracyCompletion'>]

NAME: str = 'MMLU_PRO_COT'

RESPONSE_TYPE = 'completion'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

post_process_generated_completion(completion_text, sample=None)

Return type:

str

Parameters:

• completion_text (str)

• sample (Sample | None)

stop_sequences: list[str]

class eval_framework.tasks.benchmarks.mmlu_pro.MMLU_PRO_IDK(num_fewshot=0)

Bases: MMLU_PRO

Parameters:

num_fewshot (int)

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.confidence_weighted_accuracy.ConfidenceWeightedAccuracy'>, <class 'eval_framework.metrics.loglikelihood.dcs.DistributionalCorrectnessScore'>, <class 'eval_framework.metrics.loglikelihood.ternary.TernaryScore'>]

NAME: str = 'MMLU Pro_IDK'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.mmlu_pro.MMLU_PRO_OLMES(num_fewshot=0)

Bases: MMLU_PRO

MMLU Pro with OLMES-style prompt: space before each label in the prompt (” A.”, “ B.”, …).

Parameters:

num_fewshot (int)

NAME: str = 'MMLU Pro_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

eval_framework.tasks.benchmarks.multipl_e module

MultiPL-E: translations of HumanEval and MBPP into 6 programming languages.

Corresponds to the following OLMES task suites:

multipl_e_humaneval:6lang::olmo3:n32:v2 (one task per language):

multipl_e_humaneval:{cpp,java,js,php,rs,sh}::olmo3:n32:v2

multipl_e_mbpp:6lang::olmo3:n32:v2 (one task per language):

multipl_e_mbpp:{cpp,java,js,php,rs,sh}::olmo3:n32:v2

Recommended EvalConfig settings for full OLMES replication:

repeats: 32 llm_args: {sampling_params: {temperature: 0.6, top_p: 0.6}} max_tokens: 1024 fewshot: 0

Paper: https://ieeexplore.ieee.org/abstract/document/10103177

class eval_framework.tasks.benchmarks.multipl_e.MultiPLEHumanEvalCpp(num_fewshot=0)

Bases: _BaseMPLEHumanEval

MultiPL-E HumanEval in C++ — OLMES variant (nuprl/MultiPL-E, humaneval-cpp, test split).

Corresponds to multipl_e_humaneval:cpp::olmo3:n32:v2 in oe_eval. Recommended: 0-shot, temp=0.6, top_p=0.6, repeats=32.

Parameters:

num_fewshot (int)

MULTIPL_E_LANGUAGE: str = 'cpp'

NAME: str = 'MultiPL-E HumanEval C++ OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.multipl_e.MultiPLEHumanEvalJava(num_fewshot=0)

Bases: _BaseMPLEHumanEval

MultiPL-E HumanEval in Java — OLMES variant (nuprl/MultiPL-E, humaneval-java, test split).

Corresponds to multipl_e_humaneval:java::olmo3:n32:v2 in oe_eval. Recommended: 0-shot, temp=0.6, top_p=0.6, repeats=32.

Parameters:

num_fewshot (int)

MULTIPL_E_LANGUAGE: str = 'java'

NAME: str = 'MultiPL-E HumanEval Java OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.multipl_e.MultiPLEHumanEvalJs(num_fewshot=0)

Bases: _BaseMPLEHumanEval

MultiPL-E HumanEval in JavaScript — OLMES variant (nuprl/MultiPL-E, humaneval-js, test split).

Corresponds to multipl_e_humaneval:js::olmo3:n32:v2 in oe_eval. Recommended: 0-shot, temp=0.6, top_p=0.6, repeats=32.

Parameters:

num_fewshot (int)

MULTIPL_E_LANGUAGE: str = 'js'

NAME: str = 'MultiPL-E HumanEval JS OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.multipl_e.MultiPLEHumanEvalPhp(num_fewshot=0)

Bases: _BaseMPLEHumanEval

MultiPL-E HumanEval in PHP — OLMES variant (nuprl/MultiPL-E, humaneval-php, test split).

Corresponds to multipl_e_humaneval:php::olmo3:n32:v2 in oe_eval. Recommended: 0-shot, temp=0.6, top_p=0.6, repeats=32.

Parameters:

num_fewshot (int)

MULTIPL_E_LANGUAGE: str = 'php'

NAME: str = 'MultiPL-E HumanEval PHP OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.multipl_e.MultiPLEHumanEvalRs(num_fewshot=0)

Bases: _BaseMPLEHumanEval

MultiPL-E HumanEval in Rust — OLMES variant (nuprl/MultiPL-E, humaneval-rs, test split).

Corresponds to multipl_e_humaneval:rs::olmo3:n32:v2 in oe_eval. Recommended: 0-shot, temp=0.6, top_p=0.6, repeats=32.

Parameters:

num_fewshot (int)

MULTIPL_E_LANGUAGE: str = 'rs'

NAME: str = 'MultiPL-E HumanEval Rust OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.multipl_e.MultiPLEHumanEvalSh(num_fewshot=0)

Bases: _BaseMPLEHumanEval

MultiPL-E HumanEval in Bash — OLMES variant (nuprl/MultiPL-E, humaneval-sh, test split).

Corresponds to multipl_e_humaneval:sh::olmo3:n32:v2 in oe_eval. Recommended: 0-shot, temp=0.6, top_p=0.6, repeats=32.

Parameters:

num_fewshot (int)

MULTIPL_E_LANGUAGE: str = 'sh'

NAME: str = 'MultiPL-E HumanEval Bash OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.multipl_e.MultiPLEMBPPCpp(num_fewshot=0)

Bases: _BaseMPLEMBPP

MultiPL-E MBPP in C++ — OLMES variant (nuprl/MultiPL-E, mbpp-cpp, test split).

Corresponds to multipl_e_mbpp:cpp::olmo3:n32:v2 in oe_eval. Recommended: 0-shot, temp=0.6, top_p=0.6, repeats=32.

Parameters:

num_fewshot (int)

MULTIPL_E_LANGUAGE: str = 'cpp'

NAME: str = 'MultiPL-E MBPP C++ OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.multipl_e.MultiPLEMBPPJava(num_fewshot=0)

Bases: _BaseMPLEMBPP

MultiPL-E MBPP in Java — OLMES variant (nuprl/MultiPL-E, mbpp-java, test split).

Corresponds to multipl_e_mbpp:java::olmo3:n32:v2 in oe_eval. Recommended: 0-shot, temp=0.6, top_p=0.6, repeats=32.

Parameters:

num_fewshot (int)

MULTIPL_E_LANGUAGE: str = 'java'

NAME: str = 'MultiPL-E MBPP Java OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.multipl_e.MultiPLEMBPPJs(num_fewshot=0)

Bases: _BaseMPLEMBPP

MultiPL-E MBPP in JavaScript — OLMES variant (nuprl/MultiPL-E, mbpp-js, test split).

Corresponds to multipl_e_mbpp:js::olmo3:n32:v2 in oe_eval. Recommended: 0-shot, temp=0.6, top_p=0.6, repeats=32.

Parameters:

num_fewshot (int)

MULTIPL_E_LANGUAGE: str = 'js'

NAME: str = 'MultiPL-E MBPP JS OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.multipl_e.MultiPLEMBPPPhp(num_fewshot=0)

Bases: _BaseMPLEMBPP

MultiPL-E MBPP in PHP — OLMES variant (nuprl/MultiPL-E, mbpp-php, test split).

Corresponds to multipl_e_mbpp:php::olmo3:n32:v2 in oe_eval. Recommended: 0-shot, temp=0.6, top_p=0.6, repeats=32.

Parameters:

num_fewshot (int)

MULTIPL_E_LANGUAGE: str = 'php'

NAME: str = 'MultiPL-E MBPP PHP OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.multipl_e.MultiPLEMBPPRs(num_fewshot=0)

Bases: _BaseMPLEMBPP

MultiPL-E MBPP in Rust — OLMES variant (nuprl/MultiPL-E, mbpp-rs, test split).

Corresponds to multipl_e_mbpp:rs::olmo3:n32:v2 in oe_eval. Recommended: 0-shot, temp=0.6, top_p=0.6, repeats=32.

Parameters:

num_fewshot (int)

MULTIPL_E_LANGUAGE: str = 'rs'

NAME: str = 'MultiPL-E MBPP Rust OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.multipl_e.MultiPLEMBPPSh(num_fewshot=0)

Bases: _BaseMPLEMBPP

MultiPL-E MBPP in Bash — OLMES variant (nuprl/MultiPL-E, mbpp-sh, test split).

Corresponds to multipl_e_mbpp:sh::olmo3:n32:v2 in oe_eval. Recommended: 0-shot, temp=0.6, top_p=0.6, repeats=32.

Parameters:

num_fewshot (int)

MULTIPL_E_LANGUAGE: str = 'sh'

NAME: str = 'MultiPL-E MBPP Bash OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

eval_framework.tasks.benchmarks.naturalqs_open module

class eval_framework.tasks.benchmarks.naturalqs_open.NaturalQsOpen(num_fewshot=0)

Bases: BaseTask[str]

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'google-research-datasets/nq_open'

FEWSHOT_SPLIT: str = 'train'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.completion.drop_completion.DropF1ExactMatch'>]

NAME: str = 'NaturalQsOpen'

RESPONSE_TYPE = 'completion'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'validation'

SUBJECTS: list[SubjectType] = ['no_subject']

class eval_framework.tasks.benchmarks.naturalqs_open.NaturalQsOpenBPB(num_fewshot=0)

Bases: _NaturalQsOpenChoice_Base

BPB-only variant.

Parameters:

num_fewshot (int)

NAME: str = 'NaturalQsOpenBPB'

TASK_STYLER = <eval_framework.tasks.task_style.BPBStyle object>

class eval_framework.tasks.benchmarks.naturalqs_open.NaturalQsOpenCloze(num_fewshot=0)

Bases: _NaturalQsOpenChoice_Base

Parameters:

num_fewshot (int)

NAME: str = 'NaturalQsOpenCloze'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.ClozeStyle object>

class eval_framework.tasks.benchmarks.naturalqs_open.NaturalQsOpenMC(num_fewshot=0)

Bases: _NaturalQsOpenChoice_Base

Parameters:

num_fewshot (int)

NAME: str = 'NaturalQsOpenMC'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.MCStyle object>

class eval_framework.tasks.benchmarks.naturalqs_open.NaturalQsOpenMC_OLMES(num_fewshot=0)

Bases: _NaturalQsOpenChoice_Base

NaturalQsOpenMC with OLMES-style prompt: space before each label in the prompt (” A.”, “ B.”, …).

Parameters:

num_fewshot (int)

NAME: str = 'NaturalQsOpenMC_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.MCStyle object>

eval_framework.tasks.benchmarks.piqa module

class eval_framework.tasks.benchmarks.piqa.PIQA(num_fewshot=0)

Bases: BaseTask[str]

PIQA dataset: https://huggingface.co/datasets/ybisk/piqa

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'ybisk/piqa'

FEWSHOT_SPLIT: str = 'test'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.bits_per_byte.BitsPerByteLoglikelihood'>]

NAME: str = 'PIQA'

RESPONSE_TYPE = 'loglikelihoods'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/frozen-hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'validation'

SUBJECTS: list[SubjectType] = ['no_subject']

class eval_framework.tasks.benchmarks.piqa.PIQA_IDK(num_fewshot=0)

Bases: PIQA

Parameters:

num_fewshot (int)

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.confidence_weighted_accuracy.ConfidenceWeightedAccuracy'>, <class 'eval_framework.metrics.loglikelihood.dcs.DistributionalCorrectnessScore'>, <class 'eval_framework.metrics.loglikelihood.ternary.TernaryScore'>]

NAME: str = 'PIQA_IDK'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/frozen-hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.piqa.PIQA_OLMES(num_fewshot=0)

Bases: PIQA

PIQA with OLMES-style prompt: options shown with space-prefixed labels (” A.”, “ B.”); loglikelihood over “ A”/” B”.

Parameters:

num_fewshot (int)

FEWSHOT_SPLIT: str = 'train'

NAME: str = 'PIQA_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/frozen-hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'train'

eval_framework.tasks.benchmarks.piqa_ellamind module

German PIQA (EllaMind) tasks.

https://huggingface.co/datasets/ellamind/piqa-multilingual

PIQA supplies separate easy and hard distractors. Each base class uses a _DISTRACTOR_LEVEL class attribute ("easy" or "hard") that the registered subclass overrides.

class eval_framework.tasks.benchmarks.piqa_ellamind.PIQA_ELLAMIND_BPB_DE(num_fewshot=0)

Bases: _PIQA_ELLAMIND_DE_Base

German PIQA - BPB format (distractor set is irrelevant for BPB).

Parameters:

num_fewshot (int)

NAME: str = 'PIQA_ELLAMIND_BPB_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.BPBStyle object>

class eval_framework.tasks.benchmarks.piqa_ellamind.PIQA_ELLAMIND_CLOZE_EASY_DE(num_fewshot=0)

Bases: _PIQA_ELLAMIND_DE_Base

German PIQA - Cloze format with easy distractor.

Parameters:

num_fewshot (int)

NAME: str = 'PIQA_ELLAMIND_CLOZE_EASY_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.ClozeStyle object>

class eval_framework.tasks.benchmarks.piqa_ellamind.PIQA_ELLAMIND_CLOZE_HARD_DE(num_fewshot=0)

Bases: _PIQA_ELLAMIND_DE_Base

German PIQA - Cloze format with hard distractor.

Parameters:

num_fewshot (int)

NAME: str = 'PIQA_ELLAMIND_CLOZE_HARD_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.ClozeStyle object>

class eval_framework.tasks.benchmarks.piqa_ellamind.PIQA_ELLAMIND_MC_EASY_DE(num_fewshot=0)

Bases: _PIQA_ELLAMIND_DE_Base

German PIQA - MC format with easy distractor.

Parameters:

num_fewshot (int)

NAME: str = 'PIQA_ELLAMIND_MC_EASY_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.MCStyle object>

class eval_framework.tasks.benchmarks.piqa_ellamind.PIQA_ELLAMIND_MC_HARD_DE(num_fewshot=0)

Bases: _PIQA_ELLAMIND_DE_Base

German PIQA - MC format with hard distractor.

Parameters:

num_fewshot (int)

NAME: str = 'PIQA_ELLAMIND_MC_HARD_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.MCStyle object>

eval_framework.tasks.benchmarks.sciq module

class eval_framework.tasks.benchmarks.sciq.SCIQ(num_fewshot=0)

Bases: BaseTask[str]

SciQ dataset: https://huggingface.co/datasets/allenai/sciq

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'allenai/sciq'

FEWSHOT_SPLIT: str = 'test'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.bits_per_byte.BitsPerByteLoglikelihood'>]

NAME: str = 'SciQ'

RESPONSE_TYPE = 'loglikelihoods'

SAMPLE_SPLIT: str = 'validation'

SUBJECTS: list[SubjectType] = ['no_subject']

class eval_framework.tasks.benchmarks.sciq.SCIQEvalHarness(num_fewshot=0)

Bases: SCIQ

Based on https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/sciq/sciq.yaml#L8 In the Eval Harness implementation, the instruction text includes a context passage. This passage often contains the answer, reducing the benchmark to a straightforward copy-and-paste task.

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'allenai/sciq'

FEWSHOT_SPLIT: str = 'test'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>]

NAME: str = 'SciQ Eval Harness'

RESPONSE_TYPE = 'loglikelihoods'

SAMPLE_SPLIT: str = 'validation'

SUBJECTS: list[SubjectType] = ['no_subject']

class eval_framework.tasks.benchmarks.sciq.SCIQEvalHarness_IDK(num_fewshot=0)

Bases: SCIQEvalHarness

Parameters:

num_fewshot (int)

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.confidence_weighted_accuracy.ConfidenceWeightedAccuracy'>, <class 'eval_framework.metrics.loglikelihood.dcs.DistributionalCorrectnessScore'>, <class 'eval_framework.metrics.loglikelihood.ternary.TernaryScore'>]

NAME: str = 'SciQ Eval Harness_IDK'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.sciq.SCIQ_IDK(num_fewshot=0)

Bases: SCIQ

Parameters:

num_fewshot (int)

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.confidence_weighted_accuracy.ConfidenceWeightedAccuracy'>, <class 'eval_framework.metrics.loglikelihood.dcs.DistributionalCorrectnessScore'>, <class 'eval_framework.metrics.loglikelihood.ternary.TernaryScore'>]

NAME: str = 'SciQ_IDK'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.sciq.SCIQ_OLMES(num_fewshot=0)

Bases: SCIQ

SciQ with OLMES-style prompt: options shown with space-prefixed labels (” A.”, “ B.”, “ C.”, “ D.”); loglikelihood over “ A”/” B”/” C”/” D”. Answer choices are deterministically shuffled per example.

Parameters:

num_fewshot (int)

FEWSHOT_SPLIT: str = 'train'

NAME: str = 'SciQ_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'train'

eval_framework.tasks.benchmarks.simpleqa_ellamind module

German SimpleQA (verified, EllaMind) tasks.

https://huggingface.co/datasets/ellamind/simpleqa-verified-multilingual

SimpleQA supplies separate easy and hard distractors. Each base class uses a _DISTRACTOR_LEVEL class attribute ("easy" or "hard") that the registered subclass overrides.

class eval_framework.tasks.benchmarks.simpleqa_ellamind.SIMPLEQA_ELLAMIND_BPB_DE(num_fewshot=0)

Bases: SIMPLEQA_ELLAMIND_CLOZE_EASY_DE

German SimpleQA - BPB format (distractor set is irrelevant for BPB).

Parameters:

num_fewshot (int)

NAME: str = 'SIMPLEQA_ELLAMIND_BPB_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.BPBStyle object>

class eval_framework.tasks.benchmarks.simpleqa_ellamind.SIMPLEQA_ELLAMIND_CLOZE_EASY_DE(num_fewshot=0)

Bases: _SIMPLEQA_ELLAMIND_DE_Base

German SimpleQA - Cloze format with easy distractors.

Parameters:

num_fewshot (int)

NAME: str = 'SIMPLEQA_ELLAMIND_CLOZE_EASY_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.ClozeStyle object>

class eval_framework.tasks.benchmarks.simpleqa_ellamind.SIMPLEQA_ELLAMIND_CLOZE_HARD_DE(num_fewshot=0)

Bases: _SIMPLEQA_ELLAMIND_DE_Base

German SimpleQA - Cloze format with hard distractors.

Parameters:

num_fewshot (int)

NAME: str = 'SIMPLEQA_ELLAMIND_CLOZE_HARD_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.ClozeStyle object>

class eval_framework.tasks.benchmarks.simpleqa_ellamind.SIMPLEQA_ELLAMIND_MC_EASY_DE(num_fewshot=0)

Bases: _SIMPLEQA_ELLAMIND_DE_Base

German SimpleQA - MC format with easy distractors.

Parameters:

num_fewshot (int)

NAME: str = 'SIMPLEQA_ELLAMIND_MC_EASY_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.MCStyle object>

class eval_framework.tasks.benchmarks.simpleqa_ellamind.SIMPLEQA_ELLAMIND_MC_HARD_DE(num_fewshot=0)

Bases: _SIMPLEQA_ELLAMIND_DE_Base

German SimpleQA - MC format with hard distractors.

Parameters:

num_fewshot (int)

NAME: str = 'SIMPLEQA_ELLAMIND_MC_HARD_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.MCStyle object>

eval_framework.tasks.benchmarks.siqa_ellamind module

German Social IQa (EllaMind) tasks.

https://huggingface.co/datasets/ellamind/siqa-multilingual

SIQA supplies separate easy and hard distractors. Each base class uses a _DISTRACTOR_LEVEL class attribute ("easy" or "hard") that the registered subclass overrides.

class eval_framework.tasks.benchmarks.siqa_ellamind.SIQA_ELLAMIND_BPB_DE(num_fewshot=0)

Bases: SIQA_ELLAMIND_CLOZE_EASY_DE

German SIQA - BPB format (distractor set is irrelevant for BPB).

Parameters:

num_fewshot (int)

NAME: str = 'SIQA_ELLAMIND_BPB_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.BPBStyle object>

class eval_framework.tasks.benchmarks.siqa_ellamind.SIQA_ELLAMIND_CLOZE_EASY_DE(num_fewshot=0)

Bases: _SIQA_ELLAMIND_DE_Base

German SIQA - Cloze format with easy distractors.

Parameters:

num_fewshot (int)

NAME: str = 'SIQA_ELLAMIND_CLOZE_EASY_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.ClozeStyle object>

class eval_framework.tasks.benchmarks.siqa_ellamind.SIQA_ELLAMIND_CLOZE_HARD_DE(num_fewshot=0)

Bases: _SIQA_ELLAMIND_DE_Base

German SIQA - Cloze format with hard distractors.

Parameters:

num_fewshot (int)

NAME: str = 'SIQA_ELLAMIND_CLOZE_HARD_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.ClozeStyle object>

class eval_framework.tasks.benchmarks.siqa_ellamind.SIQA_ELLAMIND_MC_EASY_DE(num_fewshot=0)

Bases: _SIQA_ELLAMIND_DE_Base

German SIQA - MC format with easy distractors.

Parameters:

num_fewshot (int)

NAME: str = 'SIQA_ELLAMIND_MC_EASY_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.MCStyle object>

class eval_framework.tasks.benchmarks.siqa_ellamind.SIQA_ELLAMIND_MC_HARD_DE(num_fewshot=0)

Bases: _SIQA_ELLAMIND_DE_Base

German SIQA - MC format with hard distractors.

Parameters:

num_fewshot (int)

NAME: str = 'SIQA_ELLAMIND_MC_HARD_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.MCStyle object>

eval_framework.tasks.benchmarks.social_iqa module

Social IQA: Commonsense reasoning about social interactions.

Dataset: allenai/social_i_qa (context, question, answerA/B/C, label 1-indexed).

class eval_framework.tasks.benchmarks.social_iqa.SocialIQACloze(num_fewshot=0)

Bases: BaseTask[str]

Social IQA cloze: loglikelihood over full answer text.

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'allenai/social_i_qa'

FEWSHOT_SPLIT: str = 'train'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.bits_per_byte.BitsPerByteLoglikelihood'>]

NAME: str = 'SocialIQACloze'

RESPONSE_TYPE = 'loglikelihoods'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'validation'

SUBJECTS: list[SubjectType] = ['no_subject']

class eval_framework.tasks.benchmarks.social_iqa.SocialIQAMC(num_fewshot=0)

Bases: SocialIQAMC_OLMES

Social IQA multiple choice: loglikelihood over “ A”/” B”/” C”. Labels in prompt have no leading space (“A.”, “B.”, “C.”); possible completions use a prefixed space (” A”, “ B”, “ C”) for tokenization consistency.

Parameters:

num_fewshot (int)

NAME: str = 'SocialIQAMC'

SAMPLE_SPLIT: str = 'validation'

class eval_framework.tasks.benchmarks.social_iqa.SocialIQAMC_OLMES(num_fewshot=0)

Bases: SocialIQACloze

Social IQA multiple choice (OLMES/oe_eval style): loglikelihood over “ A”/” B”/” C”. Uses space-prefixed labels in prompt (” A.”, “ B.”, “ C.”) for tokenization parity with oe_eval.

Parameters:

num_fewshot (int)

NAME: str = 'SocialIQAMC_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'train'

eval_framework.tasks.benchmarks.squad module

class eval_framework.tasks.benchmarks.squad.SQUAD(num_fewshot=0)

Bases: SQUAD2

Squad dataset: https://huggingface.co/datasets/rajpurkar/squad

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'rajpurkar/squad'

NAME: str = 'SQuAD'

class eval_framework.tasks.benchmarks.squad.SQUAD2(num_fewshot=0)

Bases: BaseTask[str]

Squad v2 dataset: https://huggingface.co/datasets/rajpurkar/squad_v2

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'rajpurkar/squad_v2'

FEWSHOT_SPLIT: str = 'train'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.completion.accuracy_completion.AccuracyCompletion'>, <class 'eval_framework.metrics.completion.f1.F1'>]

NAME: str = 'SQuAD2'

RESPONSE_TYPE = 'completion'

SAMPLE_SPLIT: str = 'validation'

SUBJECTS: list[SubjectType] = ['no_subject']

UNANSWERABLE_STR = 'unanswerable'

class eval_framework.tasks.benchmarks.squad.SQUAD2BPB(num_fewshot=0)

Bases: SQUAD2

SQuAD2 variant that scores loglikelihood of the gold answer text. Reports bits-per-byte on the reference answer (first acceptable answer).

Parameters:

num_fewshot (int)

METRICS = [<class 'eval_framework.metrics.loglikelihood.bits_per_byte.BitsPerByteLoglikelihood'>]

NAME: str = 'SQuAD2 BPB'

RESPONSE_TYPE = 'loglikelihoods'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

class eval_framework.tasks.benchmarks.squad.SQuAD2_MA(num_fewshot=0)

Bases: SQUAD2

SQuAD v2 with the exact system prompt used in MA training

Parameters:

num_fewshot (int)

METRICS = [<class 'eval_framework.metrics.completion.accuracy_completion.AccuracyCompletion'>, <class 'eval_framework.metrics.completion.f1.F1'>, <class 'eval_framework.metrics.completion.f1.F1SquadNormalized'>]

NAME: str = 'SQuAD2_MA'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

UNANSWERABLE_STR = 'unanswerable'

post_process_generated_completion(completion_text, sample=None)

Clean up the generated answer.

Return type:

str

Parameters:

• completion_text (str)

• sample (Sample | None)

class eval_framework.tasks.benchmarks.squad.SQuAD2_MA_NO_SYSPROMPT(num_fewshot=0)

Bases: SQuAD2_MA

Parameters:

num_fewshot (int)

class eval_framework.tasks.benchmarks.squad.SQuAD_OLMES(num_fewshot=0)

Bases: SQUAD

SQuAD variant matching OLMES implementation.

Parameters:

num_fewshot (int)

FEWSHOT_SPLIT: str = 'train'

METRICS = [<class 'eval_framework.metrics.completion.f1.F1SquadNormalized'>]

NAME: str = 'SQuAD_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'validation'

eval_framework.tasks.benchmarks.winogrande module

class eval_framework.tasks.benchmarks.winogrande.WINOGRANDE(num_fewshot=0)

Bases: BaseTask[str]

WINOGRANDE dataset: https://huggingface.co/datasets/allenai/winogrande

Parameters:

num_fewshot (int)

DATASET_PATH: str = 'allenai/winogrande'

FEWSHOT_SPLIT: str = 'train'

LANGUAGE: Language | dict[str, Language] | dict[str, tuple[Language, Language]] | None = 'English'

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.bits_per_byte.BitsPerByteLoglikelihood'>]

NAME: str = 'Winogrande'

RESPONSE_TYPE = 'loglikelihoods'

SAMPLE_SPLIT: str = 'validation'

SUBJECTS: list[SubjectType] = ['winogrande_xl']

class eval_framework.tasks.benchmarks.winogrande.WINOGRANDECloze(num_fewshot=0)

Bases: WINOGRANDE

Winogrande with Cloze-style prompt.

The big change from usual tasks is that Winogrande uses a ‘partial evaluation’, meaning for a sentence like “The _ is small” and options “garage” and “backyard”, it wants to evaluate p(suffix | prefix + option), e.g. p(“is small” | “The garage”) and p(“is small” | “The backyard”). This is conceptually different from usual tasks, where we evaluate p(options + suffix | prefix), i.e. p(“garage is small” | “The”) and p(“backyard is small” | “The”).

Mathematically, they differ by p(option + suffix | prefix) = p(option | prefix) * p(suffix | prefix + option) and thus can differ significantly, if the options have different base probabilities.

In the implementation, this creates the following issue: Normally, we have the same prompt for all options, and then check different completions. However, with the correct format, we would have different prompts for different options, and then check the same completion.

In this implementation, we fix this by having _create_samples create two samples per dataset item, and then use a custom metric (PartialEvalAccuracy) that pairs them up and compares them.

Parameters:

num_fewshot (int)

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.PartialEvalAccuracy'>]

NAME: str = 'WinograndeCloze'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

SAMPLE_SPLIT: str = 'train'

class eval_framework.tasks.benchmarks.winogrande.WINOGRANDE_IDK(num_fewshot=0)

Bases: WINOGRANDE

Parameters:

num_fewshot (int)

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.confidence_weighted_accuracy.ConfidenceWeightedAccuracy'>, <class 'eval_framework.metrics.loglikelihood.dcs.DistributionalCorrectnessScore'>, <class 'eval_framework.metrics.loglikelihood.ternary.TernaryScore'>]

NAME: str = 'Winogrande_IDK'

class eval_framework.tasks.benchmarks.winogrande.WINOGRANDE_OLMES(num_fewshot=0)

Bases: WINOGRANDE

Winogrande with OLMES-style prompt: options shown with space-prefixed labels (” A.”, “ B.”); loglikelihood over “ A”/” B”.

Parameters:

num_fewshot (int)

NAME: str = 'Winogrande_OLMES'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

eval_framework.tasks.benchmarks.winogrande_ellamind module

German Winogrande (EllaMind) tasks.

https://huggingface.co/datasets/ellamind/winogrande-multilingual

The sentence contains a blank _ to be filled by option1 or option2. All three styles (Cloze, MC, PartialEval) share this extraction logic.

class eval_framework.tasks.benchmarks.winogrande_ellamind.WINOGRANDE_ELLAMIND_CLOZE_DE(num_fewshot=0)

Bases: _WINOGRANDE_ELLAMIND_DE_Base

German Winogrande - Cloze format (like WINOGRANDE).

Prompt is the sentence prefix; model is scored over the full “option + suffix” strings as completions.

Hypothetical example:

Prompt: “George was taller than John, therefore” Completions: [” George could see more.”, “ John could see more.”]

Parameters:

num_fewshot (int)

NAME: str = 'WINOGRANDE_ELLAMIND_CLOZE_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.ClozeStyle object>

class eval_framework.tasks.benchmarks.winogrande_ellamind.WINOGRANDE_ELLAMIND_MC_DE(num_fewshot=0)

Bases: _WINOGRANDE_ELLAMIND_DE_Base

German Winogrande - MC format (like WINOGRANDE_OLMES).

Both options are listed in the prompt with space-prefixed letter labels (” A.”, “ B.”); model is scored over “ A” / “ B”.

Hypothetical example:

Prompt: “George was taller than John, thereforen A. George could see more.n B. John could see more.n” Completions: [” A”, “ B”]

Parameters:

num_fewshot (int)

NAME: str = 'WINOGRANDE_ELLAMIND_MC_DE'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

TASK_STYLER = <eval_framework.tasks.task_style.MCStyle object>

class eval_framework.tasks.benchmarks.winogrande_ellamind.WINOGRANDE_ELLAMIND_PARTIAL_EVAL_DE(num_fewshot=0)

Bases: _WINOGRANDE_ELLAMIND_DE_Base

German Winogrande - PartialEval cloze format (like WINOGRANDECloze).

Evaluates p(suffix | prefix + option) rather than p(option + suffix | prefix).

Two samples are created per dataset item - one per option - each with prompt “prefix + option” and the shared suffix as the single possible completion. Uses PartialEvalAccuracy, which pairs the two samples and compares scores.

Hypothetical example (two samples for one item):

Prompt 1: “George was taller than John, therefore George” completion: [” could see more.”] Prompt 2: “George was taller than John, therefore John” completion: [” could see more.”]

Parameters:

num_fewshot (int)

METRICS = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.PartialEvalAccuracy'>]

NAME: str = 'WINOGRANDE_ELLAMIND_PARTIAL_EVAL_DE'

RESPONSE_TYPE = 'loglikelihoods'

REVISION_LOCKFILE: Path | None = PosixPath('/home/runner/work/eval-framework/eval-framework/src/eval_framework/tasks/hf-dataset-revisions.json')

Module contents