eval_framework.tasks package¶
Submodules¶
eval_framework.tasks.base module¶
- eval_framework.tasks.base.resolve_overwrite_subjects(custom_subjects, accepted_subjects, task_name)[source]¶
Restrict accepted_subjects to the ones requested via –task-subjects.
A subject matches a token when their parts agree position by position, where “*” matches any one part – so “DE_DE,*” selects every German subject and “*” selects every scalar subject.
- Return type:
list[TypeVar(SubjectType)]- Parameters:
custom_subjects (list[str])
accepted_subjects (list[SubjectType])
task_name (str)
eval_framework.tasks.dataset_loading module¶
Fetching a benchmark’s dataset, kept out of the composed classes so they stay free of Hugging Face mechanics (revisions, cache directories, download config).
- class eval_framework.tasks.dataset_loading.DatasetLoader[source]¶
Bases:
ABCLoads a benchmark’s dataset splits for one subject.
- class eval_framework.tasks.dataset_loading.DatasetPolicy[source]¶
Bases:
ABCProduces the loader for a benchmark’s dataset, and documents where that dataset comes from.
- abstractmethod documentation()[source]¶
Markdown for the task’s
## Datasetdoc section, describing where the dataset comes from.- Return type:
str
- abstractmethod loader(custom_hf_revision)[source]¶
- Return type:
- Parameters:
custom_hf_revision (str | None)
- subject_encoded_in_column(config, column)[source]¶
By default the loaded config names the subject; call this when the subject is instead encoded in a
columnof a singleconfig(Noneloads the default config) (seeSubjectColumn).- Return type:
- Parameters:
config (str | None)
column (str)
- subset(keep, description=None)[source]¶
Restrict this policy’s dataset to the rows for which
keepreturns true (seeSubset).descriptionnames the resulting subset for the rendered dataset docs (e.g. “the diamond subset”).- Return type:
- Parameters:
keep (Callable[[dict[str, Any]], bool])
description (str | None)
- final class eval_framework.tasks.dataset_loading.FixedHfConfigLoader(dataset_path, revision, hf_config)[source]¶
Bases:
DatasetLoaderLoads a fixed HF config, ignoring the requested subject — for a dataset whose subjects are labels rather than configs (or a task with no subjects but a non-default config).
- Parameters:
dataset_path (str)
revision (str | None)
hf_config (str | None)
- final class eval_framework.tasks.dataset_loading.HfDatasetLoader(dataset_path, revision)[source]¶
Bases:
DatasetLoaderLoads one Hugging Face dataset pinned to
revision, treating the requested subject as the HF config.- Parameters:
dataset_path (str)
revision (str | None)
- final class eval_framework.tasks.dataset_loading.SubjectColumn(inner, config, column)[source]¶
Bases:
DatasetPolicyUse this in case the raw dataset encodes the subject in a column, rather than the load configuration.
- Parameters:
inner (DatasetPolicy)
config (str | None)
column (str)
- documentation()[source]¶
Markdown for the task’s
## Datasetdoc section, describing where the dataset comes from.- Return type:
str
- final class eval_framework.tasks.dataset_loading.Subset(inner, keep, description=None)[source]¶
Bases:
DatasetPolicyRestricts another policy’s dataset to the rows for which
keepreturns true, in every split.A benchmark whose items are a row-filtered subset of a larger dataset (e.g. GPQA’s diamond subset, HLE’s natively-multiple-choice subset) wraps the base policy in a
Subset.descriptionnames that subset for the rendered dataset docs; without it the docs only note that some rows are dropped.- Parameters:
inner (DatasetPolicy)
keep (Callable[[dict[str, Any]], bool])
description (str | None)
- documentation()[source]¶
Markdown for the task’s
## Datasetdoc section, describing where the dataset comes from.- Return type:
str
eval_framework.tasks.dataset_revisions module¶
Pinned Hugging Face dataset revisions.
A lock file maps dataset paths to pinned commit SHAs, so an eval runs against a reproducible
dataset revision. Benchmarks resolve their pin through a Pinned dataset policy (usually via
pinned_by_framework).
- final class eval_framework.tasks.dataset_revisions.FixedHfConfig(base, hf_config)[source]¶
Bases:
DatasetPolicyLoads a fixed HF config from an underlying
Pinnedpolicy, regardless of subject (here the subject is a label, not a config).- Parameters:
base (Pinned)
hf_config (str | None)
- documentation()[source]¶
Markdown for the task’s
## Datasetdoc section, describing where the dataset comes from.- Return type:
str
- class eval_framework.tasks.dataset_revisions.HfDatasetRevisions(revisions)[source]¶
Bases:
objectPinned revisions of Hugging Face datasets, mapping dataset path → commit SHA.
- Parameters:
revisions (dict[str, str])
- missing_references(api)[source]¶
The pins whose revisions are no longer reachable on the Hub, with the failure reason.
Intended to run on a CI schedule, so a yanked revision or a removed dataset is noticed before it breaks an eval run. An empty list means every pin is still reachable.
- Return type:
list[str]- Parameters:
api (HfApi)
- final class eval_framework.tasks.dataset_revisions.Pinned(lockfile, dataset_path)[source]¶
Bases:
DatasetPolicyPins
dataset_pathto the revision recorded for it inlockfileA CI job scans for newer available versions of these revisions, proposing PRs for updating them. This ensures the revision is up to date, while still protecting against silently changing the meaning of a benchmark.
- Parameters:
lockfile (Path)
dataset_path (str)
- documentation()[source]¶
Markdown for the task’s
## Datasetdoc section, describing where the dataset comes from.- Return type:
str
- eval_framework.tasks.dataset_revisions.pinned_by_framework(dataset_path)[source]¶
A
Pinnedpolicy bindingdataset_pathto the framework’s bundled lock file.- Return type:
- Parameters:
dataset_path (str)
- eval_framework.tasks.dataset_revisions.pinned_frozen(dataset_path)[source]¶
A
Pinnedpolicy bindingdataset_pathto the frozen lock file (revisions held fixed to keep results comparable across framework upgrades).- Return type:
- Parameters:
dataset_path (str)
- eval_framework.tasks.dataset_revisions.pinned_revision(lockfile, dataset_path)[source]¶
The commit SHA pinned for
dataset_pathinlockfile.Resolves the exact dataset revision an eval runs against, so results stay reproducible across dataset updates. Every dataset used by a task is expected to be pinned; a missing entry is a bug in the lock file and raises
KeyError.- Return type:
str- Parameters:
lockfile (Path)
dataset_path (str)
eval_framework.tasks.eval_config module¶
- class eval_framework.tasks.eval_config.EvalConfig(**data)[source]¶
Bases:
BaseConfig- Parameters:
output_dir (Path)
wandb_project (str | None)
wandb_entity (str | None)
wandb_run_id (str | None)
wandb_group (str | None)
wandb_upload_results (Annotated[bool, BeforeValidator(func=~eval_framework.tasks.eval_config.EvalConfig.<lambda>, json_schema_input_type=PydanticUndefined)])
hf_upload_dir (str | None)
hf_upload_repo (str | None)
num_fewshot (Annotated[int, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0)])])
num_samples (Annotated[int | None, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=1)])])
max_tokens (int | None)
task_name (str)
task_subjects (list[str] | None)
hf_revision (str | None)
llm_class (type[BaseLLM])
llm_args (dict[str, Any])
llm_judge_class (type[BaseLLM] | None)
judge_model_args (dict[str, Any])
randomize_judge_order (bool)
batch_size (Annotated[int, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=1)])])
description (str | None)
save_intermediate_results (Annotated[bool, BeforeValidator(func=~eval_framework.tasks.eval_config.EvalConfig.<lambda>, json_schema_input_type=PydanticUndefined)])
save_logs (Annotated[bool, BeforeValidator(func=~eval_framework.tasks.eval_config.EvalConfig.<lambda>, json_schema_input_type=PydanticUndefined)])
delete_output_dir_after_upload (Annotated[bool, BeforeValidator(func=~eval_framework.tasks.eval_config.EvalConfig.<lambda>, json_schema_input_type=PydanticUndefined)])
repeats (Annotated[int, BeforeValidator(func=~eval_framework.tasks.eval_config.EvalConfig.<lambda>, json_schema_input_type=PydanticUndefined), FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=1)])])
-
batch_size:
Annotated[int]¶
-
delete_output_dir_after_upload:
Annotated[bool]¶
-
description:
str|None¶
-
hf_revision:
str|None¶
-
hf_upload_dir:
str|None¶
-
hf_upload_repo:
str|None¶
-
judge_model_args:
dict[str,Any]¶
-
llm_args:
dict[str,Any]¶
-
max_tokens:
int|None¶
- model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'frozen': True, 'protected_namespaces': ()}¶
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
-
num_fewshot:
Annotated[int]¶
-
num_samples:
Annotated[int|None]¶
-
output_dir:
Path¶
-
randomize_judge_order:
bool¶
-
repeats:
Annotated[int]¶
-
save_intermediate_results:
Annotated[bool]¶
-
save_logs:
Annotated[bool]¶
- serialize_judge_model_args(value)[source]¶
- Return type:
dict[str,Any]- Parameters:
value (dict[str, Any])
- serialize_llm_class(value)[source]¶
Serialize the class into its fully qualified name.
- Return type:
str|None- Parameters:
value (type[BaseLLM] | None)
- serialize_llm_judge_class(value)[source]¶
Serialize the class into its fully qualified name.
- Return type:
str|None- Parameters:
value (type[BaseLLM] | None)
-
task_name:
str¶
-
task_subjects:
list[str] |None¶
- classmethod validate_judge_model_args(value)[source]¶
- Return type:
dict[str,Any]- Parameters:
value (dict[str, Any])
- classmethod validate_llm_args(value)[source]¶
- Return type:
dict[str,Any]- Parameters:
value (dict[str, Any])
-
wandb_entity:
str|None¶
-
wandb_group:
str|None¶
-
wandb_project:
str|None¶
-
wandb_run_id:
str|None¶
-
wandb_upload_results:
Annotated[bool]¶
eval_framework.tasks.lazy module¶
- class eval_framework.tasks.lazy.Lazy(id, load)[source]¶
Bases:
BenchmarkA
Benchmarkthat defers producing the real benchmark until first use.Holds an
idand aloadthunk returning the wrappedBenchmark; every other call is delegated to that benchmark, which is built (and memoized) on first access. This keepsLazyindependent of any concrete task type — the thunk owns whatever loading (e.g. importing a module) the wrapped benchmark requires.- Parameters:
id (str)
load (Callable[[], Benchmark])
- create(num_fewshot, custom_subjects, custom_hf_revision, seed=None)[source]¶
- Return type:
- Parameters:
num_fewshot (int)
custom_subjects (list[str] | None)
custom_hf_revision (str | None)
seed (int | None)
- display_name()[source]¶
Human-readable display name. Is allowed to have special characters and whitespaces.
- Return type:
str
- markdown_doc(formatters)[source]¶
Render the benchmarks’s documentation as markdown.
- Return type:
str- Parameters:
formatters (Sequence[BaseFormatter])
- metrics()[source]¶
The benchmark’s metrics
- Return type:
list[type[BaseMetric]]
- subjects(tokens=None)[source]¶
Labels of the subjects a run with these
--task-subjectstokens evaluates (no tokens = all), exactly as they appear in samples and aggregated-result keys. RaisesValueErrorfor a token no subject matches, mirroring whatcreatedoes with the same tokens.- Return type:
list[str]- Parameters:
tokens (list[str] | None)
eval_framework.tasks.markdown_doc module¶
- eval_framework.tasks.markdown_doc.markdown_doc(*, name, dataset_doc, sample_split, fewshot_split, response_type, metrics, subjects, num_fewshot, formatters, example_messages, split_sizes, possible_completions, ground_truth)[source]¶
Render a task’s documentation as markdown
- Return type:
str- Parameters:
name (str)
dataset_doc (str)
sample_split (str | None)
fewshot_split (str | None)
response_type (str)
metrics (Sequence[str])
subjects (Any)
num_fewshot (int)
formatters (Sequence[BaseFormatter])
example_messages (list[Message] | None)
split_sizes (dict[str, int] | None)
possible_completions (str | list[str] | None)
ground_truth (str | list[str] | None)
eval_framework.tasks.registry module¶
- class eval_framework.tasks.registry.Benchmark[source]¶
Bases:
ABCA benchmark is used to provide means of measuring model performance in a domain.
Benchmark act as factories for Eval. They bind all the a prior known information and enrich it with the arguments provided at runtime to create concrete instances of Eval which are used to provide measurements of the models performance.
- abstractmethod create(num_fewshot, custom_subjects, custom_hf_revision, seed=None)[source]¶
- Return type:
- Parameters:
num_fewshot (int)
custom_subjects (list[str] | None)
custom_hf_revision (str | None)
seed (int | None)
- abstractmethod display_name()[source]¶
Human-readable display name. Is allowed to have special characters and whitespaces.
- Return type:
str
- abstractmethod markdown_doc(formatters)[source]¶
Render the benchmarks’s documentation as markdown.
- Return type:
str- Parameters:
formatters (Sequence[BaseFormatter])
- abstractmethod metrics()[source]¶
The benchmark’s metrics
- Return type:
list[type[BaseMetric]]
- abstractmethod subjects(tokens=None)[source]¶
Labels of the subjects a run with these
--task-subjectstokens evaluates (no tokens = all), exactly as they appear in samples and aggregated-result keys. RaisesValueErrorfor a token no subject matches, mirroring whatcreatedoes with the same tokens.- Return type:
list[str]- Parameters:
tokens (list[str] | None)
- class eval_framework.tasks.registry.Lazy(id, load)[source]¶
Bases:
BenchmarkA
Benchmarkthat defers producing the real benchmark until first use.Holds an
idand aloadthunk returning the wrappedBenchmark; every other call is delegated to that benchmark, which is built (and memoized) on first access. This keepsLazyindependent of any concrete task type — the thunk owns whatever loading (e.g. importing a module) the wrapped benchmark requires.- Parameters:
id (str)
load (Callable[[], Benchmark])
- create(num_fewshot, custom_subjects, custom_hf_revision, seed=None)[source]¶
- Return type:
- Parameters:
num_fewshot (int)
custom_subjects (list[str] | None)
custom_hf_revision (str | None)
seed (int | None)
- display_name()[source]¶
Human-readable display name. Is allowed to have special characters and whitespaces.
- Return type:
str
- markdown_doc(formatters)[source]¶
Render the benchmarks’s documentation as markdown.
- Return type:
str- Parameters:
formatters (Sequence[BaseFormatter])
- metrics()[source]¶
The benchmark’s metrics
- Return type:
list[type[BaseMetric]]
- subjects(tokens=None)[source]¶
Labels of the subjects a run with these
--task-subjectstokens evaluates (no tokens = all), exactly as they appear in samples and aggregated-result keys. RaisesValueErrorfor a token no subject matches, mirroring whatcreatedoes with the same tokens.- Return type:
list[str]- Parameters:
tokens (list[str] | None)
- class eval_framework.tasks.registry.Registry[source]¶
Bases:
objectA registry for Benchmarks
- add(factory)[source]¶
Register a factory under the key derived from its
id().- Return type:
None- Parameters:
factory (Benchmark)
- eval_framework.tasks.registry.is_registered(name, /)[source]¶
Return True if a task is registered.
- Return type:
bool- Parameters:
name (str)
- eval_framework.tasks.registry.registered_task_names()[source]¶
Return the names of all registered tasks.
- Return type:
list[str]
eval_framework.tasks.task_loader module¶
- eval_framework.tasks.task_loader.load_extra_tasks(directory, registry=None)[source]¶
Load and register user-defined tasks from a directory of plugin modules.
Every
.pyfile underdirectory(searched recursively) is imported as a module, and any module exposing aregister_tasks(registry)function has it invoked to register its tasks.- Return type:
None- Parameters:
directory (str | PathLike)
registry (Registry | None)
- eval_framework.tasks.task_loader.load_modules_from_directory(directory)[source]¶
Recursively import every
.pyfile underdirectoryas a module.Each file is imported under a unique module name derived from its path relative to
directoryand inserted intosys.modules.- Return type:
list[ModuleType]- Parameters:
directory (str | PathLike)
eval_framework.tasks.task_names module¶
- eval_framework.tasks.task_names.register_all_tasks(registry=None)[source]¶
Register all the benchmark tasks with the eval framework
Uses global registry by default.
- Return type:
None- Parameters:
registry (Registry | None)
- eval_framework.tasks.task_names.register_arc_de_tasks(registry)[source]¶
Register arc_de benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_arc_ellamind_tasks(registry)[source]¶
Register arc_ellamind benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_arc_tasks(registry)[source]¶
Register arc benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_bigcodebench_tasks(registry)[source]¶
Register bigcodebench benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_copa_tasks(registry)[source]¶
Register copa benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_csqa_ellamind_tasks(registry)[source]¶
Register csqa_ellamind benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_csqa_tasks(registry)[source]¶
Register csqa benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_drop_tasks(registry)[source]¶
Register drop benchmark tasks (composed: DropCompletion_OLMES, DropMC_OLMES).
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_global_mmlu_tasks(registry)[source]¶
Register global_mmlu benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_goldenswag_tasks(registry)[source]¶
Register goldenswag benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_gpqa_ellamind_tasks(registry)[source]¶
Register gpqa_ellamind benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_gpqa_tasks(registry)[source]¶
Register gpqa benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_gsm8k_ellamind_tasks(registry)[source]¶
Register gsm8k_ellamind benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_gsm8k_tasks(registry)[source]¶
Register gsm8k benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_hellaswag_ellamind_tasks(registry)[source]¶
Register hellaswag_ellamind benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_hellaswag_tasks(registry)[source]¶
Register hellaswag benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_hendrycks_math_ellamind_tasks(registry)[source]¶
Register hendrycks_math_ellamind benchmark tasks (composed).
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_hle_ellamind_tasks(registry)[source]¶
Register hle_ellamind benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_humaneval_ellamind_tasks(registry)[source]¶
Register humaneval_ellamind benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_humaneval_plus_tasks(registry)[source]¶
Register humaneval_plus benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_humaneval_tasks(registry)[source]¶
Register humaneval benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_ifeval_tasks(registry)[source]¶
Register ifeval benchmark tasks (composed: IFEval, IFEvalDe).
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_lbpp_tasks(registry)[source]¶
Register lbpp benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_math_reasoning_tasks(registry)[source]¶
Register math_reasoning benchmark tasks (all composed: MATH500(_V2), AIME, GSM8KReasoning, Minerva-OLMES).
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_mbpp_ellamind_tasks(registry)[source]¶
Register mbpp_ellamind benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_mbpp_tasks(registry)[source]¶
Register mbpp benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_mmlu_pro_tasks(registry)[source]¶
Register mmlu_pro benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_mmlu_prox_tasks(registry)[source]¶
Register mmlu_prox benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_mmlu_tasks(registry)[source]¶
Register mmlu benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_multipl_e_tasks(registry)[source]¶
Register multipl_e benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_naturalqs_open_tasks(registry)[source]¶
Register naturalqs_open benchmark tasks (composed: NaturalQsOpen, NaturalQsOpenMC_OLMES).
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_piqa_ellamind_tasks(registry)[source]¶
Register piqa_ellamind benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_piqa_tasks(registry)[source]¶
Register piqa benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_sciq_tasks(registry)[source]¶
Register sciq benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_simpleqa_ellamind_tasks(registry)[source]¶
Register simpleqa_ellamind benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_siqa_ellamind_tasks(registry)[source]¶
Register siqa_ellamind benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_social_iqa_tasks(registry)[source]¶
Register social_iqa benchmark tasks.
- Return type:
None- Parameters:
registry (Registry)
- eval_framework.tasks.task_names.register_squad_tasks(registry)[source]¶
Register squad benchmark tasks (composed: SQuAD_OLMES, SQuAD2_MA, SQuAD2_MA_NO_SYSPROMPT).
- Return type:
None- Parameters:
registry (Registry)
eval_framework.tasks.task_style module¶
Task-style helpers and strategy classes for choice-based evaluation tasks.
This module provides injectable styling strategies — MCStyle, ClozeStyle, MCCompletionStyle
and BPBStyle — that render one choice item (question plus answer options) into the prompt text,
answer cue, scored candidates and ground truth. The styler also declares the response type, metrics
and metadata of its style.
Quick-start¶
A choice-based benchmark pairs a ChoiceReader (item -> raw question, choices, correct index) with
a styler via ComposedBenchmark.choice, which drives both the scored sample and the few-shot
demonstrations from the same pair:
benchmark = ComposedBenchmark.choice(
id="MyBenchmark",
reader=MyChoiceReader(),
styler=ClozeStyle(question_prefix="Question: ", cue_text="Answer:"),
sample_split="test",
fewshot_split="train",
dataset_policy=Pinned("my/dataset"),
)
Task families with MC, Cloze and BPB variants share the reader and differ only in the styler each variant’s builder injects.
- class eval_framework.tasks.task_style.BPBStyle(question_prefix='Question: ', cue_text='Answer:', trailing_newline=True, leading_space_continuations=True, initial_prompt=None)[source]¶
Bases:
ClozeStyleBPB-only styler: prompt identical to ClozeStyle, but scores only the ground-truth completion.
One LLM forward pass per sample instead of N (one per choice), making evaluation significantly faster when accuracy metrics are not needed.
- Parameters:
question_prefix (
str) – Prepended to the raw question (default"Question: ").cue_text (
str) – Assistant cue after the prompt (default"Answer:").trailing_newline (
bool) – WhenTrue(default), the instruction ends with"\n".leading_space_continuations (bool)
initial_prompt (Callable[[str], str] | None)
Assembled prompt example (3 choices):
"Question: What is the capital of France?\n" Scored completions: [" Paris"] ← ground truth only, one forward pass Ground truth: " Paris"
- get_possible_completions(choices, correct_index=None)[source]¶
Return completion candidates for scoring, or
Nonefor free generation tasks.correct_indexis only required byBPBStyle, which scores solely the ground-truth completion.MCStyleandClozeStylescore all choices and ignore it; callers may omit it when using those stylers.- Return type:
list[str]- Parameters:
choices (list[str])
correct_index (int | None)
-
metrics:
list[type[BaseMetric]] = [<class 'eval_framework.metrics.loglikelihood.bits_per_byte.BitsPerByteLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.bpb_variants.BitsPerByteVariantsLoglikelihood'>]¶
- class eval_framework.tasks.task_style.ClozeStyle(question_prefix='Question: ', cue_text='Answer:', trailing_newline=True, leading_space_continuations=True, initial_prompt=None)[source]¶
Bases:
TaskStylerCloze styler: no choices in prompt, model scored over full choice text.
Also known as “ranked classification” (RC). The prompt only shows the question; the model’s score for each full answer text determines the prediction.
- Parameters:
question_prefix (
str) – Prepended to the raw question (default"Question: ").cue_text (
str) – Assistant cue after the prompt (default"Answer:").trailing_newline (
bool) – WhenTrue(default), the instruction ends with"\n". Set toFalsefor sentence-completion tasks where the model should continue a fragment directly.initial_prompt (
Callable[[str],str] |None) – Optional subject-templated preamble prepended once at the top of the prompt. DefaultNone(no preamble).leading_space_continuations (bool)
Assembled prompt example (3 choices):
"Question: What is the capital of France?\n" Scored completions: [" Berlin", " Paris", " London"] Ground truth: " Paris"
Sentence-completion example (trailing_newline=False, cue_text=””):
"The cat sat on the" Scored completions: [" mat", " floor", " sofa"] Ground truth: " mat"
- get_cue_text()[source]¶
Return the assistant cue appended after the prompt (e.g.
"Answer:").- Return type:
str
- get_ground_truth(choices, correct_index)[source]¶
Return the ground-truth string for scoring.
- Return type:
str- Parameters:
choices (list[str])
correct_index (int)
- get_instruction_text(raw_question, choices)[source]¶
Build the instruction/prompt text from a question and answer choices.
- Return type:
str- Parameters:
raw_question (str)
choices (list[str])
- get_possible_completions(choices, correct_index=None)[source]¶
Return completion candidates for scoring, or
Nonefor free generation tasks.correct_indexis only required byBPBStyle, which scores solely the ground-truth completion.MCStyleandClozeStylescore all choices and ignore it; callers may omit it when using those stylers.- Return type:
list[str]- Parameters:
choices (list[str])
correct_index (int | None)
- initial_prompt(subject_label)[source]¶
A preamble prepended once at the top of the prompt for the given subject, or None (the default: no preamble). Subject-templated preambles (e.g. MMLU’s “… about {subject}.”) read
subject_label; fixed preambles ignore it.- Return type:
str|None- Parameters:
subject_label (str)
-
metrics:
list[type[BaseMetric]] = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyBayesianLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.bits_per_byte.BitsPerByteLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.bpb_variants.BitsPerByteVariantsLoglikelihood'>]¶
-
response_type:
ResponseType= 'loglikelihoods'¶
- class eval_framework.tasks.task_style.IdkStyle(inner, abstention_option)[source]¶
Bases:
TaskStylerAdds an explicit abstention option (e.g. “I do not know”, “?”) to a base styler’s candidates.
Use this when a benchmark rewards calibrated abstention over confident guessing: it scores the real answers together with the abstention option and reports confidence-aware metrics. Everything else — the prompt formatting and the abstention preamble (via the inner styler’s
initial_prompt) — is the inner styler’s; wrap any base styler, so both multiple-choice (abstain with an extra letter) and cloze (abstain with an extra full answer) tasks are supported.- Parameters:
inner (
TaskStyler) – The base styler doing the prompt formatting, preamble, and real-answer scoring.abstention_option (
str) – The abstention completion scored alongside the real choices, e.g." I do not know"(cloze) or" ?"(multiple-choice).
- get_cue_text()[source]¶
Return the assistant cue appended after the prompt (e.g.
"Answer:").- Return type:
str
- get_ground_truth(choices, correct_index)[source]¶
Return the ground-truth string for scoring.
- Return type:
str- Parameters:
choices (list[str])
correct_index (int)
- get_instruction_text(raw_question, choices)[source]¶
Build the instruction/prompt text from a question and answer choices.
- Return type:
str- Parameters:
raw_question (str)
choices (list[str])
- get_possible_completions(choices, correct_index=None)[source]¶
Return completion candidates for scoring, or
Nonefor free generation tasks.correct_indexis only required byBPBStyle, which scores solely the ground-truth completion.MCStyleandClozeStylescore all choices and ignore it; callers may omit it when using those stylers.- Return type:
list[str]- Parameters:
choices (list[str])
correct_index (int | None)
- initial_prompt(subject_label)[source]¶
A preamble prepended once at the top of the prompt for the given subject, or None (the default: no preamble). Subject-templated preambles (e.g. MMLU’s “… about {subject}.”) read
subject_label; fixed preambles ignore it.- Return type:
str|None- Parameters:
subject_label (str)
-
metrics:
list[type[BaseMetric]] = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyBayesianLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.confidence_weighted_accuracy.ConfidenceWeightedAccuracy'>, <class 'eval_framework.metrics.loglikelihood.dcs.DistributionalCorrectnessScore'>, <class 'eval_framework.metrics.loglikelihood.ternary.TernaryScore'>]¶
-
response_type:
ResponseType= 'loglikelihoods'¶
- class eval_framework.tasks.task_style.Language(*values)[source]¶
Bases:
EnumA language the stylers have default question/answer wording for.
Members are named by ISO 639-3 code and valued by English name
- CES = 'Czech'¶
- DAN = 'Danish'¶
- DEU = 'German'¶
- ENG = 'English'¶
- FIN = 'Finnish'¶
- FRA = 'French'¶
- HUN = 'Hungarian'¶
- ITA = 'Italian'¶
- NLD = 'Dutch'¶
- POL = 'Polish'¶
- POR = 'Portuguese'¶
- RUS = 'Russian'¶
- SPA = 'Spanish'¶
- UKR = 'Ukrainian'¶
- class eval_framework.tasks.task_style.MCCompletionStyle(question_prefix='Question: ', cue_text='Answer:', space_prefixed_labels=False)[source]¶
Bases:
TaskStylerMultiple-choice prompt style with generative completion output.
Prompt formatting is identical to
MCStyle(choices shown in prompt), but ground truth is the full answer text (for example"Paris"instead of"D"). Responses are generated viaResponseType.COMPLETIONand scored withAccuracyCompletion.This style is useful for instruction-following or CoT prompts where the model is expected to generate an answer text rather than be scored via loglikelihood over fixed label candidates.
- Parameters:
question_prefix (
str) – Prepended to the raw question (default"Question: ").cue_text (
str) – Assistant cue after the prompt (default"Answer:").space_prefixed_labels (
bool) – WhenTrue, each option line starts with a space (" A. choice"— OLMES-style). DefaultFalse.
Assembled prompt example (default settings, 4 choices):
"Question: What is the capital of France?\nA. Munich\nB. Stuttgart\nC. Berlin\nD. Paris\nAnswer:" Ground truth: "Paris"
- get_cue_text()[source]¶
Return the assistant cue appended after the prompt (e.g.
"Answer:").- Return type:
str
- get_fewshot_target_text(choices, correct_index)[source]¶
Override to insert a space between cue and answer text.
- Return type:
str- Parameters:
choices (list[str])
correct_index (int)
- get_ground_truth(choices, correct_index)[source]¶
Return the ground-truth string for scoring.
- Return type:
str- Parameters:
choices (list[str])
correct_index (int)
- get_instruction_text(raw_question, choices)[source]¶
Build the instruction/prompt text from a question and answer choices.
- Return type:
str- Parameters:
raw_question (str)
choices (list[str])
- get_possible_completions(choices, correct_index=None)[source]¶
Completion-mode tasks do not provide fixed candidate completions.
- Return type:
list[str] |None- Parameters:
choices (list[str])
correct_index (int | None)
-
metrics:
list[type[BaseMetric]] = [<class 'eval_framework.metrics.completion.accuracy_completion.AccuracyCompletion'>]¶
-
response_type:
ResponseType= 'completion'¶
- class eval_framework.tasks.task_style.MCStyle(question_prefix='Question: ', cue_text='Answer:', space_prefixed_labels=False, initial_prompt=None)[source]¶
Bases:
TaskStylerMultiple-choice styler: choices shown in prompt, model scored over letter labels.
- Parameters:
question_prefix (
str) – Prepended to the raw question (default"Question: ").cue_text (
str) – Assistant cue after the prompt (default"Answer:").space_prefixed_labels (
bool) – WhenTrue, each option line starts with a space (" A. choice"— OLMES-style). DefaultFalse.initial_prompt (
Callable[[str],str] |None) – Optional subject-templated preamble prepended once at the top of the prompt (e.g. MMLU’s “… about {subject}.”). DefaultNone(no preamble).
Assembled prompt example (default settings, 3 choices):
"Question: What is the capital of France?\nA. Berlin\nB. Paris\nC. London\n" Scored completions: [" A", " B", " C"] Ground truth: " B"
- get_cue_text()[source]¶
Return the assistant cue appended after the prompt (e.g.
"Answer:").- Return type:
str
- get_ground_truth(choices, correct_index)[source]¶
Return the ground-truth string for scoring.
- Return type:
str- Parameters:
choices (list[str])
correct_index (int)
- get_instruction_text(raw_question, choices)[source]¶
Build the instruction/prompt text from a question and answer choices.
- Return type:
str- Parameters:
raw_question (str)
choices (list[str])
- get_possible_completions(choices, correct_index=None)[source]¶
Note: correct_index is ignored for MCStyle and only used for BPBStyle.
- Return type:
list[str]- Parameters:
choices (list[str])
correct_index (int | None)
- initial_prompt(subject_label)[source]¶
A preamble prepended once at the top of the prompt for the given subject, or None (the default: no preamble). Subject-templated preambles (e.g. MMLU’s “… about {subject}.”) read
subject_label; fixed preambles ignore it.- Return type:
str|None- Parameters:
subject_label (str)
-
metrics:
list[type[BaseMetric]] = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyBayesianLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.bits_per_byte.BitsPerByteLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.bpb_variants.BitsPerByteVariantsLoglikelihood'>]¶
-
response_type:
ResponseType= 'loglikelihoods'¶
- class eval_framework.tasks.task_style.TaskStyle(*values)[source]¶
Bases:
Enum- BPB = 'bpb'¶
- CLOZE = 'cloze'¶
- MULTIPLE_CHOICE = 'multiple_choice'¶
- class eval_framework.tasks.task_style.TaskStyler[source]¶
Bases:
ABCStrategy object that controls prompt assembly and scoring for choice-based tasks.
Concrete implementations (
MCStyle,ClozeStyle) are injected into theChoiceeval kind alongside aChoiceReader, so benchmark authors only supply data access.- response_type¶
The response type the task should use (e.g. LOGLIKELIHOODS).
- metrics¶
Default metric classes for tasks using this styler.
- task_style¶
Discriminator for metadata (MULTIPLE_CHOICE or CLOZE).
- question_prefix¶
String prepended to the raw question.
- classmethod for_language(language, **kwargs)[source]¶
Factory that fills
question_prefixandcue_textfrom language defaults.Any explicitly passed keyword arguments take precedence over the defaults.
- Return type:
Self- Parameters:
language (Language)
kwargs (Any)
- abstractmethod get_cue_text()[source]¶
Return the assistant cue appended after the prompt (e.g.
"Answer:").- Return type:
str
- get_extra_metadata()[source]¶
Return styler-specific metadata to merge into the task’s metadata.
- Return type:
dict
- get_fewshot_target_text(choices, correct_index)[source]¶
Return the few-shot target string (cue + ground truth).
- Return type:
str- Parameters:
choices (list[str])
correct_index (int)
- abstractmethod get_ground_truth(choices, correct_index)[source]¶
Return the ground-truth string for scoring.
- Return type:
str- Parameters:
choices (list[str])
correct_index (int)
- abstractmethod get_instruction_text(raw_question, choices)[source]¶
Build the instruction/prompt text from a question and answer choices.
- Return type:
str- Parameters:
raw_question (str)
choices (list[str])
- abstractmethod get_possible_completions(choices, correct_index=None)[source]¶
Return completion candidates for scoring, or
Nonefor free generation tasks.correct_indexis only required byBPBStyle, which scores solely the ground-truth completion.MCStyleandClozeStylescore all choices and ignore it; callers may omit it when using those stylers.- Return type:
list[str] |None- Parameters:
choices (list[str])
correct_index (int | None)
- get_question_text(raw_question)[source]¶
Build the full question line (prefix + raw question).
Override in a subclass for non-standard question formats (e.g. HellaSwag’s
"activity: context"form).- Return type:
str- Parameters:
raw_question (str)
- initial_prompt(subject_label)[source]¶
A preamble prepended once at the top of the prompt for the given subject, or None (the default: no preamble). Subject-templated preambles (e.g. MMLU’s “… about {subject}.”) read
subject_label; fixed preambles ignore it.- Return type:
str|None- Parameters:
subject_label (str)
-
metrics:
list[type[BaseMetric]]¶
-
question_prefix:
str¶
-
response_type:
ResponseType¶
- eval_framework.tasks.task_style.answer_key_to_index(key)[source]¶
Convert a letter or 1-based integer answer key to a 0-based index.
Datasets sometimes encode the correct answer as a letter (“A”, “B”, …) and sometimes as a 1-based integer string (“1”, “2”, …). This function normalises both to a 0-based index so task code doesn’t need to branch.
- Parameters:
key (
str) – A single-character string. Either a letter (“A”-“Z”) or a digit (“1”-“9”).- Returns:
“A” or “1” → 0, “B” or “2” → 1, etc.
- Return type:
0-based index
- eval_framework.tasks.task_style.format_mc_prompt(question_text, choices, *, space_prefixed_labels=False)[source]¶
Helper function to format a question and its labeled choices into a multiple-choice prompt.
The choices are labeled A, B, C, … in order and ends with a newline.
- Parameters:
question_text (
str) – The full question string (prefix already included).choices (
list[str]) – Ordered list of answer option strings.space_prefixed_labels (
bool) – WhenTrue, each option line is prefixed with a space — “ A. choice” instead of “A. choice”. This is, e.g., the OLMES-style prompt format.
- Return type:
str- Returns:
A string of the form
"<question_text>\n[pfx]A. choice0\n[pfx]B. choice1\n".
Examples:
>>> format_mc_prompt("Question: What is 1+1?", ["1", "2", "3"]) 'Question: What is 1+1?\nA. 1\nB. 2\nC. 3\n' >>> format_mc_prompt("Question: What is 1+1?", ["1", "2"], space_prefixed_labels=True) 'Question: What is 1+1?\n A. 1\n B. 2\n'
- eval_framework.tasks.task_style.question_cue_text(language)[source]¶
- Return type:
tuple[str,str]- Parameters:
language (Language)
- eval_framework.tasks.task_style.shuffle_correct_with_distractors(correct, distractors, seed_text)[source]¶
Deterministically shuffle distractors + correct answer; return (choices, correct_index).
Calling this multiple times with the same arguments always returns the same result.
- Parameters:
correct (
str) – The correct answer string.distractors (
list[str]) – The distractor strings, e.g. wrong answer choices.seed_text (
str) – Text used as the shuffle seed (typically question + answer).
- Return type:
tuple[list[str],int]- Returns:
A tuple
(shuffled_choices, correct_index)wherecorrect_indexis the 0-based position ofcorrectinshuffled_choices.
eval_framework.tasks.utils module¶
- exception eval_framework.tasks.utils.DecodingError[source]¶
Bases:
SerializationErrorRaised when decoding a callable fails.
- exception eval_framework.tasks.utils.DockerReturnedEmptyOutput[source]¶
Bases:
ExceptionRaised when the docker returns an empty output.
- exception eval_framework.tasks.utils.EncodingError[source]¶
Bases:
SerializationErrorRaised when encoding a callable fails.
- class eval_framework.tasks.utils.ExecutionResult(success, output)[source]¶
Bases:
NamedTupleA named tuple to store the result of code execution.
- Parameters:
success (bool)
output (str)
- success¶
Indicates if the execution was successful.
- Type:
bool
- output¶
Contains the output or error messages from the execution.
- Type:
str
-
output:
str¶ Alias for field number 1
-
success:
bool¶ Alias for field number 0
- exception eval_framework.tasks.utils.SerializationError[source]¶
Bases:
ExceptionBase exception for callable serialization errors.
- eval_framework.tasks.utils.execute_python_code_with_tests(code, test_code, dockerfile, package_mapping, merge_code_fn, image, timeout, parse_output_fn)[source]¶
Executes the given code with test cases in a sandboxed environment.
- Parameters:
code (
str) – The code to be tested.test_code (
str) – The test cases to run against the code.package_mapping (
dict[str,str|None]) – Mapping of package names to install commands.merge_code_fn (
Callable[[str,str],str]) – function to merge LLM and test codeimage (
str|None) – Docker image to use.timeout (
int) – Timeout for the execution in seconds.parse_otuput_fn – function to parse docker execution output
dockerfile (str | None)
parse_output_fn (Callable[[str], ExecutionResult])
- Return type:
- Returns:
An ExecutionResult named tuple with success status and output or errors.
- eval_framework.tasks.utils.extract_imports(code)[source]¶
Extract all import statements and the imported packages from code.
- Return type:
tuple[list[str],set[str]]- Parameters:
code (str)
- eval_framework.tasks.utils.extract_python_code_from_response(llm_response)[source]¶
Extract the Python source from a chat-style LLM response that wraps its answer in markdown code fences.
The response is expected to contain a single Python solution (e.g. HumanEval, MBPP, BigCodeBench). The first matching fenced block is returned, tried in this order: a
`python block nested inside a ```markdown block, a plain ```python block, a ```markdown block, and finally any `block. If the response contains no fences at all, it is assumed to be raw code and returned unchanged.- Return type:
str- Parameters:
llm_response (str)
- eval_framework.tasks.utils.get_external_dependencies(code, package_mapping)[source]¶
Identify external dependencies in the code.
- Return type:
list[str]- Parameters:
code (str)
package_mapping (dict[str, str | None])
- eval_framework.tasks.utils.get_or_create_pool(image=None, dockerfile=None, packages=None, lang='python', min_pool_size=1, max_pool_size=1, max_container_uses=100, runtime_configs=None)[source]¶
- Return type:
ContainerPoolManager- Parameters:
image (str | None)
dockerfile (str | None)
packages (list[str] | None)
lang (str)
min_pool_size (int)
max_pool_size (int)
max_container_uses (int)
runtime_configs (dict[str, str] | None)
- eval_framework.tasks.utils.run_python_code(code, image=None, dockerfile=None, input_files=None, timeout=60, packages=None, runtime_configs=None)[source]¶
Run code in a sandboxed environment. :type code:
str:param code: The code to run. :type image:str|None:param image: Docker image to use. :type dockerfile:str|None:param dockerfile: Dockerfile to use. :type input_files:list[tuple[str,str]] |None:param input_files: pairs of host and docker paths, host files will be copied to the docker. :type timeout:int:param timeout: Timeout in seconds, 0 if no timeout. :type packages:list[str] |None:param packages: List of python packages to install with pip. :rtype:str:return: The output of the code.- Parameters:
code (str)
image (str | None)
dockerfile (str | None)
input_files (list[tuple[str, str]] | None)
timeout (int)
packages (list[str] | None)
runtime_configs (dict[str, str] | None)
- Return type:
str