eval_framework.tasks package

Submodules

eval_framework.tasks.base module

eval_framework.tasks.base.resolve_overwrite_subjects(custom_subjects, accepted_subjects, task_name)[source]

Restrict accepted_subjects to the ones requested via –task-subjects.

A subject matches a token when their parts agree position by position, where “*” matches any one part – so “DE_DE,*” selects every German subject and “*” selects every scalar subject.

Return type:

list[TypeVar(SubjectType)]

Parameters:
  • custom_subjects (list[str])

  • accepted_subjects (list[SubjectType])

  • task_name (str)

eval_framework.tasks.dataset_loading module

Fetching a benchmark’s dataset, kept out of the composed classes so they stay free of Hugging Face mechanics (revisions, cache directories, download config).

class eval_framework.tasks.dataset_loading.DatasetLoader[source]

Bases: ABC

Loads a benchmark’s dataset splits for one subject.

abstractmethod load(name)[source]

Load the splits for subject name (None = the task’s single unnamed slice). Each loader decides how a subject selects data — HfDatasetLoader treats the subject as the HF config.

Return type:

DatasetDict

Parameters:

name (str | None)

abstractmethod metadata()[source]

Dataset-identifying metadata merged into the eval’s get_metadata (e.g. the dataset path).

Return type:

dict[str, str]

class eval_framework.tasks.dataset_loading.DatasetPolicy[source]

Bases: ABC

Produces the loader for a benchmark’s dataset, and documents where that dataset comes from.

abstractmethod documentation()[source]

Markdown for the task’s ## Dataset doc section, describing where the dataset comes from.

Return type:

str

abstractmethod loader(custom_hf_revision)[source]
Return type:

DatasetLoader

Parameters:

custom_hf_revision (str | None)

subject_encoded_in_column(config, column)[source]

By default the loaded config names the subject; call this when the subject is instead encoded in a column of a single config (None loads the default config) (see SubjectColumn).

Return type:

SubjectColumn

Parameters:
  • config (str | None)

  • column (str)

subset(keep, description=None)[source]

Restrict this policy’s dataset to the rows for which keep returns true (see Subset).

description names the resulting subset for the rendered dataset docs (e.g. “the diamond subset”).

Return type:

Subset

Parameters:
  • keep (Callable[[dict[str, Any]], bool])

  • description (str | None)

final class eval_framework.tasks.dataset_loading.FixedHfConfigLoader(dataset_path, revision, hf_config)[source]

Bases: DatasetLoader

Loads a fixed HF config, ignoring the requested subject — for a dataset whose subjects are labels rather than configs (or a task with no subjects but a non-default config).

Parameters:
  • dataset_path (str)

  • revision (str | None)

  • hf_config (str | None)

load(name)[source]

Load the splits for subject name (None = the task’s single unnamed slice). Each loader decides how a subject selects data — HfDatasetLoader treats the subject as the HF config.

Return type:

DatasetDict

Parameters:

name (str | None)

metadata()[source]

Dataset-identifying metadata merged into the eval’s get_metadata (e.g. the dataset path).

Return type:

dict[str, str]

final class eval_framework.tasks.dataset_loading.HfDatasetLoader(dataset_path, revision)[source]

Bases: DatasetLoader

Loads one Hugging Face dataset pinned to revision, treating the requested subject as the HF config.

Parameters:
  • dataset_path (str)

  • revision (str | None)

load(name)[source]

Load the splits for subject name (None = the task’s single unnamed slice). Each loader decides how a subject selects data — HfDatasetLoader treats the subject as the HF config.

Return type:

DatasetDict

Parameters:

name (str | None)

metadata()[source]

Dataset-identifying metadata merged into the eval’s get_metadata (e.g. the dataset path).

Return type:

dict[str, str]

final class eval_framework.tasks.dataset_loading.SubjectColumn(inner, config, column)[source]

Bases: DatasetPolicy

Use this in case the raw dataset encodes the subject in a column, rather than the load configuration.

Parameters:
documentation()[source]

Markdown for the task’s ## Dataset doc section, describing where the dataset comes from.

Return type:

str

loader(custom_hf_revision)[source]
Return type:

DatasetLoader

Parameters:

custom_hf_revision (str | None)

final class eval_framework.tasks.dataset_loading.Subset(inner, keep, description=None)[source]

Bases: DatasetPolicy

Restricts another policy’s dataset to the rows for which keep returns true, in every split.

A benchmark whose items are a row-filtered subset of a larger dataset (e.g. GPQA’s diamond subset, HLE’s natively-multiple-choice subset) wraps the base policy in a Subset. description names that subset for the rendered dataset docs; without it the docs only note that some rows are dropped.

Parameters:
  • inner (DatasetPolicy)

  • keep (Callable[[dict[str, Any]], bool])

  • description (str | None)

documentation()[source]

Markdown for the task’s ## Dataset doc section, describing where the dataset comes from.

Return type:

str

loader(custom_hf_revision)[source]
Return type:

DatasetLoader

Parameters:

custom_hf_revision (str | None)

eval_framework.tasks.dataset_revisions module

Pinned Hugging Face dataset revisions.

A lock file maps dataset paths to pinned commit SHAs, so an eval runs against a reproducible dataset revision. Benchmarks resolve their pin through a Pinned dataset policy (usually via pinned_by_framework).

final class eval_framework.tasks.dataset_revisions.FixedHfConfig(base, hf_config)[source]

Bases: DatasetPolicy

Loads a fixed HF config from an underlying Pinned policy, regardless of subject (here the subject is a label, not a config).

Parameters:
  • base (Pinned)

  • hf_config (str | None)

documentation()[source]

Markdown for the task’s ## Dataset doc section, describing where the dataset comes from.

Return type:

str

loader(custom_hf_revision)[source]
Return type:

DatasetLoader

Parameters:

custom_hf_revision (str | None)

class eval_framework.tasks.dataset_revisions.HfDatasetRevisions(revisions)[source]

Bases: object

Pinned revisions of Hugging Face datasets, mapping dataset path → commit SHA.

Parameters:

revisions (dict[str, str])

classmethod from_file(path)[source]
Return type:

HfDatasetRevisions

Parameters:

path (Path)

missing_references(api)[source]

The pins whose revisions are no longer reachable on the Hub, with the failure reason.

Intended to run on a CI schedule, so a yanked revision or a removed dataset is noticed before it breaks an eval run. An empty list means every pin is still reachable.

Return type:

list[str]

Parameters:

api (HfApi)

num_revisions()[source]
Return type:

int

revision_for(dataset_path)[source]

The pinned commit SHA for a dataset. Raises KeyError if it is not pinned.

Return type:

str

Parameters:

dataset_path (str)

to_dict()[source]
Return type:

dict[str, str]

to_file(path)[source]
Return type:

None

Parameters:

path (Path)

update_to_latest(api)[source]

Update every pin to its dataset’s latest commit SHA.

Intended to run on CI to ensure the pinned revisions are up-to-date. If the lookup for a dataset fails, its existing pin is kept.

Return type:

None

Parameters:

api (HfApi)

final class eval_framework.tasks.dataset_revisions.Pinned(lockfile, dataset_path)[source]

Bases: DatasetPolicy

Pins dataset_path to the revision recorded for it in lockfile

A CI job scans for newer available versions of these revisions, proposing PRs for updating them. This ensures the revision is up to date, while still protecting against silently changing the meaning of a benchmark.

Parameters:
  • lockfile (Path)

  • dataset_path (str)

documentation()[source]

Markdown for the task’s ## Dataset doc section, describing where the dataset comes from.

Return type:

str

loader(custom_hf_revision)[source]
Return type:

HfDatasetLoader

Parameters:

custom_hf_revision (str | None)

with_hf_config(hf_config)[source]

Always load hf_config (None = the dataset’s default config), ignoring the subject — for an HF dataset whose subjects are labels rather than configs, or a task with a fixed non-default config.

Return type:

FixedHfConfig

Parameters:

hf_config (str | None)

eval_framework.tasks.dataset_revisions.pinned_by_framework(dataset_path)[source]

A Pinned policy binding dataset_path to the framework’s bundled lock file.

Return type:

Pinned

Parameters:

dataset_path (str)

eval_framework.tasks.dataset_revisions.pinned_frozen(dataset_path)[source]

A Pinned policy binding dataset_path to the frozen lock file (revisions held fixed to keep results comparable across framework upgrades).

Return type:

Pinned

Parameters:

dataset_path (str)

eval_framework.tasks.dataset_revisions.pinned_revision(lockfile, dataset_path)[source]

The commit SHA pinned for dataset_path in lockfile.

Resolves the exact dataset revision an eval runs against, so results stay reproducible across dataset updates. Every dataset used by a task is expected to be pinned; a missing entry is a bug in the lock file and raises KeyError.

Return type:

str

Parameters:
  • lockfile (Path)

  • dataset_path (str)

eval_framework.tasks.eval_config module

class eval_framework.tasks.eval_config.EvalConfig(**data)[source]

Bases: BaseConfig

Parameters:
  • output_dir (Path)

  • wandb_project (str | None)

  • wandb_entity (str | None)

  • wandb_run_id (str | None)

  • wandb_group (str | None)

  • wandb_upload_results (Annotated[bool, BeforeValidator(func=~eval_framework.tasks.eval_config.EvalConfig.<lambda>, json_schema_input_type=PydanticUndefined)])

  • hf_upload_dir (str | None)

  • hf_upload_repo (str | None)

  • num_fewshot (Annotated[int, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0)])])

  • num_samples (Annotated[int | None, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=1)])])

  • max_tokens (int | None)

  • task_name (str)

  • task_subjects (list[str] | None)

  • hf_revision (str | None)

  • llm_class (type[BaseLLM])

  • llm_args (dict[str, Any])

  • llm_judge_class (type[BaseLLM] | None)

  • judge_model_args (dict[str, Any])

  • randomize_judge_order (bool)

  • batch_size (Annotated[int, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=1)])])

  • description (str | None)

  • save_intermediate_results (Annotated[bool, BeforeValidator(func=~eval_framework.tasks.eval_config.EvalConfig.<lambda>, json_schema_input_type=PydanticUndefined)])

  • save_logs (Annotated[bool, BeforeValidator(func=~eval_framework.tasks.eval_config.EvalConfig.<lambda>, json_schema_input_type=PydanticUndefined)])

  • delete_output_dir_after_upload (Annotated[bool, BeforeValidator(func=~eval_framework.tasks.eval_config.EvalConfig.<lambda>, json_schema_input_type=PydanticUndefined)])

  • repeats (Annotated[int, BeforeValidator(func=~eval_framework.tasks.eval_config.EvalConfig.<lambda>, json_schema_input_type=PydanticUndefined), FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=1)])])

batch_size: Annotated[int]
delete_output_dir_after_upload: Annotated[bool]
description: str | None
hf_revision: str | None
hf_upload_dir: str | None
hf_upload_repo: str | None
judge_model_args: dict[str, Any]
llm_args: dict[str, Any]
llm_class: type[BaseLLM]
llm_judge()[source]
Return type:

BaseLLM

llm_judge_class: type[BaseLLM] | None
max_tokens: int | None
model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'frozen': True, 'protected_namespaces': ()}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_json_dump()[source]
Return type:

str

model_json_robust_subset_dump()[source]
Return type:

str

num_fewshot: Annotated[int]
num_samples: Annotated[int | None]
output_dir: Path
randomize_judge_order: bool
repeats: Annotated[int]
save_intermediate_results: Annotated[bool]
save_logs: Annotated[bool]
serialize_judge_model_args(value)[source]
Return type:

dict[str, Any]

Parameters:

value (dict[str, Any])

serialize_llm_class(value)[source]

Serialize the class into its fully qualified name.

Return type:

str | None

Parameters:

value (type[BaseLLM] | None)

serialize_llm_judge_class(value)[source]

Serialize the class into its fully qualified name.

Return type:

str | None

Parameters:

value (type[BaseLLM] | None)

serialize_output_dir(value)[source]
Return type:

str

Parameters:

value (Path)

task_name: str
task_subjects: list[str] | None
classmethod validate_judge_model_args(value)[source]
Return type:

dict[str, Any]

Parameters:

value (dict[str, Any])

classmethod validate_llm_args(value)[source]
Return type:

dict[str, Any]

Parameters:

value (dict[str, Any])

classmethod validate_output_dir(value)[source]
Return type:

Path

Parameters:

value (str | Path)

wandb_entity: str | None
wandb_group: str | None
wandb_project: str | None
wandb_run_id: str | None
wandb_upload_results: Annotated[bool]

eval_framework.tasks.lazy module

class eval_framework.tasks.lazy.Lazy(id, load)[source]

Bases: Benchmark

A Benchmark that defers producing the real benchmark until first use.

Holds an id and a load thunk returning the wrapped Benchmark; every other call is delegated to that benchmark, which is built (and memoized) on first access. This keeps Lazy independent of any concrete task type — the thunk owns whatever loading (e.g. importing a module) the wrapped benchmark requires.

Parameters:
create(num_fewshot, custom_subjects, custom_hf_revision, seed=None)[source]
Return type:

Eval

Parameters:
  • num_fewshot (int)

  • custom_subjects (list[str] | None)

  • custom_hf_revision (str | None)

  • seed (int | None)

display_name()[source]

Human-readable display name. Is allowed to have special characters and whitespaces.

Return type:

str

id()[source]

Uniquely identifies the benchmark

Return type:

str

markdown_doc(formatters)[source]

Render the benchmarks’s documentation as markdown.

Return type:

str

Parameters:

formatters (Sequence[BaseFormatter])

metrics()[source]

The benchmark’s metrics

Return type:

list[type[BaseMetric]]

response_type()[source]

The benchmark’s response type

Return type:

ResponseType

subjects(tokens=None)[source]

Labels of the subjects a run with these --task-subjects tokens evaluates (no tokens = all), exactly as they appear in samples and aggregated-result keys. Raises ValueError for a token no subject matches, mirroring what create does with the same tokens.

Return type:

list[str]

Parameters:

tokens (list[str] | None)

eval_framework.tasks.markdown_doc module

eval_framework.tasks.markdown_doc.markdown_doc(*, name, dataset_doc, sample_split, fewshot_split, response_type, metrics, subjects, num_fewshot, formatters, example_messages, split_sizes, possible_completions, ground_truth)[source]

Render a task’s documentation as markdown

Return type:

str

Parameters:
  • name (str)

  • dataset_doc (str)

  • sample_split (str | None)

  • fewshot_split (str | None)

  • response_type (str)

  • metrics (Sequence[str])

  • subjects (Any)

  • num_fewshot (int)

  • formatters (Sequence[BaseFormatter])

  • example_messages (list[Message] | None)

  • split_sizes (dict[str, int] | None)

  • possible_completions (str | list[str] | None)

  • ground_truth (str | list[str] | None)

eval_framework.tasks.registry module

class eval_framework.tasks.registry.Benchmark[source]

Bases: ABC

A benchmark is used to provide means of measuring model performance in a domain.

Benchmark act as factories for Eval. They bind all the a prior known information and enrich it with the arguments provided at runtime to create concrete instances of Eval which are used to provide measurements of the models performance.

abstractmethod create(num_fewshot, custom_subjects, custom_hf_revision, seed=None)[source]
Return type:

Eval

Parameters:
  • num_fewshot (int)

  • custom_subjects (list[str] | None)

  • custom_hf_revision (str | None)

  • seed (int | None)

abstractmethod display_name()[source]

Human-readable display name. Is allowed to have special characters and whitespaces.

Return type:

str

abstractmethod id()[source]

Uniquely identifies the benchmark

Return type:

str

abstractmethod markdown_doc(formatters)[source]

Render the benchmarks’s documentation as markdown.

Return type:

str

Parameters:

formatters (Sequence[BaseFormatter])

abstractmethod metrics()[source]

The benchmark’s metrics

Return type:

list[type[BaseMetric]]

abstractmethod response_type()[source]

The benchmark’s response type

Return type:

ResponseType

abstractmethod subjects(tokens=None)[source]

Labels of the subjects a run with these --task-subjects tokens evaluates (no tokens = all), exactly as they appear in samples and aggregated-result keys. Raises ValueError for a token no subject matches, mirroring what create does with the same tokens.

Return type:

list[str]

Parameters:

tokens (list[str] | None)

class eval_framework.tasks.registry.Lazy(id, load)[source]

Bases: Benchmark

A Benchmark that defers producing the real benchmark until first use.

Holds an id and a load thunk returning the wrapped Benchmark; every other call is delegated to that benchmark, which is built (and memoized) on first access. This keeps Lazy independent of any concrete task type — the thunk owns whatever loading (e.g. importing a module) the wrapped benchmark requires.

Parameters:
create(num_fewshot, custom_subjects, custom_hf_revision, seed=None)[source]
Return type:

Eval

Parameters:
  • num_fewshot (int)

  • custom_subjects (list[str] | None)

  • custom_hf_revision (str | None)

  • seed (int | None)

display_name()[source]

Human-readable display name. Is allowed to have special characters and whitespaces.

Return type:

str

id()[source]

Uniquely identifies the benchmark

Return type:

str

markdown_doc(formatters)[source]

Render the benchmarks’s documentation as markdown.

Return type:

str

Parameters:

formatters (Sequence[BaseFormatter])

metrics()[source]

The benchmark’s metrics

Return type:

list[type[BaseMetric]]

response_type()[source]

The benchmark’s response type

Return type:

ResponseType

subjects(tokens=None)[source]

Labels of the subjects a run with these --task-subjects tokens evaluates (no tokens = all), exactly as they appear in samples and aggregated-result keys. Raises ValueError for a token no subject matches, mirroring what create does with the same tokens.

Return type:

list[str]

Parameters:

tokens (list[str] | None)

class eval_framework.tasks.registry.Registry[source]

Bases: object

A registry for Benchmarks

add(factory)[source]

Register a factory under the key derived from its id().

Return type:

None

Parameters:

factory (Benchmark)

items()[source]

Iterate over (task name, Benchmark) pairs in the registry.

Return type:

Iterator[tuple[str, Benchmark]]

task_names()[source]

The names of all registered tasks.

Return type:

list[str]

eval_framework.tasks.registry.is_registered(name, /)[source]

Return True if a task is registered.

Return type:

bool

Parameters:

name (str)

eval_framework.tasks.registry.registered_task_names()[source]

Return the names of all registered tasks.

Return type:

list[str]

eval_framework.tasks.registry.validate_task_name(name)[source]

Pydantic-style validator for task names.

Return type:

str

Parameters:

name (str)

eval_framework.tasks.registry.with_registry(registry)[source]

Contextmanager to change the current registry.

Return type:

Generator[None, Any, None]

Parameters:

registry (Registry)

eval_framework.tasks.task_loader module

eval_framework.tasks.task_loader.load_extra_tasks(directory, registry=None)[source]

Load and register user-defined tasks from a directory of plugin modules.

Every .py file under directory (searched recursively) is imported as a module, and any module exposing a register_tasks(registry) function has it invoked to register its tasks.

Return type:

None

Parameters:
  • directory (str | PathLike)

  • registry (Registry | None)

eval_framework.tasks.task_loader.load_modules_from_directory(directory)[source]

Recursively import every .py file under directory as a module.

Each file is imported under a unique module name derived from its path relative to directory and inserted into sys.modules.

Return type:

list[ModuleType]

Parameters:

directory (str | PathLike)

eval_framework.tasks.task_loader.register_tasks_from_module(module, registry=None)[source]

Invoke register_tasks(registry) on module if it defines it.

Modules without a register_tasks entrypoint are ignored.

Return type:

None

Parameters:
  • module (ModuleType)

  • registry (Registry | None)

eval_framework.tasks.task_loader.register_tasks_from_modules(modules, registry=None)[source]

Register tasks from each module in modules (see register_tasks_from_module).

Return type:

None

Parameters:
  • modules (Sequence[ModuleType])

  • registry (Registry | None)

eval_framework.tasks.task_names module

eval_framework.tasks.task_names.register_all_tasks(registry=None)[source]

Register all the benchmark tasks with the eval framework

Uses global registry by default.

Return type:

None

Parameters:

registry (Registry | None)

eval_framework.tasks.task_names.register_arc_de_tasks(registry)[source]

Register arc_de benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_arc_ellamind_tasks(registry)[source]

Register arc_ellamind benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_arc_tasks(registry)[source]

Register arc benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_bigcodebench_tasks(registry)[source]

Register bigcodebench benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_copa_tasks(registry)[source]

Register copa benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_csqa_ellamind_tasks(registry)[source]

Register csqa_ellamind benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_csqa_tasks(registry)[source]

Register csqa benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_drop_tasks(registry)[source]

Register drop benchmark tasks (composed: DropCompletion_OLMES, DropMC_OLMES).

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_global_mmlu_tasks(registry)[source]

Register global_mmlu benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_goldenswag_tasks(registry)[source]

Register goldenswag benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_gpqa_ellamind_tasks(registry)[source]

Register gpqa_ellamind benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_gpqa_tasks(registry)[source]

Register gpqa benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_gsm8k_ellamind_tasks(registry)[source]

Register gsm8k_ellamind benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_gsm8k_tasks(registry)[source]

Register gsm8k benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_hellaswag_ellamind_tasks(registry)[source]

Register hellaswag_ellamind benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_hellaswag_tasks(registry)[source]

Register hellaswag benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_hendrycks_math_ellamind_tasks(registry)[source]

Register hendrycks_math_ellamind benchmark tasks (composed).

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_hle_ellamind_tasks(registry)[source]

Register hle_ellamind benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_humaneval_ellamind_tasks(registry)[source]

Register humaneval_ellamind benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_humaneval_plus_tasks(registry)[source]

Register humaneval_plus benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_humaneval_tasks(registry)[source]

Register humaneval benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_ifeval_tasks(registry)[source]

Register ifeval benchmark tasks (composed: IFEval, IFEvalDe).

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_lbpp_tasks(registry)[source]

Register lbpp benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_math_reasoning_tasks(registry)[source]

Register math_reasoning benchmark tasks (all composed: MATH500(_V2), AIME, GSM8KReasoning, Minerva-OLMES).

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_mbpp_ellamind_tasks(registry)[source]

Register mbpp_ellamind benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_mbpp_tasks(registry)[source]

Register mbpp benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_mmlu_pro_tasks(registry)[source]

Register mmlu_pro benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_mmlu_prox_tasks(registry)[source]

Register mmlu_prox benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_mmlu_tasks(registry)[source]

Register mmlu benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_multipl_e_tasks(registry)[source]

Register multipl_e benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_naturalqs_open_tasks(registry)[source]

Register naturalqs_open benchmark tasks (composed: NaturalQsOpen, NaturalQsOpenMC_OLMES).

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_piqa_ellamind_tasks(registry)[source]

Register piqa_ellamind benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_piqa_tasks(registry)[source]

Register piqa benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_sciq_tasks(registry)[source]

Register sciq benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_simpleqa_ellamind_tasks(registry)[source]

Register simpleqa_ellamind benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_siqa_ellamind_tasks(registry)[source]

Register siqa_ellamind benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_social_iqa_tasks(registry)[source]

Register social_iqa benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_squad_tasks(registry)[source]

Register squad benchmark tasks (composed: SQuAD_OLMES, SQuAD2_MA, SQuAD2_MA_NO_SYSPROMPT).

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_winogrande_ellamind_tasks(registry)[source]

Register winogrande_ellamind benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_names.register_winogrande_tasks(registry)[source]

Register winogrande benchmark tasks.

Return type:

None

Parameters:

registry (Registry)

eval_framework.tasks.task_style module

Task-style helpers and strategy classes for choice-based evaluation tasks.

This module provides injectable styling strategies — MCStyle, ClozeStyle, MCCompletionStyle and BPBStyle — that render one choice item (question plus answer options) into the prompt text, answer cue, scored candidates and ground truth. The styler also declares the response type, metrics and metadata of its style.

Quick-start

A choice-based benchmark pairs a ChoiceReader (item -> raw question, choices, correct index) with a styler via ComposedBenchmark.choice, which drives both the scored sample and the few-shot demonstrations from the same pair:

benchmark = ComposedBenchmark.choice(
    id="MyBenchmark",
    reader=MyChoiceReader(),
    styler=ClozeStyle(question_prefix="Question: ", cue_text="Answer:"),
    sample_split="test",
    fewshot_split="train",
    dataset_policy=Pinned("my/dataset"),
)

Task families with MC, Cloze and BPB variants share the reader and differ only in the styler each variant’s builder injects.

class eval_framework.tasks.task_style.BPBStyle(question_prefix='Question: ', cue_text='Answer:', trailing_newline=True, leading_space_continuations=True, initial_prompt=None)[source]

Bases: ClozeStyle

BPB-only styler: prompt identical to ClozeStyle, but scores only the ground-truth completion.

One LLM forward pass per sample instead of N (one per choice), making evaluation significantly faster when accuracy metrics are not needed.

Parameters:
  • question_prefix (str) – Prepended to the raw question (default "Question: ").

  • cue_text (str) – Assistant cue after the prompt (default "Answer:").

  • trailing_newline (bool) – When True (default), the instruction ends with "\n".

  • leading_space_continuations (bool)

  • initial_prompt (Callable[[str], str] | None)

Assembled prompt example (3 choices):

"Question: What is the capital of France?\n"

Scored completions: [" Paris"]  ← ground truth only, one forward pass
Ground truth:        " Paris"
get_possible_completions(choices, correct_index=None)[source]

Return completion candidates for scoring, or None for free generation tasks.

correct_index is only required by BPBStyle, which scores solely the ground-truth completion. MCStyle and ClozeStyle score all choices and ignore it; callers may omit it when using those stylers.

Return type:

list[str]

Parameters:
  • choices (list[str])

  • correct_index (int | None)

metrics: list[type[BaseMetric]] = [<class 'eval_framework.metrics.loglikelihood.bits_per_byte.BitsPerByteLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.bpb_variants.BitsPerByteVariantsLoglikelihood'>]
task_style: TaskStyle = 'bpb'
class eval_framework.tasks.task_style.ClozeStyle(question_prefix='Question: ', cue_text='Answer:', trailing_newline=True, leading_space_continuations=True, initial_prompt=None)[source]

Bases: TaskStyler

Cloze styler: no choices in prompt, model scored over full choice text.

Also known as “ranked classification” (RC). The prompt only shows the question; the model’s score for each full answer text determines the prediction.

Parameters:
  • question_prefix (str) – Prepended to the raw question (default "Question: ").

  • cue_text (str) – Assistant cue after the prompt (default "Answer:").

  • trailing_newline (bool) – When True (default), the instruction ends with "\n". Set to False for sentence-completion tasks where the model should continue a fragment directly.

  • initial_prompt (Callable[[str], str] | None) – Optional subject-templated preamble prepended once at the top of the prompt. Default None (no preamble).

  • leading_space_continuations (bool)

Assembled prompt example (3 choices):

"Question: What is the capital of France?\n"

Scored completions: [" Berlin", " Paris", " London"]
Ground truth: " Paris"

Sentence-completion example (trailing_newline=False, cue_text=””):

"The cat sat on the"

Scored completions: [" mat", " floor", " sofa"]
Ground truth: " mat"
get_cue_text()[source]

Return the assistant cue appended after the prompt (e.g. "Answer:").

Return type:

str

get_ground_truth(choices, correct_index)[source]

Return the ground-truth string for scoring.

Return type:

str

Parameters:
  • choices (list[str])

  • correct_index (int)

get_instruction_text(raw_question, choices)[source]

Build the instruction/prompt text from a question and answer choices.

Return type:

str

Parameters:
  • raw_question (str)

  • choices (list[str])

get_possible_completions(choices, correct_index=None)[source]

Return completion candidates for scoring, or None for free generation tasks.

correct_index is only required by BPBStyle, which scores solely the ground-truth completion. MCStyle and ClozeStyle score all choices and ignore it; callers may omit it when using those stylers.

Return type:

list[str]

Parameters:
  • choices (list[str])

  • correct_index (int | None)

initial_prompt(subject_label)[source]

A preamble prepended once at the top of the prompt for the given subject, or None (the default: no preamble). Subject-templated preambles (e.g. MMLU’s “… about {subject}.”) read subject_label; fixed preambles ignore it.

Return type:

str | None

Parameters:

subject_label (str)

metrics: list[type[BaseMetric]] = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyBayesianLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.bits_per_byte.BitsPerByteLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.bpb_variants.BitsPerByteVariantsLoglikelihood'>]
response_type: ResponseType = 'loglikelihoods'
task_style: TaskStyle = 'cloze'
class eval_framework.tasks.task_style.IdkStyle(inner, abstention_option)[source]

Bases: TaskStyler

Adds an explicit abstention option (e.g. “I do not know”, “?”) to a base styler’s candidates.

Use this when a benchmark rewards calibrated abstention over confident guessing: it scores the real answers together with the abstention option and reports confidence-aware metrics. Everything else — the prompt formatting and the abstention preamble (via the inner styler’s initial_prompt) — is the inner styler’s; wrap any base styler, so both multiple-choice (abstain with an extra letter) and cloze (abstain with an extra full answer) tasks are supported.

Parameters:
  • inner (TaskStyler) – The base styler doing the prompt formatting, preamble, and real-answer scoring.

  • abstention_option (str) – The abstention completion scored alongside the real choices, e.g. " I do not know" (cloze) or " ?" (multiple-choice).

get_cue_text()[source]

Return the assistant cue appended after the prompt (e.g. "Answer:").

Return type:

str

get_ground_truth(choices, correct_index)[source]

Return the ground-truth string for scoring.

Return type:

str

Parameters:
  • choices (list[str])

  • correct_index (int)

get_instruction_text(raw_question, choices)[source]

Build the instruction/prompt text from a question and answer choices.

Return type:

str

Parameters:
  • raw_question (str)

  • choices (list[str])

get_possible_completions(choices, correct_index=None)[source]

Return completion candidates for scoring, or None for free generation tasks.

correct_index is only required by BPBStyle, which scores solely the ground-truth completion. MCStyle and ClozeStyle score all choices and ignore it; callers may omit it when using those stylers.

Return type:

list[str]

Parameters:
  • choices (list[str])

  • correct_index (int | None)

initial_prompt(subject_label)[source]

A preamble prepended once at the top of the prompt for the given subject, or None (the default: no preamble). Subject-templated preambles (e.g. MMLU’s “… about {subject}.”) read subject_label; fixed preambles ignore it.

Return type:

str | None

Parameters:

subject_label (str)

metrics: list[type[BaseMetric]] = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyBayesianLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.confidence_weighted_accuracy.ConfidenceWeightedAccuracy'>, <class 'eval_framework.metrics.loglikelihood.dcs.DistributionalCorrectnessScore'>, <class 'eval_framework.metrics.loglikelihood.ternary.TernaryScore'>]
response_type: ResponseType = 'loglikelihoods'
class eval_framework.tasks.task_style.Language(*values)[source]

Bases: Enum

A language the stylers have default question/answer wording for.

Members are named by ISO 639-3 code and valued by English name

CES = 'Czech'
DAN = 'Danish'
DEU = 'German'
ENG = 'English'
FIN = 'Finnish'
FRA = 'French'
HUN = 'Hungarian'
ITA = 'Italian'
NLD = 'Dutch'
POL = 'Polish'
POR = 'Portuguese'
RUS = 'Russian'
SPA = 'Spanish'
UKR = 'Ukrainian'
class eval_framework.tasks.task_style.MCCompletionStyle(question_prefix='Question: ', cue_text='Answer:', space_prefixed_labels=False)[source]

Bases: TaskStyler

Multiple-choice prompt style with generative completion output.

Prompt formatting is identical to MCStyle (choices shown in prompt), but ground truth is the full answer text (for example "Paris" instead of "D"). Responses are generated via ResponseType.COMPLETION and scored with AccuracyCompletion.

This style is useful for instruction-following or CoT prompts where the model is expected to generate an answer text rather than be scored via loglikelihood over fixed label candidates.

Parameters:
  • question_prefix (str) – Prepended to the raw question (default "Question: ").

  • cue_text (str) – Assistant cue after the prompt (default "Answer:").

  • space_prefixed_labels (bool) – When True, each option line starts with a space (" A. choice" — OLMES-style). Default False.

Assembled prompt example (default settings, 4 choices):

"Question: What is the capital of France?\nA. Munich\nB. Stuttgart\nC. Berlin\nD. Paris\nAnswer:"

Ground truth: "Paris"
get_cue_text()[source]

Return the assistant cue appended after the prompt (e.g. "Answer:").

Return type:

str

get_fewshot_target_text(choices, correct_index)[source]

Override to insert a space between cue and answer text.

Return type:

str

Parameters:
  • choices (list[str])

  • correct_index (int)

get_ground_truth(choices, correct_index)[source]

Return the ground-truth string for scoring.

Return type:

str

Parameters:
  • choices (list[str])

  • correct_index (int)

get_instruction_text(raw_question, choices)[source]

Build the instruction/prompt text from a question and answer choices.

Return type:

str

Parameters:
  • raw_question (str)

  • choices (list[str])

get_possible_completions(choices, correct_index=None)[source]

Completion-mode tasks do not provide fixed candidate completions.

Return type:

list[str] | None

Parameters:
  • choices (list[str])

  • correct_index (int | None)

metrics: list[type[BaseMetric]] = [<class 'eval_framework.metrics.completion.accuracy_completion.AccuracyCompletion'>]
response_type: ResponseType = 'completion'
task_style: TaskStyle = 'multiple_choice'
class eval_framework.tasks.task_style.MCStyle(question_prefix='Question: ', cue_text='Answer:', space_prefixed_labels=False, initial_prompt=None)[source]

Bases: TaskStyler

Multiple-choice styler: choices shown in prompt, model scored over letter labels.

Parameters:
  • question_prefix (str) – Prepended to the raw question (default "Question: ").

  • cue_text (str) – Assistant cue after the prompt (default "Answer:").

  • space_prefixed_labels (bool) – When True, each option line starts with a space (" A. choice" — OLMES-style). Default False.

  • initial_prompt (Callable[[str], str] | None) – Optional subject-templated preamble prepended once at the top of the prompt (e.g. MMLU’s “… about {subject}.”). Default None (no preamble).

Assembled prompt example (default settings, 3 choices):

"Question: What is the capital of France?\nA. Berlin\nB. Paris\nC. London\n"

Scored completions: [" A", " B", " C"]
Ground truth:  " B"
get_cue_text()[source]

Return the assistant cue appended after the prompt (e.g. "Answer:").

Return type:

str

get_ground_truth(choices, correct_index)[source]

Return the ground-truth string for scoring.

Return type:

str

Parameters:
  • choices (list[str])

  • correct_index (int)

get_instruction_text(raw_question, choices)[source]

Build the instruction/prompt text from a question and answer choices.

Return type:

str

Parameters:
  • raw_question (str)

  • choices (list[str])

get_possible_completions(choices, correct_index=None)[source]

Note: correct_index is ignored for MCStyle and only used for BPBStyle.

Return type:

list[str]

Parameters:
  • choices (list[str])

  • correct_index (int | None)

initial_prompt(subject_label)[source]

A preamble prepended once at the top of the prompt for the given subject, or None (the default: no preamble). Subject-templated preambles (e.g. MMLU’s “… about {subject}.”) read subject_label; fixed preambles ignore it.

Return type:

str | None

Parameters:

subject_label (str)

metrics: list[type[BaseMetric]] = [<class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyNormLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.accuracy_loglikelihood.AccuracyBayesianLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.bits_per_byte.BitsPerByteLoglikelihood'>, <class 'eval_framework.metrics.loglikelihood.bpb_variants.BitsPerByteVariantsLoglikelihood'>]
response_type: ResponseType = 'loglikelihoods'
task_style: TaskStyle = 'multiple_choice'
class eval_framework.tasks.task_style.TaskStyle(*values)[source]

Bases: Enum

BPB = 'bpb'
CLOZE = 'cloze'
MULTIPLE_CHOICE = 'multiple_choice'
class eval_framework.tasks.task_style.TaskStyler[source]

Bases: ABC

Strategy object that controls prompt assembly and scoring for choice-based tasks.

Concrete implementations (MCStyle, ClozeStyle) are injected into the Choice eval kind alongside a ChoiceReader, so benchmark authors only supply data access.

response_type

The response type the task should use (e.g. LOGLIKELIHOODS).

metrics

Default metric classes for tasks using this styler.

task_style

Discriminator for metadata (MULTIPLE_CHOICE or CLOZE).

question_prefix

String prepended to the raw question.

classmethod for_language(language, **kwargs)[source]

Factory that fills question_prefix and cue_text from language defaults.

Any explicitly passed keyword arguments take precedence over the defaults.

Return type:

Self

Parameters:
abstractmethod get_cue_text()[source]

Return the assistant cue appended after the prompt (e.g. "Answer:").

Return type:

str

get_extra_metadata()[source]

Return styler-specific metadata to merge into the task’s metadata.

Return type:

dict

get_fewshot_target_text(choices, correct_index)[source]

Return the few-shot target string (cue + ground truth).

Return type:

str

Parameters:
  • choices (list[str])

  • correct_index (int)

abstractmethod get_ground_truth(choices, correct_index)[source]

Return the ground-truth string for scoring.

Return type:

str

Parameters:
  • choices (list[str])

  • correct_index (int)

abstractmethod get_instruction_text(raw_question, choices)[source]

Build the instruction/prompt text from a question and answer choices.

Return type:

str

Parameters:
  • raw_question (str)

  • choices (list[str])

abstractmethod get_possible_completions(choices, correct_index=None)[source]

Return completion candidates for scoring, or None for free generation tasks.

correct_index is only required by BPBStyle, which scores solely the ground-truth completion. MCStyle and ClozeStyle score all choices and ignore it; callers may omit it when using those stylers.

Return type:

list[str] | None

Parameters:
  • choices (list[str])

  • correct_index (int | None)

get_question_text(raw_question)[source]

Build the full question line (prefix + raw question).

Override in a subclass for non-standard question formats (e.g. HellaSwag’s "activity: context" form).

Return type:

str

Parameters:

raw_question (str)

initial_prompt(subject_label)[source]

A preamble prepended once at the top of the prompt for the given subject, or None (the default: no preamble). Subject-templated preambles (e.g. MMLU’s “… about {subject}.”) read subject_label; fixed preambles ignore it.

Return type:

str | None

Parameters:

subject_label (str)

metrics: list[type[BaseMetric]]
question_prefix: str
response_type: ResponseType
task_style: TaskStyle
final with_abstention_option(abstention_option)[source]

Wrap this styler so the model may also abstain with abstention_option (scored alongside the real answers, with confidence-aware metrics). See IdkStyle.

Return type:

IdkStyle

Parameters:

abstention_option (str)

eval_framework.tasks.task_style.answer_key_to_index(key)[source]

Convert a letter or 1-based integer answer key to a 0-based index.

Datasets sometimes encode the correct answer as a letter (“A”, “B”, …) and sometimes as a 1-based integer string (“1”, “2”, …). This function normalises both to a 0-based index so task code doesn’t need to branch.

Parameters:

key (str) – A single-character string. Either a letter (“A”-“Z”) or a digit (“1”-“9”).

Returns:

“A” or “1” → 0, “B” or “2” → 1, etc.

Return type:

0-based index

eval_framework.tasks.task_style.format_mc_prompt(question_text, choices, *, space_prefixed_labels=False)[source]

Helper function to format a question and its labeled choices into a multiple-choice prompt.

The choices are labeled A, B, C, … in order and ends with a newline.

Parameters:
  • question_text (str) – The full question string (prefix already included).

  • choices (list[str]) – Ordered list of answer option strings.

  • space_prefixed_labels (bool) – When True, each option line is prefixed with a space — “ A. choice” instead of “A. choice”. This is, e.g., the OLMES-style prompt format.

Return type:

str

Returns:

A string of the form "<question_text>\n[pfx]A. choice0\n[pfx]B. choice1\n".

Examples:

>>> format_mc_prompt("Question: What is 1+1?", ["1", "2", "3"])
'Question: What is 1+1?\nA. 1\nB. 2\nC. 3\n'
>>> format_mc_prompt("Question: What is 1+1?", ["1", "2"], space_prefixed_labels=True)
'Question: What is 1+1?\n A. 1\n B. 2\n'
eval_framework.tasks.task_style.question_cue_text(language)[source]
Return type:

tuple[str, str]

Parameters:

language (Language)

eval_framework.tasks.task_style.shuffle_correct_with_distractors(correct, distractors, seed_text)[source]

Deterministically shuffle distractors + correct answer; return (choices, correct_index).

Calling this multiple times with the same arguments always returns the same result.

Parameters:
  • correct (str) – The correct answer string.

  • distractors (list[str]) – The distractor strings, e.g. wrong answer choices.

  • seed_text (str) – Text used as the shuffle seed (typically question + answer).

Return type:

tuple[list[str], int]

Returns:

A tuple (shuffled_choices, correct_index) where correct_index is the 0-based position of correct in shuffled_choices.

eval_framework.tasks.utils module

class eval_framework.tasks.utils.CallableSerializer[source]

Bases: object

static decode(fn_str)[source]
Return type:

Callable[..., Any]

Parameters:

fn_str (str)

static encode(fn)[source]
Return type:

str

Parameters:

fn (Callable[[...], Any])

exception eval_framework.tasks.utils.DecodingError[source]

Bases: SerializationError

Raised when decoding a callable fails.

exception eval_framework.tasks.utils.DockerReturnedEmptyOutput[source]

Bases: Exception

Raised when the docker returns an empty output.

exception eval_framework.tasks.utils.EncodingError[source]

Bases: SerializationError

Raised when encoding a callable fails.

class eval_framework.tasks.utils.ExecutionResult(success, output)[source]

Bases: NamedTuple

A named tuple to store the result of code execution.

Parameters:
  • success (bool)

  • output (str)

success

Indicates if the execution was successful.

Type:

bool

output

Contains the output or error messages from the execution.

Type:

str

output: str

Alias for field number 1

success: bool

Alias for field number 0

exception eval_framework.tasks.utils.SerializationError[source]

Bases: Exception

Base exception for callable serialization errors.

eval_framework.tasks.utils.close_pools()[source]
Return type:

None

eval_framework.tasks.utils.execute_python_code_with_tests(code, test_code, dockerfile, package_mapping, merge_code_fn, image, timeout, parse_output_fn)[source]

Executes the given code with test cases in a sandboxed environment.

Parameters:
  • code (str) – The code to be tested.

  • test_code (str) – The test cases to run against the code.

  • package_mapping (dict[str, str | None]) – Mapping of package names to install commands.

  • merge_code_fn (Callable[[str, str], str]) – function to merge LLM and test code

  • image (str | None) – Docker image to use.

  • timeout (int) – Timeout for the execution in seconds.

  • parse_otuput_fn – function to parse docker execution output

  • dockerfile (str | None)

  • parse_output_fn (Callable[[str], ExecutionResult])

Return type:

ExecutionResult

Returns:

An ExecutionResult named tuple with success status and output or errors.

eval_framework.tasks.utils.extract_imports(code)[source]

Extract all import statements and the imported packages from code.

Return type:

tuple[list[str], set[str]]

Parameters:

code (str)

eval_framework.tasks.utils.extract_python_code_from_response(llm_response)[source]

Extract the Python source from a chat-style LLM response that wraps its answer in markdown code fences.

The response is expected to contain a single Python solution (e.g. HumanEval, MBPP, BigCodeBench). The first matching fenced block is returned, tried in this order: a `python block nested inside a ```markdown block, a plain ```python block, a ```markdown block, and finally any ` block. If the response contains no fences at all, it is assumed to be raw code and returned unchanged.

Return type:

str

Parameters:

llm_response (str)

eval_framework.tasks.utils.get_docker_address()[source]
Return type:

str

eval_framework.tasks.utils.get_external_dependencies(code, package_mapping)[source]

Identify external dependencies in the code.

Return type:

list[str]

Parameters:
  • code (str)

  • package_mapping (dict[str, str | None])

eval_framework.tasks.utils.get_n_letters(n)[source]
Return type:

list[str]

Parameters:

n (int)

eval_framework.tasks.utils.get_or_create_pool(image=None, dockerfile=None, packages=None, lang='python', min_pool_size=1, max_pool_size=1, max_container_uses=100, runtime_configs=None)[source]
Return type:

ContainerPoolManager

Parameters:
  • image (str | None)

  • dockerfile (str | None)

  • packages (list[str] | None)

  • lang (str)

  • min_pool_size (int)

  • max_pool_size (int)

  • max_container_uses (int)

  • runtime_configs (dict[str, str] | None)

eval_framework.tasks.utils.run_python_code(code, image=None, dockerfile=None, input_files=None, timeout=60, packages=None, runtime_configs=None)[source]

Run code in a sandboxed environment. :type code: str :param code: The code to run. :type image: str | None :param image: Docker image to use. :type dockerfile: str | None :param dockerfile: Dockerfile to use. :type input_files: list[tuple[str, str]] | None :param input_files: pairs of host and docker paths, host files will be copied to the docker. :type timeout: int :param timeout: Timeout in seconds, 0 if no timeout. :type packages: list[str] | None :param packages: List of python packages to install with pip. :rtype: str :return: The output of the code.

Parameters:
  • code (str)

  • image (str | None)

  • dockerfile (str | None)

  • input_files (list[tuple[str, str]] | None)

  • timeout (int)

  • packages (list[str] | None)

  • runtime_configs (dict[str, str] | None)

Return type:

str

eval_framework.tasks.utils.unittest_merge_snippets(code, test_code)[source]
Return type:

str

Parameters:
  • code (str)

  • test_code (str)

Module contents