eval_framework.benchmarks.utils package

Submodules

eval_framework.benchmarks.utils.ellamind_languages module

class eval_framework.benchmarks.utils.ellamind_languages.EllamindLanguage(language, config)[source]

Bases: object

A language of the EllaMind datasets: the framework language and the dataset config holding it.

Parameters:
config: str
language: Language
eval_framework.benchmarks.utils.ellamind_languages.ellamind_language(code)[source]

The language and dataset config of a language token, e.g. ellamind_language("DE").

Return type:

EllamindLanguage

Parameters:

code (Literal['CS', 'DA', 'DE', 'ES', 'FI', 'FR', 'IT', 'NL', 'PL'])

eval_framework.benchmarks.utils.global_mmlu_languages module

Localized prompt wording for Global-MMLU: one GlobalMMLULocale per language config.

class eval_framework.benchmarks.utils.global_mmlu_languages.GlobalMMLULocale(*, subject_names, preamble_template, question_label, answer_label)[source]

Bases: object

One language of Global-MMLU: the subject names, the preamble and the question/answer labels.

Parameters:
  • subject_names (dict[str, str])

  • preamble_template (str)

  • question_label (str)

  • answer_label (str)

answer_label: str
preamble(subject)[source]
Return type:

str

Parameters:

subject (str)

preamble_template: str
question_label: str
subject_names: dict[str, str]

Module contents