eval_frameworkΒΆ
- eval_framework package
- Subpackages
- eval_framework.benchmarks package
- Submodules
- eval_framework.benchmarks.arc module
- eval_framework.benchmarks.arc_de module
- eval_framework.benchmarks.arc_ellamind module
- eval_framework.benchmarks.bigcodebench module
- eval_framework.benchmarks.copa module
- eval_framework.benchmarks.cot module
- eval_framework.benchmarks.csqa module
- eval_framework.benchmarks.csqa_ellamind module
- eval_framework.benchmarks.drop module
- eval_framework.benchmarks.global_mmlu module
- eval_framework.benchmarks.goldenswag module
- eval_framework.benchmarks.gpqa module
- eval_framework.benchmarks.gpqa_ellamind module
- eval_framework.benchmarks.gsm8k module
- eval_framework.benchmarks.gsm8k_ellamind module
- eval_framework.benchmarks.hellaswag module
- eval_framework.benchmarks.hellaswag_ellamind module
- eval_framework.benchmarks.hendrycks_math_ellamind module
- eval_framework.benchmarks.hle_ellamind module
- eval_framework.benchmarks.humaneval module
- eval_framework.benchmarks.humaneval_ellamind module
- eval_framework.benchmarks.humaneval_plus module
- eval_framework.benchmarks.ifeval module
- eval_framework.benchmarks.math_reasoning module
- eval_framework.benchmarks.mbpp module
- eval_framework.benchmarks.mbpp_ellamind module
- eval_framework.benchmarks.mmlu module
- eval_framework.benchmarks.mmlu_pro module
- eval_framework.benchmarks.multipl_e module
- eval_framework.benchmarks.naturalqs_open module
- eval_framework.benchmarks.piqa module
- eval_framework.benchmarks.piqa_ellamind module
- eval_framework.benchmarks.sciq module
- eval_framework.benchmarks.simpleqa_ellamind module
- eval_framework.benchmarks.siqa_ellamind module
- eval_framework.benchmarks.social_iqa module
- eval_framework.benchmarks.squad module
- eval_framework.benchmarks.winogrande module
- eval_framework.benchmarks.winogrande_ellamind module
- Module contents
- eval_framework.context package
- eval_framework.llm package
- eval_framework.metrics package
- eval_framework.result_processors package
- eval_framework.tasks package
- Subpackages
- Submodules
- eval_framework.tasks.base module
- eval_framework.tasks.dataset_loading module
- eval_framework.tasks.dataset_revisions module
- eval_framework.tasks.eval_config module
- eval_framework.tasks.lazy module
- eval_framework.tasks.markdown_doc module
- eval_framework.tasks.registry module
- eval_framework.tasks.task_loader module
- eval_framework.tasks.task_names module
- eval_framework.tasks.task_style module
- eval_framework.tasks.utils module
- Module contents
- eval_framework.benchmarks package
- Submodules
- eval_framework.answer module
- eval_framework.base_config module
- eval_framework.choices module
- eval_framework.composed module
- eval_framework.contract module
- eval_framework.eval_kind module
- eval_framework.evaluation_generator module
- eval_framework.exceptions module
- eval_framework.fewshot module
- eval_framework.logger module
- eval_framework.main module
- eval_framework.response_generator module
- eval_framework.run module
- eval_framework.run_direct module
- eval_framework.subjects module
- eval_framework.suite module
MetricSourceSuiteAggregateSuiteResultTaskSuiteTaskSuite.aggregatesTaskSuite.batch_sizeTaskSuite.extra_llm_argsTaskSuite.get_hyperparam_overrides()TaskSuite.hf_revisionTaskSuite.is_leafTaskSuite.load()TaskSuite.load_from_py()TaskSuite.load_from_yaml()TaskSuite.max_tokensTaskSuite.model_configTaskSuite.nameTaskSuite.num_fewshotTaskSuite.num_samplesTaskSuite.repeatsTaskSuite.task_nameTaskSuite.task_subjectsTaskSuite.tasksTaskSuite.temperatureTaskSuite.top_kTaskSuite.top_pTaskSuite.validate_suite()
compute_aggregates()parse_strings_to_task_or_suite()resolve_to_evalconfig_kwargs()run_suite()save_suite_results()
- Module contents
- Subpackages