Eval-Framework Documentation¶
A suite of curated benchmarks for evaluating **L**arge **L**anguage **M**odels (LLMs)
Benchmarks in this suite are:
Validated: Benchmarks implemented in this suite are required to have reproduced published results
Legal: The data is legal to exist and to use. Some datasets are subject to third party licenses
Comprehensive: For a list of supported benchmarks execute the CLI tool or check the benchmarks source directory. https://github.com/Aleph-Alpha-Research/eval-framework/tree/main/src/eval_framework/benchmarks
Quick Start¶
Running a benchmark locally.
Clone the repository
git clone https://github.com/Aleph-Alpha-Research/eval-framework/tree/main
cd eval-framework
Setup the Python environment. We recommend using uv https://docs.astral.sh/uv/getting-started/installation/:
uv sync
By default Open AIs API is used. You can explicitly specify an OpenAI compatible endpoint by
passing base_url=http://host:port/v1 together with the –llm-args options. Some metrics make
use of the additional information provided by vLLM and won’t work with OpenAI. In this exapmle we
stick with the default. Therfore the OPENAI_API_KEY environment variable needs to contain an API
token.
uv run eval_framework \
--llm-args model_name="gpt-4o-mini" \
--task-name "MMLU" \
--task-subjects "abstract_algebra" \
--output-dir ./eval_results \
--num-fewshot 5 \
--num-samples 10
Documentation Overview¶
Getting Started
User Guides
Testing Guidelines
API Reference
Citation & License¶
If you use eval-framework in research:
@software{eval_framework,
title={Aleph Alpha Eval Framework},
year={2025},
url={https://github.com/Aleph-Alpha-Research/eval-framework}
}
Licensed under the [Apache License 2.0](LICENSE).
Funding¶
This project has received funding from the European Union’s Digital Europe Programme under grant agreement No. 101195233 (OpenEuroLLM). The contents of this publication are the sole responsibility of the OpenEuroLLM consortium.