Eval-Framework Documentation

A suite of curated benchmarks for evaluating **L**arge **L**anguage **M**odels (LLMs)

Benchmarks in this suite are:

Quick Start

Running a benchmark locally.

Clone the repository

git clone https://github.com/Aleph-Alpha-Research/eval-framework/tree/main
cd eval-framework

Setup the Python environment. We recommend using uv https://docs.astral.sh/uv/getting-started/installation/:

uv sync

By default Open AIs API is used. You can explicitly specify an OpenAI compatible endpoint by passing base_url=http://host:port/v1 together with the –llm-args options. Some metrics make use of the additional information provided by vLLM and won’t work with OpenAI. In this exapmle we stick with the default. Therfore the OPENAI_API_KEY environment variable needs to contain an API token.

uv run eval_framework \
     --llm-args model_name="gpt-4o-mini" \
     --task-name "MMLU" \
     --task-subjects "abstract_algebra" \
     --output-dir ./eval_results \
     --num-fewshot 5 \
     --num-samples 10

Documentation Overview

Getting Started

Testing Guidelines

API Reference

Citation & License

If you use eval-framework in research:

@software{eval_framework,
  title={Aleph Alpha Eval Framework},
  year={2025},
  url={https://github.com/Aleph-Alpha-Research/eval-framework}
}

Licensed under the [Apache License 2.0](LICENSE).

Funding

This project has received funding from the European Union’s Digital Europe Programme under grant agreement No. 101195233 (OpenEuroLLM). The contents of this publication are the sole responsibility of the OpenEuroLLM consortium.