Installation

This guide provides detailed installation instructions and dependency information for the eval-framework.

1. Install uv

Follow the official installation instructions.

2. Install Eval Framework

Clone the repository and install all dependencies:

# Clone the repository
git clone https://github.com/Aleph-Alpha-Research/eval-framework/tree/main
cd eval-framework

# Install all dependencies
uv sync

3. Test Your Installation

The example below talks to the OpenAI API and needs OPENAI_API_KEY to be set. Any OpenAI-compatible server (for example vLLM) can be targeted by adding base_url=http://host:port/v1 to --llm-args.

uv run eval_framework \
    --llm-args model_name="gpt-4o-mini" \
    --task-name "MMLU" \
    --task-subjects "abstract_algebra" "anatomy" \
    --output-dir ./eval_results \
    --num-fewshot 5 \
    --num-samples 10

Environment Configuration

Environment Variables

Create a .env file in the project root:

# API Keys (if using external models)
HF_TOKEN="your_huggingface_token"        # For private HuggingFace models
OPENAI_API_KEY="your_openai_key"         # For OpenAI models as judges

Docker Installation

Available Dockerfiles

Dockerfile

Purpose

src/eval_framework/tasks/Dockerfile_codebench

Specialized for BigCodeBench coding tasks

Build from Source

Main Evaluation Framework

# Build main image (uses Dockerfile)
docker build -t eval-framework .

# Run with GPU support
docker run -it --gpus all -v $(pwd):/workspace eval-framework

Specialized Builds

# BigCodeBench coding tasks (build context must be the tasks directory so COPY utils.py resolves)
docker build -f src/eval_framework/tasks/Dockerfile_codebench -t eval-framework-codebench src/eval_framework/tasks/

PyPI installation

It is also possible to download Eval-Framework through pip:

pip install eval-framework

However, we recommend using the uv solver to avoid many dependency version issues, so:

uv pip install eval-framework

This allows you to run an evaluation without going through uv run:

eval_framework \
    --llm-args model_name="gpt-4o-mini" \
    --task-name "MMLU" \
    --task-subjects "abstract_algebra" "anatomy" \
    --output-dir ./eval_results \
    --num-fewshot 5 \
    --num-samples 10