Using the CLI

The Eval-Framework CLI provides a flexible interface for evaluating LLMs across a wide range of benchmarks. Whether you’re running evaluations locally or in a distributed environment, the CLI allows you to configure tasks, models, and metrics with ease.

Command Structure

uv run eval_framework [OPTIONS]

Model Selection

The evaluated model is always eval_framework.llm.openai.OpenAIModel. Configure it through --llm-args, which must include model_name (e.g. --llm-args model_name="gpt-4o-mini" base_url="http://localhost:8000/v1").

Execution Configuration

--llm-args [LLM_ARGS ...] Arguments to pass to the OpenAIModel constructor as key=value pairs. model_name is required.

--task-name TASK_NAME The name of the task to evaluate.

--output-dir OUTPUT_DIR The path for evaluation outputs.

--num-samples NUM_SAMPLES The number of samples per subject to evaluate.

--num-fewshot NUM_FEWSHOT The number of fewshot examples to use.

--max-tokens The maximum number of tokens to generate for each sample. Overwrites any task default value.

--batch-size BATCH_SIZE Size of batch of samples to send to the LLM for evaluation in parallel. Use 1 for sequential running (default).

--task-suite PATH_TO_TASK_SUITE_FILE Path to a suite file (.yaml or .py) defining a set of tasks to run. Mutually exclusive with --task-name. Hyperparameter overrides and score aggregation methods can be configured within the suite file.

Task Configuration

--task-subjects TASK_SUBJECTS [TASK_SUBJECTS ...] The subjects of the task to evaluate. If empty, all subjects are evaluated. Subjects in the form of tuples can be specified in a comma-delimited way, possibly using wildcard * in some dimensions of a tuple. Examples: "DE_DE, *" or "FR_FR, astronomy"

--hf-revision HF_REVISION A tag name, a branch name, or commit hash for the task HF dataset.

Judge Models

The LLM judge is always an eval_framework.llm.openai.OpenAIModel.

--judge-model-args JUDGE_MODEL_ARGS Arguments to pass to the judge OpenAIModel constructor as key=value pairs; model_name is required (e.g. model_name="gpt-4o"). Setting this enables the LLM judge, which is required for tasks scored with LLM-judge metrics.

Logging & Tracking

--wandb-project WANDB_PROJECT The name of the Weights & Biases project to log runs to.

--wandb-entity WANDB_ENTITY The name of the Weights & Biases entity to log runs to. Defaults to the user’s default entity.

--wandb-run-id WANDB_RUN_ID The ID of an existing Weights & Biases run to resume. If not given, creates a new run. If given and exists, will continue the run but will overwrite the python command logged in WandB.

--wandb-upload-results or --no-wandb-upload-results Whether to upload results as an artifact to Weights & Biases (default: True). Needs --wandb-project to be set.

--description DESCRIPTION Description of the run. This will be added to the metadata of the run to help with bookkeeping.

--verbosity VERBOSITY Set the logging verbosity level: 0=CRITICAL, 1=INFO, 2=DEBUG.

-h, --help Show help message and exit.

Running Models Behind an OpenAI-Compatible Endpoint

Any model served through an OpenAI-compatible API (the OpenAI API itself, vLLM, TGI, …) can be evaluated with the OpenAIModel class. The API key is read from OPENAI_API_KEY; pass base_url to target a self-hosted server:

uv run eval_framework \
    --llm-args model_name="Qwen/Qwen3-0.6B" base_url="http://localhost:8000/v1" \
    --task-name "MMLU" \
    --task-subjects "abstract_algebra" \
    --output-dir ./eval_results \
    --num-fewshot 5 \
    --num-samples 10

Every keyword argument of the OpenAIModel constructor can be set through --llm-args; see the Model Arguments guide for the full list.

Running a Task Suite

Task suites let you run multiple tasks in one command and aggregate their scores. You can define a suite in a .yaml file:

# my_suite.yaml
name: my_suite
tasks:
  - tasks: MMLU
    num_samples: 100
    num_fewshot: 5
  - tasks: GSM8K
    num_samples: 50
    num_fewshot: 0
aggregates:
  - name: avg_accuracy
    metric: accuracy
    method: mean

Or equivalently in a .py file:

# my_suite.py
from eval_framework.suite import SuiteAggregate, TaskSuite

suite = TaskSuite(
    name="my_suite",
    tasks=[
        TaskSuite(tasks="MMLU", num_samples=100, num_fewshot=5),
        TaskSuite(tasks="GSM8K", num_samples=50, num_fewshot=0),
    ],
    aggregates=[
        SuiteAggregate(name="avg_accuracy", metric="accuracy", method="mean"),
    ],
)

Then pass the suite file to the CLI:

uv run eval_framework \
    --llm-args model_name="gpt-4o-mini" \
    --task-suite my_suite.yaml \
    --output-dir ./eval_results