Using the CLI¶
The Eval-Framework CLI provides a flexible interface for evaluating LLMs across a wide range of benchmarks. Whether you’re running evaluations locally or in a distributed environment, the CLI allows you to configure tasks, models, and metrics with ease.
Command Structure¶
uv run eval_framework [OPTIONS]
Model Selection¶
The evaluated model is always eval_framework.llm.openai.OpenAIModel. Configure it through --llm-args, which must
include model_name (e.g. --llm-args model_name="gpt-4o-mini" base_url="http://localhost:8000/v1").
Execution Configuration¶
--llm-args [LLM_ARGS ...]
Arguments to pass to the OpenAIModel constructor as key=value pairs. model_name is required.
--task-name TASK_NAME
The name of the task to evaluate.
--output-dir OUTPUT_DIR
The path for evaluation outputs.
--num-samples NUM_SAMPLES
The number of samples per subject to evaluate.
--num-fewshot NUM_FEWSHOT
The number of fewshot examples to use.
--max-tokens
The maximum number of tokens to generate for each sample. Overwrites any task default value.
--batch-size BATCH_SIZE
Size of batch of samples to send to the LLM for evaluation in parallel. Use 1 for sequential running (default).
--task-suite PATH_TO_TASK_SUITE_FILE
Path to a suite file (.yaml or .py) defining a set of tasks to run. Mutually exclusive with --task-name. Hyperparameter overrides and score aggregation methods can be configured within the suite file.
Task Configuration¶
--task-subjects TASK_SUBJECTS [TASK_SUBJECTS ...]
The subjects of the task to evaluate. If empty, all subjects are evaluated. Subjects in the form of tuples can be specified in a comma-delimited way, possibly using wildcard * in some dimensions of a tuple.
Examples: "DE_DE, *" or "FR_FR, astronomy"
--hf-revision HF_REVISION
A tag name, a branch name, or commit hash for the task HF dataset.
Judge Models¶
The LLM judge is always an eval_framework.llm.openai.OpenAIModel.
--judge-model-args JUDGE_MODEL_ARGS
Arguments to pass to the judge OpenAIModel constructor as key=value pairs; model_name is required (e.g.
model_name="gpt-4o"). Setting this enables the LLM judge, which is required for tasks scored with LLM-judge metrics.
Logging & Tracking¶
--wandb-project WANDB_PROJECT
The name of the Weights & Biases project to log runs to.
--wandb-entity WANDB_ENTITY
The name of the Weights & Biases entity to log runs to. Defaults to the user’s default entity.
--wandb-run-id WANDB_RUN_ID
The ID of an existing Weights & Biases run to resume. If not given, creates a new run. If given and exists, will continue the run but will overwrite the python command logged in WandB.
--wandb-upload-results or --no-wandb-upload-results
Whether to upload results as an artifact to Weights & Biases (default: True). Needs --wandb-project to be set.
--description DESCRIPTION
Description of the run. This will be added to the metadata of the run to help with bookkeeping.
--verbosity VERBOSITY
Set the logging verbosity level: 0=CRITICAL, 1=INFO, 2=DEBUG.
-h, --help
Show help message and exit.
Running Models Behind an OpenAI-Compatible Endpoint¶
Any model served through an OpenAI-compatible API (the OpenAI API itself, vLLM, TGI, …) can be evaluated with the
OpenAIModel class. The API key is read from OPENAI_API_KEY; pass base_url to target a self-hosted server:
uv run eval_framework \
--llm-args model_name="Qwen/Qwen3-0.6B" base_url="http://localhost:8000/v1" \
--task-name "MMLU" \
--task-subjects "abstract_algebra" \
--output-dir ./eval_results \
--num-fewshot 5 \
--num-samples 10
Every keyword argument of the OpenAIModel constructor can be set through --llm-args; see the
Model Arguments guide for the full list.
Running a Task Suite¶
Task suites let you run multiple tasks in one command and aggregate their scores. You can define a suite in a .yaml file:
# my_suite.yaml
name: my_suite
tasks:
- tasks: MMLU
num_samples: 100
num_fewshot: 5
- tasks: GSM8K
num_samples: 50
num_fewshot: 0
aggregates:
- name: avg_accuracy
metric: accuracy
method: mean
Or equivalently in a .py file:
# my_suite.py
from eval_framework.suite import SuiteAggregate, TaskSuite
suite = TaskSuite(
name="my_suite",
tasks=[
TaskSuite(tasks="MMLU", num_samples=100, num_fewshot=5),
TaskSuite(tasks="GSM8K", num_samples=50, num_fewshot=0),
],
aggregates=[
SuiteAggregate(name="avg_accuracy", metric="accuracy", method="mean"),
],
)
Then pass the suite file to the CLI:
uv run eval_framework \
--llm-args model_name="gpt-4o-mini" \
--task-suite my_suite.yaml \
--output-dir ./eval_results