Testing

This repository contains a large and diverse test suite. To keep iteration fast, tests are split into fast PR tests and slow/advanced tests. Contributors should generally run only the fast tests locally unless reproducing a specific failure.

Test Tiers

1. Fast / PR Tests

  • Runs on: Every push to main, pull requests, and merge queue.

  • Runtime: ~20 minutes total

  • Purpose: Ensure code correctness for PRs without running the heaviest tests.

  • Includes:

    • Linting, pre-commit, type checks (2 min)

    • Tag setup and HuggingFace datasets cache pull (2 min)

    • Docker image build (5 min)

    • UV install dependency tests (1 min)

    • CPU tests excluding slow/external tests (3 min)

    • CPU slow tests (3–4 min)

    • Formatter hash tests (3 min)

    • GPU tests / optional extras (12 min)

Recommended local command: For most contributors, running the CPU fast tests is sufficient:

# Run the tests that PR CI runs
uv run --all-extras pytest -n auto --max-worker-restart=0 -v \
    -m "not gpu and not cpu_slow and not external_api and not formatter_hash"

2. Advanced / GPU Tests

  • Runs on: PR workflow (test-docker-gpu)

  • Runtime: ~12 min

  • Purpose: Run GPU tests or all optional extras together. Typically only required if debugging GPU-specific issues.

  • Includes:

    • GPU tests excluding CPU-slow / external API / vllm (12 min)

    • Optional extras (vllm, mistral) installations

Recommended local command (advanced users):

uv run --exact --all-extras pytest -v --noconftest tests/tests_eval_framework/installs/

⚠️ Warning: Running GPU/full extras locally may take significant time and requires a GPU.

CI as Source of Truth

The authoritative definition of which tests belong to each tier is encoded in the GitHub workflows:

  • tests.yml → PR tests, CPU and GPU tests, linting

Tips for Contributors

  • Run fast PR tests before pushing code.

  • Do not attempt to run the full suite unless reproducing a CI failure.

  • CI automatically runs GPU and slow tests on PRs.