Contributing to Eval Framework¶

Thank you for your interest in contributing to the Eval Framework! We welcome contributions from the community and are grateful for your support.

How to Contribute¶

We welcome several types of contributions:

Bug fixes: Help us identify and fix issues
Feature implementations: Add new functionality to the framework
Documentation improvements: Enhance or clarify existing documentation
Performance optimizations: Make the framework faster and more efficient
Examples and tutorials: Help others learn how to use the framework

Getting Started¶

Fork the repository: Click the “Fork” button on the GitHub repository page to create your own copy of the project.

Clone your fork locally:

git clone https://github.com/your-username/eval-framework.git
cd eval-framework

Add the upstream repository:

git remote add upstream https://github.com/Aleph-Alpha-Research/eval-framework.git

Install uv if you haven’t already following these instructions
Install the project dependencies:
```
uv sync --all-extras
```

Install pre-commit:

uv tool install pre-commit
uv run pre-commit install

Documentation Task documentation under docs/tasks/ is auto-generated by CI on every merge to main. You do not need to run this manually.

Submitting Changes¶

Pull Request Process¶

Create a new branch for your feature or bugfix:

git checkout -b feature/your-feature-name
# or
git checkout -b fix/issue-description

Make your changes and add tests for them:

# Make your code changes
# Add tests for your new functionality in the tests/ directory
git add .
git commit -m "Add meaningful commit message"

Make sure your code lints:
```
pre-commit run --all-files
```

Push to your fork:

git push origin feature/your-feature-name

Create a Pull Request: Go to GitHub, navigate to your fork, and create a Pull Request from your branch to the original repository’s main branch. Fill in the PR description and submit.
Ensure your PR title is inline with conventional commits so that release-please can auto-log changes to main since the last release.

Reporting Issues¶

Bug Reports¶

When reporting bugs, please include:

Clear title: Summarize the issue briefly
Environment details: OS, Python version, package versions
Reproduction steps: Step-by-step instructions to reproduce
Expected behavior: What should happen
Actual behavior: What actually happens
Error messages: Include full error messages and stack traces
Additional context: Screenshots, logs, or other relevant information

Feature Requests¶

When requesting features:

Use case: Describe why this feature would be useful
Proposed solution: How you think it should work
Additional context: Any other relevant information

Releasing a New Version¶

The eval_framework package follows semantic versioning specification. That is, starting with the first 1.0.0 release we aim for backwards-compatible changes within minor version changes and compatibility-breaking changes only within major version.

Automated Release¶

We use release-please to automate our package releases. release-please.yml runs on every push to main. It opens or updates a PR that logs changes committed since the last release. To release:

Wait for the release-please PR to reflect your changes (CI runs automatically when the PR is opened or updated)
Approve and merge the PR to main
release.yml publishes to PyPI and the Docker registry

Behind the scenes, merging also runs release-please.yml once more to create the GitHub release and tag; that event is what triggers release.yml. You do not need to run or trigger anything yourself beyond the merge.

Release cadence is controlled by when the release PR is merged. To release at most once per day with no manual steps, add a scheduled workflow that auto-merges the release-please PR when CI passes.

Manual Release¶

To launch a new release, please follow these steps:

Increase the project.version number in the pyproject.toml either manually, or through uv version --bump={major,minor,patch}
Adapt the CHANGELOG.md file to include the new version information
Merge these changes to the main branch
Create a new release through Github
- Click on “Create a new release”
- Create the appropriate tag on the main branch. That is, if the package version is X.Y.Z the tag must be vX.Y.Z or the release workflow will fail.
- Set the title of the release equivalent to the version number (X.Y.Z)
- Click on Generate release notes
- Manually copy the highlights from the changelog on top of the auto-generated notes
- Click on Publish Release
Update the project version to an incremented dev release by running uv version --bump patch --bump dev. Merge this change to main.

This will create a new version tag and run the release workflow. Open the Github Actions panel and look for the release workflow. Once things are ready, you will have to approve publishing to PyPi.

If things go wrong…¶

When a release workflow fails, the best way is to go to the Release page and delete the release and also delete the corresponding tag on the tag page. Then fix the workflow and re-release the package.

License¶

By contributing, you agree that your contributions will be licensed under the same license as the project. See the LICENSE file in the root directory for details.