Overview Dataloading

To evaluate models on benchmarks, we define custom tasks that inherit from BaseTask to handle dataset loading and formatting. The framework supports two main evaluation types: completion tasks (text generation) and loglikelihood tasks (multiple choice ranking).

Core Data Types

The framework uses different data types based on the evaluation approach:

from eval_framework.shared.types import Completion, Loglikelihood, RawCompletion, RawLoglikelihood
from template_formatting.formatter import Message, Role


# For completion tasks (text generation)
class Completion(BaseModel):
    completion_text: str  # Generated text from the model
    # Additional fields based on actual implementation


# For loglikelihood tasks (multiple choice)
class Loglikelihood(BaseModel):
    loglikelihoods: list[float]  # Probability scores for each choice
    # Additional fields based on actual implementation


# Raw response types from LLMs
class RawCompletion(BaseModel):
    text: str  # Raw generated text
    # Additional fields based on actual implementation


class RawLoglikelihood(BaseModel):
    loglikelihoods: list[float]  # Raw probability scores
    # Additional fields based on actual implementation

Message Structure

Each prompt is structured as a sequence of messages using the template formatting system:

from template_formatting.formatter import Message, Role


class Message(BaseModel):
    role: Role  # SYSTEM, USER, or ASSISTANT
    content: str  # Message content
    # Additional fields based on actual formatter implementation

Task Implementation Pattern

Custom tasks inherit from BaseTask and implement specific methods based on their evaluation type:

For Completion Tasks:

from eval_framework.tasks.base import BaseTask
from eval_framework.models.sample import ResponseType


class MyCompletionTask(BaseTask[str]):
    NAME = "My Task"
    DATASET_PATH = "dataset_name"
    RESPONSE_TYPE = ResponseType.COMPLETION

    def _get_instruction_text(self, item: dict) -> str:
        """Format the question/instruction."""
        return f"Question: {item['question']}"

    def _get_ground_truth(self, item: dict) -> str:
        """Return the expected answer."""
        return item["answer"]

For Loglikelihood Tasks:

class MyLoglikelihoodTask(BaseTask[str]):
    NAME = "My Task"
    DATASET_PATH = "dataset_name"
    RESPONSE_TYPE = ResponseType.LOGLIKELIHOODS

    def _get_instruction_text(self, item: dict) -> str:
        """Format the question without choices."""
        return f"Question: {item['question']}"

    def _get_ground_truth(self, item: dict) -> str:
        """Return the correct answer choice."""
        return item["choices"][item["answer_idx"]]

    def _get_possible_completions(self, item: dict) -> list[str]:
        """Return all answer choices for ranking."""
        return item["choices"]

Few-Shot Example Construction

The framework automatically constructs few-shot prompts using these methods:

# Example prompt construction for a two-shot scenario
def construct_prompt(self, item: dict) -> list[Message]:
    messages = []

    # 1. System prompt (optional)
    if system_prompt := self._get_system_prompt_text(item):
        messages.append(Message(role=Role.SYSTEM, content=system_prompt))

    # 2. Few-shot examples
    fewshot_examples = self._sample_fewshot_examples(item)
    for example in fewshot_examples:
        # User instruction
        messages.append(Message(role=Role.USER, content=self._get_instruction_text(example)))
        # Assistant response
        messages.append(Message(role=Role.ASSISTANT, content=self._get_fewshot_target_text(example)))

    # 3. Actual instruction
    messages.append(Message(role=Role.USER, content=self._get_instruction_text(item)))

    # 4. Response cue (optional)
    if cue := self._get_cue_text(item):
        messages.append(Message(role=Role.ASSISTANT, content=cue))

    return messages

Example: Geography Quiz

Here’s how a complete geography quiz task might look:

messages = [
    Message(Role.SYSTEM, "Answer geography questions accurately."),
    Message(Role.USER, "Question: What is the capital of Germany?"),
    Message(Role.ASSISTANT, "Answer: Berlin"),
    Message(Role.USER, "Question: What is the capital of France?"),
    Message(Role.ASSISTANT, "Answer: Paris"),
    Message(Role.USER, "Question: What is the capital of Italy?"),
    Message(Role.ASSISTANT, "Answer:"),
]

For completion tasks, the model generates the complete answer: " Rome"

For loglikelihood tasks, the model ranks options: [" Rome", " Madrid", " Athens", " Vienna"]