eval_framework.llm package

Submodules

eval_framework.llm.base module

class eval_framework.llm.base.BaseLLM[source]

Bases: ABC

generate(samples, stop_sequences=None, max_tokens=None, temperature=None, top_p=None, response_format=None)[source]

Generates a model response for each sample.

Uses ‘generate_from_samples’ to generate responses if implemented, otherwise falls back to ‘generate_from_messages’.

Return type:

list[RawCompletion]

Parameters:
  • samples (list[ResultCandidate])

  • stop_sequences (list[str] | None)

  • max_tokens (int | None)

  • temperature (float | None)

  • top_p (float | None)

  • response_format (ResponseFormatText | ResponseFormatJSONSchema | ResponseFormatJSONObject | None)

abstractmethod generate_from_messages(messages, stop_sequences=None, max_tokens=None, temperature=None, top_p=None, response_format=None)[source]

stop_sequences and max_tokens are injected by the task if exist. They should be overwritten or extended with the properties of the model. This includes but is not limited to the stop tokens by the evaluated checkpoint (e.g. <|eot_id|> for an instruction finetuned Llama3.1, <|endoftext|> for a pretrained Llama3.1).

This function is expected to raise errors which are caught and reported when running the eval. Please also make sure to raise an error in case of sequence length issues. We expect to always raise an error if something impedes the expected completion of a task.

Important! The completion is expected to be detokenized and to NOT contain special tokens. Backends must also cut the completion at the first occurrence of a stop sequence and return the text before it; the stop sequence itself must not be part of the completion. Answer extraction relies on this and does no stripping of its own.

Returns: List[RawCompletion]

Return type:

list[RawCompletion]

Parameters:
  • messages (list[Sequence[Message]])

  • stop_sequences (list[str] | None)

  • max_tokens (int | None)

  • temperature (float | None)

  • top_p (float | None)

  • response_format (ResponseFormatText | ResponseFormatJSONSchema | ResponseFormatJSONObject | None)

generate_from_samples(samples, stop_sequences=None, max_tokens=None, temperature=None, top_p=None, response_format=None)[source]

stop_sequences and max_tokens are injected by the task if exist. They should be overwritten or extended with the properties of the model. This includes but is not limited to the stop tokens by the evaluated checkpoint (e.g. <|eot_id|> for an instruction finetuned Llama3.1, <|endoftext|> for a pretrained Llama3.1).

This function is expected to raise errors which are caught and reported when running the eval. Please also make sure to raise an error in case of sequence length issues. We expect to always raise an error if something impedes the expected completion of a task.

Important! The completion is expected to be detokenized and to NOT contain special tokens. Backends must also cut the completion at the first occurrence of a stop sequence and return the text before it; the stop sequence itself must not be part of the completion. Answer extraction relies on this and does no stripping of its own.

Returns: List[RawCompletion]

Return type:

list[RawCompletion]

Parameters:
  • samples (list[ResultCandidate])

  • stop_sequences (list[str] | None)

  • max_tokens (int | None)

  • temperature (float | None)

  • top_p (float | None)

  • response_format (ResponseFormatText | ResponseFormatJSONSchema | ResponseFormatJSONObject | None)

is_retryable(exc)[source]

Whether with_retry() repeats a call that raised exc: by default any failure at all.

Return type:

bool

Parameters:

exc (Exception)

abstractmethod logprobs(samples)[source]

This function is expected to raise errors which are caught and reported when running the eval. Please also make sure to raise an error in case of sequence length issues. We expect to always raise an error if something prevents the expected completion of a task.

Return type:

list[RawLoglikelihood]

Parameters:

samples (list[ResultCandidate])

property name: str

This property is used to name the results folder and identify the eval results. Overwrite this property in the subclass with e.g. the checkpoint name/huggingface model name.

post_process_completion(completion, sample)[source]

Model-specific post-processing of generated completions.

Override this method to apply model-specific cleanup or transformations (e.g., removing specific artifacts such as reasoning traces, handling special tokens).

Parameters:
  • completion (str) – The raw completion string from the model

  • sample (ResultCandidate) – The sample that was used to generate the completion

Return type:

str

Returns:

The post-processed completion string

with_retry(policy=None)[source]

This model with every generation and logprob call repeated while is_retryable allows.

Return type:

BaseLLM

Parameters:

policy (RetryPolicy | None)

eval_framework.llm.models module

This is just a default model file with API-backed models.

Please define your own model file externally and pass it to the eval-framework entrypoint to use it.

eval_framework.llm.openai module

exception eval_framework.llm.openai.EmptyCompletionError[source]

Bases: ValueError

class eval_framework.llm.openai.OpenAIEmbeddingModel(model_name='text-embedding-3-large', formatter=None, api_key=None, organization=None, base_url=None)[source]

Bases: BaseLLM

Parameters:
  • model_name (str)

  • formatter (BaseFormatter | None)

  • api_key (str | None)

  • organization (str | None)

  • base_url (str | None)

generate_embeddings(messages)[source]
Return type:

list[list[float]]

Parameters:

messages (list[Sequence[Message]])

generate_from_messages(messages, stop_sequences=None, max_tokens=None, temperature=None, top_p=None, response_format=None)[source]

stop_sequences and max_tokens are injected by the task if exist. They should be overwritten or extended with the properties of the model. This includes but is not limited to the stop tokens by the evaluated checkpoint (e.g. <|eot_id|> for an instruction finetuned Llama3.1, <|endoftext|> for a pretrained Llama3.1).

This function is expected to raise errors which are caught and reported when running the eval. Please also make sure to raise an error in case of sequence length issues. We expect to always raise an error if something impedes the expected completion of a task.

Important! The completion is expected to be detokenized and to NOT contain special tokens. Backends must also cut the completion at the first occurrence of a stop sequence and return the text before it; the stop sequence itself must not be part of the completion. Answer extraction relies on this and does no stripping of its own.

Returns: List[RawCompletion]

Return type:

list[RawCompletion]

Parameters:
  • messages (list[Sequence[Message]])

  • stop_sequences (list[str] | None)

  • max_tokens (int | None)

  • temperature (float | None)

  • top_p (float | None)

  • response_format (ResponseFormatText | ResponseFormatJSONSchema | ResponseFormatJSONObject | None)

logprobs(samples)[source]

This function is expected to raise errors which are caught and reported when running the eval. Please also make sure to raise an error in case of sequence length issues. We expect to always raise an error if something prevents the expected completion of a task.

Return type:

list[RawLoglikelihood]

Parameters:

samples (list[ResultCandidate])

class eval_framework.llm.openai.OpenAIModel(model_name=None, formatter=None, temperature=None, top_p=None, api_key=None, organization=None, base_url=None, bytes_per_token=None)[source]

Bases: BaseLLM

LLM wrapper for OpenAI API providing text/chat completions and log-probability evaluation output.

Parameters:
  • model_name (str | None)

  • formatter (BaseFormatter | None)

  • temperature (float | None)

  • top_p (float | None)

  • api_key (str | None)

  • organization (str | None)

  • base_url (str | None)

  • bytes_per_token (float | None)

BYTES_PER_TOKEN: float = 4.0
DEFAULT_FORMATTER: Callable[[], BaseFormatter] | None = None
LLM_NAME: str | None = None
generate_from_messages(messages, stop_sequences=None, max_tokens=None, temperature=None, top_p=None, response_format=None)[source]

Generate completions for a list of message sequences concurrently.

Uses text completion API when a formatter is configured, otherwise uses chat completion API.

Parameters:
  • messages (list[Sequence[Message]]) – Sequence of messages.

  • stop_sequences (list[str] | None) – Optional list of stop sequences.

  • max_tokens (int | None) – Optional maximum number of tokens to generate.

  • temperature (float | None) – Sampling temperature.

  • top_p (float | None) – Nucleus sampling probability mass (0.0 to 1.0). Overrides instance default if provided.

  • response_format (Union[ResponseFormatText, ResponseFormatJSONSchema, ResponseFormatJSONObject, None]) – Optional OpenAI response_format payload for structured output, e.g. {“type”: “json_schema”, “json_schema”: {…}}. Only used by the chat completion API.

Return type:

list[RawCompletion]

Returns:

List of RawCompletion objects containing prompts and completions.

is_retryable(exc)[source]

Whether with_retry() repeats a call that raised exc: by default any failure at all.

Return type:

bool

Parameters:

exc (Exception)

logprobs(samples)[source]

Compute total log-probabilities for possible completions given each sample’s prompt.

Note

Requires a vLLM server, which returns prompt_logprobs on the completions API. All prompt and choice pairs are sent in a single batched request.

Return type:

list[RawLoglikelihood]

Parameters:

samples (list[ResultCandidate])

eval_framework.llm.retry module

Repeating an LLM’s transport calls while its provider fails transiently.

exception eval_framework.llm.retry.LLMUnavailable[source]

Bases: Exception

The model kept failing transiently until the retries ran out, the last failure as __cause__.

class eval_framework.llm.retry.RetryPolicy(attempts=5, backoff_s=8.0, max_backoff_s=60.0)[source]

Bases: object

How often and how patiently one call is repeated before the model is given up on.

The pause before a retry doubles from backoff_s and is capped at max_backoff_s.

Parameters:
  • attempts (int)

  • backoff_s (float)

  • max_backoff_s (float)

attempts: int = 5
backoff_s: float = 8.0
max_backoff_s: float = 60.0
pause_before(attempt)[source]
Return type:

float

Parameters:

attempt (int)

class eval_framework.llm.retry.RetryingLLM(inner, policy)[source]

Bases: BaseLLM

A model whose generation and logprob calls are repeated while its is_retryable allows.

Parameters:
generate_from_messages(messages, stop_sequences=None, max_tokens=None, temperature=None, top_p=None, response_format=None)[source]

stop_sequences and max_tokens are injected by the task if exist. They should be overwritten or extended with the properties of the model. This includes but is not limited to the stop tokens by the evaluated checkpoint (e.g. <|eot_id|> for an instruction finetuned Llama3.1, <|endoftext|> for a pretrained Llama3.1).

This function is expected to raise errors which are caught and reported when running the eval. Please also make sure to raise an error in case of sequence length issues. We expect to always raise an error if something impedes the expected completion of a task.

Important! The completion is expected to be detokenized and to NOT contain special tokens. Backends must also cut the completion at the first occurrence of a stop sequence and return the text before it; the stop sequence itself must not be part of the completion. Answer extraction relies on this and does no stripping of its own.

Returns: List[RawCompletion]

Return type:

list[RawCompletion]

Parameters:
  • messages (list[Sequence[Message]])

  • stop_sequences (list[str] | None)

  • max_tokens (int | None)

  • temperature (float | None)

  • top_p (float | None)

  • response_format (ResponseFormatText | ResponseFormatJSONSchema | ResponseFormatJSONObject | None)

generate_from_samples(samples, stop_sequences=None, max_tokens=None, temperature=None, top_p=None, response_format=None)[source]

stop_sequences and max_tokens are injected by the task if exist. They should be overwritten or extended with the properties of the model. This includes but is not limited to the stop tokens by the evaluated checkpoint (e.g. <|eot_id|> for an instruction finetuned Llama3.1, <|endoftext|> for a pretrained Llama3.1).

This function is expected to raise errors which are caught and reported when running the eval. Please also make sure to raise an error in case of sequence length issues. We expect to always raise an error if something impedes the expected completion of a task.

Important! The completion is expected to be detokenized and to NOT contain special tokens. Backends must also cut the completion at the first occurrence of a stop sequence and return the text before it; the stop sequence itself must not be part of the completion. Answer extraction relies on this and does no stripping of its own.

Returns: List[RawCompletion]

Return type:

list[RawCompletion]

Parameters:
  • samples (list[ResultCandidate])

  • stop_sequences (list[str] | None)

  • max_tokens (int | None)

  • temperature (float | None)

  • top_p (float | None)

  • response_format (ResponseFormatText | ResponseFormatJSONSchema | ResponseFormatJSONObject | None)

is_retryable(exc)[source]

Whether with_retry() repeats a call that raised exc: by default any failure at all.

Return type:

bool

Parameters:

exc (Exception)

logprobs(samples)[source]

This function is expected to raise errors which are caught and reported when running the eval. Please also make sure to raise an error in case of sequence length issues. We expect to always raise an error if something prevents the expected completion of a task.

Return type:

list[RawLoglikelihood]

Parameters:

samples (list[ResultCandidate])

property name: str

This property is used to name the results folder and identify the eval results. Overwrite this property in the subclass with e.g. the checkpoint name/huggingface model name.

post_process_completion(completion, sample)[source]

Model-specific post-processing of generated completions.

Override this method to apply model-specific cleanup or transformations (e.g., removing specific artifacts such as reasoning traces, handling special tokens).

Parameters:
  • completion (str) – The raw completion string from the model

  • sample (ResultCandidate) – The sample that was used to generate the completion

Return type:

str

Returns:

The post-processed completion string

with_retry(policy=None)[source]

This model with every generation and logprob call repeated while is_retryable allows.

Return type:

BaseLLM

Parameters:

policy (RetryPolicy | None)

Module contents