eval_framework.llm package¶
Submodules¶
eval_framework.llm.base module¶
- class eval_framework.llm.base.BaseLLM[source]¶
Bases:
ABC- generate(samples, stop_sequences=None, max_tokens=None, temperature=None, top_p=None, response_format=None)[source]¶
Generates a model response for each sample.
Uses ‘generate_from_samples’ to generate responses if implemented, otherwise falls back to ‘generate_from_messages’.
- Return type:
list[RawCompletion]- Parameters:
samples (list[ResultCandidate])
stop_sequences (list[str] | None)
max_tokens (int | None)
temperature (float | None)
top_p (float | None)
response_format (ResponseFormatText | ResponseFormatJSONSchema | ResponseFormatJSONObject | None)
- abstractmethod generate_from_messages(messages, stop_sequences=None, max_tokens=None, temperature=None, top_p=None, response_format=None)[source]¶
stop_sequences and max_tokens are injected by the task if exist. They should be overwritten or extended with the properties of the model. This includes but is not limited to the stop tokens by the evaluated checkpoint (e.g. <|eot_id|> for an instruction finetuned Llama3.1, <|endoftext|> for a pretrained Llama3.1).
This function is expected to raise errors which are caught and reported when running the eval. Please also make sure to raise an error in case of sequence length issues. We expect to always raise an error if something impedes the expected completion of a task.
Important! The completion is expected to be detokenized and to NOT contain special tokens. Backends must also cut the completion at the first occurrence of a stop sequence and return the text before it; the stop sequence itself must not be part of the completion. Answer extraction relies on this and does no stripping of its own.
Returns: List[RawCompletion]
- Return type:
list[RawCompletion]- Parameters:
messages (list[Sequence[Message]])
stop_sequences (list[str] | None)
max_tokens (int | None)
temperature (float | None)
top_p (float | None)
response_format (ResponseFormatText | ResponseFormatJSONSchema | ResponseFormatJSONObject | None)
- generate_from_samples(samples, stop_sequences=None, max_tokens=None, temperature=None, top_p=None, response_format=None)[source]¶
stop_sequences and max_tokens are injected by the task if exist. They should be overwritten or extended with the properties of the model. This includes but is not limited to the stop tokens by the evaluated checkpoint (e.g. <|eot_id|> for an instruction finetuned Llama3.1, <|endoftext|> for a pretrained Llama3.1).
This function is expected to raise errors which are caught and reported when running the eval. Please also make sure to raise an error in case of sequence length issues. We expect to always raise an error if something impedes the expected completion of a task.
Important! The completion is expected to be detokenized and to NOT contain special tokens. Backends must also cut the completion at the first occurrence of a stop sequence and return the text before it; the stop sequence itself must not be part of the completion. Answer extraction relies on this and does no stripping of its own.
Returns: List[RawCompletion]
- Return type:
list[RawCompletion]- Parameters:
samples (list[ResultCandidate])
stop_sequences (list[str] | None)
max_tokens (int | None)
temperature (float | None)
top_p (float | None)
response_format (ResponseFormatText | ResponseFormatJSONSchema | ResponseFormatJSONObject | None)
- is_retryable(exc)[source]¶
Whether
with_retry()repeats a call that raisedexc: by default any failure at all.- Return type:
bool- Parameters:
exc (Exception)
- abstractmethod logprobs(samples)[source]¶
This function is expected to raise errors which are caught and reported when running the eval. Please also make sure to raise an error in case of sequence length issues. We expect to always raise an error if something prevents the expected completion of a task.
- Return type:
list[RawLoglikelihood]- Parameters:
samples (list[ResultCandidate])
- property name: str¶
This property is used to name the results folder and identify the eval results. Overwrite this property in the subclass with e.g. the checkpoint name/huggingface model name.
- post_process_completion(completion, sample)[source]¶
Model-specific post-processing of generated completions.
Override this method to apply model-specific cleanup or transformations (e.g., removing specific artifacts such as reasoning traces, handling special tokens).
- Parameters:
completion (
str) – The raw completion string from the modelsample (
ResultCandidate) – The sample that was used to generate the completion
- Return type:
str- Returns:
The post-processed completion string
- with_retry(policy=None)[source]¶
This model with every generation and logprob call repeated while is_retryable allows.
- Return type:
- Parameters:
policy (RetryPolicy | None)
eval_framework.llm.models module¶
This is just a default model file with API-backed models.
Please define your own model file externally and pass it to the eval-framework entrypoint to use it.
eval_framework.llm.openai module¶
- class eval_framework.llm.openai.OpenAIEmbeddingModel(model_name='text-embedding-3-large', formatter=None, api_key=None, organization=None, base_url=None)[source]¶
Bases:
BaseLLM- Parameters:
model_name (str)
formatter (BaseFormatter | None)
api_key (str | None)
organization (str | None)
base_url (str | None)
- generate_embeddings(messages)[source]¶
- Return type:
list[list[float]]- Parameters:
messages (list[Sequence[Message]])
- generate_from_messages(messages, stop_sequences=None, max_tokens=None, temperature=None, top_p=None, response_format=None)[source]¶
stop_sequences and max_tokens are injected by the task if exist. They should be overwritten or extended with the properties of the model. This includes but is not limited to the stop tokens by the evaluated checkpoint (e.g. <|eot_id|> for an instruction finetuned Llama3.1, <|endoftext|> for a pretrained Llama3.1).
This function is expected to raise errors which are caught and reported when running the eval. Please also make sure to raise an error in case of sequence length issues. We expect to always raise an error if something impedes the expected completion of a task.
Important! The completion is expected to be detokenized and to NOT contain special tokens. Backends must also cut the completion at the first occurrence of a stop sequence and return the text before it; the stop sequence itself must not be part of the completion. Answer extraction relies on this and does no stripping of its own.
Returns: List[RawCompletion]
- Return type:
list[RawCompletion]- Parameters:
messages (list[Sequence[Message]])
stop_sequences (list[str] | None)
max_tokens (int | None)
temperature (float | None)
top_p (float | None)
response_format (ResponseFormatText | ResponseFormatJSONSchema | ResponseFormatJSONObject | None)
- logprobs(samples)[source]¶
This function is expected to raise errors which are caught and reported when running the eval. Please also make sure to raise an error in case of sequence length issues. We expect to always raise an error if something prevents the expected completion of a task.
- Return type:
list[RawLoglikelihood]- Parameters:
samples (list[ResultCandidate])
- class eval_framework.llm.openai.OpenAIModel(model_name=None, formatter=None, temperature=None, top_p=None, api_key=None, organization=None, base_url=None, bytes_per_token=None)[source]¶
Bases:
BaseLLMLLM wrapper for OpenAI API providing text/chat completions and log-probability evaluation output.
- Parameters:
model_name (str | None)
formatter (BaseFormatter | None)
temperature (float | None)
top_p (float | None)
api_key (str | None)
organization (str | None)
base_url (str | None)
bytes_per_token (float | None)
-
BYTES_PER_TOKEN:
float= 4.0¶
-
DEFAULT_FORMATTER:
Callable[[],BaseFormatter] |None= None¶
-
LLM_NAME:
str|None= None¶
- generate_from_messages(messages, stop_sequences=None, max_tokens=None, temperature=None, top_p=None, response_format=None)[source]¶
Generate completions for a list of message sequences concurrently.
Uses text completion API when a formatter is configured, otherwise uses chat completion API.
- Parameters:
messages (
list[Sequence[Message]]) – Sequence of messages.stop_sequences (
list[str] |None) – Optional list of stop sequences.max_tokens (
int|None) – Optional maximum number of tokens to generate.temperature (
float|None) – Sampling temperature.top_p (
float|None) – Nucleus sampling probability mass (0.0 to 1.0). Overrides instance default if provided.response_format (
Union[ResponseFormatText,ResponseFormatJSONSchema,ResponseFormatJSONObject,None]) – Optional OpenAI response_format payload for structured output, e.g. {“type”: “json_schema”, “json_schema”: {…}}. Only used by the chat completion API.
- Return type:
list[RawCompletion]- Returns:
List of RawCompletion objects containing prompts and completions.
- is_retryable(exc)[source]¶
Whether
with_retry()repeats a call that raisedexc: by default any failure at all.- Return type:
bool- Parameters:
exc (Exception)
- logprobs(samples)[source]¶
Compute total log-probabilities for possible completions given each sample’s prompt.
Note
Requires a vLLM server, which returns prompt_logprobs on the completions API. All prompt and choice pairs are sent in a single batched request.
- Return type:
list[RawLoglikelihood]- Parameters:
samples (list[ResultCandidate])
eval_framework.llm.retry module¶
Repeating an LLM’s transport calls while its provider fails transiently.
Bases:
ExceptionThe model kept failing transiently until the retries ran out, the last failure as
__cause__.
- class eval_framework.llm.retry.RetryPolicy(attempts=5, backoff_s=8.0, max_backoff_s=60.0)[source]¶
Bases:
objectHow often and how patiently one call is repeated before the model is given up on.
The pause before a retry doubles from
backoff_sand is capped atmax_backoff_s.- Parameters:
attempts (int)
backoff_s (float)
max_backoff_s (float)
-
attempts:
int= 5¶
-
backoff_s:
float= 8.0¶
-
max_backoff_s:
float= 60.0¶
- class eval_framework.llm.retry.RetryingLLM(inner, policy)[source]¶
Bases:
BaseLLMA model whose generation and logprob calls are repeated while its
is_retryableallows.- Parameters:
inner (BaseLLM)
policy (RetryPolicy)
- generate_from_messages(messages, stop_sequences=None, max_tokens=None, temperature=None, top_p=None, response_format=None)[source]¶
stop_sequences and max_tokens are injected by the task if exist. They should be overwritten or extended with the properties of the model. This includes but is not limited to the stop tokens by the evaluated checkpoint (e.g. <|eot_id|> for an instruction finetuned Llama3.1, <|endoftext|> for a pretrained Llama3.1).
This function is expected to raise errors which are caught and reported when running the eval. Please also make sure to raise an error in case of sequence length issues. We expect to always raise an error if something impedes the expected completion of a task.
Important! The completion is expected to be detokenized and to NOT contain special tokens. Backends must also cut the completion at the first occurrence of a stop sequence and return the text before it; the stop sequence itself must not be part of the completion. Answer extraction relies on this and does no stripping of its own.
Returns: List[RawCompletion]
- Return type:
list[RawCompletion]- Parameters:
messages (list[Sequence[Message]])
stop_sequences (list[str] | None)
max_tokens (int | None)
temperature (float | None)
top_p (float | None)
response_format (ResponseFormatText | ResponseFormatJSONSchema | ResponseFormatJSONObject | None)
- generate_from_samples(samples, stop_sequences=None, max_tokens=None, temperature=None, top_p=None, response_format=None)[source]¶
stop_sequences and max_tokens are injected by the task if exist. They should be overwritten or extended with the properties of the model. This includes but is not limited to the stop tokens by the evaluated checkpoint (e.g. <|eot_id|> for an instruction finetuned Llama3.1, <|endoftext|> for a pretrained Llama3.1).
This function is expected to raise errors which are caught and reported when running the eval. Please also make sure to raise an error in case of sequence length issues. We expect to always raise an error if something impedes the expected completion of a task.
Important! The completion is expected to be detokenized and to NOT contain special tokens. Backends must also cut the completion at the first occurrence of a stop sequence and return the text before it; the stop sequence itself must not be part of the completion. Answer extraction relies on this and does no stripping of its own.
Returns: List[RawCompletion]
- Return type:
list[RawCompletion]- Parameters:
samples (list[ResultCandidate])
stop_sequences (list[str] | None)
max_tokens (int | None)
temperature (float | None)
top_p (float | None)
response_format (ResponseFormatText | ResponseFormatJSONSchema | ResponseFormatJSONObject | None)
- is_retryable(exc)[source]¶
Whether
with_retry()repeats a call that raisedexc: by default any failure at all.- Return type:
bool- Parameters:
exc (Exception)
- logprobs(samples)[source]¶
This function is expected to raise errors which are caught and reported when running the eval. Please also make sure to raise an error in case of sequence length issues. We expect to always raise an error if something prevents the expected completion of a task.
- Return type:
list[RawLoglikelihood]- Parameters:
samples (list[ResultCandidate])
- property name: str¶
This property is used to name the results folder and identify the eval results. Overwrite this property in the subclass with e.g. the checkpoint name/huggingface model name.
- post_process_completion(completion, sample)[source]¶
Model-specific post-processing of generated completions.
Override this method to apply model-specific cleanup or transformations (e.g., removing specific artifacts such as reasoning traces, handling special tokens).
- Parameters:
completion (
str) – The raw completion string from the modelsample (
ResultCandidate) – The sample that was used to generate the completion
- Return type:
str- Returns:
The post-processed completion string
- with_retry(policy=None)[source]¶
This model with every generation and logprob call repeated while is_retryable allows.
- Return type:
- Parameters:
policy (RetryPolicy | None)