Model Arguments¶
The Eval-Framework provides a set of model wrapper classes that standardize how LLMs are loaded, formatted, and used for evaluation. Each wrapper manages specific model backends, such as OpenAI-compatible endpoints.
The following sections describe the constructor arguments for each model class, highlighting configuration options, defaults, and their purpose. Understanding these arguments allows you to customize evaluation behavior, token limits, concurrency, and model-specific settings.
OpenAIModel — Constructor Arguments¶
OpenAIModel is a wrapper for OpenAI’s API models (e.g., GPT-4, GPT-3.5) that integrates with the evaluation framework.
It manages model configuration, authentication, and request parameters for the OpenAI client.
OpenAIModel Constructor Argument Reference¶
Argument |
Type |
Description |
Default |
|---|---|---|---|
|
|
Name of the OpenAI model to use (e.g. |
|
|
|
Explicit formatter instance used to convert chat messages into model prompts. If not provided, falls back to |
|
|
|
Sampling temperature controlling output randomness ( |
|
|
|
OpenAI API key. If not provided, defaults to the |
|
|
|
Optional OpenAI organization ID for multi-org API usage or billing separation. |
|
|
|
Custom API base URL, e.g., for Azure OpenAI endpoints or local proxies. |
|
|
|
Used to scale token-based limits based on model tokenizer density. See Deep Dive: bytes_per_token. |
|
Deep Dive: bytes_per_token¶
What it is¶
bytes_per_token is a scalar that adjusts generation limits (max_tokens) based on the model’s tokenizer characteristics.
Different models tokenize text differently — some produce more tokens per byte, some fewer.
This parameter helps keep generation length consistent across models by normalizing the token budget.
How it works internally¶
if bytes_per_token is not None:
bytes_per_token_scalar = 4.0 / bytes_per_token
else:
bytes_per_token_scalar = 4.0 / BYTES_PER_TOKEN # defaults to 4.0 / 4.0 = 1.0
The constant
BYTES_PER_TOKEN = 4.0is a heuristic from OpenAI’s tokenizer documentation.The scalar is then applied when calculating generation limits:
scaled_max_tokens = ceil(max_tokens * bytes_per_token_scalar)
This ensures that, for models with different token byte sizes, output lengths remain roughly comparable in bytes or visible characters.
Why it matters¶
Without this scaling, a model that uses shorter tokens would produce longer outputs (more tokens for the same byte size),
and a model with longer tokens would produce shorter outputs.
bytes_per_token compensates for that, aligning models to a common byte-level generation length.
Example calculations¶
Scenario |
bytes_per_token |
Computed Scalar (4.0 / bpt) |
max_tokens=100 → scaled |
Effect |
|---|---|---|---|---|
Default (no override) |
|
|
|
No change. |
Denser tokenizer (2 bytes/token) |
|
|
|
Model can generate twice as many tokens. |
Sparser tokenizer (8 bytes/token) |
|
|
|
Model generates half as many tokens. |
Recommended usage¶
Leave unset (
None) for most models — default behavior is fine.If you empirically know your tokenizer’s average bytes/token, pass it explicitly:
model = OpenAIModel(model_name="my-model", bytes_per_token=3.2)
Common approximate values:
GPT-family BPE: 3–4 bytes/token
SentencePiece or WordPiece (smaller vocab): 2–3 bytes/token
Character-level tokenizers: 1–2 bytes/token