Schemas

Short unique identifier of the scenario

Description of the scenario (task, data)

Extra metadata (e.g., whether this is a question answering or commonsense task)

Where the scenario subclass for self is defined.

Which instance we're evaluating

Which reference of the instance we're evaluating (if any)

Which request mode ("original" or "calibration") of the instance we're evaluating (if any) (for ADAPT_MULTIPLE_CHOICE_SEPARATE_CALIBRATED)

Which training set this request is for

How to map the completion text back to a real output (e.g., for multiple choice, "B" => "the second choice")

The request that is actually made

The result of the request (filled in when the request is executed)

Number of training instances (i.e., in-context examples)

Whether the prompt (instructions + test input) is truncated to fit the model's context window.

The number of initial tokens that will be ignored when computing language modeling metrics

Output of some post-processing step that is needed for the metric to understand the request Should match the annotator's name to an Annotation (usually a list of dictionaries for each completion) Example: parsing, rendering an image based on the text completion, etc.

The input

References that helps us evaluate

Split (e.g., train, valid, test)

Sub split (e.g. toxic, non-toxic)

Used to group Instances that were created from a particular Instance through data augmentation

Description of the Perturbation that was applied when creating this Instance

Perturbed input as defined by contrast sets (if available)

References for the perturbed input above (if available)

Return the first correct reference.

Return all correct references.

The output

Extra metadata (e.g., whether it's correct/factual/toxic)

Name of the Perturbation

Whether a perturbation is relevant to robustness. Will be used to aggregate perturbations metrics

Whether a perturbation is relevant to fairness. Will be used to aggregate perturbations metrics

Which types of Instances we are evaluating, to be populated during metric evaluation. PERTURBATION_PERTURBED (default) means we are evaluating on perturbed instances, PERTURBATION_ORIGINAL means we are evaluating the unperturbed version of instances where this perturbation applies, and, PERTURBATION_WORST means the the minimum metric between the two.

Seed added to instance_id when generating perturbation

Which model deployment to query -> Determines the Client. Refers to a deployment in the model deployment registry.

Which model to use -> Determines the Engine. Refers to a model metadata in the model registry.

Whether to query embedding instead of text response

What prompt do condition the language model on

Temperature parameter that governs diversity

Generate this many completions (by sampling from the model)

Take this many highest probability candidates per token in the completion

Maximum number of tokens to generate (per completion)

Stop generating once we hit one of these strings.

Should prompt be included as a prefix of each completion? (e.g., for evaluating perplexity of the prompt)

Same from tokens that occupy this probability mass (nucleus sampling)

Penalize repetition (OpenAI & Writer only)

Penalize repetition (OpenAI & Writer only)

Used to control randomness. Expect different responses for the same request but with different values for random.

Used for chat models. (OpenAI only for now). if messages is specified for a chat model, the prompt is ignored. Otherwise, the client should convert the prompt into a message.

Multimodal prompt with media objects interleaved (e.g., text, video, image, text, ...)

Parameters for image generation.

Returns the model host (referring to the deployment). Not to be confused with the model creator organization (referring to the model).

Returns the model engine (referring to the model). This is often the same as self.model_deploymentl.split("/")[1], but not always. For example, one model could be served on several servers (each with a different model_deployment) In that case we would have for example: 'aws/bloom-1', 'aws/bloom-2', 'aws/bloom-3' => 'bloom' This is why we need to keep track of the model engine with the model metadata. Example: 'openai/davinci' => 'davinci'

Whether the request was successful

Fixed dimensional embedding corresponding to the entire prompt

List of completion

Whether the request was actually cached

How long did the request take?

When was the request sent? We keep track of when the request was made because the underlying model or inference procedure backing the API might change over time. The integer represents the current time in seconds since the Epoch (January 1, 1970).

If success is false, what was the error?

Describes how to treat errors in the request.

Batch size (TogetherClient only)

How long it took to process the batch? (TogetherClient only)

Which replication

Statistics computed from the predicted output

This is the population variance, not the sample variance.

See https://towardsdatascience.com/variance-sample-vs-population-3ddbd29e498a for details.

This is the population standard deviation, not the sample standard deviation.

See https://towardsdatascience.com/variance-sample-vs-population-3ddbd29e498a for details.