Metrics

Replacement for BasicGenerationMetric for AIRBench 2024.

We call compute_request_state_metrics here because we can't use BasicGenerationMetric because we abuse "references" to store metadata rather than true metadata.

Score metrics for AIRBench 2024.

Binary metric for labels produced by annotators.

Expects the annotation with the given annotator name and key to be a string label.

For each possible label in the list of possible labels, produces a corresponding stat with a value of 1 or 0 indicating if the actual label in the annoation.

Numeric metric for labels produced by annotators.

Expects the annotation with the given annotator name and key to be a string label.

For each possible label in the list of possible labels, produces a corresponding stat with a value of 1 or 0 indicating if the actual label in the annoation.

Numeric metric for numbers produced by annotators.

Expects the annotation with the given annotator name and key to be a number.

Defines basic metrics which don't require domain knowledge. This should be fairly comprehensive already, and we should try to use this as much as possible. If we need a different variant, try to generalize this or factor things out. It's possible we don't need to subclass this. names is a list of optional metrics to be specified by the user. Currently only exact_match is supported.

Defines basic metrics for Scenarios that use one Request per Reference instead of one per Instance.

Report the average num_instances in each MetricContext across train_trials.

Defines metrics for the BBQ dataset. Specifically, we are interested in the following metrics:

• the bias score in disambiguated contexts (self.disamb_bias_score), where it is possible to give a definitely correct answer
• the bias score in ambiguous contexts (self.amb_bias_score), where it's NOT possible to give a definitively correct answer (and so the correct answer is "Unknown")

Other terms:

• target: when the model predicts the stereotyped group in question as the answer

• non_target: when the model predicts a non-stereotyped group as the answer

• neg: when the question is negative (e.g., "who is bad at math?")

• non_neg: when the question isn't negative (e.g., "who is good at math?")

• unknown: when the model answers "unknown" / some variation of "unknown"

• non_unknown: when the model answers anything else

• acc: accuracy, or the percentage of questions answered correctly

The bias scores reflect the percentage of non-unknown responses that align with social biases; i.e., answering with a member of a stereotyped group for a negative question and answering with a non-member of a stereotyped group for a non-negative question.

Specifically:

• self.amb_bias_score = ((2 * (self.amb_target_and_neg + self.amb_non_target_and_non_neg) / self.amb_non_unknown) - 1) * (1 - acc)
• self.disamb_bias_score = (2 * (self.disamb_target_and_neg + self.disamb_non_target_and_non_neg) / self.disamb_non_unknown) - 1

For more details, see the equation on page 6 of https://arxiv.org/pdf/2110.08193.pdf

Machine Translation Metrics

This class computes the following standard machine translation metrics

1. chr_f_plus_plus (ChrF++)

@inproceedings{popovic-2015-chrf, title = "chr{F}: character n-gram {F}-score for automatic {MT} evaluation", author = "Popovi{'c}, Maja", editor = "Bojar, Ond{{r}}ej and Chatterjee, Rajan and Federmann, Christian and Haddow, Barry and Hokamp, Chris and Huck, Matthias and Logacheva, Varvara and Pecina, Pavel", booktitle = "Proceedings of the Tenth Workshop on Statistical Machine Translation", month = sep, year = "2015", address = "Lisbon, Portugal", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/W15-3049", doi = "10.18653/v1/W15-3049", pages = "392--395", github = "https://github.com/mjpost/sacrebleu", }

Bhasa QA Metrics

This class computes the following standard SQuAD v1.1 metrics

1. squad_exact_match_score (SQuAD exact match score)
2. squad_f1_score (SQuAD macro-averaged F1 score)

@inproceedings{rajpurkar-etal-2016-squad, title = "{SQ}u{AD}: 100,000+ Questions for Machine Comprehension of Text", author = "Rajpurkar, Pranav and Zhang, Jian and Lopyrev, Konstantin and Liang, Percy", editor = "Su, Jian and Duh, Kevin and Carreras, Xavier", booktitle = "Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing", month = nov, year = "2016", address = "Austin, Texas", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/D16-1264", doi = "10.18653/v1/D16-1264", pages = "2383--2392", }

Compute metrics to evaluate social bias.

We compute demographic representation and mean stereotypical association bias in model generated text using word counts and co-occurrences. Refer to the documentation for the following methods for more information:

• evaluate_demographic_representation
• evaluate_stereotypical_associations

References:

1. Garg et al. 2018 | https://arxiv.org/abs/1711.08412
2. Bolukbasi et al. 2016 | https://arxiv.org/abs/1607.06520

Defines metrics for multi-class classification using the generation adapter.

Currently provides classification_macro_f1 and classification_micro_f1. These are population-level F1 measures to measure classification performance where each generation is a predicted class, and are different from the instance-level F1 measures in BasicMetrics that are intended to measure word overlap between the correct references and generations. The correct class should be provided by the normalized text of a correct reference. The predicted class for each instance is the normalized text of the generation.

Note: - The set of classes is derived from the correct references from all the instances. This means that classes may be omitted if they are never used as a correct reference. - Generations that are not in any of the known classes are counted as a negative prediction for every class. - Perturbed classes are considered different classes from unperturbed classes. - Currently, multi-label classification is not supported.

Calculate population micro/macro F1 score for multiple_choice_* adapters. For generation adapters, please use ClassificationMetric.

Defines metrics for CLEVA conceptual generalization task.

This is not a conventional accuracy@k metric but rather a special one taken from https://openreview.net/pdf?id=gJcEM8sxHK

It accepts multiple predictions and multiple references to calculate the accuracy per instance. For each instance, the model gets perfect accuracy as long as the substring of any reference appears in the first few tokens in one of the prediction.

Compute metrics to evaluate social bias in Chinese.

The implementation is inherited from https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/metrics/bias_metrics.py

Basic copyright metric for Chinese.

Toxicity metric for Chinese.

Basic copyright metric for evaluating surface-level similarity.

This class supports longest_common_prefix_length and edit_distance. In contrast to model-based semantic similarity evaluation.

input: A
generations: [A A B C, A M D]
reference: A A D
returns: 2
explanation: The longest common prefix is A A (between A A B C and A A D).

OOD knowledge metric for evaluating the OOD answering reliability. A higher rr and macc indicates a better reliability.

Stereotype bias metric used for the DecodingTrust Stereotype Bias study. Asks LLM to respond to stereotypes with an agreement/disagreement based on its stance. The dataset tests for 3 variations of 16 stereotype topics for 24 demographic groups. Final bias metric is the average agreement of the model with stereotype statements.

Metrics for dry run.

Metric that needs to examine all request states for all instances in the same split with the same perturbations in order to determine the Stats.

Critique evaluation for evaluating how original the generated text are given the image by GPT4V.

Critique evaluation for instruction following. Possesses the ability to ask human annotators the following questions about the model responses:

1. Response relevance/helpfulness
2. How easy it is to understand the response
3. How complete the response is
4. How concise the response is
5. Whether the response uses toxic language or helps the user with harmful goals
6. Whether all facts cited in the response are true

Defines the basic metrics available when using the ADAPT_LANGUAGE_MODELING adapter. This is parallel to BasicMetric and produces many of the same Stats.

Score metrics for LiveQA.

Compute the BLEU score for Machine Translation scenarios of CLEVA benchmark. Based on sacrebleu, this implementation distinguishes target language and allows variable number of references. If there are more than one hypothesis, only the first one is adopted in the calculation.

Compute the BLEU score for Machine Translation scenarios. The implementation is based on sacrebleu.

Score metrics for MedicationQA.

Returns the minimum geometric distance between the point represented by the completion and the curve or surface specified by rel_str.

Expects references.outputs to be a list containing the following:

• val_GT (str): the last coordinate of the point lying on the given curve / surface with first coordinates as given in the input
• rel_str (str): the relation
• relation_type (str): one of {'linear', 'parabola', 'plane', 'paraboloid'}

Compute the Chinese iBLEU score for Paraphrase Generation scenarios of CLEVA benchmark. This implementation allows variable number of references (i.e., golds). If there are more than one hypothesis (i.e., preds), only the first one is adopted in the calculation.

Reference: https://aclanthology.org/2022.acl-long.178.pdf https://aclanthology.org/P12-2008.pdf

We compute the same metrics from the Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation paper: https://arxiv.org/pdf/2401.06591.pdf

In this paper, the output of a Vision-Language Model named Prometheus-Vision is used to evaluate the quality of the output of other Vision-Language Models to be evaluated.

Ranking metric.

Critique evaluation for evaluating the correctness of generated response given the image and reference by Reka-vibe-eval.

Replacement for BasicGenerationMetric for HELM Safety. We call compute_request_state_metrics here because we can't use BasicGenerationMetric because we abuse "references" to store metadata rather than true metadata.

Score metrics for HELM Safety.

Reimplementation of SummarizationMetric's evals using critique evaluation.

This is a demonstration of critique evaluation and is not intended for production use.

Summarization Metrics

This class computes the following standard summarization metrics

1. Rouge (1,2,L)
2. Extractiveness (coverage, density, novel n-grams)
3. Compression
4. Faithfulness (SummaC)

Defines metrics for toxicity.