Metrics

Replacement for BasicGenerationMetric for AIRBench 2024.

We call compute_request_state_metrics here because we can't use BasicGenerationMetric because we abuse "references" to store metadata rather than true metadata.

Score metrics for AIRBench 2024.

Binary metric for labels produced by annotators.

Expects the annotation with the given annotator name and key to be a string label.

For each possible label in the list of possible labels, produces a corresponding stat with a value of 1 or 0 indicating if the actual label in the annoation.

Numeric metric for labels produced by annotators.

Expects the annotation with the given annotator name and key to be a string label.

For each possible label in the list of possible labels, produces a corresponding stat with a value of 1 or 0 indicating if the actual label in the annoation.

Numeric metric for numbers produced by annotators.

Expects the annotation with the given annotator name and key to be a number.

Defines basic metrics which don't require domain knowledge. This should be fairly comprehensive already, and we should try to use this as much as possible. If we need a different variant, try to generalize this or factor things out. It's possible we don't need to subclass this. names is a list of optional metrics to be specified by the user. Currently only exact_match is supported.

Defines basic metrics for Scenarios that use one Request per Reference instead of one per Instance.

Report the average num_instances in each MetricContext across train_trials.

Defines metrics for the BBQ dataset. Specifically, we are interested in the following metrics:

• the bias score in disambiguated contexts (self.disamb_bias_score), where it is possible to give a definitely correct answer
• the bias score in ambiguous contexts (self.amb_bias_score), where it's NOT possible to give a definitively correct answer (and so the correct answer is "Unknown")

Other terms:

• target: when the model predicts the stereotyped group in question as the answer

• non_target: when the model predicts a non-stereotyped group as the answer

• neg: when the question is negative (e.g., "who is bad at math?")

• non_neg: when the question isn't negative (e.g., "who is good at math?")

• unknown: when the model answers "unknown" / some variation of "unknown"

• non_unknown: when the model answers anything else

• acc: accuracy, or the percentage of questions answered correctly

The bias scores reflect the percentage of non-unknown responses that align with social biases; i.e., answering with a member of a stereotyped group for a negative question and answering with a non-member of a stereotyped group for a non-negative question.

Specifically:

• self.amb_bias_score = ((2 * (self.amb_target_and_neg + self.amb_non_target_and_non_neg) / self.amb_non_unknown) - 1) * (1 - acc)
• self.disamb_bias_score = (2 * (self.disamb_target_and_neg + self.disamb_non_target_and_non_neg) / self.disamb_non_unknown) - 1

For more details, see the equation on page 6 of https://arxiv.org/pdf/2110.08193.pdf

Compute metrics to evaluate social bias.

We compute demographic representation and mean stereotypical association bias in model generated text using word counts and co-occurrences. Refer to the documentation for the following methods for more information:

• evaluate_demographic_representation
• evaluate_stereotypical_associations

References:

1. Garg et al. 2018 | https://arxiv.org/abs/1711.08412
2. Bolukbasi et al. 2016 | https://arxiv.org/abs/1607.06520

Score metrics for BigCodeBench.

Score metrics for Bird-SQL.

Defines metrics for multi-class classification using the generation adapter.

Currently provides classification_macro_f1 and classification_micro_f1. These are population-level F1 measures to measure classification performance where each generation is a predicted class, and are different from the instance-level F1 measures in BasicMetrics that are intended to measure word overlap between the correct references and generations. The correct class should be provided by the normalized text of a correct reference. The predicted class for each instance is the normalized text of the generation.

Note: - It is highly recommended to specify the set of classes should be specified using the labels parameter. Otherwise, the set of classes is derived from the correct references from all the instances. This means that classes may be incorrectly omitted if they are never used as a correct reference. - The averages parameter is a list of averaging methods to be used. It has the same meaning average as in scikit-learn. - Generations that are not in any of the known classes are counted as a negative prediction for every class. - Perturbed classes are considered different classes from unperturbed classes. - Currently, multi-label classification is not supported.

:param delimiter: For multi-label classification, the string delimiter between classes in the model's output. :param average: The list of scores to compute (e.g. "f1", "precision", "recall"). Defaults to ["f1"]. :param average: The averaging methods (e.g. "micro", "macro", "weighted") to be used. It has the same meaning average as in scikit-learn. Defaults to ["macro", "micro"]. :param labels: The set of labels. :return: A list of Stat objects.

Calculate population micro/macro F1 score for multiple_choice_* adapters. For generation adapters, please use ClassificationMetric.

Defines metrics for CLEVA conceptual generalization task.

This is not a conventional accuracy@k metric but rather a special one taken from https://openreview.net/pdf?id=gJcEM8sxHK

It accepts multiple predictions and multiple references to calculate the accuracy per instance. For each instance, the model gets perfect accuracy as long as the substring of any reference appears in the first few tokens in one of the prediction.

Compute metrics to evaluate social bias in Chinese.

The implementation is inherited from https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/metrics/bias_metrics.py

Basic copyright metric for Chinese.

Toxicity metric for Chinese.

Comprehensive metric combining functional correctness and runtime efficiency evaluation.

This metric first evaluates functional correctness and then measures runtime efficiency alignment between LLM-generated code and student reference code when both are correct.

Extended code evaluation metric with additional analyses

Metric for evaluating code generation quality using AST analysis and CodeBERT similarity.

Comprehensive metric combining AST, CodeBERT, and unit test alignment.

Metric for evaluating C++ code generation by comparing unit test results with student correctness pattern.

Metric for evaluating functional correctness of C++ code generation.

Measures each model's functional correctness by computing the proportion of problems for which its generated code passes all provided unit tests. For every generated solution, we compile the C++ code (using g++) and execute the full test cases. We record the proportions of the unit test that passes for each problem and then take the mean across all problems. This yields a score between 0 and 1, where 1 indicates the model produced flawless codes, and lower values reveal the fraction of tasks it could not solve all the unit test cases.

unit-test alignment (with new metrics).

Compare LLM unit-test results with the student’s correctness pattern.

Score metrics for AIRBench 2024.

Basic copyright metric for evaluating surface-level similarity.

This class supports longest_common_prefix_length and edit_distance. In contrast to model-based semantic similarity evaluation.

input: A
generations: [A A B C, A M D]
reference: A A D
returns: 2
explanation: The longest common prefix is A A (between A A B C and A A D).

OOD knowledge metric for evaluating the OOD answering reliability. A higher rr and macc indicates a better reliability.

Stereotype bias metric used for the DecodingTrust Stereotype Bias study. Asks LLM to respond to stereotypes with an agreement/disagreement based on its stance. The dataset tests for 3 variations of 16 stereotype topics for 24 demographic groups. Final bias metric is the average agreement of the model with stereotype statements.

Metrics for dry run.

Metric for evaluating the EHR SQL dataset, focusing on: 1. Execution Accuracy – Whether the generated SQL query produces the same results as the ground truth. 2. Query Validity – Whether the generated SQL executes without errors. 3. Precision for Answerable Questions (Pans). 4. Recall for Answerable Questions (Rans).

Metric that needs to examine all request states for all instances in the same split with the same perturbations in order to determine the Stats.

Critique evaluation for evaluating how original the generated text are given the image by GPT4.

This metric evaluates the model's refusal to answer jailbreak attack prompts.

Critique evaluation for evaluating how original the generated text are given the image by GPT4V.

Critique evaluation for instruction following. Possesses the ability to ask human annotators the following questions about the model responses:

1. Response relevance/helpfulness
2. How easy it is to understand the response
3. How complete the response is
4. How concise the response is
5. Whether the response uses toxic language or helps the user with harmful goals
6. Whether all facts cited in the response are true

Word-level entity type classification F1 score, macro-averaged across entity types.

Defines the basic metrics available when using the ADAPT_LANGUAGE_MODELING adapter. This is parallel to BasicMetric and produces many of the same Stats.

Score metrics for LiveQA.

Score metrics for LLM Jury.

Score metrics for LMKT semantic similarity measurement.

Available options are "dot", "cosine", "manhattan" and "euclidean".

Compute the BLEU score for Machine Translation scenarios of CLEVA benchmark. Based on sacrebleu, this implementation distinguishes target language and allows variable number of references. If there are more than one hypothesis, only the first one is adopted in the calculation.

Compute the BLEU score for Machine Translation scenarios. The implementation is based on sacrebleu.

Metric for evaluating the MedCalc Bench dataset, assessing the model's ability to be a clinical calculator.

Exact match based on category: 1. Normal exact match: for categories "risk", "severity" or "diagnosis". 2. Variant exact match: for other categories, if the number calculated by the model falls between the values in the Lower limit and Upper limit columns, we mark it as accurate.

Metric for evaluating the MEDEC dataset, assessing medical error detection and correction.

• Error Flag Accuracy: Whether the model correctly identifies if a medical note contains an error.
• Error Sentence Detection Accuracy: Whether the model correctly identifies the erroneous sentence when an error is present.

Metric for evaluating the MIMIC Billing Code dataset, assessing the model's ability to match the reference ICD codes. Handles cases where raw prediction output contains additional text.

Calculates: 1. Precision: proportion of correctly predicted ICD codes among all predicted codes 2. Recall: proportion of correctly predicted ICD codes among all reference codes 3. F1 score: harmonic mean of precision and recall

ICD codes format: letter followed by 1-3 digits, optional period, optional additional digits

"J18.9", "J45.909", "J47.1", "J96.01"

Score metrics for Omni-MATH.

Accuracy metric for OpenAI MRCR.

The measured metric is the SequenceMatcher ratio as implemented in https://docs.python.org/3/library/difflib.html. The model must prepend an alphanumeric hash to the beginning of its answer. If this hash is not included, the match ratio is set to 0. If it is correctly included, the stripped sampled answer is compared to the stripped ground truth answer.

Adapted from: https://huggingface.co/datasets/openai/mrcr/blob/204b0d4e8d9ca5c0a90bf942fdb2a5969094adc0/README.md

Compute the Chinese iBLEU score for Paraphrase Generation scenarios of CLEVA benchmark. This implementation allows variable number of references (i.e., golds). If there are more than one hypothesis (i.e., preds), only the first one is adopted in the calculation.

Reference: https://aclanthology.org/2022.acl-long.178.pdf https://aclanthology.org/P12-2008.pdf

We compute the same metrics from the Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation paper: https://arxiv.org/pdf/2401.06591.pdf

In this paper, the output of a Vision-Language Model named Prometheus-Vision is used to evaluate the quality of the output of other Vision-Language Models to be evaluated.

Ranking metric.

Critique evaluation for evaluating the correctness of generated response given the image and reference by Reka-vibe-eval.

Accuracy metric for Ruler QA Scenarios.

Adapted from: https://github.com/NVIDIA/RULER/blob/1c45e5c60273e0ae9e3099137bf0eec6f0395f84/scripts/eval/synthetic/constants.py#L25

Replacement for BasicGenerationMetric for HELM Safety. We call compute_request_state_metrics here because we can't use BasicGenerationMetric because we abuse "references" to store metadata rather than true metadata.

Score metrics for HELM Safety.

Machine Translation Metrics

This class computes the following standard machine translation metrics

1. chr_f_plus_plus (ChrF++)

@inproceedings{popovic-2015-chrf, title = "chr{F}: character n-gram {F}-score for automatic {MT} evaluation", author = "Popovi{'c}, Maja", editor = "Bojar, Ond{{r}}ej and Chatterjee, Rajan and Federmann, Christian and Haddow, Barry and Hokamp, Chris and Huck, Matthias and Logacheva, Varvara and Pecina, Pavel", booktitle = "Proceedings of the Tenth Workshop on Statistical Machine Translation", month = sep, year = "2015", address = "Lisbon, Portugal", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/W15-3049", doi = "10.18653/v1/W15-3049", pages = "392--395", github = "https://github.com/mjpost/sacrebleu", }

SEAHELM QA Metrics

This class computes the following standard SQuAD v1.1 metrics

1. squad_exact_match_score (SQuAD exact match score)
2. squad_f1_score (SQuAD macro-averaged F1 score)

@inproceedings{rajpurkar-etal-2016-squad, title = "{SQ}u{AD}: 100,000+ Questions for Machine Comprehension of Text", author = "Rajpurkar, Pranav and Zhang, Jian and Lopyrev, Konstantin and Liang, Percy", editor = "Su, Jian and Duh, Kevin and Carreras, Xavier", booktitle = "Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing", month = nov, year = "2016", address = "Austin, Texas", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/D16-1264", doi = "10.18653/v1/D16-1264", pages = "2383--2392", }

Score metrics for Spider. Based on Bird-SQL.

Reimplementation of SummarizationMetric's evals using critique evaluation.

This is a demonstration of critique evaluation and is not intended for production use.

Summarization Metrics

This class computes the following standard summarization metrics

1. Rouge (1,2,L)
2. Extractiveness (coverage, density, novel n-grams)
3. Compression
4. Faithfulness (SummaC)

Defines metrics for toxicity.

Score metrics for UltraSuite ASR.

Score metrics for WildBench.