HEIM (Text-to-image Model Evaluation)

To run HEIM, follow these steps:

1. Create a run specs configuration file. For example, to evaluate Stable Diffusion v1.4 against the MS-COCO scenario, run:

echo 'entries: [{description: "mscoco:model=huggingface/stable-diffusion-v1-4", priority: 1}]' > run_entries.conf

1. Run the benchmark with certain number of instances (e.g., 10 instances): helm-run --conf-paths run_entries.conf --suite heim_v1 --max-eval-instances 10

Examples of run specs configuration files can be found here. We used this configuration file to produce results of the paper.