","element":"span"},{"text":". The image description is ","element":"span"},{"style":{"fontStyle":"italic"},"text":"<im-age_description>","element":"span"},{"text":". What is the meme poster trying to convey? Answer:","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"can ","element":"span"},{"text":"be ","element":"span"},{"text":"downloaded ","element":"span"},{"text":"from ","element":"span"},{"href":"https://github.com/eujhwang/meme-cap/tree/main","text":"https://github.com/eujhwang/meme-cap/tree/main","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"No license is available for the dataset.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information is present. However, offensive content may be present in the images due to the dataset containing meme data.","element":"span"}],[{"style":{"width":"61%"},"width":1507,"height":168,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/24-2.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Splits: ","element":"span"},{"text":"We use the ’dev’ split from ","element":"span"},{"href":"https://www.kaggle.com/datasets/parthplc/facebook-hateful-meme-dataset/data","text":"https://www.kaggle.com/datasets/parthplc/facebook- ","element":"a"},{"href":"https://www.kaggle.com/datasets/parthplc/facebook-hateful-meme-dataset/data","text":"hateful-meme-dataset/data","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"You are given an image. In the image, the text phrase that you will be given and the image are innocuous when considered by themselves. The semantic content of the meme becomes mean only when the text phrase and image are considered together. Text phrase: ","element":"span"},{"style":{"fontStyle":"italic"},"text":"<text_phrase> ","element":"span"},{"text":"You have to judge if the combination of image and text is hateful or not. Always begin your answer with either ’yes’ or ’no’ with ’yes’ indicating that the meme is hateful and ’no’ if it is not hateful. Answer:","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"downloaded ","element":"span"},{"text":"from ","element":"span"},{"href":"https://www.kaggle.com/datasets/parthplc/facebook-hateful-meme-dataset/data","text":"https://www.kaggle.com/datasets/parthplc/facebook-hateful-meme-dataset/data","element":"a"}],[{"style":{"width":"91%"},"width":1445,"height":37,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/25-0.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information is present. However, offensive content may be present in the images since it is the goal of the dataset to train a detector for offensiveness given multimodal meme inputs.","element":"span"}],[{"text":"20. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"I","element":"span"},{"style":{"fontWeight":"bold"},"text":"N","element":"span"},{"style":{"fontWeight":"bold"},"text":"ATURALIST ","element":"span"},{"text":"is an image classification dataset for 5000 wildlife species of plants and animals. Images and labels are sourced from iNaturalist website ","element":"span"},{"text":"*","element":"span"},{"text":". We evaluate the models by asking them to identify the species present in the given image. We do not provide it with possible classes as the dataset spans over a set of 5000 species.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"We evaluate the model on the validation split provided in the 2021 edition of the dataset.","element":"span"}],[{"style":{"width":"80%"},"width":1283,"height":37,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/25-1.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Access restrictions: ","element":"span"},{"text":"We use the dataset available at ","element":"span"},{"href":"https://ml-inat-competition-datasets.s3.amazonaws.com/2021/val.tar.gz","text":"https://ml-inat-competition- ","element":"a"},{"href":"https://ml-inat-competition-datasets.s3.amazonaws.com/2021/val.tar.gz","text":"datasets.s3.amazonaws.com/2021/val.tar.gz","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"Work ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"under ","element":"span"},{"text":"MIT ","element":"span"},{"text":"license ","element":"span"},{"href":"https://github.com/visipedia/inat_comp/blob/master/LICENSE","text":"https://github.com/visipedia/inat_comp/blob/master/LICENSE","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"text":"21. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"N","element":"span"},{"style":{"fontWeight":"bold"},"text":"LVR ","element":"span"},{"text":"consists of image-text pairs for visual reasoning. Images are created by generating objects and their properties randomly. These images are then given to a crowd worker to describe the image in a sentence.","element":"span"}],[{"style":{"width":"75%"},"width":1200,"height":37,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/25-2.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"Given this image along with a question about the image, please answer the question with only the word ’true’ or ’false’. Question: ","element":"span"},{"style":{"fontStyle":"italic"},"text":"<question>","element":"span"}],[{"style":{"width":"84%"},"width":1332,"height":103,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/25-3.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"text":"22. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"N","element":"span"},{"style":{"fontWeight":"bold"},"text":"LVR","element":"span"},{"style":{"fontWeight":"bold"},"text":"2 ","element":"span"},{"text":"extends NLVR to real-world photographs, and captions for these photographs. Images are retrieved using search queries from the ILSVRC2014 ImageNet challenge ","element":"span"},{"text":"*","element":"span"},{"text":". Crowdworkers are used to write the captions for the images. For this dataset, each data point has two images and a sentence that talks about the images. We concatenate the two images so that we pass a single image in the model.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"Evaluation ","element":"span"},{"text":"is ","element":"span"},{"text":"performed ","element":"span"},{"text":"on ","element":"span"},{"text":"the ","element":"span"},{"text":"dev ","element":"span"},{"text":"split ","element":"span"},{"text":"from ","element":"span"},{"href":"https://github.com/lil-lab/nlvr/tree/master/nlvr2","text":"https://github.com/lil- ","element":"a"},{"href":"https://github.com/lil-lab/nlvr/tree/master/nlvr2","text":"lab/nlvr/tree/master/nlvr2","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"You are given an image and a related text, use the image as context and reply with true or false only Text: ","element":"span"},{"style":{"fontStyle":"italic"},"text":"<text> ","element":"span"},{"text":"Answer:","element":"span"}],[{"style":{"width":"86%"},"width":1368,"height":37,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/25-4.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"Images have licenses CC BY-SA 2.0 ","element":"span"},{"href":"https://creativecommons.org/licenses/by-sa/2.0/deed.en","text":"https://creativecommons.org/licenses/by- ","element":"a"},{"href":"https://creativecommons.org/licenses/by-sa/2.0/deed.en","text":"sa/2.0/deed.en, ","element":"a"},{"text":"CC ","element":"span"},{"text":"BY ","element":"span"},{"text":"3.0 ","element":"span"},{"href":"https://creativecommons.org/licenses/by/3.0/deed.en","text":"https://creativecommons.org/licenses/by/3.0/deed.en, ","element":"a"},{"text":"CC ","element":"span"},{"text":"BY-SA ","element":"span"},{"text":"3.0 ","element":"span"},{"href":"https://creativecommons.org/licenses/by-sa/3.0/deed.en","text":"https://creativecommons.org/licenses/by-sa/3.0/deed.en, ","element":"a"},{"text":"CC0 ","element":"span"},{"text":"1.0 ","element":"span"},{"href":"https://creativecommons.org/publicdomain/zero/1.0/deed.en","text":"https://creativecommons.org/publicdomain/zero/1.0/deed.en, ","element":"a"},{"text":"CC ","element":"span"},{"text":"BY-SA ","element":"span"},{"text":"4.0 ","element":"span"},{"href":"https://creativecommons.org/licenses/by-sa/4.0/deed.en","text":"https://creativecommons.org/licenses/by-sa/4.0/deed.en, ","element":"a"},{"text":"CC0 ","element":"span"},{"href":"https://creativecommons.org/public-domain/cc0/","text":"https://creativecommons.org/public-domain/cc0/ ","element":"a"},{"text":"and ","element":"span"},{"text":"Pixabay ","element":"span"},{"href":"https://pixabay.com/service/terms/","text":"https://pixabay.com/service/terms/","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"style":{"width":"61%"},"width":1511,"height":168,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/26-0.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"’val’ split is used from ","element":"span"},{"href":"https://visualcommonsense.com/download/","text":"https://visualcommonsense.com/download/","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"Question: ","element":"span"},{"style":{"fontStyle":"italic"},"text":"<question> ","element":"span"},{"text":"Choose from the below choices: ","element":"span"},{"style":{"fontStyle":"italic"},"text":"<choices>","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"download ","element":"span"},{"text":"from ","element":"span"},{"href":"https://visualcommonsense.com/download/","text":"https://visualcommonsense.com/download/","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"The dataset is provided in license as ","element":"span"},{"href":"https://visualcommonsense.com/license/","text":"https://visualcommonsense.com/license/","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"style":{"width":"61%"},"width":1512,"height":255,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/26-1.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"Test set from ","element":"span"},{"href":"https://huggingface.co/datasets/facebook/winoground","text":"https://huggingface.co/datasets/facebook/winoground ","element":"a"},{"text":"is used.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"You are given an image and a text. Answer yes if the text matches the image and no if the text does not match the image. Text: ","element":"span"},{"style":{"fontStyle":"italic"},"text":"<text> ","element":"span"},{"text":"Answer:","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"downloaded ","element":"span"},{"text":"from ","element":"span"},{"href":"https://huggingface.co/datasets/facebook/winoground","text":"https://huggingface.co/datasets/facebook/winoground","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"Authors of the dataset have Getty image license ","element":"span"},{"href":"https://www.gettyimages.in/eula","text":"https://www.gettyimages.in/eula.","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"style":{"width":"61%"},"width":1510,"height":168,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/26-2.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Split","element":"span"},{"text":": We use the dataset from ","element":"span"},{"href":"https://www.kaggle.com/datasets/happyyang/nwpu-data-set","text":"https://www.kaggle.com/datasets/happyyang/nwpu-data-set","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"Image is given to you. Classify if the image belongs to one of the following classes: ’basketball_court’, ’overpass’, ’ground_track_field’, ’church’, ’chaparral’, ’forest’, ’parking_lot’, ’golf_course’, ’baseball_diamond’, ’meadow’, ’beach’,’sparse_residential’, ’desert’, ’terrace’, ’palace’, ’bridge’, ’commercial_area’, ’stadium’, ’runway’, ’lake’, ’railway’, ’tennis_court’, ’ship’, ’intersection’, ’river’, ’freeway’, ’airplane’, ’industrial_area’, ’mountain’, ’storage_tank’, ’cloud’, ’roundabout’, ’wetland’, ’mobile_home_park’, ’island’, ’harbor’, ’railway_station’, ’medium_residential’, ’sea_ice’, ’thermal_power_station’, ’snowberg’, ’circular_farmland’, ’airport’, ’dense_residential’, ’rectangular_farmland’. Choose a class from the above classes.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"downloaded ","element":"span"},{"text":"from ","element":"span"},{"href":"https://www.kaggle.com/datasets/happyyang/nwpu-data-set","text":"https://www.kaggle.com/datasets/happyyang/nwpu-data-set","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"No license is provided with the dataset.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"style":{"width":"61%"},"width":1515,"height":125,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/26-3.png","element":"img"}],[{"text":"grounded in the image. Each question is associated with a series of steps that need to be followed to get the answer as well as a scene graph that captures objects, attributes, and relations in the image","element":"span"}],[{"style":{"width":"97%"},"width":1545,"height":38,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/27-0.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"You are given an image and a question. Answer the question in a single word. Question: ","element":"span"},{"style":{"fontStyle":"italic"},"text":"<question>","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"download ","element":"span"},{"text":"from ","element":"span"},{"href":"https://cs.stanford.edu/people/dorarad/gqa/download.html","text":"https://cs.stanford.edu/people/dorarad/gqa/download.html","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"The images in the dataset come with a CC BY 4.0 DEED license ","element":"span"},{"href":"https://creativecommons.org/licenses/by/4.0/deed.en","text":"https://creativecommons.org/licenses/by/4.0/deed.en","element":"a"}],[{"text":"27. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"O","element":"span"},{"style":{"fontWeight":"bold"},"text":"PEN","element":"span"},{"style":{"fontWeight":"bold"},"text":"P","element":"span"},{"style":{"fontWeight":"bold"},"text":"ATH ","element":"span"},{"text":"is a dataset created from Twitter and other public sources. Each image has a natural language description, and the dataset is sourced from tweets across 32 hashtag sub-specialty categories in pathology.","element":"span"}],[{"style":{"width":"41%"},"width":661,"height":37,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/27-1.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"Choose from the below choices, Given image is a hematoxylin and eosin image of: cancer-associated stroma, adipose tissue, debris, lymphocytes, mucus, background, normal colon mucosa, colorectal adenocarcinoma epithelium, smooth muscle","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access restrictions: ","element":"span"},{"text":"The dataset is available to download from huggingface datasets ","element":"span"},{"href":"https://huggingface.co/datasets/akshayg08/OpenPath","text":"https://huggingface.co/datasets/akshayg08/OpenPath","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"under ","element":"span"},{"text":"CC ","element":"span"},{"text":"BY-NC ","element":"span"},{"text":"4.0 ","element":"span"},{"text":"license. ","element":"span"},{"href":"https://creativecommons.org/licenses/by-nc/4.0/","text":"https://creativecommons.org/licenses/by-nc/4.0/","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"text":"28. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"IRFL ","element":"span"},{"text":"is an image-text dataset for figurative language. The dataset consists of three broad categories: idioms, similes, and metaphors. Metaphors and similes were collected from online lists whereas idioms were collected from MAGPIE corpus[","element":"span"},{"href":"#id-108","referenceIndex":33,"text":"33","element":"a"},{"text":"]. Since the MAGPIE corpus did not contain definitions for idioms, definitions were crawled from online dictionaries to search for figurative images. Google images were used for searching the images for idioms using these definitions. For similes and metaphors, annotators were used for definitions, and images were searched on the internet. For our evaluation, we use simile categorization. For each data point, one simile and four images are given. We modify this task to evaluate one image at a time, so a pair of an image and similes are passed to the model to see if they match or not.","element":"span"}],[{"style":{"width":"59%"},"width":941,"height":38,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/27-2.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"You are given a simile and a picture along with the simile. You have to say if the simile matches the given picture. Answer the following question in a single word with a yes or no. Simile: ","element":"span"},{"style":{"fontStyle":"italic"},"text":"<simile> ","element":"span"},{"text":"Answer:","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access restrictions: ","element":"span"},{"text":"Dataset is available for download from ","element":"span"},{"href":"https://github.com/irfl-dataset/IRFL","text":"https://github.com/irfl- ","element":"a"},{"href":"https://github.com/irfl-dataset/IRFL","text":"dataset/IRFL","element":"a"}],[{"style":{"width":"50%"},"width":794,"height":37,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/27-3.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"text":"29. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"S","element":"span"},{"style":{"fontWeight":"bold"},"text":"CREEN","element":"span"},{"style":{"fontWeight":"bold"},"text":"2W","element":"span"},{"style":{"fontWeight":"bold"},"text":"ORDS ","element":"span"},{"text":"is a mobile UI summarization dataset consisting of images from RicoSCA","element":"span"},{"href":"#id-109","referenceIndex":64,"text":"[64] ","element":"a"},{"text":"dataset. A total of 85 annotators were used to describe the image.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"We ","element":"span"},{"text":"use ","element":"span"},{"text":"the ","element":"span"},{"text":"test ","element":"span"},{"text":"split ","element":"span"},{"text":"from ","element":"span"},{"href":"https://github.com/google-research-datasets/screen2words/tree/main","text":"https://github.com/google-research- ","element":"a"},{"href":"https://github.com/google-research-datasets/screen2words/tree/main","text":"datasets/screen2words/tree/main","element":"a"}],[{"style":{"width":"85%"},"width":1359,"height":37,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/27-4.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Access restrictions: ","element":"span"},{"text":"The dataset is available to download from ","element":"span"},{"href":"https://github.com/google-research-datasets/screen2words","text":"https://github.com/google- ","element":"a"},{"href":"https://github.com/google-research-datasets/screen2words","text":"research-datasets/screen2words","element":"a"}],[{"style":{"width":"59%"},"width":1464,"height":300,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/28-0.png","element":"img"}],[{"text":"Figure 6: Multimodal skills are the basic building blocks central to solving problems, spanning information integrated across modalities at different granularities, different ways modalities might interact to create new information, reasoning, and external knowledge.","element":"figcaption","subtype":"caption"}],[{"style":{"width":"56%"},"width":1374,"height":220,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/28-1.png","element":"img"}],[{"text":"Figure 7: Multimodal information flow studies how the content changes across the two modalities for the task, such as through cross-modal translation, editing, querying, and fusion.","element":"figcaption","subtype":"caption"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"No license is provided with the dataset.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"style":{"width":"61%"},"width":1512,"height":125,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/28-2.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"We use the COCO subset from the Localized Narratives Dataset [","element":"span"},{"href":"#id-110","referenceIndex":87,"text":"87","element":"a"},{"text":"] containing 8,573 samples. The ground truth images are used from the MSCOCO (17) validation set.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"Generate an Image based on the provided caption. Caption:","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"download ","element":"span"},{"text":"from ","element":"span"},{"href":"https://google.github.io/localized-narratives/","text":"https://google.github.io/localized-narratives/","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"under ","element":"span"},{"text":"CC ","element":"span"},{"text":"BY-NC ","element":"span"},{"text":"4.0 ","element":"span"},{"text":"license. ","element":"span"},{"href":"https://creativecommons.org/licenses/by-nc/4.0/","text":"https://creativecommons.org/licenses/by-nc/4.0/","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"id":"id-104","style":{"width":"64%"},"width":1589,"height":786,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/28-3.png","element":"img"}],[{"text":"Table 6: Inter-Annotator agreement scores for stage 1 annotation.","element":"figcaption","subtype":"caption"}],[{"style":{"width":"55%"},"width":880,"height":460,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/29-0.png","element":"img"}],[{"text":"Table 7: Categorization after aggregating human annotations.","element":"figcaption","subtype":"caption"}],[{"id":"id-111","style":{"width":"70%"},"width":1116,"height":1141,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/29-1.png","element":"img"}],[{"text":"As per the annotations, we aggregate the annotations for each dataset across each dimension and calculate the maximum occurrence of annotation across all dimensions to categorize the datasets presented in Table ","element":"span"},{"href":"#id-111","text":"7. ","element":"a"},{"text":"We also consider ‘Neutral Reasoning’ and ‘Less Reasoning’ to be the same category and label them as ‘Less Reasoning’ before aggregating over the annotations. However, we see that the inter-annotator scores have low agreement, and some annotations go against the definitions above in the section ","element":"span"},{"text":"2. ","element":"span"},{"text":"Hence, we carry out an additional round of the annotation process using GPT-4V, and explain the process below.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"A.2.2 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Categorization stage 2: Automatic annotation with human verification ","element":"span"},{"text":"After the first stage was done, we found that most of the annotations were reliable but there were some cases where annotators misunderstood the definitions and tasks which led to low agreement values. For the second stage of the annotation process, we query GPT-4V for categorization of datapoints into dimensions to supplement the human annotations we obtained in the first stage. For each dataset, we consider three samples from each dataset for a total of 87 data points for categorization spread out across six sets. For each data point, we ask the model the same questions as asked to the human annotators above and obtain the categorization across the dimensions. For some questions, the model","element":"span"}],[{"text":"Table 8: Categorization after aggregating GPT-4V annotations. Cases where ’-’ is present are due to the model ","element":"figcaption","subtype":"caption"},{"text":"not providing an answer, citing a lack of information available for evaluating the input. We ignore such cases for categorization.","element":"figcaption","subtype":"caption"}],[{"style":{"width":"77%"},"width":1116,"height":1141,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/30-0.png","element":"img"}],[{"text":"Table 9: Inter-annotator agreement scores for stage 2 annotations.","element":"figcaption","subtype":"caption"}],[{"style":{"width":"64%"},"width":1587,"height":1134,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/30-1.png","element":"img"}],[{"text":"Table 10: Final dataset categorization.","element":"figcaption","subtype":"caption"}],[{"id":"id-112","style":{"width":"76%"},"width":1206,"height":1141,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/31-0.png","element":"img"}],[{"text":"(4) indicates that authors of this work have categorized the dataset for the dimension. As we can see from Table ","element":"span"},{"href":"#id-112","text":"10, ","element":"a"},{"text":"the majority of categories are agreed upon both by human annotators and GPT-4V, indicating reliability. There are only a few with (4), indicating that authors had to provide the final categorization due to dimensions that were hard to understand by non-experts in multimodal learning and by GPT-4V.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"A.2.4 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Details on annotation and participants","element":"span"}],[{"text":"The annotations in stages 1 (human annotation) and 2 (automatic inference with human verification) are all university students with some knowledge of machine learning. There were 10 sets of annotations each evaluated by two annotators for a total of 20 annotators. All participation in user studies was voluntary and done for pay at a level consistent with research participation at our university (15 dollars an hour). The annotations in stage 3 (final check) are done by 5 experts in the multimodal machine learning community for a final verification in case of misunderstandings in the first two stages.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"A.3 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Modeling categorizations and details ","element":"span"},{"text":"We also evaluate the performance of the models based on various modeling decisions. To achieve this, we categorize the models into various classes based on the following properties:","element":"span"}],[{"text":"1. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Interleaved modality training: ","element":"span"},{"text":"In the multi-modal setting, models are broadly trained/fine-tuned either by separately processing individual modalities using modality-specific encoders followed by fusion, or by interleaving the raw modalities first and then processing the interleaved input together.","element":"span"}],[{"text":"2. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Instruction Tuning: ","element":"span"},{"text":"Generative multimodal models can be trained/fine-tuned using objectives such as image-text matching, image-grounded text generation [","element":"span"},{"href":"#id-52","referenceIndex":62,"text":"62","element":"a"},{"text":"], etc., to generate relevant outputs. However, recently such generative models are also instruction instruction-tuned in order to generate outputs that resemble human responses. Therefore, we also categorise the models based on whether instruction tuning is employed or not.","element":"span"}],[{"text":"3. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Architecture: ","element":"span"},{"text":"For training multi-modal models, parameters can either be initialized using a pre-trained model and then are fine-tuned/kept frozen, or are initialized randomly and trained in an end-to-end fashion. Based on this choice, we categorize models into two classes -fine-tuned and trained from scratch.","element":"span"}],[{"text":"4. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Training Data Size: ","element":"span"},{"text":"The amount of data used for training the models, plays an important role in the performance and generalization of the model. Based on the size of the training data (in our work, the number of image-text or image-image samples), we categorize the models into three categories - Small, Medium, and Large.","element":"span"}],[{"text":"5. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Number of Parameters: ","element":"span"},{"text":"Model size is an important modeling decision as it affects the performance of the model, cost and efficiency of training, and the inference time. Hence, we also categorize the models based on both the total and trainable number of parameters, and compare the performance across these categories.","element":"span"}],[{"text":"6. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Diversity in Training Data: ","element":"span"},{"text":"Training multimodal models on data from different tasks, improves the diversity of the training data and may help the models to perform well on multiple tasks. By categorizing the models based on the diversity of the training data used, we evaluate the effect of using data from diverse tasks.","element":"span"}],[{"id":"id-102","style":{"width":"64%"},"width":1587,"height":506,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/32-0.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"License: ","element":"span"},{"text":"The ","element":"span"},{"text":"model ","element":"span"},{"text":"comes ","element":"span"},{"text":"with ","element":"span"},{"text":"BSD-3 ","element":"span"},{"text":"Clause ","element":"span"},{"href":"https://github.com/salesforce/LAVIS/blob/main/LICENSE.txt","text":"https://github.com/salesforce/LAVIS/blob/main/LICENSE.txt","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Access restrictions: ","element":"span"},{"text":"The model is available to use from the LAVIS repository ","element":"span"},{"href":"https://github.com/salesforce/LAVIS","text":"https://github.com/salesforce/LAVIS","element":"a"}],[{"style":{"width":"61%"},"width":1495,"height":338,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/32-1.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"License: ","element":"span"},{"text":"The ","element":"span"},{"text":"model ","element":"span"},{"text":"comes ","element":"span"},{"text":"with ","element":"span"},{"text":"BSD-3 ","element":"span"},{"text":"Clause ","element":"span"},{"href":"https://github.com/salesforce/LAVIS/blob/main/LICENSE.txt","text":"https://github.com/salesforce/LAVIS/blob/main/LICENSE.txt","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Access restrictions: ","element":"span"},{"text":"The model is available to use from the LAVIS repository ","element":"span"},{"href":"https://github.com/salesforce/LAVIS","text":"https://github.com/salesforce/LAVIS","element":"a"}],[{"style":{"width":"61%"},"width":1494,"height":253,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/32-2.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"License: ","element":"span"},{"text":"The model comes with BSD-3 Clause ","element":"span"},{"href":"https://github.com/Vision-CAIR/MiniGPT-4/blob/main/LICENSE.md","text":"https://github.com/Vision-CAIR/MiniGPT- ","element":"a"},{"href":"https://github.com/Vision-CAIR/MiniGPT-4/blob/main/LICENSE.md","text":"4/blob/main/LICENSE.md","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Access restrictions: ","element":"span"},{"text":"The model is available to use from ","element":"span"},{"href":"https://github.com/Vision-CAIR/MiniGPT-4/tree/main","text":"https://github.com/Vision- ","element":"a"},{"href":"https://github.com/Vision-CAIR/MiniGPT-4/tree/main","text":"CAIR/MiniGPT-4/tree/main","element":"a"}],[{"text":"4. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"O","element":"span"},{"style":{"fontWeight":"bold"},"text":"PEN","element":"span"},{"style":{"fontWeight":"bold"},"text":"F","element":"span"},{"style":{"fontWeight":"bold"},"text":"LAMINGO ","element":"span"},{"text":"is an open-source reproduction of the Flamingo [","element":"span"},{"href":"#id-113","referenceIndex":3,"text":"3","element":"a"},{"text":"] models. Unlike models that can only take one input image per sample (e.g., BLIP2, MiniGPT-4), OpenFlamingo can handle multiple images by interleaving images and texts. The architecture comprises of pre-trained Vision and Language encoder/decoder, where the layers of the pre-trained LLM are augmented with the vision encoder outputs which allows for cross-modal attention. All the pre-trained components are kept frozen except for the cross-modal attention component. For evaluation, we use the ","element":"span"},{"style":{"fontFamily":"monospace"},"text":"OpenFlamingo-3B-vitl-mpt1b ","element":"span"},{"text":"model from the OpenFlamingo Github Repository ","element":"span"},{"text":"*","element":"span"},{"text":". The chosen models has 1.4B trainable parameters and a total of 3.2B parameters. It is trained using 180M image-text samples.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"License: ","element":"span"},{"text":"Work ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"under ","element":"span"},{"text":"MIT ","element":"span"},{"text":"License ","element":"span"},{"href":"https://github.com/mlfoundations/open_flamingo/blob/main/LICENSE","text":"https://github.com/mlfoundations/open_flamingo/blob/main/LICENSE","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"model ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"use ","element":"span"},{"text":"from ","element":"span"},{"href":"https://github.com/mlfoundations/open_flamingo","text":"https://github.com/mlfoundations/open_flamingo","element":"a"}],[{"text":"5. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"LL","element":"span"},{"style":{"fontWeight":"bold"},"text":"A","element":"span"},{"style":{"fontWeight":"bold"},"text":"MA-A","element":"span"},{"style":{"fontWeight":"bold"},"text":"DAPTER ","element":"span"},{"text":"is based on the architecture of LLaMA Adapter [","element":"span"},{"href":"#id-114","referenceIndex":125,"text":"125","element":"a"},{"text":"] which augments the text tokens with learnable adaptation prompts. In addition to this, LLaMA Adapter V2 uses early fusion to add visual knowledge to the decoding LLM. The architecture uses both early fusion and late fusion, and while fine-tuning, all the pre-trained components are frozen except for the bias layers of the LLM, Visual Projection Layer and the zero-initialized cross attention module. We evaluate the ","element":"span"},{"style":{"fontFamily":"monospace"},"text":"BIAS-LORA-7B ","element":"span"},{"text":"model which uses LLaMA-7B as the decoder","element":"span"},{"text":"*","element":"span"},{"text":". The model is instruction tuned using 619K samples, and has 14M trainable parameters.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"License: ","element":"span"},{"text":"Work ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"under ","element":"span"},{"text":"GNU ","element":"span"},{"text":"General ","element":"span"},{"text":"public ","element":"span"},{"text":"license ","element":"span"},{"href":"https://github.com/OpenGVLab/LLaMA-Adapter/blob/main/LICENSE","text":"https://github.com/OpenGVLab/LLaMA-Adapter/blob/main/LICENSE","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"Model ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"use ","element":"span"},{"text":"from ","element":"span"},{"href":"https://github.com/OpenGVLab/LLaMA-Adapter","text":"https://github.com/OpenGVLab/LLaMA-Adapter","element":"a"}],[{"text":"6. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"E","element":"span"},{"style":{"fontWeight":"bold"},"text":"MU ","element":"span"},{"text":"is a large multimodal model trained using interleaved video, image and text data, trained in an autoregressive manner to predict the next token in the multimodal sequence. With the ability to produce the next visual token, Emu is also able to generate images and has been evaluated on the Magic Brush dataset in this work. The architecture uses pre-trained encoder and a decoding LLM such as LLaMA. EMU is first pre-trained using interleaved video, image, and text data, and all the parameters are updated during the pre-training. In the second stage, emu is further instruction-tuned. However, in this work we only evaluate the pre-trained version of Emu. We evaluate the Emu-14B model pre-trained using 82M samples.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"License: ","element":"span"},{"text":"Work ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"under ","element":"span"},{"text":"Apache ","element":"span"},{"text":"2.0 ","element":"span"},{"text":"license ","element":"span"},{"href":"https://github.com/baaivision/Emu/blob/main/LICENSE","text":"https://github.com/baaivision/Emu/blob/main/LICENSE","element":"a"}],[{"style":{"width":"90%"},"width":1441,"height":36,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/33-0.png","element":"img"}],[{"text":"7. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"F","element":"span"},{"style":{"fontWeight":"bold"},"text":"UYU","element":"span"},{"style":{"fontWeight":"bold"},"text":"-8B ","element":"span"},{"text":"is a decoder only architecture where the image patches are linearly projected into the first layer of the transformer architecture. Fuyu’s architecture is same as that of Persimmon-8B ","element":"span"},{"text":"*","element":"span"},{"text":", and we use the details of Persimmon-8B to categorise Fuyu into the model categories. Persimmon-8B has 9.3B parameters and is trained from scratch. In our work we evaluate the pre-trained model as the instruction tuned models aren’t available and the pre-training data sources and sizes are unknown. We evaluate the Fuyu-8B model available through HuggingFace ","element":"span"},{"text":"*","element":"span"},{"text":".","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"License: ","element":"span"},{"text":"Work is available under Creative Commons Attribution Non Commercial 4.0 International license ","element":"span"},{"href":"https://spdx.org/licenses/CC-BY-NC-4.0","text":"https://spdx.org/licenses/CC-BY-NC-4.0","element":"a"}],[{"style":{"width":"99%"},"width":1584,"height":61,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/33-1.png","element":"img"}],[{"href":"https://huggingface.co/adept/fuyu-8b","text":"https://huggingface.co/adept/fuyu-8b","element":"a"}],[{"style":{"width":"60%"},"width":1490,"height":295,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/34-0.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"License: ","element":"span"},{"text":"Work ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"under ","element":"span"},{"text":"MIT ","element":"span"},{"text":"License ","element":"span"},{"href":"https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/mit.md","text":"https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/mit.md","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"model ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"use ","element":"span"},{"text":"from ","element":"span"},{"href":"https://huggingface.co/microsoft/kosmos-2-patch14-224","text":"https://huggingface.co/microsoft/kosmos-2-patch14-224","element":"a"}],[{"style":{"width":"61%"},"width":1495,"height":383,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/34-1.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"License: ","element":"span"},{"text":"Work is available under MIT License ","element":"span"},{"href":"https://github.com/X-PLUG/mPLUG-Owl/blob/main/LICENSE","text":"https://github.com/X-PLUG/mPLUG- ","element":"a"},{"href":"https://github.com/X-PLUG/mPLUG-Owl/blob/main/LICENSE","text":"Owl/blob/main/LICENSE","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Access restrictions: ","element":"span"},{"text":"The model is available to use from ","element":"span"},{"href":"https://github.com/X-PLUG/mPLUG-Owl","text":"https://github.com/X- ","element":"a"},{"href":"https://github.com/X-PLUG/mPLUG-Owl","text":"PLUG/mPLUG-Owl","element":"a"}],[{"style":{"width":"61%"},"width":1507,"height":248,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/34-2.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"License: ","element":"span"},{"text":"None","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"model ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"via ","element":"span"},{"text":"OpenAI’s ","element":"span"},{"text":"API ","element":"span"},{"href":"https://platform.openai.com/docs/guides/vision","text":"https://platform.openai.com/docs/guides/vision","element":"a"}],[{"style":{"width":"61%"},"width":1512,"height":339,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/34-3.png","element":"img"}],[{"text":"’HARM_CATEGORY_DANGEROUS_CONTENT’ ","element":"span"},{"text":"and ","element":"span"},{"text":"set ","element":"span"},{"text":"the ","element":"span"},{"text":"threshold ","element":"span"},{"text":"to ’BLOCK_NONE’ provided by the API ","element":"span"},{"text":"*","element":"span"},{"text":".","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"License: ","element":"span"},{"text":"None","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access: ","element":"span"},{"text":"Available via API ","element":"span"},{"href":"https://ai.google.dev/gemini-api/docs/models/gemini","text":"https://ai.google.dev/gemini-api/docs/models/gemini","element":"a"}],[{"text":"Table 11: Evaluation metrics supported in HEMM ","element":"figcaption","subtype":"caption"},{"id":"id-115","text":"Metric ","element":"figcaption","subtype":"caption"},{"text":"Task ","element":"figcaption","subtype":"caption"},{"text":"Modalities","element":"figcaption","subtype":"caption"}],[{"style":{"width":"47%"},"width":760,"height":330,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/35-0.png","element":"img"}]]},{"heading":"B Experimental Details","paragraphs":[[{"style":{"fontWeight":"bold"},"text":"B.1 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Evaluation metrics","element":"span"}],[{"text":"We present our results on BARTScore [","element":"span"},{"href":"#id-44","referenceIndex":122,"text":"122","element":"a"},{"text":"] as models under our evaluation generate noisy free from text, however, we also support other text generation metrics under our evaluation suite listed in the Table ","element":"span"},{"href":"#id-115","text":"11 ","element":"a"},{"text":"below.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"B.2 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Evaluation protocol","element":"span"}],[{"text":"HEMM supports image generation tasks, models and metrics. However, currently there are only 2 image generation tasks (LNCOCO and M","element":"span"},{"text":"AGIC ","element":"span"},{"text":"B","element":"span"},{"text":"RUSH","element":"span"},{"text":") and 1 model (E","element":"span"},{"text":"MU","element":"span"},{"text":") that supports image generation. Hence, we perform all our evaluation on the remaining 28 text generation tasks and report the results on the image generation tasks in Appendix ","element":"span"},{"text":"C.","element":"span"}],[{"text":"Note: Since HEMM contains models that are unable to process multiple images in the same input, we modify the W","element":"span"},{"text":"INOGROUND ","element":"span"},{"text":"and IRFL tasks (as per ","element":"span"},{"href":"#id-101","text":"A.1) ","element":"a"},{"text":"in order to have a single image-text pair as input for each sample.","element":"span"}],[{"text":"For each dataset, we use the same prompts across all models as shown in Section ","element":"span"},{"href":"#id-104","text":"C, ","element":"a"},{"text":"for standardization, however, there can be a scenario where these models perform better with other prompts or scenarios and may perform poorly under our scenarios or prompts in our evaluation.","element":"span"}],[{"text":"For each dataset, the computed metrics for the models are normalized on a scale of 0 to 1, 0 corresponds to the model achieving the lowest score on that dataset, and 1 corresponds to the performance achieve by exactly generating the ground truth. For BERTScore [","element":"span"},{"href":"#id-116","referenceIndex":126,"text":"126","element":"a"},{"text":"], ROUGE [","element":"span"},{"href":"#id-117","referenceIndex":70,"text":"70","element":"a"},{"text":"], and RefCLIPScore [","element":"span"},{"href":"#id-118","referenceIndex":36,"text":"36","element":"a"},{"text":"] the maximum value is set to 1. BARTScore [","element":"span"},{"href":"#id-44","referenceIndex":122,"text":"122","element":"a"},{"text":"] uses the log of probabilities. Following [","element":"span"},{"href":"#id-45","referenceIndex":16,"text":"16","element":"a"},{"text":"], we calculate the maximum value for each dataset separately as BARTScore(r, r) where r is the ground truth sentence.","element":"span"}],[{"text":"Since details regarding training type for G","element":"span"},{"text":"EMINI ","element":"span"},{"text":"and GPT-4V, and modality processing for GPT-4V are not revealed, we do not use the scores from these models while evaluating the performance for the training type and modality processing dimensions. Further, for H","element":"span"},{"text":"ATEFUL ","element":"span"},{"text":"M","element":"span"},{"text":"EMES","element":"span"},{"text":", O","element":"span"},{"text":"PEN","element":"span"},{"text":"P","element":"span"},{"text":"ATH","element":"span"},{"text":", and M","element":"span"},{"text":"EMOTION ","element":"span"},{"text":"datasets, GPT-4V did not respond and generated ","element":"span"},{"style":{"fontStyle":"italic"},"text":"can’t provide assistance ","element":"span"},{"text":"and ","element":"span"},{"style":{"fontStyle":"italic"},"text":"\"indeterminate\" ","element":"span"},{"text":"for many samples. Hence, we exclude the results of GPT-4V on these datasets during evaluation.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"B.3 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Significance tests","element":"span"}],[{"text":"While comparing performance across categories in each dimension, we perform paired t-tests to determine the significance of the results. For datasets, specifically, for each category, we calculate the average performance of each of the 11 models on all the datasets in a category (","element":"span"},{"style":{"height":9.18},"width":28.28,"height":22.96,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/35-1.png","element":"img","alt":"ci","inline":true},{"text":") to create a vector ","element":"span"},{"style":{"height":16.08},"width":131.28,"height":40.2,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/35-2.png","element":"img","alt":"vi ∈ R11","inline":true},{"text":". Next, we performed pairwise t-tests between these vectors to determine the significance of the results. The p-values obtained through the t-tests are presented in Table ","element":"span"},{"href":"#id-119","text":"13. ","element":"a"},{"text":"We find that the difference between the performance on different categories is statistically significant (p-value < 0.05) for real-world use cases, multimodal interaction, external knowledge, and information flow dimensions, which explains that these are particularly difficult dimensions for today’s multimodal model.","element":"span"}],[{"text":"We also conducted t-tests for various categories in each of the modeling dimensions. For all models in a category (","element":"span"},{"style":{"height":9.18},"width":28.24,"height":22.96,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/35-3.png","element":"img","alt":"ci","inline":true},{"text":"), we use their average performance on each of the 28 datasets to construct a vector ","element":"span"},{"style":{"height":16.1},"width":139.36,"height":40.24,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/35-4.png","element":"img","alt":"wi ∈ R28","inline":true},{"text":". We then perform pair-wise t-tests across all the categories for all dimensions. As mentioned","element":"span"}],[{"text":"Table 12: Hyperparameters used for running inference for various models. Temperature for GPT-4V and Beam ","element":"figcaption","subtype":"caption"},{"text":"Size for GPT-4V and G","element":"figcaption","subtype":"caption"},{"text":"EMINI ","element":"figcaption","subtype":"caption"},{"text":"are unknown. We also report the average inference time in seconds for an image-text input. For each models, we take the average of inference times across all the datasets.","element":"figcaption","subtype":"caption"}],[{"id":"id-58","style":{"width":"64%"},"width":1588,"height":2347,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/36-0.png","element":"img"}],[{"id":"id-119","text":"Table 13: Standard deviation and p-values (from paired t-tests) across categories for each dataset dimension. On ","element":"figcaption","subtype":"caption"},{"text":"average, models achieve significantly higher scores on Multimedia and Affect as compared to other use cases. The p-values for Reasoning and Granularity dimensions are higher than 0.05, indicating that there is no category significantly more challenging than the rest.","element":"figcaption","subtype":"caption"}],[{"style":{"width":"84%"},"width":1347,"height":1408,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/37-0.png","element":"img"}]]},{"heading":"C All Results","paragraphs":[[{"text":"Due to query limits for GPT-4V and G","element":"span"},{"text":"EMINI","element":"span"},{"text":", we evaluated the two models only on 100 samples per dataset, and for a fair comparison, we performed our analysis using the outputs of all the models on those 100 samples. In this section, we present the results and analysis on the whole evaluation set using the outputs of all the models except GPT-4V and G","element":"span"},{"text":"EMINI","element":"span"},{"text":". Further, since our analysis was based on text-generation tasks, we present here the results on the image-generation tasks - M","element":"span"},{"text":"AGIC ","element":"span"},{"text":"B","element":"span"},{"text":"RUSH ","element":"span"},{"text":"and LNCOCO. Specifically, we evaluated E","element":"span"},{"text":"MU ","element":"span"},{"text":"(only model in HEMM that can generate images) on both tasks. We find the MSE and the CLIP-I score between the generated and the ground truth image for M","element":"span"},{"text":"AGIC ","element":"span"},{"text":"B","element":"span"},{"text":"RUSH ","element":"span"},{"text":"to be 0.17 and 0.54. For the LNCOCO dataset, the MSE and CLIP-I score are 0.18 and 0.50.","element":"span"}],[{"text":"Note: due to high inference time of some models (e.g., M","element":"span"},{"text":"INI","element":"span"},{"text":"-GPT-4, E","element":"span"},{"text":"MU","element":"span"},{"text":", O","element":"span"},{"text":"PEN","element":"span"},{"text":"F","element":"span"},{"text":"LAMINGO","element":"span"},{"text":"), missing image URLs in the N","element":"span"},{"text":"LVR","element":"span"},{"text":"2 dataset, and compute restrictions for larger evaluation sets like MM-IMD","element":"span"},{"text":"B","element":"span"},{"text":", V","element":"span"},{"text":"ISUAL ","element":"span"},{"text":"G","element":"span"},{"text":"ENOME","element":"span"},{"text":", and ","element":"span"},{"text":"I","element":"span"},{"text":"N","element":"span"},{"text":"ATURALIST","element":"span"},{"text":", we use the results from the same 100 samples used for evaluation in Section ","element":"span"},{"text":"4.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"C.1 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Dataset and model comparisons Dataset comparisons: ","element":"span"},{"text":"On average, the models achieve the highest scores on IRFL (0.53), W","element":"span"},{"text":"INOGROUND ","element":"span"},{"text":"(0.42), and N","element":"span"},{"text":"LVR ","element":"span"},{"text":"(0.40) datasets. Healthcare, Science, and HCI datasets are the most challenging use cases for the models with the average scores being the lowest for D","element":"span"},{"text":"ECIMER ","element":"span"},{"text":"(0.05),","element":"span"}],[{"text":"Table 14: Standard deviation and p-values for categories in various modeling dimensions. Models in the ","element":"figcaption","subtype":"caption"},{"text":"best-performing category in each dimension, receive significantly higher scores than the other categories.","element":"figcaption","subtype":"caption"}],[{"style":{"width":"64%"},"width":1589,"height":2430,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/38-0.png","element":"img"}],[{"text":"Table 15: Comparisons on different dataset categories. ","element":"figcaption","subtype":"caption"},{"text":"30 multimodal datasets are split into various groups based on their real-world use case, type of multimodal interaction, presence of reasoning and external knowledge, granularity of alignment, and types of information flow. Performance is measured via the mean BARTscore across 9 multimodal models.","element":"figcaption","subtype":"caption"}],[{"text":"Table 16: Comparisons on different modeling decisions. We group models based on the modeling and training decisions, including how they process modalities, their parameter counts, model architecture, training data size and diversity, and the presence of instruction tuning. Performance is measured via the mean BARTscore across all 30 tested multimodal datasets.","element":"figcaption","subtype":"caption"}],[{"style":{"width":"99%"},"width":1584,"height":908,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/39-0.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Multimodal Skills 4: External Knowledge ","element":"span"},{"text":"Average performance of models on tasks requiring external knowledge (0.20) is significantly lower than tasks not requiring knowledge (0.24). For example, on average, models perform better on N","element":"span"},{"text":"LVR","element":"span"},{"text":", FER-2013 and W","element":"span"},{"text":"INOGROUND ","element":"span"},{"text":"that do not require external knowledge as compared to tasks like ","element":"span"},{"text":"I","element":"span"},{"text":"N","element":"span"},{"text":"ATURALIST ","element":"span"},{"text":"and S","element":"span"},{"text":"LAKE ","element":"span"},{"text":"which require external knowledge to identify appropriate species or organs in the image.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Multimodal Skills 5: Information flow ","element":"span"},{"text":"Models achieve significantly lower average score on translation datasets (0.17) as compared to querying (0.20) and fusion (0.29) datasets. Lower scores on translation dataset is due to the presence of highly challenging datasets such as D","element":"span"},{"text":"ECIMER ","element":"span"},{"text":"which requires domain knowledge of molecules to generate the correct textual sequence.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"C.3 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Modeling trends","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Model scale: ","element":"span"},{"text":"Since we do not consider GPT-4V and G","element":"span"},{"text":"EMINI ","element":"span"},{"text":"for analysis in this section, there are no models in the large category. Amongst small and medium models, we find no significant difference (p-value = 0.45) between the average performance of models from the two categories with small and medium models receiving 0.21 and 0.23 average scores respectively.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Pretraining data scale: ","element":"span"},{"text":"On average, models with medium pretraining data achieve the highest score (0.28) as compared to the models pretrained with small (0.16) or large (0.21) scale data. Although the average score of models trained with large pretraining data is lower as compared to models trained with medium pretraining data, we find that the former models perform better on tasks such as IRFL, W","element":"span"},{"text":"INOGROUND","element":"span"},{"text":", M","element":"span"},{"text":"EME","element":"span"},{"text":"C","element":"span"},{"text":"AP","element":"span"},{"text":", and D","element":"span"},{"text":"ECIMER ","element":"span"},{"text":"which require complex reasoning and external knowledge.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Diversity of pretraining data: ","element":"span"},{"text":"Models trained with diverse pretraining data (0.24) perform better than models trained only on image-captioning datasets (0.20). The p-value for the paired t-test is 0.01 indicating that the difference is significant. On average, we find that models pretrained with diverse data achieve better scores on knowledge-intensive tasks such as ","element":"span"},{"text":"I","element":"span"},{"text":"N","element":"span"},{"text":"ATURALIST ","element":"span"},{"text":"and OK-VQA with improvements in average scores up to 0.21.","element":"span"}],[{"id":"id-49","style":{"width":"64%"},"width":1580,"height":701,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/40-0.png","element":"img"}],[{"text":"Figure 8: Model outputs on samples from E","element":"figcaption","subtype":"caption"},{"text":"NRICO","element":"figcaption","subtype":"caption"},{"text":", VQARAD, ","element":"figcaption","subtype":"caption"},{"text":"I","element":"figcaption","subtype":"caption"},{"text":"N","element":"figcaption","subtype":"caption"},{"text":"ATURALIST","element":"figcaption","subtype":"caption"},{"text":", and S","element":"figcaption","subtype":"caption"},{"text":"CIENCE","element":"figcaption","subtype":"caption"},{"text":"QA. In (a), all the models struggle to reason about the use of the ","element":"figcaption","subtype":"caption"},{"style":{"fontStyle":"italic"},"text":"zip code ","element":"figcaption","subtype":"caption"},{"text":"field in the UI, which will be used to ","element":"figcaption","subtype":"caption"},{"style":{"fontStyle":"italic"},"text":"search ","element":"figcaption","subtype":"caption"},{"text":"the TV provider. Example (b) underscores the complexity faced by models in interpreting medical images, particularly evident in their inability to recognize the absence of a kidney in the radiology image. As shown in (c), the highly fine-grained ","element":"figcaption","subtype":"caption"},{"text":"I","element":"figcaption","subtype":"caption"},{"text":"N","element":"figcaption","subtype":"caption"},{"text":"ATURALIST ","element":"figcaption","subtype":"caption"},{"text":"dataset is very challenging and none of the models can determine the species of the insect. In (d), all models provide incorrect responses when tasked with identifying the colony’s name, illustrating the challenges posed by tasks requiring external knowledge.","element":"figcaption","subtype":"caption"}],[{"style":{"width":"64%"},"width":1586,"height":782,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/40-1.png","element":"img"}],[{"text":"1. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Challenging datasets","element":"span"},{"text":": Health, HCI, and Science are all relatively difficult use cases for today’s multimodal foundation models, which are statistically significantly harder than Multimedia and Affective Computing use cases. In particular, images of scientific diagrams, satellite images, medical images, memes, and rich social interactions pose challenges. It is therefore important to evaluate multimodal models on a diverse range of input modalities and output tasks to get a better measure of generalization performance.","element":"span"}],[{"text":"2. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Multimodal interactions","element":"span"},{"text":": Models perform better on redundant interactions but struggle when visual information is not directly referenced by text (i.e., uniqueness or synergy). Future benchmarks should contain richer multimodal interactions beyond redundancy, such as in analyzing sarcasm, humor, memes, science, environment, and education. These can serve as better test beds for multimodal models and enable their applications towards real-world multimodal interactions.","element":"span"}],[{"text":"3. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Reasoning, fine-grained, and knowledge","element":"span"},{"text":": We need better datasets that test for complex reasoning and fine-grained alignment - current ones do not pose enough challenges to today’s models, with no significant performance differences with or without reasoning and fine-grained alignment.","element":"span"}],[{"text":"We do find that tasks requiring external knowledge are significantly harder than no knowledge; bridging this gap can be a promising direction for multimodal research.","element":"span"}],[{"text":"4. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Model and data size","element":"span"},{"text":": Perhaps unsurprisingly, larger scales of data and models improve the average score across the board, with significant improvements of up to 75% as compared to medium-sized models. Training on diverse data sources also improves over models that only pretrain on images and captions. The tasks that show the most improvement are ","element":"span"},{"text":"I","element":"span"},{"text":"N","element":"span"},{"text":"ATURALIST ","element":"span"},{"text":"and M","element":"span"},{"text":"EME","element":"span"},{"text":"C","element":"span"},{"text":"AP ","element":"span"},{"text":"which are knowledge-intensive and require complex reasoning.","element":"span"}],[{"text":"5. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Model training","element":"span"},{"text":": Instruction-tuned models performed better than those with only supervised fine-tuning. Cross-modal translation (image-to-text) tasks show the most improvements (e.g., D","element":"span"},{"text":"ECIMER","element":"span"},{"text":", M","element":"span"},{"text":"EME","element":"span"},{"text":"C","element":"span"},{"text":"AP","element":"span"},{"text":", and S","element":"span"},{"text":"CREEN","element":"span"},{"text":"2W","element":"span"},{"text":"ORDS","element":"span"},{"text":"). However, some instruction-tuned models still struggle to follow the instructions (e.g., generating a caption when asked to classify an image, or generating long responses when asked to answer in a few words). Instruction tuning using larger datasets with diverse instructions can help alleviate this problem.","element":"span"}]]}],"_version":"3.3.4"},"paperNode":"$21:props:children:props:children:0:props:product"}]] </article></body></html>

34:[["$","audio",null,{"id":"tts"}],["$","$L39",null,{"paperID":"97825","publisher":"neurips","paperJSON":{"title":"HEMM: Holistic Evaluation of Multimodal Foundation Models","paperID":"97825","avgLineHeight":10.89,"imgScale":4,"sections":[{"heading":"Abstract","paragraphs":[[{"text":"Multimodal foundation models that can holistically process text alongside images, video, audio, and other sensory modalities are increasingly used in a variety of real-world applications. However, it is challenging to characterize and study progress in multimodal foundation models, given the range of possible modeling decisions, tasks, and domains. In this paper, we introduce Holistic Evaluation of Multimodal Models (HEMM) to systematically evaluate the capabilities of multimodal foundation models across a set of 3 dimensions: basic skills, information flow, and real-world use cases. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Basic multimodal skills ","element":"span"},{"text":"are internal abilities required to solve problems, such as learning interactions across modalities, fine-grained alignment, multi-step reasoning, and the ability to handle external knowledge. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Information flow ","element":"span"},{"text":"studies how multimodal content changes during a task through querying, translation, editing, and fusion. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Use cases ","element":"span"},{"text":"span domain-specific challenges introduced in real-world multimedia, affective computing, natural sciences, healthcare, and human-computer interaction applications. Through comprehensive experiments across the 30 tasks in HEMM, we (1) identify key ","element":"span"},{"style":{"fontStyle":"italic"},"text":"dataset dimensions ","element":"span"},{"text":"(e.g., basic skills, information flows, and use cases) that pose challenges to today’s models, and (2) distill performance trends regarding how different ","element":"span"},{"style":{"fontStyle":"italic"},"text":"modeling dimensions ","element":"span"},{"text":"(e.g., scale, pre-training data, multimodal alignment, pre-training, and instruction tuning objectives) influence performance. Our conclusions regarding challenging multimodal interactions, use cases, and tasks requiring reasoning and external knowledge, the benefits of data and model scale, and the impacts of instruction tuning yield actionable insights for future work in multimodal foundation models.","element":"span"}]]},{"heading":"1 Introduction","paragraphs":[[{"text":"Building upon rapid progress in large-scale language and vision pretraining [","element":"span"},{"href":"#id-0","referenceIndex":24,"text":"24","element":"a"},{"text":", ","element":"span"},{"href":"#id-1","referenceIndex":69,"text":"69","element":"a"},{"text":", ","element":"span"},{"href":"#id-2","referenceIndex":106,"text":"106","element":"a"},{"text":"], the new generation of multimodal foundation models is increasing adept at learning interactions between modalities [","element":"span"},{"href":"#id-3","referenceIndex":83,"text":"83","element":"a"},{"text":"], enables both static prediction and dynamic interaction [","element":"span"},{"href":"#id-4","referenceIndex":55,"text":"55","element":"a"},{"text":"], and even shows emergent properties never seen before in pretraining corpora [","element":"span"},{"href":"#id-5","referenceIndex":60,"text":"60","element":"a"},{"text":"]. Previous standards for benchmarking multimodal models based on collections of modality and task-specific datasets [","element":"span"},{"href":"#id-6","referenceIndex":8,"text":"8","element":"a"},{"text":", ","element":"span"},{"href":"#id-7","referenceIndex":57,"text":"57","element":"a"},{"text":", ","element":"span"},{"href":"#id-8","referenceIndex":29,"text":"29","element":"a"},{"text":", ","element":"span"},{"href":"#id-9","referenceIndex":66,"text":"66","element":"a"},{"text":"] are increasingly insufficient in light of these general capabilities. In order to study fundamental questions regarding ","element":"span"},{"style":{"fontStyle":"italic"},"text":"why ","element":"span"},{"text":"multimodal foundation models exhibit certain behaviors, ","element":"span"},{"style":{"fontStyle":"italic"},"text":"when ","element":"span"},{"text":"they perform well in the real world, and ","element":"span"},{"style":{"fontStyle":"italic"},"text":"which ","element":"span"},{"text":"modeling paradigms are most effective, there is a need for a holistic evaluation scheme beyond individual datasets or contexts.","element":"span"}],[{"text":"To address this need, we contribute ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Holistic Evaluation of Multimodal Models (HEMM)","element":"span"},{"text":", visualized in Figure ","element":"span"},{"href":"#id-10","text":"1. ","element":"a"},{"text":"HEMM, as an evaluation framework, goes beyond conventional lists of datasets to emphasize holistic benchmarking at three levels. The first level benchmarks ","element":"span"},{"style":{"fontStyle":"italic"},"text":"basic multimodal skills","element":"span"},{"text":": fundamental internal abilities required to address multimodal problems, such as interactions between","element":"span"}],[{"id":"id-10","style":{"width":"79%"},"width":1260,"height":681,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/1-0.png","element":"img"}],[{"text":"Figure 1: HEMM is an evaluation framework that characterizes multimodal models along several dimensions (size, architecture, pretraining objective, fine-tuning objective, training data) and emphasizes holistic benchmarking of these models at three disentangled levels: basic skills, information flow, and use cases.","element":"figcaption","subtype":"caption"}],[{"text":"redundant, unique, and synergistic features [","element":"span"},{"href":"#id-11","referenceIndex":26,"text":"26","element":"a"},{"text":", ","element":"span"},{"href":"#id-12","referenceIndex":68,"text":"68","element":"a"},{"text":"], alignment of fine-grained and coarse-grained information [","element":"span"},{"href":"#id-13","referenceIndex":104,"text":"104","element":"a"},{"text":"], reasoning across compositional features [","element":"span"},{"href":"#id-14","referenceIndex":115,"text":"115","element":"a"},{"text":"], and integration of external knowledge [","element":"span"},{"href":"#id-15","referenceIndex":90,"text":"90","element":"a"},{"text":"]. The second level benchmarks ","element":"span"},{"style":{"fontStyle":"italic"},"text":"information flow","element":"span"},{"text":": how multimodal information transforms during tasks such as querying [","element":"span"},{"href":"#id-16","referenceIndex":98,"text":"98","element":"a"},{"text":"], translation [","element":"span"},{"href":"#id-17","referenceIndex":109,"text":"109","element":"a"},{"text":"], editing [","element":"span"},{"href":"#id-18","referenceIndex":108,"text":"108","element":"a"},{"text":"], and fusion [","element":"span"},{"href":"#id-5","referenceIndex":60,"text":"60","element":"a"},{"text":"]. The third level benchmarks ","element":"span"},{"style":{"fontStyle":"italic"},"text":"multimodal use cases","element":"span"},{"text":": how models perform in real-world challenges across domains, including multimedia, affective computing, natural sciences, healthcare, and human-computer interaction (HCI). Together, these three levels taxonomize a wide spectrum of ","element":"span"},{"text":"30 ","element":"span"},{"text":"image-text datasets, enabling HEMM to serve as a holistic framework to evaluate multimodal models.","element":"span"}],[{"text":"To aid in HEMM evaluation, we also present a new categorization of models spanning key ","element":"span"},{"style":{"fontStyle":"italic"},"text":"modeling decisions","element":"span"},{"text":", such as model size and modality processing (e.g., interleaved inputs), and ","element":"span"},{"style":{"fontStyle":"italic"},"text":"training decisions","element":"span"},{"text":", such as pretraining and fine-tuning objectives. We (1) identify key ","element":"span"},{"style":{"fontStyle":"italic"},"text":"dataset dimensions ","element":"span"},{"text":"(e.g., basic skills, information flows, and use cases) that pose challenges to today’s models, and (2) distill performance trends regarding how different ","element":"span"},{"style":{"fontStyle":"italic"},"text":"modeling and training decisions ","element":"span"},{"text":"(e.g., scale, pre-training data, multimodal alignment, pre-training, and instruction tuning objectives) influence downstream task performance. Our analysis yields tangible directions for future work, including challenging multimodal skills, tasks, and use cases, impacts of diversity and scale, and guidelines on modeling architectures and training objectives. HEMM is publicly available at ","element":"span"},{"style":{"fontFamily":"monospace"},"text":"anon","element":"span"},{"text":", and encourages community involvement in its expansion of datasets, annotations, models, and evaluation metrics.","element":"span"}]]},{"heading":"2 Key Benchmarking Principles and Datasets in HEMM","paragraphs":[[{"text":"HEMM includes 30 datasets summarized in Table ","element":"span"},{"href":"#id-19","text":"1. ","element":"a"},{"text":"These datasets require different multimodal skills to solve, display different types of multimodal information flow, and belong to different real-world use cases with domain-specific challenges.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"2.1 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Basic multimodal skills","element":"span"}],[{"text":"Multimodal skills are internal abilities required to solve multimodal tasks, such as learning interactions across modalities, fine-grained alignment, multi-step reasoning, and using external knowledge.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Multimodal interactions ","element":"span"},{"text":"study how modality information is integrated for a multimodal task [","element":"span"},{"href":"#id-1","referenceIndex":69,"text":"69","element":"a"},{"text":", ","element":"span"},{"href":"#id-20","referenceIndex":77,"text":"77","element":"a"},{"text":", ","element":"span"},{"href":"#id-21","referenceIndex":52,"text":"52","element":"a"},{"text":", ","element":"span"},{"href":"#id-22","referenceIndex":9,"text":"9","element":"a"},{"text":"], which can be ","element":"span"},{"style":{"fontStyle":"italic"},"text":"redundant","element":"span"},{"text":": shared between modalities, such as smiling while telling a humorous joke [","element":"span"},{"href":"#id-23","referenceIndex":43,"text":"43","element":"a"},{"text":", ","element":"span"},{"href":"#id-24","referenceIndex":89,"text":"89","element":"a"},{"text":"], ","element":"span"},{"style":{"fontStyle":"italic"},"text":"unique","element":"span"},{"text":": present in only one of the modalities [","element":"span"},{"href":"#id-25","referenceIndex":35,"text":"35","element":"a"},{"text":", ","element":"span"},{"href":"#id-26","referenceIndex":54,"text":"54","element":"a"},{"text":"], and ","element":"span"},{"style":{"fontStyle":"italic"},"text":"synergistic","element":"span"},{"text":": emergence of new information from both modalities, such as conveying sarcasm through conflicting verbal and nonverbal cues [","element":"span"},{"href":"#id-27","referenceIndex":15,"text":"15","element":"a"},{"text":", ","element":"span"},{"href":"#id-12","referenceIndex":68,"text":"68","element":"a"},{"text":"]. Datasets with high referential information between modalities test for redundancy, such as in VQA, and translation on N","element":"span"},{"text":"O","element":"span"},{"text":"C","element":"span"},{"text":"APS","element":"span"},{"text":". Tasks with uniqueness or synergy include understanding movie posters (MM-IMD","element":"span"},{"text":"B","element":"span"},{"text":"), memes (M","element":"span"},{"text":"EME","element":"span"},{"text":"C","element":"span"},{"text":"AP","element":"span"},{"text":"), figurative language (IRFL), facial expressions (FER-2013), and cartoons (N","element":"span"},{"text":"EW ","element":"span"},{"text":"Y","element":"span"},{"text":"ORKER ","element":"span"},{"text":"C","element":"span"},{"text":"ARTOON","element":"span"},{"text":").","element":"span"}],[{"id":"id-19","text":"Table 1: HEMM includes a comprehensive suite of ","element":"figcaption","subtype":"caption"},{"text":"30 ","element":"figcaption","subtype":"caption"},{"text":"datasets to benchmark multimodal foundation models. We categorize each dataset based on the ","element":"figcaption","subtype":"caption"},{"style":{"fontStyle":"italic"},"text":"basic multimodal skills ","element":"figcaption","subtype":"caption"},{"text":"needed to solve them – the type of multimodal interaction, granularity of multimodal alignment, level of reasoning, and need for external knowledge, how ","element":"figcaption","subtype":"caption"},{"style":{"fontStyle":"italic"},"text":"information flows ","element":"figcaption","subtype":"caption"},{"text":"between modalities, and the real-world ","element":"figcaption","subtype":"caption"},{"style":{"fontStyle":"italic"},"text":"use cases ","element":"figcaption","subtype":"caption"},{"text":"they impact.","element":"figcaption","subtype":"caption"}],[{"style":{"width":"64%"},"width":1586,"height":1152,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/2-0.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Granularity of multimodal alignment ","element":"span"},{"text":"involves identifying alignment across elements in different modalities. For example, answering a question might require a model to perform fine-grained alignment to reference one specific object out of many possible objects in an image. Tasks that explicitly test for fine-grained alignment include localized reasoning on V","element":"span"},{"text":"ISUAL ","element":"span"},{"text":"G","element":"span"},{"text":"ENOME","element":"span"},{"text":", W","element":"span"},{"text":"INOGROUND","element":"span"},{"text":", while tasks that emphasize coarse-grained alignment (e.g., making a prediction relevant to a whole image) include interpreting cartoon images ","element":"span"},{"href":"#id-28","referenceIndex":37,"text":"[37]","element":"a"},{"text":", movie posters ","element":"span"},{"href":"#id-29","referenceIndex":5,"text":"[5]","element":"a"},{"text":", and memes ","element":"span"},{"href":"#id-30","referenceIndex":46,"text":"[46, ","element":"a"},{"href":"#id-24","referenceIndex":89,"text":"89, ","element":"a"},{"href":"#id-23","referenceIndex":43,"text":"43]","element":"a"},{"text":".","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Reasoning and external knowledge ","element":"span"},{"text":"involve the combination of local pieces of information to form increasingly rich and complex multimodal representations. For example, being able to perform multi-hop inference from Wikipedia text and images [","element":"span"},{"href":"#id-31","referenceIndex":76,"text":"76","element":"a"},{"text":"] or solving science questions given visual diagrams and executing multiple logical steps [","element":"span"},{"href":"#id-32","referenceIndex":75,"text":"75","element":"a"},{"text":"]. Tasks like W","element":"span"},{"text":"INOGROUND ","element":"span"},{"text":"explicitly test for reasoning and tasks like OK-VQA are designed to assess external knowledge.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"2.2 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Multimodal information flow","element":"span"}],[{"text":"Multimodal information flow studies how information transforms across tasks, including cross-modal translation, editing, querying, and fusion.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Cross-modal translation ","element":"span"},{"text":"exploits shared information by mapping data in one modality to another. Examples include translating from text to image for image generation (e.g., LNCOCO) and translating from image to text for image captioning (e.g., N","element":"span"},{"text":"O","element":"span"},{"text":"C","element":"span"},{"text":"APS","element":"span"},{"text":", S","element":"span"},{"text":"CREEN","element":"span"},{"text":"2W","element":"span"},{"text":"ORDS","element":"span"},{"text":").","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Cross-modal editing ","element":"span"},{"text":"involves semantically editing data in one modality according to another modality (e.g., given an image, following a natural language instruction to \"change the background from day to night\"). The model takes in the original image (with potentially more reference images), along with a task description specifying the edit, and outputs the edited image. We use the M","element":"span"},{"text":"AGIC ","element":"span"},{"text":"B","element":"span"},{"text":"RUSH ","element":"span"},{"text":"dataset to test cross-modal editing.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Cross-modal querying ","element":"span"},{"text":"involves a model’s ability to answer natural language questions that query specific information about an input. The model takes in the original image, a text description, the","element":"span"}],[{"id":"id-40","text":"Table 2: Models used in HEMM, ranked from small to large, and categorized by #Param (model size), Data ","element":"figcaption","subtype":"caption"},{"text":"Size (pretraining data size), Data Diversity (pretraining data diversity), Training Type (end-to-end training or frozen alignment), INST (instruction tuning), Modality Proc (interleaved or separate modality inputs).","element":"figcaption","subtype":"caption"}],[{"style":{"width":"84%"},"width":1334,"height":477,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/3-0.png","element":"img"}],[{"text":"query, and must output the desired answer (typically in natural language). Querying can be done for visual scenes (GQA), environmental indicators (R","element":"span"},{"text":"ESISC","element":"span"},{"text":"45), and medical data (VQARAD).","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Multimodal fusion ","element":"span"},{"text":"aims to learn interactions to combine information from different modalities, such as classifying diseases given x-ray images and medical tests, or detecting humor from cartoon images and captions. Multimodal fusion takes in the image, text, and a description of the task, and then outputs a prediction, which can include affective states like humor in N","element":"span"},{"text":"EW ","element":"span"},{"text":"Y","element":"span"},{"text":"ORKER ","element":"span"},{"text":"C","element":"span"},{"text":"ARTOON","element":"span"},{"text":", hate speech detection in H","element":"span"},{"text":"ATEFUL ","element":"span"},{"text":"M","element":"span"},{"text":"EMES","element":"span"},{"text":", or in science problems (S","element":"span"},{"text":"CIENCE","element":"span"},{"text":"QA).","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"2.3 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Real-world Use Cases","element":"span"}],[{"text":"Each use case is drawn from a real-world application with their own specific challenges.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Multimedia ","element":"span"},{"text":"includes efficient search, retrieval, indexing, and generation of digital content. Multimedia tasks in HEMM include question answering about images and videos (VQA, VCR), multimedia captioning (F","element":"span"},{"text":"LICKR","element":"span"},{"text":"30K, N","element":"span"},{"text":"O","element":"span"},{"text":"C","element":"span"},{"text":"APS","element":"span"},{"text":"), compositional visual reasoning (W","element":"span"},{"text":"INOGROUND","element":"span"},{"text":", N","element":"span"},{"text":"LVR","element":"span"},{"text":"), understanding cartoons, movie posters (MM-IMD","element":"span"},{"text":"B","element":"span"},{"text":"), memes (M","element":"span"},{"text":"EME","element":"span"},{"text":"C","element":"span"},{"text":"AP ","element":"span"},{"text":"and M","element":"span"},{"text":"EMOTION","element":"span"},{"text":"), and figurative language (IRFL), and editing images (M","element":"span"},{"text":"AGIC ","element":"span"},{"text":"B","element":"span"},{"text":"RUSH","element":"span"},{"text":").","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Affective computing ","element":"span"},{"text":"aims to perceive human affective states (emotions, sentiment, personalities, humor, sarcasm, social interactions) [","element":"span"},{"href":"#id-33","referenceIndex":86,"text":"86","element":"a"},{"text":"], and is important for building emotionally and sociallyintelligent AI [","element":"span"},{"href":"#id-34","referenceIndex":56,"text":"56","element":"a"},{"text":", ","element":"span"},{"href":"#id-35","referenceIndex":78,"text":"78","element":"a"},{"text":"] and human-AI interaction [","element":"span"},{"href":"#id-4","referenceIndex":55,"text":"55","element":"a"},{"text":"]. HEMM includes N","element":"span"},{"text":"EW ","element":"span"},{"text":"Y","element":"span"},{"text":"ORKER ","element":"span"},{"text":"C","element":"span"},{"text":"ARTOON ","element":"span"},{"text":"(cartoon images and captions), H","element":"span"},{"text":"ATEFUL ","element":"span"},{"text":"M","element":"span"},{"text":"EMES ","element":"span"},{"text":"(hateful content in memes), FER-2013 for facial expressions, M","element":"span"},{"text":"EME","element":"span"},{"text":"C","element":"span"},{"text":"AP ","element":"span"},{"text":"for meme captioning, and M","element":"span"},{"text":"EMOTION ","element":"span"},{"text":"for emotions in memes.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Natural sciences ","element":"span"},{"text":"aims to deepen our knowledge of physical, chemical, biological, and environmental sciences. These can involve satellite images, chemical bonds, land and agriculture use, wildlife, and specific scientific terminologye [","element":"span"},{"href":"#id-36","referenceIndex":101,"text":"101","element":"a"},{"text":"]. Tasks in HEMM include S","element":"span"},{"text":"CIENCE","element":"span"},{"text":"QA testing different science topics and R","element":"span"},{"text":"ESISC","element":"span"},{"text":"45 for land scene classification.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Healthcare ","element":"span"},{"text":"involves integrating multimodal signals such as lab tests, imaging reports, and doctorpatient interactions to help doctors interpret high-dimensional data and assist them in diagnosis ","element":"span"},{"href":"#id-37","referenceIndex":48,"text":"[48, ","element":"a"},{"href":"#id-38","referenceIndex":51,"text":"51","element":"a"},{"text":"]. We include processing text reports and medical images in the form of P","element":"span"},{"text":"ATH","element":"span"},{"text":"VQA for pathology, VQARAD for radiology images, and S","element":"span"},{"text":"LAKE ","element":"span"},{"text":"for medical visual question answering.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"HCI ","element":"span"},{"text":"involves user design, usability, user experience, and other challenges related to humans interacting with computers [","element":"span"},{"href":"#id-39","referenceIndex":81,"text":"81","element":"a"},{"text":"]. HCI tasks can involve visual information such as screen layouts, user actions, and feedback mechanisms. HCI tasks in HEMM include E","element":"span"},{"text":"NRICO ","element":"span"},{"text":"for classifying mobile UI designs and S","element":"span"},{"text":"CREEN","element":"span"},{"text":"2W","element":"span"},{"text":"ORDS ","element":"span"},{"text":"for UI screen content summarization.","element":"span"}]]},{"heading":"3 Key Modeling Principles and Models in HEMM","paragraphs":[[{"text":"Table ","element":"span"},{"href":"#id-40","text":"2 ","element":"a"},{"text":"summarizes the 11 models we evaluate in HEMM, which span different numbers of parameters, model architectures, training datasets, pretraining objectives, and fine-tuning objectives.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"3.1 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Modeling decisions","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Model parameters ","element":"span"},{"text":"Parameters can vary greatly across different multimodal models, from 100M params to approximately 1000B params. We consider models with total number of parameters less","element":"span"}],[{"id":"id-47","text":"Table 3: Performance on different dataset dimensions, ","element":"figcaption","subtype":"caption"},{"text":"as measured via the mean BARTscore on each dataset across all 11 tested multimodal models.","element":"figcaption","subtype":"caption"}],[{"style":{"width":"31%"},"width":768,"height":874,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/4-0.png","element":"img"}],[{"text":"Table 4: Performance on different modeling decisions, as measured via the mean BARTscore for each model across all 30 tested multimodal datasets.","element":"figcaption","subtype":"caption"}],[{"style":{"width":"53%"},"width":768,"height":876,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/4-1.png","element":"img"}],[{"text":"than or equal to 4B (e.g., I","element":"span"},{"text":"NSTRUCT","element":"span"},{"text":"-BLIP) as ","element":"span"},{"style":{"fontStyle":"italic"},"text":"small","element":"span"},{"text":", whereas those having more than 4B parameters (e.g., F","element":"span"},{"text":"UYU","element":"span"},{"text":"-8B) are considered ","element":"span"},{"style":{"fontStyle":"italic"},"text":"medium","element":"span"},{"text":". GPT-4V and G","element":"span"},{"text":"EMINI ","element":"span"},{"text":"are considered ","element":"span"},{"style":{"fontStyle":"italic"},"text":"large","element":"span"},{"text":".","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Modality processing ","element":"span"},{"text":"Some multimodal models (e.g., F","element":"span"},{"text":"UYU","element":"span"},{"text":"-8B) support interleaved inputs like “","element":"span"},{"style":{"fontFamily":"monospace"},"text":" This is a very cute dog. This is a very cute cat.","element":"span"},{"text":"”, unlike models that only support separate image and text queries (e.g., BLIP-2, M","element":"span"},{"text":"INI","element":"span"},{"text":"-GPT-4).","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"3.2 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Training Characteristics","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Training type ","element":"span"},{"text":"End-to-end training involves fine-tuning unimodal encoders, pretrained LLMs, and a multimodal model jointly, as seen in E","element":"span"},{"text":"MU","element":"span"},{"text":", F","element":"span"},{"text":"UYU","element":"span"},{"text":"-8B, etc. Another category operates by freezing unimodal encoders and LLM, and then training only a mapping that aligns frozen image features with frozen LLM features. These trainable mappings include Q-former [","element":"span"},{"href":"#id-41","referenceIndex":22,"text":"22","element":"a"},{"text":"] (used in I","element":"span"},{"text":"NSTRUCT","element":"span"},{"text":"-BLIP), linear layers [","element":"span"},{"href":"#id-42","referenceIndex":128,"text":"128","element":"a"},{"text":", ","element":"span"},{"href":"#id-43","referenceIndex":92,"text":"92","element":"a"},{"text":"] (used in M","element":"span"},{"text":"INI","element":"span"},{"text":"-GPT-4), and attention blocks used in O","element":"span"},{"text":"PEN","element":"span"},{"text":"F","element":"span"},{"text":"LAMINGO","element":"span"},{"text":".","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Size of pre-training data ","element":"span"},{"text":"We consider the total size of pre-training data used for training, including instruction and supervised data. E","element":"span"},{"text":"MU ","element":"span"},{"text":"has ","element":"span"},{"style":{"fontStyle":"italic"},"text":"small ","element":"span"},{"text":"data scale, with less than 100M training data points. F","element":"span"},{"text":"UYU","element":"span"},{"text":"-8B has ","element":"span"},{"style":{"fontStyle":"italic"},"text":"medium ","element":"span"},{"text":"data-scale, with more than 100M training data points. While GPT-4V and G","element":"span"},{"text":"EMINI ","element":"span"},{"text":"do not release data sizes, we estimate their size to be much larger than other models and therefore are considered to have ","element":"span"},{"style":{"fontStyle":"italic"},"text":"large ","element":"span"},{"text":"data scale.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Diversity of pre-training data ","element":"span"},{"text":"We consider the diversity of multimodal tasks used for training, including visual QA, visual conversations, and interleaved images and text. I","element":"span"},{"text":"NSTRUCT","element":"span"},{"text":"-BLIP and E","element":"span"},{"text":"MU ","element":"span"},{"text":"are pre-trained on diverse data, in contrast to LL","element":"span"},{"text":"A","element":"span"},{"text":"MA-A","element":"span"},{"text":"DAPTER","element":"span"},{"text":", O","element":"span"},{"text":"PEN","element":"span"},{"text":"F","element":"span"},{"text":"LAMINGO","element":"span"},{"text":", etc., which only use image captioning data for training.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Instruction tuning ","element":"span"},{"text":"By transforming supervised tasks into an ‘instruction’ format, instruction tuning has been shown to benefit performance and improve the controllability of LLMs. M","element":"span"},{"text":"INI","element":"span"},{"text":"-GPT-4 and I","element":"span"},{"text":"NSTRUCT","element":"span"},{"text":"-BLIP include an instruction tuning stage, while models like BLIP-2 do not.","element":"span"}]]},{"heading":"4 Experiments","paragraphs":[[{"text":"In this section, we discuss extensive experiments conducted to holistically evaluate the performance of multimodal foundation models based on HEMM.","element":"span"}],[{"id":"id-48","style":{"width":"99%"},"width":1583,"height":493,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/5-0.png","element":"img"}],[{"text":"Figure 2: Responses of GPT-4V and G","element":"figcaption","subtype":"caption"},{"text":"EMINI ","element":"figcaption","subtype":"caption"},{"text":"on samples from the science category. These failure cases show that the models lack domain knowledge and are unable to correctly translate the images of molecules to the SMILES notations (a). Example (b) shows that the models struggle on tasks requiring complex reasoning, failing to comprehend the relation between the force and the size of the magnets. In (c), all models except GPT-4V are unable to capture the fine-grained details and misclassify the image as an airport instead of a runway.","element":"figcaption","subtype":"caption"}],[{"id":"id-99","style":{"fontWeight":"bold"},"text":"4.1 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Experimental setup","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Individual metrics ","element":"span"},{"text":"For all text generation tasks, we use the established natural language generation evaluation metric BARTScore [","element":"span"},{"href":"#id-44","referenceIndex":122,"text":"122","element":"a"},{"text":"], which was found to have the highest correlation with human judgement [","element":"span"},{"href":"#id-44","referenceIndex":122,"text":"122","element":"a"},{"text":"]. We compute BARTScore(r, c), where r is the reference and c is the candidate. It can be interpreted as the probability of generating the candidate sentence from the reference. For example, a model might caption an image with the following generated candidate: ","element":"span"},{"style":{"fontStyle":"italic"},"text":"A row of violins hanging on a wall.","element":"span"},{"text":". The reference (ground truth) of ","element":"span"},{"style":{"fontStyle":"italic"},"text":"A painting of 5 cello’s with a green background ","element":"span"},{"text":"would be used to compute the BARTScore with respect to c.","element":"span"}],[{"id":"id-100","style":{"fontWeight":"bold"},"text":"Aggregating metrics ","element":"span"},{"text":"To aggregate scores across multiple tasks or models, we normalize scores using min-max scaling. Following Chang et al. ","element":"span"},{"href":"#id-45","referenceIndex":16,"text":"[16]","element":"a"},{"text":", ","element":"span"},{"style":{"fontStyle":"italic"},"text":"min ","element":"span"},{"text":"represents the score of the worst multi-modal model and ","element":"span"},{"style":{"fontStyle":"italic"},"text":"max ","element":"span"},{"text":"represents the identity score BARTScore(r, r), where r is the ground truth. Subsequently, these normalized scores in the 0 to 1 range can be interpreted as a percentage of model performance relative to the ground truth.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Computation ","element":"span"},{"text":"Since GPT-4V and G","element":"span"},{"text":"EMINI ","element":"span"},{"text":"have query limits, we evaluate their performance on 100 random samples for each dataset (2800 total data points). For a fair comparison with other models, we present the results and findings below based on the performance of those 100 samples per dataset. In Appendix ","element":"span"},{"text":"C ","element":"span"},{"text":"we present the results of the other models on the full evaluation sets. We evaluate all the models on a single NVIDIA A100 80GB GPU with the inference time for a single image-text pair ranging from 0.1 seconds to 63.7 seconds. We report the average inference times for the models across all datasets and include additional details on the evaluation protocol in Appendix ","element":"span"},{"text":"B.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"4.2 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Main results","element":"span"}],[{"text":"We summarize our main results here and include full details in Appendix ","element":"span"},{"text":"C. ","element":"span"},{"text":"We first explain performance trends across the datasets in HEMM, before explaining performance differences across different multimodal foundation models and their design decisions.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"4.2.1 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Performance across dataset dimensions","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Overall comparisons ","element":"span"},{"text":"We summarize overall trends in Figure ","element":"span"},{"href":"#id-46","text":"3 ","element":"a"},{"text":"and Table ","element":"span"},{"href":"#id-47","text":"3. ","element":"a"},{"text":"On average, models perform better on multimedia datasets, with IRFL (0.58), N","element":"span"},{"text":"LVR ","element":"span"},{"text":"(0.50), and W","element":"span"},{"text":"INOGROUND ","element":"span"},{"text":"(0.49) showing the highest scores. The lowest scores are for Healthcare, HCI, and Science use cases, such as on D","element":"span"},{"text":"ECIMER ","element":"span"},{"text":"(0.07), ","element":"span"},{"text":"I","element":"span"},{"text":"N","element":"span"},{"text":"ATURALIST ","element":"span"},{"text":"(0.08), E","element":"span"},{"text":"NRICO ","element":"span"},{"text":"(0.12), P","element":"span"},{"text":"ATH","element":"span"},{"text":"VQA (0.15), and M","element":"span"},{"text":"EME","element":"span"},{"text":"C","element":"span"},{"text":"AP ","element":"span"},{"text":"(0.32). For predicting molecular structures on D","element":"span"},{"text":"ECIMER","element":"span"},{"text":", models are not able to generate correct chemical notations (in Simplified Molecular Input Line Entry System notation) and instead only generate names of individual atoms or compounds (see Figure ","element":"span"},{"href":"#id-48","text":"2)","element":"a"},{"text":". Other challenging datasets include ","element":"span"},{"text":"I","element":"span"},{"text":"N","element":"span"},{"text":"ATURALIST ","element":"span"},{"text":"due to fine-grained visual differences between 5000 species of plants and animals, and healthcare datasets that require intricate analysis of pathology images to identify organs, tissues, and anomalies (see Figure ","element":"span"},{"href":"#id-49","text":"8)","element":"a"},{"text":". Datasets related to memes were also challenging (0.32 and 0.38 on M","element":"span"},{"text":"EME","element":"span"},{"text":"C","element":"span"},{"text":"AP ","element":"span"},{"href":"#id-23","referenceIndex":43,"text":"[43] ","element":"a"},{"text":"and M","element":"span"},{"text":"EMOTION ","element":"span"},{"href":"#id-24","referenceIndex":89,"text":"[89]","element":"a"},{"text":"), requiring knowledge about current events, pop culture, and metaphors beyond literal meanings.","element":"span"}],[{"id":"id-50","style":{"width":"64%"},"width":1581,"height":796,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/6-0.png","element":"img"}],[{"text":"Figure 4: Tasks requiring commonsense and compositional reasoning are challenging. In (a), GPT-4V and G","element":"figcaption","subtype":"caption"},{"text":"EMINI ","element":"figcaption","subtype":"caption"},{"text":"are unable to employ social commonsense to analyze the relationships between the two people. Example (b) demonstrates the models’ difficulty in composing information from both modalities, leading to their failure to comprehend the scenario where ","element":"figcaption","subtype":"caption"},{"style":{"fontStyle":"italic"},"text":"a tree smashed into the car ","element":"figcaption","subtype":"caption"},{"text":"(not a car smashed into the tree). In (c), all models except GPT-4V fail to grasp the visual metaphors and the juxtaposition of the two scenarios.","element":"figcaption","subtype":"caption"}],[{"style":{"fontWeight":"bold"},"text":"Multimodal skills 1: Interactions ","element":"span"},{"text":"The average scores for redundant, unique, and synergistic interactions are 0.29, 0.20, and 0.33. One reason for lower uniqueness scores is the presence of highly challenging visual datasets like D","element":"span"},{"text":"ECIMER ","element":"span"},{"text":"and E","element":"span"},{"text":"NRICO","element":"span"},{"text":". On average, the easiest tasks in redundancy are N","element":"span"},{"text":"LVR ","element":"span"},{"text":"(0.50) and W","element":"span"},{"text":"INOGROUND ","element":"span"},{"text":"(0.49). The hardest datasets in uniqueness are ","element":"span"},{"text":"I","element":"span"},{"text":"N","element":"span"},{"text":"ATURALIST ","element":"span"},{"text":"(0.08) and D","element":"span"},{"text":"EC","element":"span"},{"text":"- ","element":"span"},{"text":"IMER ","element":"span"},{"text":"(0.07), and in synergy are M","element":"span"},{"text":"EME","element":"span"},{"text":"C","element":"span"},{"text":"AP ","element":"span"},{"text":"(0.14) and M","element":"span"},{"text":"EMOTION ","element":"span"},{"text":"(0.21).","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Multimodal skills 2: Granularity ","element":"span"},{"text":"We do not find that fine-grained datasets are significantly harder than those with coarse-grained alignment. Tasks requiring fine-grained alignment between image and text like GQA and W","element":"span"},{"text":"INOGROUND ","element":"span"},{"text":"achieve a score of 0.26, while those only needing coarse-grained alignment (e.g., E","element":"span"},{"text":"NRICO","element":"span"},{"text":", S","element":"span"},{"text":"CIENCE","element":"span"},{"text":"QA) are still quite challenging (score: 0.27).","element":"span"}],[{"id":"id-46","style":{"width":"54%"},"width":788,"height":658,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/6-1.png","element":"img"}],[{"text":"Figure 3: Average scores are higher for multimedia datasets as compared to other use cases, and lowest for healthcare, HCI, and science. The models struggle on ","element":"figcaption","subtype":"caption"},{"text":"I","element":"figcaption","subtype":"caption"},{"text":"N","element":"figcaption","subtype":"caption"},{"text":"ATURALIST","element":"figcaption","subtype":"caption"},{"text":", D","element":"figcaption","subtype":"caption"},{"text":"ECIMER","element":"figcaption","subtype":"caption"},{"text":", E","element":"figcaption","subtype":"caption"},{"text":"NRICO","element":"figcaption","subtype":"caption"},{"text":", P","element":"figcaption","subtype":"caption"},{"text":"ATH","element":"figcaption","subtype":"caption"},{"text":"VQA, and M","element":"figcaption","subtype":"caption"},{"text":"EME","element":"figcaption","subtype":"caption"},{"text":"C","element":"figcaption","subtype":"caption"},{"text":"AP ","element":"figcaption","subtype":"caption"},{"text":"which require external knowledge, fine-grained alignment, and complex reasoning.","element":"figcaption","subtype":"caption"}],[{"style":{"fontWeight":"bold"},"text":"Multimodal skills 3: Reasoning ","element":"span"},{"text":"We do not find a significant difference between the perfor-","element":"span"}],[{"text":"mance of the models on tasks requiring more (average score = 0.275) or less reasoning (average score ","element":"span"},{"style":{"height":14.4},"width":1184.48,"height":36,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/6-2.png","element":"img","alt":"= 0.268). The most challenging datasets requiring less reasoning include I","inline":true},{"text":"N","element":"span"},{"text":"ATURALIST ","element":"span"},{"text":"(0.08) and E","element":"span"},{"text":"NRICO ","element":"span"},{"text":"(0.12) due to challenges in fine-grained visual perception and external knowledge, while there are also several challenging datasets requiring more complex reasoning like VCR (0.34) and M","element":"span"},{"text":"EME","element":"span"},{"text":"C","element":"span"},{"text":"AP ","element":"span"},{"text":"(0.14), where the models encounter difficulties with samples requiring commonsense and compositional reasoning (See Figure ","element":"span"},{"href":"#id-50","text":"4 ","element":"a"},{"text":"for examples).","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Multimodal skills 4: External knowledge ","element":"span"},{"text":"The average performance on tasks requiring external knowledge is 0.23, compared to 0.30 for those not requiring external knowledge. For example, I","element":"span"},{"text":"NSTRUCT","element":"span"},{"text":"-BLIP performs well on W","element":"span"},{"text":"INOGROUND ","element":"span"},{"text":"and VCR that do not require external knowledge but struggles more on knowledge-intensive tasks e.g., ","element":"span"},{"text":"I","element":"span"},{"text":"N","element":"span"},{"text":"ATURALIST","element":"span"},{"text":", which requires knowledge","element":"span"}],[{"text":"about characteristics of a vast number of species, and S","element":"span"},{"text":"LAKE","element":"span"},{"text":", where medical knowledge is needed to identify the abnormalities in organs.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Multimodal Skills 5: Information flow ","element":"span"},{"text":"Translation has the lowest average score amongst all types of information flow (0.19), whereas the average scores on querying and fusion are 0.26 and 0.33 respectively. The low performance on translation is due to the presence of challenging datasets like D","element":"span"},{"text":"ECIMER ","element":"span"},{"text":"and S","element":"span"},{"text":"CREEN","element":"span"},{"text":"2W","element":"span"},{"text":"ORDS ","element":"span"},{"text":"requiring mapping images of chemicals and screenshots into text. Although the average score for fusion is high, the performance on some datasets is still quite low, such as I","element":"span"},{"text":"NSTRUCT","element":"span"},{"text":"-BLIP achieving a score of only 0.04 on M","element":"span"},{"text":"EME","element":"span"},{"text":"C","element":"span"},{"text":"AP ","element":"span"},{"text":"and 0.15 on MM-IMD","element":"span"},{"text":"B","element":"span"},{"text":".","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"4.2.2 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Performance across modeling dimensions","element":"span"}],[{"text":"We now compare different modeling decisions and training objectives in Table ","element":"span"},{"href":"#id-47","text":"4.","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Overall comparisons across models ","element":"span"},{"text":"G","element":"span"},{"text":"EMINI ","element":"span"},{"text":"[","element":"span"},{"href":"#id-51","referenceIndex":97,"text":"97","element":"a"},{"text":"] (0.44), I","element":"span"},{"text":"NSTRUCT","element":"span"},{"text":"-BLIP [","element":"span"},{"href":"#id-41","referenceIndex":22,"text":"22","element":"a"},{"text":"] (0.41), BLIP-2 [","element":"span"},{"href":"#id-52","referenceIndex":62,"text":"62","element":"a"},{"text":"] (0.41), and GPT-4V [","element":"span"},{"href":"#id-53","referenceIndex":1,"text":"1","element":"a"},{"text":"] (0.40) achieve the best average performance across all tasks. The low scores of GPT-4V as compared to G","element":"span"},{"text":"EMINI ","element":"span"},{"text":"and I","element":"span"},{"text":"NSTRUCT","element":"span"},{"text":"-BLIP are due to its generation of keywords like “Indeterminate”, “Uncertain”, and “Unknown” on datasets like VQA and GQA, perhaps due to its alignment process. Further, on some datasets related to Memes (e.g., H","element":"span"},{"text":"ATEFUL ","element":"span"},{"text":"M","element":"span"},{"text":"EMES","element":"span"},{"text":") and Health (e.g., S","element":"span"},{"text":"LAKE","element":"span"},{"text":"), GPT-4V refrains from answering the questions and instead generates a response saying ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Cannot assist with the request","element":"span"},{"text":". O","element":"span"},{"text":"PEN","element":"span"},{"text":"F","element":"span"},{"text":"LAMINGO ","element":"span"},{"text":"[","element":"span"},{"href":"#id-54","referenceIndex":6,"text":"6","element":"a"},{"text":"] (0.06), E","element":"span"},{"text":"MU ","element":"span"},{"text":"[","element":"span"},{"href":"#id-55","referenceIndex":95,"text":"95","element":"a"},{"text":"] (0.11) have the lowest average scores. From their generations, we find that these models struggle to follow the instructions for challenging datasets like D","element":"span"},{"text":"ECIMER ","element":"span"},{"text":"and E","element":"span"},{"text":"NRICO","element":"span"},{"text":", and generate hallucinated responses. Moreover, with relatively easier datasets such as F","element":"span"},{"text":"LICKR","element":"span"},{"text":"30K, the captions produced by E","element":"span"},{"text":"MU ","element":"span"},{"text":"and O","element":"span"},{"text":"PEN","element":"span"},{"text":"F","element":"span"},{"text":"LAMINGO ","element":"span"},{"text":"tend to fixate on specific objects rather than providing a comprehensive description of the scene, often leading to instances of hallucination related to these objects. As a result, these models rank lowest on many datasets, receiving a normalized score of 0.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Model scale ","element":"span"},{"text":"We find that the performance of larger models (both total and trainable parameters) is significantly better than the models with a medium or small number of parameters (Figure ","element":"span"},{"href":"#id-56","text":"5)","element":"a"},{"text":". When grouped based on the total number of parameters, the average scores achieved by large, medium, and small models are 0.42, 0.24, and 0.23 respectively. The difference between the performance of large and medium models is significant (p-value for paired t-Test < 0.001). In particular, large models showed the most improvement on MM-IMD","element":"span"},{"text":"B","element":"span"},{"text":", M","element":"span"},{"text":"EME","element":"span"},{"text":"-C","element":"span"},{"text":"AP","element":"span"},{"text":", and H","element":"span"},{"text":"ATEFUL ","element":"span"},{"text":"M","element":"span"},{"text":"EMES ","element":"span"},{"text":"datasets, which fall into the category of tasks requiring synergistic interactions. On average, the large models perform the","element":"span"}],[{"id":"id-56","style":{"width":"59%"},"width":942,"height":512,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/7-0.png","element":"img"}],[{"text":"Figure 5: On average, large models are better than small and medium models (p-values < 0.001). I","element":"figcaption","subtype":"caption"},{"text":"NSTRUCT","element":"figcaption","subtype":"caption"},{"text":"-BLIP and BLIP-2 are outliers - despite having fewer params, they achieve relatively high performance, even close to GPT-4V and G","element":"figcaption","subtype":"caption"},{"text":"EMINI","element":"figcaption","subtype":"caption"},{"text":".","element":"figcaption","subtype":"caption"}],[{"text":"best on synergistic tasks with a score of 0.53 compared to 0.30 for medium and 0.23 for small models. For instance, on the MM-IMD","element":"span"},{"text":"B ","element":"span"},{"text":"dataset, we observed significant gains in performance when increasing model size: from 0.15 for I","element":"span"},{"text":"NSTRUCT","element":"span"},{"text":"-BLIP (small) to 0.36 for BLIP-2 (medium) and 0.48 for G","element":"span"},{"text":"EMINI ","element":"span"},{"text":"(large).","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Pretraining data scale ","element":"span"},{"text":"Average scores of the models in ","element":"span"},{"style":{"fontStyle":"italic"},"text":"large ","element":"span"},{"text":"and ","element":"span"},{"style":{"fontStyle":"italic"},"text":"medium ","element":"span"},{"text":"data size categories are 0.31 and 0.30 respectively, whereas models with ","element":"span"},{"style":{"fontStyle":"italic"},"text":"small ","element":"span"},{"text":"pretraining data achieve a significantly lower score of 0.17. We also find that for all datasets, the average score of models with ","element":"span"},{"style":{"fontStyle":"italic"},"text":"medium ","element":"span"},{"text":"pretraining data is higher than the models with ","element":"span"},{"style":{"fontStyle":"italic"},"text":"small ","element":"span"},{"text":"pretraining data. For instance, on the W","element":"span"},{"text":"INOGROUND ","element":"span"},{"text":"dataset which requires fine-grained alignment between the modalities, the maximum scores achieved by the models with ","element":"span"},{"style":{"fontStyle":"italic"},"text":"medium ","element":"span"},{"text":"and ","element":"span"},{"style":{"fontStyle":"italic"},"text":"small ","element":"span"},{"text":"pretraining data are 0.45 and 0.80. We also find a significant gap between the maximum scores achieved by the models in the ","element":"span"},{"style":{"fontStyle":"italic"},"text":"medium ","element":"span"},{"text":"(maximum score - 0.18) and ","element":"span"},{"style":{"fontStyle":"italic"},"text":"small ","element":"span"},{"text":"categories (maximum score - 0.70), on the N","element":"span"},{"text":"LVR","element":"span"},{"text":"2 dataset for visual reasoning.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Diversity of pre-training data ","element":"span"},{"text":"On average, models trained on diverse datasets perform better (score: 0.30) than models trained only on image captioning datasets (score: 0.21). Diverse training","element":"span"}],[{"text":"data allows the models to share learned knowledge and generalize across different tasks. For example, models pretrained with diverse datasets perform significantly better on the knowledge-intensive ","element":"span"},{"text":"I","element":"span"},{"text":"N","element":"span"},{"text":"ATURALIST ","element":"span"},{"text":"task, such as BLIP-2 (non-diverse) scoring 0.08 and G","element":"span"},{"text":"EMINI ","element":"span"},{"text":"scoring 0.24. For the M","element":"span"},{"text":"EME","element":"span"},{"text":"C","element":"span"},{"text":"AP ","element":"span"},{"text":"dataset which requires external knowledge and complex reasoning, we observe that BLIP-2 (non-diverse) scores 0.06 and ","element":"span"},{"text":"M","element":"span"},{"text":"PLUG-O","element":"span"},{"text":"WL ","element":"span"},{"text":"(diverse) scores 0.21.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Instruction tuning vs supervised fine-tuning ","element":"span"},{"text":"On average, instruction-tuned models (average score of 0.30) performed better than the models trained using only supervised fine-tuning (average score of 0.22). The top 3 tasks with the largest performance gap between instruction-tuned and non-instruction-tuned models are D","element":"span"},{"text":"ECIMER","element":"span"},{"text":", M","element":"span"},{"text":"EME","element":"span"},{"text":"C","element":"span"},{"text":"AP","element":"span"},{"text":", and S","element":"span"},{"text":"CREEN","element":"span"},{"text":"2W","element":"span"},{"text":"ORDS","element":"span"},{"text":", with improvements of 0.15, 0.09, and 0.09 respectively. We also observe that translation tasks (image-to-text) (e.g., F","element":"span"},{"text":"LICKR","element":"span"},{"text":"30K, N","element":"span"},{"text":"O","element":"span"},{"text":"C","element":"span"},{"text":"APS","element":"span"},{"text":") benefit from instruction tuning, where the models generate more accurate and detailed captions after human instruction.","element":"span"}],[{"id":"id-103","style":{"fontWeight":"bold"},"text":"4.3 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Human evaluation","element":"span"}],[{"text":"To assess how well HEMM aligns with human preferences, we performed human preference-based evaluation, following Chiang et al. ","element":"span"},{"href":"#id-57","referenceIndex":19,"text":"[19]","element":"a"},{"text":", where annotators are shown the outputs of two different models for the same inputs and choose the better output or a tie option. Across 1000 pairwise comparisons by 5 annotators, the pairwise rankings are used to calculate each model’s average win rate and Elo rating (see Appendix ","element":"span"},{"href":"#id-58","text":"B.5 ","element":"a"},{"text":"for calculation details).","element":"span"}],[{"text":"The models ranked by Elo ratings are G","element":"span"},{"text":"EM","element":"span"},{"text":"- ","element":"span"},{"text":"INI ","element":"span"},{"text":"(1074), GPT-4V (1057), BLIP-2 (1033), and I","element":"span"},{"text":"NSTRUCT","element":"span"},{"text":"-BLIP (1032) (see Table ","element":"span"},{"href":"#id-59","text":"5)","element":"a"},{"text":". The top 4 models based on the Elo Rating are the same as the top 4 models ranked by BARTScore. Elo Rating of GPT-4V is better than BLIP-2 and I","element":"span"},{"text":"NSTRUCT","element":"span"},{"text":"-BLIP. However, the average BARTScore for GPT-4V (0.40) is lower than I","element":"span"},{"text":"NSTRUCT","element":"span"},{"text":"-BLIP (0.42) and BLIP-2 (0.41). We also find Elo Rating of bottom two models to be consistent with BARTScore rankings - E","element":"span"},{"text":"MU ","element":"span"},{"text":"(0.11) and O","element":"span"},{"text":"PEN","element":"span"},{"text":"F","element":"span"},{"text":"LAMINGO ","element":"span"},{"text":"(0.06).","element":"span"}],[{"text":"Table 5: Average win rate and Elo Rating of 11 mod- ","element":"figcaption","subtype":"caption"},{"id":"id-59","text":"els calculated based on the human evaluation of 1000 ","element":"figcaption","subtype":"caption"},{"text":"pair-wise comparisons of model responses. Elo rating is reported as the median over 1000 runs with shuffled battle sequences and an initial rating of 1000 for each model. Top 4 and bottom 2 models identified by Elo Rating are consistent with those found by Average BARTScore.","element":"figcaption","subtype":"caption"}],[{"style":{"width":"54%"},"width":790,"height":572,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/8-0.png","element":"img"}]]},{"heading":"5 Related Work","paragraphs":[[{"style":{"fontWeight":"bold"},"text":"Multimodal machine learning ","element":"span"},{"text":"brings unique challenges for ML research due to the heterogeneity between modalities and the interconnections found between them [","element":"span"},{"href":"#id-1","referenceIndex":69,"text":"69","element":"a"},{"text":"]. It has inspired many theoretical studies in data heterogeneity and interactions [","element":"span"},{"href":"#id-60","referenceIndex":25,"text":"25","element":"a"},{"text":"], as well as diverse applications in multimedia [","element":"span"},{"href":"#id-61","referenceIndex":44,"text":"44","element":"a"},{"text":", ","element":"span"},{"href":"#id-62","referenceIndex":14,"text":"14","element":"a"},{"text":", ","element":"span"},{"href":"#id-63","referenceIndex":88,"text":"88","element":"a"},{"text":"], affective computing [","element":"span"},{"href":"#id-33","referenceIndex":86,"text":"86","element":"a"},{"text":"], robotics [","element":"span"},{"href":"#id-64","referenceIndex":47,"text":"47","element":"a"},{"text":"], finance [","element":"span"},{"href":"#id-65","referenceIndex":39,"text":"39","element":"a"},{"text":"],","element":"span"}],[{"text":"HCI ","element":"span"},{"href":"#id-60","referenceIndex":25,"text":"[25, ","element":"a"},{"href":"#id-66","referenceIndex":82,"text":"82]","element":"a"},{"text":", education ","element":"span"},{"href":"#id-67","referenceIndex":12,"text":"[12] ","element":"a"},{"text":"and healthcare ","element":"span"},{"href":"#id-68","referenceIndex":80,"text":"[80, ","element":"a"},{"href":"#id-69","referenceIndex":110,"text":"110]","element":"a"},{"text":".","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Evaluation frameworks for multimodal models ","element":"span"},{"text":"have significantly shaped the multimodal research landscape, through holistic [","element":"span"},{"href":"#id-7","referenceIndex":57,"text":"57","element":"a"},{"text":", ","element":"span"},{"href":"#id-9","referenceIndex":66,"text":"66","element":"a"},{"text":"] and domain-specific benchmarks [","element":"span"},{"href":"#id-70","referenceIndex":31,"text":"31","element":"a"},{"text":", ","element":"span"},{"href":"#id-71","referenceIndex":28,"text":"28","element":"a"},{"text":"]. Recent benchmarks have focused on testing the capabilities of multimodal foundation models, such as MME [","element":"span"},{"href":"#id-8","referenceIndex":29,"text":"29","element":"a"},{"text":"], MMBench [","element":"span"},{"href":"#id-72","referenceIndex":73,"text":"73","element":"a"},{"text":"], LVLM-ehub [","element":"span"},{"href":"#id-73","referenceIndex":111,"text":"111","element":"a"},{"text":"], SEED-Bench [","element":"span"},{"href":"#id-74","referenceIndex":59,"text":"59","element":"a"},{"text":"], Touchstone [","element":"span"},{"href":"#id-75","referenceIndex":7,"text":"7","element":"a"},{"text":"], Mm-vet [","element":"span"},{"href":"#id-76","referenceIndex":121,"text":"121","element":"a"},{"text":"], ReFormEval [","element":"span"},{"href":"#id-77","referenceIndex":65,"text":"65","element":"a"},{"text":"], VisIT-Bench [","element":"span"},{"href":"#id-78","referenceIndex":11,"text":"11","element":"a"},{"text":"], FLAVA [","element":"span"},{"href":"#id-79","referenceIndex":45,"text":"45","element":"a"},{"text":"]. Other benchmarks focus on evaluating hallucination [","element":"span"},{"href":"#id-80","referenceIndex":21,"text":"21","element":"a"},{"text":"] and applications in medicine [","element":"span"},{"href":"#id-81","referenceIndex":113,"text":"113","element":"a"},{"text":"] and autonomous driving [","element":"span"},{"href":"#id-82","referenceIndex":107,"text":"107","element":"a"},{"text":"]. These benchmarks contain many tasks, but without the systematic taxonomy and comprehensiveness that HEMM provides.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Multimodal foundation models ","element":"span"},{"text":"are promising foundations for the future of AI, with impressive reasoning [","element":"span"},{"href":"#id-32","referenceIndex":75,"text":"75","element":"a"},{"text":"], interactive dialogue [","element":"span"},{"href":"#id-83","referenceIndex":49,"text":"49","element":"a"},{"text":"], and few-shot generalization abilities [","element":"span"},{"href":"#id-84","referenceIndex":100,"text":"100","element":"a"},{"text":"]. These models can be pre-trained (typically with image-text self-supervised learning) and fine-tuned for downstream tasks [","element":"span"},{"href":"#id-85","referenceIndex":63,"text":"63","element":"a"},{"text":", ","element":"span"},{"href":"#id-86","referenceIndex":74,"text":"74","element":"a"},{"text":", ","element":"span"},{"href":"#id-87","referenceIndex":91,"text":"91","element":"a"},{"text":", ","element":"span"},{"href":"#id-88","referenceIndex":67,"text":"67","element":"a"},{"text":"], or based on adapting language models with vision to enable text generation conditioned on images ","element":"span"},{"href":"#id-89","referenceIndex":61,"text":"[61, ","element":"a"},{"href":"#id-90","referenceIndex":105,"text":"105]","element":"a"},{"text":". Cross-modal transformer architectures have emerged as a popular backbone due to their suitability for both language and image data [","element":"span"},{"href":"#id-91","referenceIndex":17,"text":"17","element":"a"},{"text":", ","element":"span"},{"href":"#id-92","referenceIndex":99,"text":"99","element":"a"},{"text":"]. Additionally, composable","element":"span"}],[{"text":"models [","element":"span"},{"href":"#id-93","referenceIndex":96,"text":"96","element":"a"},{"text":"] and mixtures of experts [","element":"span"},{"href":"#id-94","referenceIndex":120,"text":"120","element":"a"},{"text":"] can be used to further generate combinations of output modalities.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Adapting language models for multimodality ","element":"span"},{"text":"is another promising approach where frozen models are aligned on both vision and language to generate text from multimodal inputs [","element":"span"},{"href":"#id-42","referenceIndex":128,"text":"128","element":"a"},{"text":", ","element":"span"},{"href":"#id-52","referenceIndex":62,"text":"62","element":"a"},{"text":", ","element":"span"},{"href":"#id-95","referenceIndex":118,"text":"118","element":"a"},{"text":", ","element":"span"},{"href":"#id-17","referenceIndex":109,"text":"109","element":"a"},{"text":"]. These approaches typically use parameter-efficient modules like LLaMA-Adapter V2 [","element":"span"},{"href":"#id-96","referenceIndex":30,"text":"30","element":"a"},{"text":"] and MAGMA [","element":"span"},{"href":"#id-97","referenceIndex":27,"text":"27","element":"a"},{"text":"] for efficient finetuning. Vision-language instruction tuning has also emerged as a useful technique, as it allows the models to better follow human instructions [","element":"span"},{"href":"#id-98","referenceIndex":112,"text":"112","element":"a"},{"text":", ","element":"span"},{"href":"#id-42","referenceIndex":128,"text":"128","element":"a"},{"text":"]. Our goal is to make HEMM the most comprehensive benchmark to study the current and future generation of multimodal foundation models, and for the community to continuously contribute to its expansion.","element":"span"}]]},{"heading":"6 Conclusion","paragraphs":[[{"text":"Holistic Evaluation of Multimodal Models (HEMM) is a framework for benchmarking multimodal foundation models. Through a new taxonomy of multimodal skills, information flow, and real-world use cases, HEMM enables comprehensive analysis of multimodal models. HEMM is publicly available, will be regularly updated, and encourages community involvement in its expansion.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Limitations and social impact ","element":"span"},{"text":"The evaluation of multimodal models is done only on a subset of all possible skills, information, and use cases in the world. Future work can improve the categorization of datasets into skills, information, and use cases, and discover new dimensions that pose challenges to multimodal models. Such evaluation is critical to ensure that models are sufficiently robust when deployed in real-world scenarios, to prevent unexpected and unintended consequences. Future work should also add new metrics to HEMM measuring real-world societal concerns such as fairness, robustness, social biases, privacy, and efficiency of multimodal models.","element":"span"}]]},{"heading":"Acknowledgements","paragraphs":[[{"text":"This material is based upon work partially supported by National Science Foundation awards 1722822 and 1750439, National Institutes of Health awards R01MH125740, R01MH132225, R01MH096951 and R21MH130767, and Meta. PPL is supported in part by a Siebel Scholarship and a Waibel Presidential Fellowship. RS is supported in part by ONR grant N000142312368 and DARPA FA87502321015. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors, and no official endorsement should be inferred. We are also grateful to NVIDIA’s GPU support.","element":"span"}]]},{"heading":"References","paragraphs":[[{"id":"id-53","text":"[1] ","element":"span"},{"text":"Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2303.08774","element":"span"},{"text":", 2023.","element":"span"}],[{"text":"[2] ","element":"span"},{"text":"Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Proceedings of the IEEE International Conference on Computer Vision","element":"span"},{"text":", pages 8948–8957, 2019.","element":"span"}],[{"id":"id-113","text":"[3] ","element":"span"},{"text":"Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Advances in neural information processing systems","element":"span"},{"text":", 35:23716–23736, 2022.","element":"span"}],[{"text":"[4] ","element":"span"},{"text":"Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Proceedings of the IEEE international conference on computer vision","element":"span"},{"text":", pages 2425–2433, 2015.","element":"span"}],[{"id":"id-29","text":"[5] ","element":"span"},{"text":"John Arevalo, Thamar Solorio, Manuel Montes-y Gómez, and Fabio A González. Gated multimodal units for information fusion. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:1702.01992","element":"span"},{"text":", 2017.","element":"span"}],[{"id":"id-54","text":"[6] ","element":"span"},{"text":"Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2308.01390","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-75","text":"[7] ","element":"span"},{"text":"Shuai Bai, Shusheng Yang, Jinze Bai, Peng Wang, Xingxuan Zhang, Junyang Lin, Xinggang Wang, Chang Zhou, and Jingren Zhou. Touchstone: Evaluating vision-language models by language models. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2308.16890","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-6","text":"[8] Eslam Mohamed Bakr, Pengzhan Sun, Xiaogian Shen, Faizan Farooq Khan, Li Erran Li, and Mohamed ","element":"span"},{"text":"Elhoseiny. Hrs-bench: Holistic, reliable and scalable benchmark for text-to-image models. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Proceedings of the IEEE/CVF International Conference on Computer Vision","element":"span"},{"text":", pages 20041–20053, 2023.","element":"span"}],[{"id":"id-22","text":"[9] John Bateman. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Text and image: A critical introduction to the visual/verbal divide","element":"span"},{"text":". Routledge, 2014.","element":"span"}],[{"text":"[10] ","element":"span"},{"text":"Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sa˘gnak Ta¸sırlar. Introducing our multimodal models, 2023. URL ","element":"span"},{"href":"https://www.adept.ai/blog/fuyu-8b","style":{"fontFamily":"monospace"},"text":"https://www.adept.ai/blog/ ","element":"a"},{"href":"https://www.adept.ai/blog/fuyu-8b","style":{"fontFamily":"monospace"},"text":"fuyu-8b","element":"a"},{"text":".","element":"span"}],[{"id":"id-78","text":"[11] ","element":"span"},{"text":"Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, and Ludwig Schimdt. Visit-bench: A benchmark for vision-language instruction following inspired by real-world use. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2308.06595","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-67","text":"[12] ","element":"span"},{"text":"Paulo Blikstein and Marcelo Worsley. Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Journal of Learning Analytics","element":"span"},{"text":", 3(2): 220–238, 2016.","element":"span"}],[{"text":"[13] ","element":"span"},{"text":"Henning Otto Brinkhaus, Achim Zielesny, Christoph Steinbeck, and Kohulan Rajan. Decimer—hand-drawn molecule images dataset. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Journal of Cheminformatics","element":"span"},{"text":", 14(1):1–4, 2022.","element":"span"}],[{"id":"id-62","text":"[14] ","element":"span"},{"text":"Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. Revisiting the\" video\" in video-language understanding. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","element":"span"},{"text":", pages 2917–2927, 2022.","element":"span"}],[{"id":"id-27","text":"[15] ","element":"span"},{"text":"Yitao Cai, Huiyu Cai, and Xiaojun Wan. Multi-modal sarcasm detection in Twitter with hierarchical fusion model. In Anna Korhonen, David Traum, and Lluís Màrquez, editors, ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","element":"span"},{"text":", pages 2506–2515, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1239. URL ","element":"span"},{"href":"https://aclanthology.org/P19-1239","style":{"fontFamily":"monospace"},"text":"https: ","element":"a"},{"href":"https://aclanthology.org/P19-1239","style":{"fontFamily":"monospace"},"text":"//aclanthology.org/P19-1239","element":"a"},{"text":".","element":"span"}],[{"id":"id-45","text":"[16] ","element":"span"},{"text":"Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. Webqa: Multihop and multimodal qa. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","element":"span"},{"text":", pages 16495–16504, 2022.","element":"span"}],[{"id":"id-91","text":"[17] ","element":"span"},{"text":"Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"European conference on computer vision","element":"span"},{"text":", pages 104–120. Springer, 2020.","element":"span"}],[{"text":"[18] ","element":"span"},{"text":"Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Proceedings of the IEEE","element":"span"},{"text":", 105(10):1865–1883, 2017.","element":"span"}],[{"id":"id-57","text":"[19] ","element":"span"},{"text":"Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2403.04132","element":"span"},{"text":", 2024.","element":"span"}],[{"text":"[20] ","element":"span"},{"text":"Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Advances in neural information processing systems","element":"span"},{"text":", 30, 2017.","element":"span"}],[{"id":"id-80","text":"[21] ","element":"span"},{"text":"Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao. Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2311.03287","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-41","text":"[22] ","element":"span"},{"text":"Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning, June 2023. URL ","element":"span"},{"href":"http://arxiv.org/abs/2305.06500","style":{"fontFamily":"monospace"},"text":"http://arxiv.org/abs/2305.06500","element":"a"},{"text":". arXiv:2305.06500 [cs].","element":"span"}],[{"id":"id-107","text":"[23] ","element":"span"},{"text":"Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design applications. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Proceedings of the 30th annual ACM symposium on user interface software and technology","element":"span"},{"text":", pages 845–854, 2017.","element":"span"}],[{"id":"id-0","text":"[24] ","element":"span"},{"text":"Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. A survey of vision-language pre-trained models. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2202.10936","element":"span"},{"text":", 2022.","element":"span"}],[{"id":"id-60","text":"[25] ","element":"span"},{"text":"Bruno Dumas, Denis Lalanne, and Sharon Oviatt. Multimodal interfaces: A survey of principles, models and frameworks. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Human machine interaction: Research results of the mmi program","element":"span"},{"text":", pages 3–26. Springer, 2009.","element":"span"}],[{"id":"id-11","text":"[26] ","element":"span"},{"text":"Bruno Dumas, Jonathan Pirau, and Denis Lalanne. Modelling fusion of modalities in multimodal interactive systems with mmmm. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Proceedings of the 19th ACM International Conference on Multimodal Interaction","element":"span"},{"text":", pages 288–296, 2017.","element":"span"}],[{"id":"id-97","text":"[27] ","element":"span"},{"text":"Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank. Magma–multimodal augmentation of generative models through adapter-based finetuning. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2112.05253","element":"span"},{"text":", 2021.","element":"span"}],[{"id":"id-71","text":"[28] ","element":"span"},{"text":"Francis Ferraro, Nasrin Mostafazadeh, Ting-Hao Huang, Lucy Vanderwende, Jacob Devlin, Michel Galley, and Margaret Mitchell. A survey of current datasets for vision and language research. In Lluís Màrquez, Chris Callison-Burch, and Jian Su, editors, ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","element":"span"},{"text":", pages 207–213, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/v1/D15-1021. URL ","element":"span"},{"href":"https://aclanthology.org/D15-1021","style":{"fontFamily":"monospace"},"text":"https://aclanthology. ","element":"a"},{"href":"https://aclanthology.org/D15-1021","style":{"fontFamily":"monospace"},"text":"org/D15-1021","element":"a"},{"text":".","element":"span"}],[{"id":"id-8","text":"[29] ","element":"span"},{"text":"Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2306.13394","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-96","text":"[30] ","element":"span"},{"text":"Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2304.15010","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-70","text":"[31] ","element":"span"},{"text":"Dimitris Gkoumas, Qiuchi Li, Christina Lioma, Yijun Yu, and Dawei Song. What makes the difference? an empirical comparison of fusion strategies for multimodal language analysis. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Information Fusion","element":"span"},{"text":", 66: 184–197, 2021.","element":"span"}],[{"text":"[32] ","element":"span"},{"text":"Ian J Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, et al. Challenges in representation learning: A report on three machine learning contests. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Neural Information Processing: 20th International Conference, ICONIP 2013, Daegu, Korea, November 3-7, 2013. Proceedings, Part III 20","element":"span"},{"text":", pages 117–124. Springer, 2013.","element":"span"}],[{"id":"id-108","text":"[33] ","element":"span"},{"text":"Hessel Haagsma, Johan Bos, and Malvina Nissim. Magpie: A large corpus of potentially idiomatic expressions. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"12th Language Resources and Evaluation Conference: LREC 2020","element":"span"},{"text":", pages 279–287. European Language Resources Association (ELRA), 2020.","element":"span"}],[{"text":"[34] ","element":"span"},{"text":"F Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Acm transactions on interactive intelligent systems (tiis)","element":"span"},{"text":", 5(4):1–19, 2015.","element":"span"}],[{"id":"id-25","text":"[35] ","element":"span"},{"text":"Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2003.10286","element":"span"},{"text":", 2020.","element":"span"}],[{"id":"id-118","text":"[36] ","element":"span"},{"text":"Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2104.08718","element":"span"},{"text":", 2021.","element":"span"}],[{"id":"id-28","text":"[37] ","element":"span"},{"text":"Jack Hessel, Ana Marasovi´c, Jena D Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff, and Yejin Choi. Do androids laugh at electric sheep? humor\" understanding\" benchmarks from the new yorker caption contest. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2209.06293","element":"span"},{"text":", 2022.","element":"span"}],[{"text":"[38] ","element":"span"},{"text":"Micah Hodosh, Peter Young, and Julia Hockenmaier. Framing image description as a ranking task: Data, models and evaluation metrics. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Journal of Artificial Intelligence Research","element":"span"},{"text":", 47:853–899, 2013.","element":"span"}],[{"id":"id-65","text":"[39] ","element":"span"},{"text":"Markus A Höllerer, Dennis Jancsary, and Maria Grafström. ‘a picture is worth a thousand words’: Multimodal sensemaking of the global financial crisis. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Organization Studies","element":"span"},{"text":", 39(5-6):617–644, 2018.","element":"span"}],[{"text":"[40] ","element":"span"},{"text":"Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Aligning perception with language models. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Advances in Neural Information Processing Systems","element":"span"},{"text":", 36, 2024.","element":"span"}],[{"text":"[41] ","element":"span"},{"text":"Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas Montine, and James Zou. Leveraging medical twitter to build a visual–language foundation model for pathology ai. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"bioRxiv","element":"span"},{"text":", pages 2023–03, 2023.","element":"span"}],[{"text":"[42] ","element":"span"},{"text":"Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","element":"span"},{"text":", pages 6700–6709, 2019.","element":"span"}],[{"id":"id-23","text":"[43] ","element":"span"},{"text":"EunJeong Hwang and Vered Shwartz. Memecap: A dataset for captioning and interpreting memes. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2305.13703","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-61","text":"[44] ","element":"span"},{"text":"Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"European Conference on Computer Vision","element":"span"},{"text":", pages 105–124. Springer, 2022.","element":"span"}],[{"id":"id-79","text":"[45] ","element":"span"},{"text":"Douwe Kiela. Grounding, meaning and foundation models: Adventures in multimodal machine learning. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Proceedings of the 30th ACM International Conference on Multimedia","element":"span"},{"text":", pages 5–5, 2022.","element":"span"}],[{"id":"id-30","text":"[46] ","element":"span"},{"text":"Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Advances in neural information processing systems","element":"span"},{"text":", 33:2611–2624, 2020.","element":"span"}],[{"id":"id-64","text":"[47] ","element":"span"},{"text":"Elsa A Kirchner, Stephen H Fairclough, and Frank Kirchner. Embedded multimodal interfaces in robotics: applications, future trends, and societal implications. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"The Handbook of Multimodal-Multisensor Interfaces: Language Processing, Software, Commercialization, and Emerging Directions-Volume 3","element":"span"},{"text":", pages 523–576. 2019.","element":"span"}],[{"id":"id-37","text":"[48] ","element":"span"},{"text":"Adrienne Kline, Hanyin Wang, Yikuan Li, Saya Dennis, Meghan Hutch, Zhenxing Xu, Fei Wang, Feixiong Cheng, and Yuan Luo. Multimodal machine learning in precision health: A scoping review. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"npj Digital Medicine","element":"span"},{"text":", 5(1):171, 2022.","element":"span"}],[{"id":"id-83","text":"[49] ","element":"span"},{"text":"Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and outputs. 2023.","element":"span"}],[{"text":"[50] ","element":"span"},{"text":"Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"International journal of computer vision","element":"span"},{"text":", 123:32–73, 2017.","element":"span"}],[{"id":"id-38","text":"[51] ","element":"span"},{"text":"Felix Krones, Umar Marikkar, Guy Parsons, Adam Szmul, and Adam Mahdi. Review of multimodal machine learning approaches in healthcare. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2402.02460","element":"span"},{"text":", 2024.","element":"span"}],[{"id":"id-21","text":"[52] ","element":"span"},{"text":"Julia Kruk, Jonah Lubin, Karan Sikka, Xiao Lin, Dan Jurafsky, and Ajay Divakaran. Integrating text and image: Determining multimodal document intent in instagram posts. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","element":"span"},{"text":", pages 4622–4632, 2019.","element":"span"}],[{"text":"[53] ","element":"span"},{"text":"Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Scientific data","element":"span"},{"text":", 5(1):1–10, 2018.","element":"span"}],[{"id":"id-26","text":"[54] ","element":"span"},{"text":"Jason J Lau, Soumya Gayen, Dina Demner, and Asma Ben Abacha. Visual question answering in radiology (vqa-rad), Feb 2019. URL ","element":"span"},{"style":{"fontFamily":"monospace"},"text":"osf.io/89kps","element":"span"},{"text":".","element":"span"}],[{"id":"id-4","text":"[55] ","element":"span"},{"text":"Mina Lee, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus, Ashwin Paranjape, Ines Gerard-Ursin, Xiang Lisa Li, Faisal Ladhak, Frieda Rong, et al. Evaluating human-language model interaction. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2212.09746","element":"span"},{"text":", 2022.","element":"span"}],[{"id":"id-34","text":"[56] ","element":"span"},{"text":"Sangmin Lee, Bolin Lai, Fiona Ryan, Bikram Boote, and James M Rehg. ","element":"span"},{"text":"Modeling multimodal social interactions: New challenges and baselines with densely aligned representations. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2403.02090","element":"span"},{"text":", 2024.","element":"span"}],[{"id":"id-7","text":"[57] ","element":"span"},{"text":"Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Benita Teufel, Marco Bellagente, et al. Holistic evaluation of text-to-image models. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2311.04287","element":"span"},{"text":", 2023.","element":"span"}],[{"text":"[58] ","element":"span"},{"text":"Luis A Leiva, Asutosh Hota, and Antti Oulasvirta. Enrico: A high-quality dataset for topic modeling of mobile ui designs. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Proc. MobileHCI extended abstracts","element":"span"},{"text":", 2020.","element":"span"}],[{"id":"id-74","text":"[59] ","element":"span"},{"text":"Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2307.16125","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-5","text":"[60] ","element":"span"},{"text":"Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, and Jianfeng Gao. Multimodal foundation models: From specialists to general-purpose assistants. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2309.10020","element":"span"},{"text":", 1, 2023.","element":"span"}],[{"id":"id-89","text":"[61] ","element":"span"},{"text":"Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"International Conference on Machine Learning","element":"span"},{"text":", pages 12888–12900. PMLR, 2022.","element":"span"}],[{"id":"id-52","text":"[62] ","element":"span"},{"text":"Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2301.12597","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-85","text":"[63] ","element":"span"},{"text":"Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:1908.03557","element":"span"},{"text":", 2019.","element":"span"}],[{"id":"id-109","text":"[64] ","element":"span"},{"text":"Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile ui action sequences. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2005.03776","element":"span"},{"text":", 2020.","element":"span"}],[{"id":"id-77","text":"[65] ","element":"span"},{"text":"Zejun Li, Ye Wang, Mengfei Du, Qingwen Liu, Binhao Wu, Jiwen Zhang, Chengxing Zhou, Zhihao Fan, Jie Fu, Jingjing Chen, et al. Reform-eval: Evaluating large vision language models via unified re-formulation of task-oriented benchmarks. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2310.02569","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-9","text":"[66] ","element":"span"},{"text":"Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Yufan Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al. Multibench: Multiscale benchmarks for multimodal representation learning. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)","element":"span"},{"text":", 2021.","element":"span"}],[{"id":"id-88","text":"[67] ","element":"span"},{"text":"Paul Pu Liang, Yiwei Lyu, Xiang Fan, Jeffrey Tsaw, Yudong Liu, Shentong Mo, Dani Yogatama, Louis-Philippe Morency, and Ruslan Salakhutdinov. High-modality multimodal transformer: Quantifying modality & interaction heterogeneity for high-modality representation learning. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2203.01311","element":"span"},{"text":", 2022.","element":"span"}],[{"id":"id-12","text":"[68] ","element":"span"},{"text":"Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Faisal Mahmood, Ruslan Salakhutdinov, and Louis-Philippe Morency. Quantifying & modeling multimodal interactions: An information decomposition framework. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Advances in Neural Information Processing Systems","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-1","text":"[69] ","element":"span"},{"text":"Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations & trends in multimodal machine learning: Principles, challenges, and open questions. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"ACM Computing Surveys","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-117","text":"[70] ","element":"span"},{"text":"Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Text summarization branches out","element":"span"},{"text":", pages 74–81, 2004.","element":"span"}],[{"id":"id-105","text":"[71] ","element":"span"},{"text":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13","element":"span"},{"text":", pages 740–755. Springer, 2014.","element":"span"}],[{"text":"[72] ","element":"span"},{"text":"Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI)","element":"span"},{"text":", pages 1650–1654. IEEE, 2021.","element":"span"}],[{"id":"id-72","text":"[73] ","element":"span"},{"text":"Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2307.06281","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-86","text":"[74] ","element":"span"},{"text":"Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Advances in neural information processing systems","element":"span"},{"text":", 32, 2019.","element":"span"}],[{"id":"id-32","text":"[75] ","element":"span"},{"text":"Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"The 36th Conference on Neural Information Processing Systems (NeurIPS)","element":"span"},{"text":", 2022.","element":"span"}],[{"id":"id-31","text":"[76] ","element":"span"},{"text":"Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Conference on Computer Vision and Pattern Recognition (CVPR)","element":"span"},{"text":", 2019.","element":"span"}],[{"id":"id-20","text":"[77] ","element":"span"},{"text":"Emily E Marsh and Marilyn Domas White. A taxonomy of relationships between images and text. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Journal of documentation","element":"span"},{"text":", 2003.","element":"span"}],[{"id":"id-35","text":"[78] ","element":"span"},{"text":"Leena Mathur, Paul Pu Liang, and Louis-Philippe Morency. Advancing social intelligence in ai agents: Technical challenges and open questions. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2404.11023","element":"span"},{"text":", 2024.","element":"span"}],[{"id":"id-106","text":"[79] ","element":"span"},{"text":"George A Miller. Wordnet: a lexical database for english. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Communications of the ACM","element":"span"},{"text":", 38(11):39–41, 1995.","element":"span"}],[{"id":"id-68","text":"[80] ","element":"span"},{"text":"Christian Moro, Jessica Smith, and Zane Stromberga. Multimodal learning in health sciences and medicine: Merging technologies to enhance student learning and communication. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Biomedical Visualisation: Volume 5","element":"span"},{"text":", pages 71–78, 2019.","element":"span"}],[{"id":"id-39","text":"[81] ","element":"span"},{"text":"Brad A Myers. A brief history of human-computer interaction technology. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"interactions","element":"span"},{"text":", 5(2):44–54, 1998.","element":"span"}],[{"id":"id-66","text":"[82] ","element":"span"},{"text":"Zeljko Obrenovic and Dusan Starcevic. Modeling multimodal human-computer interaction. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Computer","element":"span"},{"text":", 37(9):65–72, 2004.","element":"span"}],[{"id":"id-3","text":"[83] ","element":"span"},{"text":"Christian Otto, Matthias Springstein, Avishek Anand, and Ralph Ewerth. Understanding, categorizing and predicting semantic image-text relations. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Proceedings of the 2019 on International Conference on Multimedia Retrieval","element":"span"},{"text":", pages 168–176, 2019.","element":"span"}],[{"text":"[84] ","element":"span"},{"text":"Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Advances in neural information processing systems","element":"span"},{"text":", 35:27730–27744, 2022.","element":"span"}],[{"text":"[85] ","element":"span"},{"text":"Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2306.14824","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-33","text":"[86] Rosalind W Picard. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Affective computing","element":"span"},{"text":". MIT press, 2000.","element":"span"}],[{"id":"id-110","text":"[87] ","element":"span"},{"text":"Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and language with localized narratives. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16","element":"span"},{"text":", pages 647–664. Springer, 2020.","element":"span"}],[{"id":"id-63","text":"[88] ","element":"span"},{"text":"Vignesh Ramanathan, Percy Liang, and Li Fei-Fei. Video event understanding using natural language descriptions. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Proceedings of the IEEE international conference on computer vision","element":"span"},{"text":", pages 905–912, 2013.","element":"span"}],[{"id":"id-24","text":"[89] ","element":"span"},{"text":"Chhavi Sharma, William Paka, Scott, Deepesh Bhageria, Amitava Das, Soujanya Poria, Tanmoy Chakraborty, and Björn Gambäck. Task Report: Memotion Analysis 1.0 @SemEval 2020: The VisuoLingual Metaphor! In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020)","element":"span"},{"text":", Barcelona, Spain, Sep 2020. Association for Computational Linguistics.","element":"span"}],[{"id":"id-15","text":"[90] ","element":"span"},{"text":"Sheng Shen, Chunyuan Li, Xiaowei Hu, Yujia Xie, Jianwei Yang, Pengchuan Zhang, Zhe Gan, Lijuan Wang, Lu Yuan, Ce Liu, et al. K-lite: Learning transferable visual models with external knowledge. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Advances in Neural Information Processing Systems","element":"span"},{"text":", 35:15558–15573, 2022.","element":"span"}],[{"id":"id-87","text":"[91] ","element":"span"},{"text":"Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:1908.08530","element":"span"},{"text":", 2019.","element":"span"}],[{"id":"id-43","text":"[92] ","element":"span"},{"text":"Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2305.16355","element":"span"},{"text":", 2023.","element":"span"}],[{"text":"[93] ","element":"span"},{"text":"Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. A corpus of natural language for visual reasoning. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","element":"span"},{"text":", pages 217–223, 2017.","element":"span"}],[{"text":"[94] ","element":"span"},{"text":"Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:1811.00491","element":"span"},{"text":", 2018.","element":"span"}],[{"id":"id-55","text":"[95] ","element":"span"},{"text":"Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. ","element":"span"},{"text":"Generative pretraining in multimodality. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2307.05222","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-93","text":"[96] ","element":"span"},{"text":"Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2305.11846","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-51","text":"[97] ","element":"span"},{"text":"Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2312.11805","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-16","text":"[98] ","element":"span"},{"text":"Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","element":"span"},{"text":", pages 5238– 5248, 2022.","element":"span"}],[{"id":"id-92","text":"[99] ","element":"span"},{"text":"Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","element":"span"},{"text":", pages 6558–6569, 2019.","element":"span"}],[{"id":"id-84","text":"[100] ","element":"span"},{"text":"Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Advances in Neural Information Processing Systems","element":"span"},{"text":", 34: 200–212, 2021.","element":"span"}],[{"id":"id-36","text":"[101] ","element":"span"},{"text":"Devis Tuia, Benjamin Kellenberger, Sara Beery, Blair R Costelloe, Silvia Zuffi, Benjamin Risse, Alexander Mathis, Mackenzie W Mathis, Frank Van Langevelde, Tilo Burghardt, et al. Perspectives in machine learning for wildlife conservation. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Nature communications","element":"span"},{"text":", 13(1):1–15, 2022.","element":"span"}],[{"text":"[102] ","element":"span"},{"text":"Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset-supplementary material. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Reptilia","element":"span"},{"text":", 32(400):1–3, 2017.","element":"span"}],[{"text":"[103] ","element":"span"},{"text":"Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. ","element":"span"},{"text":"Screen2words: Automatic mobile ui summarization with multimodal learning. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"The 34th Annual ACM Symposium on User Interface Software and Technology","element":"span"},{"text":", pages 498–510, 2021.","element":"span"}],[{"id":"id-13","text":"[104] ","element":"span"},{"text":"Fei Wang, Liang Ding, Jun Rao, Ye Liu, Li Shen, and Changxing Ding. Can linguistic knowledge improve multimodal alignment in vision-language pretraining? ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2308.12898","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-90","text":"[105] ","element":"span"},{"text":"Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2205.14100","element":"span"},{"text":", 2022.","element":"span"}],[{"id":"id-2","text":"[106] ","element":"span"},{"text":"Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2208.10442","element":"span"},{"text":", 2022.","element":"span"}],[{"id":"id-82","text":"[107] ","element":"span"},{"text":"Licheng Wen, Xuemeng Yang, Daocheng Fu, Xiaofeng Wang, Pinlong Cai, Xin Li, Tao Ma, Yingxuan Li, Linran Xu, Dengke Shang, Zheng Zhu, Shaoyan Sun, Yeqi Bai, Xinyu Cai, Min Dou, Shuanglu Hu, and Botian Shi. On the road with gpt-4v(ision): Early explorations of visual-language model on autonomous driving, 2023.","element":"span"}],[{"id":"id-18","text":"[108] ","element":"span"},{"text":"Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2303.04671","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-17","text":"[109] ","element":"span"},{"text":"Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2309.05519","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-69","text":"[110] ","element":"span"},{"text":"Keyang Xu, Mike Lam, Jingzhi Pang, Xin Gao, Charlotte Band, Piyush Mathur, Frank Papay, Ashish K Khanna, Jacek B Cywinski, Kamal Maheshwari, et al. Multimodal machine learning for automated icd coding. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Machine learning for healthcare conference","element":"span"},{"text":", pages 197–215. PMLR, 2019.","element":"span"}],[{"id":"id-73","text":"[111] ","element":"span"},{"text":"Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2306.09265","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-98","text":"[112] ","element":"span"},{"text":"Zhiyang Xu, Ying Shen, and Lifu Huang. Multiinstruct: Improving multi-modal zero-shot learning via instruction tuning. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2212.10773","element":"span"},{"text":", 2022.","element":"span"}],[{"id":"id-81","text":"[113] ","element":"span"},{"text":"Zhiling Yan, Kai Zhang, Rong Zhou, Lifang He, Xiang Li, and Lichao Sun. Multimodal chatgpt for medical applications: an experimental study of gpt-4v. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2310.19061","element":"span"},{"text":", 2023.","element":"span"}],[{"text":"[114] ","element":"span"},{"text":"Yi Yang and Shawn Newsam. Bag-of-visual-words and spatial extensions for land-use classification. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Proceedings of the 18th SIGSPATIAL international conference on advances in geographic information systems","element":"span"},{"text":", pages 270–279, 2010.","element":"span"}],[{"id":"id-14","text":"[115] ","element":"span"},{"text":"Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2303.11381","element":"span"},{"text":", 2023.","element":"span"}],[{"text":"[116] ","element":"span"},{"text":"Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2304.14178","element":"span"},{"text":", 2023.","element":"span"}],[{"text":"[117] ","element":"span"},{"text":"Ron Yosef, Yonatan Bitton, and Dafna Shahaf. Irfl: Image recognition of figurative language. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2303.15445","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-95","text":"[118] ","element":"span"},{"text":"Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2310.07704","element":"span"},{"text":", 2023.","element":"span"}],[{"text":"[119] ","element":"span"},{"text":"Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"TACL","element":"span"},{"text":", 2:67–78, 2014.","element":"span"}],[{"id":"id-94","text":"[120] ","element":"span"},{"text":"Haofei Yu, Paul Pu Liang, Ruslan Salakhutdinov, and Louis-Philippe Morency. Mixture of multimodal interaction experts. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"UniReps: the First Workshop on Unifying Representations in Neural Models","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-76","text":"[121] ","element":"span"},{"text":"Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2308.02490","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-44","text":"[122] ","element":"span"},{"text":"Weizhe Yuan, Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text as text generation. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Advances in Neural Information Processing Systems","element":"span"},{"text":", 34:27263–27277, 2021.","element":"span"}],[{"text":"[123] ","element":"span"},{"text":"Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","element":"span"},{"text":", June 2019.","element":"span"}],[{"text":"[124] ","element":"span"},{"text":"Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guided image editing. In ","element":"span"},{"style":{"fontStyle":"italic"},"text":"Advances in Neural Information Processing Systems","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-114","text":"[125] ","element":"span"},{"text":"Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2303.16199","element":"span"},{"text":", 2023.","element":"span"}],[{"id":"id-116","text":"[126] ","element":"span"},{"text":"Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:1904.09675","element":"span"},{"text":", 2019.","element":"span"}],[{"text":"[127] ","element":"span"},{"text":"Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"International Journal of Computer Vision","element":"span"},{"text":", 127:302–321, 2019.","element":"span"}],[{"id":"id-42","text":"[128] ","element":"span"},{"text":"Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. ","element":"span"},{"style":{"fontStyle":"italic"},"text":"arXiv preprint arXiv:2304.10592","element":"span"},{"text":", 2023.","element":"span"}]]},{"heading":"Checklist","paragraphs":[[{"text":"1. For all authors...","element":"span"}],[{"text":"(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? ","element":"span"},{"text":"[Yes]","element":"span"}],[{"text":"(b) Did you describe the limitations of your work? ","element":"span"},{"text":"[Yes] ","element":"span"},{"text":"We have included limitations in Section ","element":"span"},{"text":"6.","element":"span"}],[{"text":"(c) Did you discuss any potential negative societal impacts of your work? ","element":"span"},{"text":"[Yes] ","element":"span"},{"text":"We have included potential negative societal impacts in Section ","element":"span"},{"text":"6.","element":"span"}],[{"text":"(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? ","element":"span"},{"text":"[Yes]","element":"span"}],[{"text":"2. If you are including theoretical results...","element":"span"}],[{"text":"(a) Did you state the full set of assumptions of all theoretical results? ","element":"span"},{"text":"[N/A] ","element":"span"},{"text":"We do not present any theoretical results in our work.","element":"span"}],[{"text":"(b) Did you include complete proofs of all theoretical results? ","element":"span"},{"text":"[N/A] ","element":"span"},{"text":"We do not present any theoretical results in our work, hence there are no proofs.","element":"span"}],[{"text":"3. If you ran experiments (e.g. for benchmarks)...","element":"span"}],[{"text":"(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? ","element":"span"},{"text":"[Yes] ","element":"span"},{"text":"We have included code in the supplemental material.","element":"span"}],[{"text":"(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? ","element":"span"},{"text":"[Yes] ","element":"span"},{"text":"We provide the experimental details in Section ","element":"span"},{"href":"#id-99","text":"4.1 ","element":"a"},{"text":"and in Appendix ","element":"span"},{"text":"B.","element":"span"}],[{"text":"(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? ","element":"span"},{"text":"[Yes] ","element":"span"},{"text":"We report results with mean and standard deviation from running multiple times in the appendix. Using multiple runs we also compute statistical significance for all dataset and model performance comparisons, all the results in the main paper are only highlighted if they are statistically significant according to p-value.","element":"span"}],[{"text":"(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? ","element":"span"},{"text":"[Yes] ","element":"span"},{"text":"We provide details about compute and the type of resources used in Section ","element":"span"},{"href":"#id-100","text":"4.1.","element":"a"}],[{"text":"4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...","element":"span"}],[{"text":"(a) If your work uses existing assets, did you cite the creators? ","element":"span"},{"text":"[Yes] ","element":"span"},{"text":"We cite all existing models, datasets, and work we used in the references.","element":"span"}],[{"text":"(b) Did you mention the license of the assets? ","element":"span"},{"text":"[Yes] ","element":"span"},{"text":"The license of all the assets used in our work has been mentioned in Appendix ","element":"span"},{"href":"#id-101","text":"A.1 ","element":"a"},{"text":"and ","element":"span"},{"href":"#id-102","text":"A.4.","element":"a"}],[{"text":"(c) Did you include any new assets either in the supplemental material or as a URL? ","element":"span"},{"text":"[Yes]","element":"span"}],[{"text":"We have included full links to the dataset, models, and code in the supplementary material.","element":"span"}],[{"text":"(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating? ","element":"span"},{"text":"[Yes] ","element":"span"},{"text":"We provide access information of the datasets in Appendix ","element":"span"},{"href":"#id-101","text":"A.1. ","element":"a"},{"text":"To the best of our knowledge, all of these datasets are collected with consent from participating users, especially in the healthcare domain where user data is sensitive. Best practices for de-identification of user data were followed by these datasets. The dataset for facial expression recognition (FER-2013) contains human faces collected through Google image search queries, so user consent was not directly obtained, but the authors of FER-2013 have ensured that their dataset follows fair use guidelines and there is no personally identifiable information released.","element":"span"}],[{"text":"(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? ","element":"span"},{"text":"[Yes] ","element":"span"},{"text":"Yes, we have included all dataset details in Appendix ","element":"span"},{"href":"#id-101","text":"A.1 ","element":"a"},{"text":"including if the individual datasets in HEMM contain personally","element":"span"}],[{"text":"identifiable information or offensive content. To the best of our knowledge, all potentially identifiable information in all datasets (especially in those from medical settings or human social data) has been removed and completely de-identified. The dataset for facial expression recognition (FER-2013) contains human faces collected through Google image search queries, but does not contain any identifying information about user identities and backgrounds. Finally, the Hateful Memes dataset contains offensive content, since that is the goal of the research.","element":"span"}],[{"text":"5. If you used crowdsourcing or conducted research with human subjects...","element":"span"}],[{"text":"(a) Did you include the full text of instructions given to participants and screenshots, if applicable? ","element":"span"},{"text":"[Yes] ","element":"span"},{"text":"We provide the instructions regarding annotations in Section ","element":"span"},{"href":"#id-103","text":"4.3 ","element":"a"},{"text":"and in Appendix ","element":"span"},{"href":"#id-104","text":"A.2.","element":"a"}],[{"text":"(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? ","element":"span"},{"text":"[Yes] ","element":"span"},{"text":"Based on direct communication with our institution’s IRB office, this line of research is exempt from IRB, and the information obtained during our study is recorded in such a manner that the identity of the human subjects cannot readily be ascertained, directly or through identifiers linked to the subjects. There is no potential risk to participants and we do not collect any identifiable information from annotators.","element":"span"}],[{"text":"(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? ","element":"span"},{"text":"[Yes] ","element":"span"},{"text":"We include participant details in Appendix ","element":"span"},{"href":"#id-104","text":"A.2.","element":"a"}]]},{"heading":"Appendix A HEMM Details","paragraphs":[[{"id":"id-101","style":{"fontWeight":"bold"},"text":"A.1 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Individual dataset details","element":"span"}],[{"text":"In this section we provide the details of the tasks and datasets chosen for the HEMM benchmark: we describe the split used to evaluate the models, any prepossessing applied to the samples, and their access restrictions and licenses.","element":"span"}],[{"text":"1. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"VQA ","element":"span"},{"text":"dataset consists of samples of an image and a corresponding free-form, open-ended question. To answer the questions, the models need to perform fine-grained recognition of objects and activities. Some of the samples require commonsense reasoning to correctly answer the questions. Most of the samples in the dataset have \"yes\" or \"no\" answers.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"We evaluate on the real images validation set which comprises of a total of 244,302 questions.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"You are given an image and a question. Answer the question in a single word. Question: ","element":"span"},{"style":{"fontStyle":"italic"},"text":"","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"download ","element":"span"},{"text":"from ","element":"span"},{"href":"https://visualqa.org/vqa_v1_download.html","text":"https://visualqa.org/vqa_v1_download.html","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"The images in the dataset come with a CC BY 4.0 DEED license ","element":"span"},{"href":"https://creativecommons.org/licenses/by/4.0/deed.en","text":"https://creativecommons.org/licenses/by/4.0/deed.en","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content present in the dataset.","element":"span"}],[{"text":"2. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"N","element":"span"},{"style":{"fontWeight":"bold"},"text":"O","element":"span"},{"style":{"fontWeight":"bold"},"text":"C","element":"span"},{"style":{"fontWeight":"bold"},"text":"APS ","element":"span"},{"text":"dataset is a large scale image captioning dataset. Training data for this dataset consists of Image-Caption pairs from COCO dataset [","element":"span"},{"href":"#id-105","referenceIndex":71,"text":"71","element":"a"},{"text":"] as well as images and labels from Open Images. Many objects seen in the test set have very few associated captions from the training set making it a robust benchmark for image captioning.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"Evaluation is performed on the validation set which consists of 4500 images.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"You are given an image. This image might contain a lot of objects. You have to generate a caption for the image but the caption should just be a single sentence. Please do not generate more than one sentences. Caption:","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access restrictions: ","element":"span"},{"text":"The dataset is available to download from ","element":"span"},{"href":"https://nocaps.org/download","text":"https://nocaps.org/download","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"Images belonging to the dataset are available under CC BY 2.0 ","element":"span"},{"href":"https://creativecommons.org/licenses/by/2.0/deed.en","text":"https://creativecommons.org/licenses/by/2.0/deed.en ","element":"a"},{"text":"license which permits to redistribute the data.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content present in the dataset.","element":"span"}],[{"text":"3. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"D","element":"span"},{"style":{"fontWeight":"bold"},"text":"ECIMER ","element":"span"},{"text":"dataset is a hand-drawn molecule image dataset consisting of chemical structure as the images and their SMILES representation as the strings. This SMILES representation stands for ’Simplified Molecular Input Line Entry System’, which depicts the threedimensional structure of the chemical into a string of symbols. In order to solve this task, the model should have an understanding of structure of the chemical and how these structures are depicted in the given format.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"The dataset consists of 5088 images over which evaluation has been performed.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"Simplified molecular-input line-entry system (SMILES) notation of the given molecule:","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"download ","element":"span"},{"text":"from ","element":"span"},{"href":"https://zenodo.org/records/7617107","text":"https://zenodo.org/records/7617107","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"The dataset is available under Creative Commons Attribution 4.0 International License ","element":"span"},{"href":"https://creativecommons.org/licenses/by/4.0/deed.en","text":"https://creativecommons.org/licenses/by/4.0/deed.en, ","element":"a"},{"text":"which permits use and sharing of data.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content present in the dataset.","element":"span"}],[{"style":{"width":"60%"},"width":1493,"height":212,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/20-0.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Splits: ","element":"span"},{"text":"A total of 6992 images were used.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"Question: Given the Meme and the following caption, Caption:","element":"span"},{"style":{"fontStyle":"italic"},"text":"","element":"span"},{"text":". How funny is the meme? Choose from the following comma separated options: funny, very funny, not funny, hilarious.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"download ","element":"span"},{"text":"from ","element":"span"},{"href":"https://www.kaggle.com/datasets/williamscott701/memotion-dataset-7k","text":"https://www.kaggle.com/datasets/williamscott701/memotion-dataset-7k","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"The dataset is available under Creative Commons Attribution 4.0 International License ","element":"span"},{"href":"https://creativecommons.org/licenses/by/4.0/deed.en","text":"https://creativecommons.org/licenses/by/4.0/deed.en ","element":"a"},{"text":"which allows sharing of data.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information is present in the data. Offensive content is present in the dataset in some meme images.","element":"span"}],[{"style":{"width":"60%"},"width":1491,"height":212,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/20-1.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"A total of 4.24k questions from the test set.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"You are given a question and few choices. There is context provided with the image which will help you to understand the image. To answer the question, you have been given lecture notes. You can use these lecture notes, image, and context to answer the question. There are some choices given to you which are comma-separated. You have to select which choice best answers the question. Generate choice as it is from the given choices. lecture: ","element":"span"},{"style":{"fontStyle":"italic"},"text":" ","element":"span"},{"text":"question: ","element":"span"},{"style":{"fontStyle":"italic"},"text":" ","element":"span"},{"text":"context: ","element":"span"},{"style":{"fontStyle":"italic"},"text":" ","element":"span"},{"text":"choices: ","element":"span"},{"style":{"fontStyle":"italic"},"text":" ","element":"span"},{"text":"Answer:","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"download ","element":"span"},{"text":"from ","element":"span"},{"href":"https://huggingface.co/datasets/derek-thomas/ScienceQA","text":"https://huggingface.co/datasets/derek-thomas/ScienceQA","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"Dataset is distributed under the CC BY-NC-SA (Attribution-NonCommercial-ShareAlike) ","element":"span"},{"href":"https://creativecommons.org/licenses/by-nc-sa/4.0/deed.en","text":"https://creativecommons.org/licenses/by-nc-sa/4.0/deed.en ","element":"a"},{"text":"which allows sharing data.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"style":{"width":"61%"},"width":1495,"height":168,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/20-2.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"We use the test set of this dataset which consists of 2070 questions.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"Answer the question in a single word, Question: ","element":"span"},{"style":{"fontStyle":"italic"},"text":"","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"download ","element":"span"},{"text":"from ","element":"span"},{"href":"https://huggingface.co/datasets/BoKelvin/SLAKE","text":"https://huggingface.co/datasets/BoKelvin/SLAKE","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"Images under this dataset are available in CC-BY-SA 4.0 license ","element":"span"},{"href":"https://creativecommons.org/licenses/by-sa/4.0/deed.en","text":"https://creativecommons.org/licenses/by-sa/4.0/deed.en ","element":"a"},{"text":"which allows sharing data.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"text":"7. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"V","element":"span"},{"style":{"fontWeight":"bold"},"text":"ISUAL ","element":"span"},{"style":{"fontWeight":"bold"},"text":"G","element":"span"},{"style":{"fontWeight":"bold"},"text":"ENOME ","element":"span"},{"text":"dataset is a visual question-answering dataset that grounds visual concepts to language. Visual Genome provides a formal representation of an image, as relationships between objects in the image are depicted with the help of a scene graph. WordNet [","element":"span"},{"href":"#id-106","referenceIndex":79,"text":"79","element":"a"},{"text":"] is used to canonicalize objects, attributes, and relationships in each image.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"We use the splits available from ","element":"span"},{"href":"https://homes.cs.washington.edu/~ranjay/visualgenome/data/dataset/question_answers.json.zip","text":"https://homes.cs.washington.edu/ ran- ","element":"a"},{"href":"https://homes.cs.washington.edu/~ranjay/visualgenome/data/dataset/question_answers.json.zip","text":"jay/visualgenome/data/dataset/question_answers.json.zip","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"You are given an image and a question. Answer the question in a single word only. Question: ","element":"span"},{"style":{"fontStyle":"italic"},"text":"","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"download ","element":"span"},{"text":"from ","element":"span"},{"href":"https://homes.cs.washington.edu/~ranjay/visualgenome/api.html","text":"https://homes.cs.washington.edu/ ranjay/visualgenome/api.html","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"The data is available under Creative Commons Attribution 4.0 International License ","element":"span"},{"href":"https://creativecommons.org/licenses/by/4.0/deed.en","text":"https://creativecommons.org/licenses/by/4.0/deed.en","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"text":"8. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"P","element":"span"},{"style":{"fontWeight":"bold"},"text":"ATH","element":"span"},{"style":{"fontWeight":"bold"},"text":"VQA ","element":"span"},{"text":"is a visual QA dataset based on pathology images, PathVQA consists of images taken from pathology textbooks and online digital libraries, with question-answer pairs generated from captions using a question generation pipeline. Each pathology image is coupled with a question-answer pair.","element":"span"}],[{"style":{"width":"46%"},"width":734,"height":37,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/21-0.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"You are given a radiology image and a question. Answer the question in a single word. Question: ","element":"span"},{"style":{"fontStyle":"italic"},"text":"","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access restrictions: ","element":"span"},{"text":"The data is available to download from ","element":"span"},{"href":"https://github.com/UCSD-AI4H/PathVQA","text":"https://github.com/UCSD- ","element":"a"},{"href":"https://github.com/UCSD-AI4H/PathVQA","text":"AI4H/PathVQA","element":"a"}],[{"style":{"width":"51%"},"width":817,"height":30,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/21-1.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"text":"9. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"UCM","element":"span"},{"style":{"fontWeight":"bold"},"text":"ERCED LAND USE ","element":"span"},{"text":"is another dataset for land use classification which has 21 classes. Images from the USGS National Map Urban Area Imagery were extracted manually, which involves various urban areas around the country. We include all the possible classes in the prompt so the model can choose from them.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"We ","element":"span"},{"text":"evaluate ","element":"span"},{"text":"on ","element":"span"},{"text":"the ","element":"span"},{"text":"validation ","element":"span"},{"text":"split ","element":"span"},{"text":"present ","element":"span"},{"text":"in ","element":"span"},{"href":"https://www.kaggle.com/datasets/apollo2506/landuse-scene-classification","text":"https://www.kaggle.com/datasets/apollo2506/landuse-scene-classification.","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"Image is given to you. Classify if the image belongs to one of the following classes: mediumresidential, buildings, tenniscourt, denseresidential, baseballdiamond, intersection, harbor, parkinglot, river, overpass, mobilehomepark, runway, forest, beach, freeway, airplane, storagetanks, chaparral, golfcourse, sparseresidential, agricultural. Choose a class from the above classes.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"download from ","element":"span"},{"href":"http://weegee.vision.ucmerced.edu/datasets/landuse.html","text":"http://weegee.vision.ucmerced.edu/datasets/landuse.html ","element":"a"},{"text":"or ","element":"span"},{"href":"https://www.kaggle.com/datasets/apollo2506/landuse-scene-classification","text":"https://www.kaggle.com/datasets/apollo2506/landuse-scene-classification","element":"a"}],[{"style":{"width":"51%"},"width":817,"height":30,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/21-2.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"text":"10. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"E","element":"span"},{"style":{"fontWeight":"bold"},"text":"NRICO ","element":"span"},{"text":"is a topic modeling dataset for mobile UI screens. It is an enhanced version of RICO dataset [","element":"span"},{"href":"#id-107","referenceIndex":23,"text":"23","element":"a"},{"text":"] where samples were ranked as a good or bad design example by two human annotators. UI classes in the dataset consist of interfaces such as calculator, camera, chat, news, profile, etc from which the model has to choose for a particular image.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"We ","element":"span"},{"text":"evaluate ","element":"span"},{"text":"on ","element":"span"},{"text":"the ","element":"span"},{"text":"dataset ","element":"span"},{"text":"provided ","element":"span"},{"text":"in ","element":"span"},{"href":"http://userinterfaces.aalto.fi/enrico/resources/screenshots.zip","text":"http://userinterfaces.aalto.fi/enrico/resources/screenshots.zip","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"Given a screenshot of the user interface of a mobile application. Choose the most appropriate design topic from the following comma-separated choices: bare, dialer, camera, chat, editor, form, gallery, list, login, maps, mediaplayer, menu, modal, news, other, profile, search, settings, terms, tutorial","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"download ","element":"span"},{"text":"from ","element":"span"},{"href":"https://github.com/luileito/enrico","text":"https://github.com/luileito/enrico","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"comes ","element":"span"},{"text":"under ","element":"span"},{"text":"MIT ","element":"span"},{"text":"license ","element":"span"},{"href":"https://github.com/luileito/enrico/blob/master/LICENSE","text":"https://github.com/luileito/enrico/blob/master/LICENSE","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"style":{"width":"61%"},"width":1508,"height":211,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/22-0.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"We evaluate on the test split.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"Given the movie poster and the corresponding plot of the movie, choose the appropriate genres from the following comma-separated genres: drama, comedy, romance, thriller, crime, action, adventure, horror, documentry, mystery, sci-fi, fantasy, family, biography, war, history, music, animation, musical, western, sport, short, film-noir. Plot: ","element":"span"},{"style":{"fontStyle":"italic"},"text":" ","element":"span"},{"text":"Note that a movie can belong to more than one genres, provide all the suitable genres seperated by commas.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access restrictions: ","element":"span"},{"text":"It is a public dataset free to download by the research community from ","element":"span"},{"href":"http://lisi1.unal.edu.co/mmimdb/","text":"http://lisi1.unal.edu.co/mmimdb/ ","element":"a"},{"text":"and ","element":"span"},{"href":"https://github.com/johnarevalo/gmu-mmimdb/","text":"https://github.com/johnarevalo/gmu-mmimdb/","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"The dataset comes under MIT license ","element":"span"},{"href":"https://github.com/johnarevalo/gmu-mmimdb/blob/master/LICENSE","text":"https://github.com/johnarevalo/gmu- ","element":"a"},{"href":"https://github.com/johnarevalo/gmu-mmimdb/blob/master/LICENSE","text":"mmimdb/blob/master/LICENSE","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"style":{"width":"61%"},"width":1508,"height":203,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/22-1.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"We ","element":"span"},{"text":"evaluate ","element":"span"},{"text":"on ","element":"span"},{"text":"the ","element":"span"},{"text":"test ","element":"span"},{"text":"set ","element":"span"},{"text":"present ","element":"span"},{"text":"in ","element":"span"},{"href":"https://huggingface.co/datasets/flaviagiammarino/vqa-rad/viewer/default/test","text":"https://huggingface.co/datasets/flaviagiammarino/vqa-rad/viewer/default/test ","element":"a"},{"text":"which consists of 451 questions.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"You are given a radiology image and a question. Answer the question in a single word. Question:","element":"span"},{"style":{"fontStyle":"italic"},"text":"","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"at ","element":"span"},{"href":"https://huggingface.co/datasets/flaviagiammarino/vqa-rad/viewer","text":"https://huggingface.co/datasets/flaviagiammarino/vqa-rad/viewer","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"The dataset is available under Creative Commons Attribution 4.0 International License ","element":"span"},{"href":"https://creativecommons.org/licenses/by/4.0/deed.en","text":"https://creativecommons.org/licenses/by/4.0/deed.en","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"style":{"width":"61%"},"width":1504,"height":88,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/22-2.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"We evaluate the dataset on the test split.","element":"span"}],[{"style":{"width":"27%"},"width":440,"height":37,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/23-0.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"download ","element":"span"},{"text":"from ","element":"span"},{"href":"https://www.kaggle.com/datasets/hsankesara/flickr-image-dataset","text":"https://www.kaggle.com/datasets/hsankesara/flickr-image-dataset","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"under ","element":"span"},{"text":"CC0: ","element":"span"},{"text":"Public ","element":"span"},{"text":"Domain ","element":"span"},{"text":"License ","element":"span"},{"href":"https://creativecommons.org/publicdomain/zero/1.0/deed.en","text":"https://creativecommons.org/publicdomain/zero/1.0/deed.en","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"text":"14. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"FER-2013 ","element":"span"},{"text":"is a classic dataset for facial expression recognition, where each image has to be classified into 7 labels. Images for this dataset were obtained from Google images, by searching them using Google Search API. OpenCV was used to get bounding boxes for faces in each of the images.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"We ","element":"span"},{"text":"evaluate ","element":"span"},{"text":"on ","element":"span"},{"text":"the ","element":"span"},{"text":"test ","element":"span"},{"text":"dataset ","element":"span"},{"text":"present ","element":"span"},{"text":"in ","element":"span"},{"href":"https://www.kaggle.com/datasets/msambare/fer2013","text":"https://www.kaggle.com/datasets/msambare/fer2013","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"Given the photo of a face, determine the face expression, choose from the following choices: angry, disgust, fear, happy, neutral, sad, surprise. Answer in a single word.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"download ","element":"span"},{"text":"from ","element":"span"},{"href":"https://www.kaggle.com/datasets/msambare/fer2013","text":"https://www.kaggle.com/datasets/msambare/fer2013","element":"a"}],[{"style":{"width":"49%"},"width":787,"height":37,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/23-1.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"This dataset contains human faces collected through Google image search queries but does not contain any identifying information about user identities and backgrounds. No offensive content is present in the dataset.","element":"span"}],[{"text":"15. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"NY C","element":"span"},{"style":{"fontWeight":"bold"},"text":"ARTOON ","element":"span"},{"text":"is collected from the weekly New Yorker magazine cartoon captioning contest ","element":"span"},{"text":"*","element":"span"},{"text":", where readers are tasked to give a humorous caption for a cartoon image and the funniest captions are selected based on public votes. The dataset is formulated based on taking in the image and caption to predict how funny the pair is based on the normalized number of votes. Given an image and its caption, we ask the model if the caption is humorous or not. Each image has multiple caption choices with votes for the caption being not funny, somewhat funny, funny. We select the funniest caption to have a ground truth answer as ’yes’ when prompted for evaluation. The next four funniest captions are selected to have ground truth answers as ’no’ when prompted for evaluation.","element":"span"}],[{"style":{"width":"82%"},"width":1312,"height":38,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/23-2.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"You are given a cartoon image and a caption. start the answer with yes if the caption is funny or No if the caption is not funny. Caption: ","element":"span"},{"style":{"fontStyle":"italic"},"text":"","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"download ","element":"span"},{"text":"from ","element":"span"},{"href":"https://github.com/nextml/caption-contest-data","text":"https://github.com/nextml/caption-contest-data","element":"a"}],[{"style":{"width":"50%"},"width":794,"height":37,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/23-3.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"text":"16. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"OK-VQA ","element":"span"},{"text":"is a visual question-answering task that requires outside knowledge and reasoning to answer questions. Images for this dataset are taken from the COCO dataset[","element":"span"},{"href":"#id-105","referenceIndex":71,"text":"71","element":"a"},{"text":"] and MTurk ","element":"span"},{"text":"* ","element":"span"},{"text":"is used for labeling questions. A specific instruction is given to the workers to label questions that require knowledge outside the image. In this dataset, questions are of open-ended type.","element":"span"}],[{"style":{"width":"80%"},"width":1275,"height":37,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/23-4.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"You are given an image and a question. Answer the question in a single word. Question: ","element":"span"},{"style":{"fontStyle":"italic"},"text":"","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"Dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"download ","element":"span"},{"text":"from ","element":"span"},{"href":"https://okvqa.allenai.org/download.html","text":"https://okvqa.allenai.org/download.html","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"Dataset ","element":"span"},{"text":"consists ","element":"span"},{"text":"of ","element":"span"},{"text":"images ","element":"span"},{"text":"which ","element":"span"},{"text":"have ","element":"span"},{"text":"CC ","element":"span"},{"text":"BY ","element":"span"},{"text":"4.0 ","element":"span"},{"text":"DEED ","element":"span"},{"href":"https://creativecommons.org/licenses/by/4.0/deed.en","text":"https://creativecommons.org/licenses/by/4.0/deed.en","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"style":{"width":"61%"},"width":1512,"height":212,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/24-0.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"We evaluate on the test set available from ","element":"span"},{"href":"https://osu-nlp-group.github.io/MagicBrush/","text":"https://osu-nlp-group.github.io/MagicBrush/","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"Edit the given image based on the provided instruction. Instruction: ","element":"span"},{"style":{"fontStyle":"italic"},"text":"","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access restrictions: ","element":"span"},{"text":"The dataset is available to download from ","element":"span"},{"href":"https://osu-nlp-group.github.io/MagicBrush/","text":"https://osu-nlp- ","element":"a"},{"href":"https://osu-nlp-group.github.io/MagicBrush/","text":"group.github.io/MagicBrush/","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"comes ","element":"span"},{"text":"under ","element":"span"},{"text":"CC ","element":"span"},{"text":"BY ","element":"span"},{"text":"4.0 ","element":"span"},{"text":"license ","element":"span"},{"href":"https://creativecommons.org/licenses/by/4.0/deed.en","text":"https://creativecommons.org/licenses/by/4.0/deed.en","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"style":{"width":"61%"},"width":1507,"height":212,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/24-1.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"We evaluate on the test set from ","element":"span"},{"href":"https://github.com/eujhwang/meme-cap/tree/main","text":"https://github.com/eujhwang/meme-cap/tree/main","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"This is a meme with the title ","element":"span"},{"style":{"fontStyle":"italic"},"text":"","element":"span"},{"text":". The image description is ","element":"span"},{"style":{"fontStyle":"italic"},"text":"<im-age_description>","element":"span"},{"text":". What is the meme poster trying to convey? Answer:","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"can ","element":"span"},{"text":"be ","element":"span"},{"text":"downloaded ","element":"span"},{"text":"from ","element":"span"},{"href":"https://github.com/eujhwang/meme-cap/tree/main","text":"https://github.com/eujhwang/meme-cap/tree/main","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"No license is available for the dataset.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information is present. However, offensive content may be present in the images due to the dataset containing meme data.","element":"span"}],[{"style":{"width":"61%"},"width":1507,"height":168,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/24-2.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Splits: ","element":"span"},{"text":"We use the ’dev’ split from ","element":"span"},{"href":"https://www.kaggle.com/datasets/parthplc/facebook-hateful-meme-dataset/data","text":"https://www.kaggle.com/datasets/parthplc/facebook- ","element":"a"},{"href":"https://www.kaggle.com/datasets/parthplc/facebook-hateful-meme-dataset/data","text":"hateful-meme-dataset/data","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"You are given an image. In the image, the text phrase that you will be given and the image are innocuous when considered by themselves. The semantic content of the meme becomes mean only when the text phrase and image are considered together. Text phrase: ","element":"span"},{"style":{"fontStyle":"italic"},"text":"<text_phrase> ","element":"span"},{"text":"You have to judge if the combination of image and text is hateful or not. Always begin your answer with either ’yes’ or ’no’ with ’yes’ indicating that the meme is hateful and ’no’ if it is not hateful. Answer:","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"downloaded ","element":"span"},{"text":"from ","element":"span"},{"href":"https://www.kaggle.com/datasets/parthplc/facebook-hateful-meme-dataset/data","text":"https://www.kaggle.com/datasets/parthplc/facebook-hateful-meme-dataset/data","element":"a"}],[{"style":{"width":"91%"},"width":1445,"height":37,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/25-0.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information is present. However, offensive content may be present in the images since it is the goal of the dataset to train a detector for offensiveness given multimodal meme inputs.","element":"span"}],[{"text":"20. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"I","element":"span"},{"style":{"fontWeight":"bold"},"text":"N","element":"span"},{"style":{"fontWeight":"bold"},"text":"ATURALIST ","element":"span"},{"text":"is an image classification dataset for 5000 wildlife species of plants and animals. Images and labels are sourced from iNaturalist website ","element":"span"},{"text":"*","element":"span"},{"text":". We evaluate the models by asking them to identify the species present in the given image. We do not provide it with possible classes as the dataset spans over a set of 5000 species.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"We evaluate the model on the validation split provided in the 2021 edition of the dataset.","element":"span"}],[{"style":{"width":"80%"},"width":1283,"height":37,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/25-1.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Access restrictions: ","element":"span"},{"text":"We use the dataset available at ","element":"span"},{"href":"https://ml-inat-competition-datasets.s3.amazonaws.com/2021/val.tar.gz","text":"https://ml-inat-competition- ","element":"a"},{"href":"https://ml-inat-competition-datasets.s3.amazonaws.com/2021/val.tar.gz","text":"datasets.s3.amazonaws.com/2021/val.tar.gz","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"Work ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"under ","element":"span"},{"text":"MIT ","element":"span"},{"text":"license ","element":"span"},{"href":"https://github.com/visipedia/inat_comp/blob/master/LICENSE","text":"https://github.com/visipedia/inat_comp/blob/master/LICENSE","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"text":"21. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"N","element":"span"},{"style":{"fontWeight":"bold"},"text":"LVR ","element":"span"},{"text":"consists of image-text pairs for visual reasoning. Images are created by generating objects and their properties randomly. These images are then given to a crowd worker to describe the image in a sentence.","element":"span"}],[{"style":{"width":"75%"},"width":1200,"height":37,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/25-2.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"Given this image along with a question about the image, please answer the question with only the word ’true’ or ’false’. Question: ","element":"span"},{"style":{"fontStyle":"italic"},"text":"<question>","element":"span"}],[{"style":{"width":"84%"},"width":1332,"height":103,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/25-3.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"text":"22. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"N","element":"span"},{"style":{"fontWeight":"bold"},"text":"LVR","element":"span"},{"style":{"fontWeight":"bold"},"text":"2 ","element":"span"},{"text":"extends NLVR to real-world photographs, and captions for these photographs. Images are retrieved using search queries from the ILSVRC2014 ImageNet challenge ","element":"span"},{"text":"*","element":"span"},{"text":". Crowdworkers are used to write the captions for the images. For this dataset, each data point has two images and a sentence that talks about the images. We concatenate the two images so that we pass a single image in the model.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"Evaluation ","element":"span"},{"text":"is ","element":"span"},{"text":"performed ","element":"span"},{"text":"on ","element":"span"},{"text":"the ","element":"span"},{"text":"dev ","element":"span"},{"text":"split ","element":"span"},{"text":"from ","element":"span"},{"href":"https://github.com/lil-lab/nlvr/tree/master/nlvr2","text":"https://github.com/lil- ","element":"a"},{"href":"https://github.com/lil-lab/nlvr/tree/master/nlvr2","text":"lab/nlvr/tree/master/nlvr2","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"You are given an image and a related text, use the image as context and reply with true or false only Text: ","element":"span"},{"style":{"fontStyle":"italic"},"text":"<text> ","element":"span"},{"text":"Answer:","element":"span"}],[{"style":{"width":"86%"},"width":1368,"height":37,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/25-4.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"Images have licenses CC BY-SA 2.0 ","element":"span"},{"href":"https://creativecommons.org/licenses/by-sa/2.0/deed.en","text":"https://creativecommons.org/licenses/by- ","element":"a"},{"href":"https://creativecommons.org/licenses/by-sa/2.0/deed.en","text":"sa/2.0/deed.en, ","element":"a"},{"text":"CC ","element":"span"},{"text":"BY ","element":"span"},{"text":"3.0 ","element":"span"},{"href":"https://creativecommons.org/licenses/by/3.0/deed.en","text":"https://creativecommons.org/licenses/by/3.0/deed.en, ","element":"a"},{"text":"CC ","element":"span"},{"text":"BY-SA ","element":"span"},{"text":"3.0 ","element":"span"},{"href":"https://creativecommons.org/licenses/by-sa/3.0/deed.en","text":"https://creativecommons.org/licenses/by-sa/3.0/deed.en, ","element":"a"},{"text":"CC0 ","element":"span"},{"text":"1.0 ","element":"span"},{"href":"https://creativecommons.org/publicdomain/zero/1.0/deed.en","text":"https://creativecommons.org/publicdomain/zero/1.0/deed.en, ","element":"a"},{"text":"CC ","element":"span"},{"text":"BY-SA ","element":"span"},{"text":"4.0 ","element":"span"},{"href":"https://creativecommons.org/licenses/by-sa/4.0/deed.en","text":"https://creativecommons.org/licenses/by-sa/4.0/deed.en, ","element":"a"},{"text":"CC0 ","element":"span"},{"href":"https://creativecommons.org/public-domain/cc0/","text":"https://creativecommons.org/public-domain/cc0/ ","element":"a"},{"text":"and ","element":"span"},{"text":"Pixabay ","element":"span"},{"href":"https://pixabay.com/service/terms/","text":"https://pixabay.com/service/terms/","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"style":{"width":"61%"},"width":1511,"height":168,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/26-0.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"’val’ split is used from ","element":"span"},{"href":"https://visualcommonsense.com/download/","text":"https://visualcommonsense.com/download/","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"Question: ","element":"span"},{"style":{"fontStyle":"italic"},"text":"<question> ","element":"span"},{"text":"Choose from the below choices: ","element":"span"},{"style":{"fontStyle":"italic"},"text":"<choices>","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"download ","element":"span"},{"text":"from ","element":"span"},{"href":"https://visualcommonsense.com/download/","text":"https://visualcommonsense.com/download/","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"The dataset is provided in license as ","element":"span"},{"href":"https://visualcommonsense.com/license/","text":"https://visualcommonsense.com/license/","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"style":{"width":"61%"},"width":1512,"height":255,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/26-1.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"Test set from ","element":"span"},{"href":"https://huggingface.co/datasets/facebook/winoground","text":"https://huggingface.co/datasets/facebook/winoground ","element":"a"},{"text":"is used.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"You are given an image and a text. Answer yes if the text matches the image and no if the text does not match the image. Text: ","element":"span"},{"style":{"fontStyle":"italic"},"text":"<text> ","element":"span"},{"text":"Answer:","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"downloaded ","element":"span"},{"text":"from ","element":"span"},{"href":"https://huggingface.co/datasets/facebook/winoground","text":"https://huggingface.co/datasets/facebook/winoground","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"Authors of the dataset have Getty image license ","element":"span"},{"href":"https://www.gettyimages.in/eula","text":"https://www.gettyimages.in/eula.","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"style":{"width":"61%"},"width":1510,"height":168,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/26-2.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Split","element":"span"},{"text":": We use the dataset from ","element":"span"},{"href":"https://www.kaggle.com/datasets/happyyang/nwpu-data-set","text":"https://www.kaggle.com/datasets/happyyang/nwpu-data-set","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"Image is given to you. Classify if the image belongs to one of the following classes: ’basketball_court’, ’overpass’, ’ground_track_field’, ’church’, ’chaparral’, ’forest’, ’parking_lot’, ’golf_course’, ’baseball_diamond’, ’meadow’, ’beach’,’sparse_residential’, ’desert’, ’terrace’, ’palace’, ’bridge’, ’commercial_area’, ’stadium’, ’runway’, ’lake’, ’railway’, ’tennis_court’, ’ship’, ’intersection’, ’river’, ’freeway’, ’airplane’, ’industrial_area’, ’mountain’, ’storage_tank’, ’cloud’, ’roundabout’, ’wetland’, ’mobile_home_park’, ’island’, ’harbor’, ’railway_station’, ’medium_residential’, ’sea_ice’, ’thermal_power_station’, ’snowberg’, ’circular_farmland’, ’airport’, ’dense_residential’, ’rectangular_farmland’. Choose a class from the above classes.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"downloaded ","element":"span"},{"text":"from ","element":"span"},{"href":"https://www.kaggle.com/datasets/happyyang/nwpu-data-set","text":"https://www.kaggle.com/datasets/happyyang/nwpu-data-set","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"No license is provided with the dataset.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"style":{"width":"61%"},"width":1515,"height":125,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/26-3.png","element":"img"}],[{"text":"grounded in the image. Each question is associated with a series of steps that need to be followed to get the answer as well as a scene graph that captures objects, attributes, and relations in the image","element":"span"}],[{"style":{"width":"97%"},"width":1545,"height":38,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/27-0.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"You are given an image and a question. Answer the question in a single word. Question: ","element":"span"},{"style":{"fontStyle":"italic"},"text":"<question>","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"download ","element":"span"},{"text":"from ","element":"span"},{"href":"https://cs.stanford.edu/people/dorarad/gqa/download.html","text":"https://cs.stanford.edu/people/dorarad/gqa/download.html","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"The images in the dataset come with a CC BY 4.0 DEED license ","element":"span"},{"href":"https://creativecommons.org/licenses/by/4.0/deed.en","text":"https://creativecommons.org/licenses/by/4.0/deed.en","element":"a"}],[{"text":"27. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"O","element":"span"},{"style":{"fontWeight":"bold"},"text":"PEN","element":"span"},{"style":{"fontWeight":"bold"},"text":"P","element":"span"},{"style":{"fontWeight":"bold"},"text":"ATH ","element":"span"},{"text":"is a dataset created from Twitter and other public sources. Each image has a natural language description, and the dataset is sourced from tweets across 32 hashtag sub-specialty categories in pathology.","element":"span"}],[{"style":{"width":"41%"},"width":661,"height":37,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/27-1.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"Choose from the below choices, Given image is a hematoxylin and eosin image of: cancer-associated stroma, adipose tissue, debris, lymphocytes, mucus, background, normal colon mucosa, colorectal adenocarcinoma epithelium, smooth muscle","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access restrictions: ","element":"span"},{"text":"The dataset is available to download from huggingface datasets ","element":"span"},{"href":"https://huggingface.co/datasets/akshayg08/OpenPath","text":"https://huggingface.co/datasets/akshayg08/OpenPath","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"under ","element":"span"},{"text":"CC ","element":"span"},{"text":"BY-NC ","element":"span"},{"text":"4.0 ","element":"span"},{"text":"license. ","element":"span"},{"href":"https://creativecommons.org/licenses/by-nc/4.0/","text":"https://creativecommons.org/licenses/by-nc/4.0/","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"text":"28. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"IRFL ","element":"span"},{"text":"is an image-text dataset for figurative language. The dataset consists of three broad categories: idioms, similes, and metaphors. Metaphors and similes were collected from online lists whereas idioms were collected from MAGPIE corpus[","element":"span"},{"href":"#id-108","referenceIndex":33,"text":"33","element":"a"},{"text":"]. Since the MAGPIE corpus did not contain definitions for idioms, definitions were crawled from online dictionaries to search for figurative images. Google images were used for searching the images for idioms using these definitions. For similes and metaphors, annotators were used for definitions, and images were searched on the internet. For our evaluation, we use simile categorization. For each data point, one simile and four images are given. We modify this task to evaluate one image at a time, so a pair of an image and similes are passed to the model to see if they match or not.","element":"span"}],[{"style":{"width":"59%"},"width":941,"height":38,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/27-2.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"You are given a simile and a picture along with the simile. You have to say if the simile matches the given picture. Answer the following question in a single word with a yes or no. Simile: ","element":"span"},{"style":{"fontStyle":"italic"},"text":"<simile> ","element":"span"},{"text":"Answer:","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access restrictions: ","element":"span"},{"text":"Dataset is available for download from ","element":"span"},{"href":"https://github.com/irfl-dataset/IRFL","text":"https://github.com/irfl- ","element":"a"},{"href":"https://github.com/irfl-dataset/IRFL","text":"dataset/IRFL","element":"a"}],[{"style":{"width":"50%"},"width":794,"height":37,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/27-3.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"text":"29. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"S","element":"span"},{"style":{"fontWeight":"bold"},"text":"CREEN","element":"span"},{"style":{"fontWeight":"bold"},"text":"2W","element":"span"},{"style":{"fontWeight":"bold"},"text":"ORDS ","element":"span"},{"text":"is a mobile UI summarization dataset consisting of images from RicoSCA","element":"span"},{"href":"#id-109","referenceIndex":64,"text":"[64] ","element":"a"},{"text":"dataset. A total of 85 annotators were used to describe the image.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"We ","element":"span"},{"text":"use ","element":"span"},{"text":"the ","element":"span"},{"text":"test ","element":"span"},{"text":"split ","element":"span"},{"text":"from ","element":"span"},{"href":"https://github.com/google-research-datasets/screen2words/tree/main","text":"https://github.com/google-research- ","element":"a"},{"href":"https://github.com/google-research-datasets/screen2words/tree/main","text":"datasets/screen2words/tree/main","element":"a"}],[{"style":{"width":"85%"},"width":1359,"height":37,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/27-4.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Access restrictions: ","element":"span"},{"text":"The dataset is available to download from ","element":"span"},{"href":"https://github.com/google-research-datasets/screen2words","text":"https://github.com/google- ","element":"a"},{"href":"https://github.com/google-research-datasets/screen2words","text":"research-datasets/screen2words","element":"a"}],[{"style":{"width":"59%"},"width":1464,"height":300,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/28-0.png","element":"img"}],[{"text":"Figure 6: Multimodal skills are the basic building blocks central to solving problems, spanning information integrated across modalities at different granularities, different ways modalities might interact to create new information, reasoning, and external knowledge.","element":"figcaption","subtype":"caption"}],[{"style":{"width":"56%"},"width":1374,"height":220,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/28-1.png","element":"img"}],[{"text":"Figure 7: Multimodal information flow studies how the content changes across the two modalities for the task, such as through cross-modal translation, editing, querying, and fusion.","element":"figcaption","subtype":"caption"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"No license is provided with the dataset.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"style":{"width":"61%"},"width":1512,"height":125,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/28-2.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Split: ","element":"span"},{"text":"We use the COCO subset from the Localized Narratives Dataset [","element":"span"},{"href":"#id-110","referenceIndex":87,"text":"87","element":"a"},{"text":"] containing 8,573 samples. The ground truth images are used from the MSCOCO (17) validation set.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Prompt used: ","element":"span"},{"text":"Generate an Image based on the provided caption. Caption:","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"download ","element":"span"},{"text":"from ","element":"span"},{"href":"https://google.github.io/localized-narratives/","text":"https://google.github.io/localized-narratives/","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Licenses: ","element":"span"},{"text":"The ","element":"span"},{"text":"dataset ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"under ","element":"span"},{"text":"CC ","element":"span"},{"text":"BY-NC ","element":"span"},{"text":"4.0 ","element":"span"},{"text":"license. ","element":"span"},{"href":"https://creativecommons.org/licenses/by-nc/4.0/","text":"https://creativecommons.org/licenses/by-nc/4.0/","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Ethical considerations: ","element":"span"},{"text":"No personally identifiable information or offensive content is present in the dataset.","element":"span"}],[{"id":"id-104","style":{"width":"64%"},"width":1589,"height":786,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/28-3.png","element":"img"}],[{"text":"Table 6: Inter-Annotator agreement scores for stage 1 annotation.","element":"figcaption","subtype":"caption"}],[{"style":{"width":"55%"},"width":880,"height":460,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/29-0.png","element":"img"}],[{"text":"Table 7: Categorization after aggregating human annotations.","element":"figcaption","subtype":"caption"}],[{"id":"id-111","style":{"width":"70%"},"width":1116,"height":1141,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/29-1.png","element":"img"}],[{"text":"As per the annotations, we aggregate the annotations for each dataset across each dimension and calculate the maximum occurrence of annotation across all dimensions to categorize the datasets presented in Table ","element":"span"},{"href":"#id-111","text":"7. ","element":"a"},{"text":"We also consider ‘Neutral Reasoning’ and ‘Less Reasoning’ to be the same category and label them as ‘Less Reasoning’ before aggregating over the annotations. However, we see that the inter-annotator scores have low agreement, and some annotations go against the definitions above in the section ","element":"span"},{"text":"2. ","element":"span"},{"text":"Hence, we carry out an additional round of the annotation process using GPT-4V, and explain the process below.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"A.2.2 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Categorization stage 2: Automatic annotation with human verification ","element":"span"},{"text":"After the first stage was done, we found that most of the annotations were reliable but there were some cases where annotators misunderstood the definitions and tasks which led to low agreement values. For the second stage of the annotation process, we query GPT-4V for categorization of datapoints into dimensions to supplement the human annotations we obtained in the first stage. For each dataset, we consider three samples from each dataset for a total of 87 data points for categorization spread out across six sets. For each data point, we ask the model the same questions as asked to the human annotators above and obtain the categorization across the dimensions. For some questions, the model","element":"span"}],[{"text":"Table 8: Categorization after aggregating GPT-4V annotations. Cases where ’-’ is present are due to the model ","element":"figcaption","subtype":"caption"},{"text":"not providing an answer, citing a lack of information available for evaluating the input. We ignore such cases for categorization.","element":"figcaption","subtype":"caption"}],[{"style":{"width":"77%"},"width":1116,"height":1141,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/30-0.png","element":"img"}],[{"text":"Table 9: Inter-annotator agreement scores for stage 2 annotations.","element":"figcaption","subtype":"caption"}],[{"style":{"width":"64%"},"width":1587,"height":1134,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/30-1.png","element":"img"}],[{"text":"Table 10: Final dataset categorization.","element":"figcaption","subtype":"caption"}],[{"id":"id-112","style":{"width":"76%"},"width":1206,"height":1141,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/31-0.png","element":"img"}],[{"text":"(4) indicates that authors of this work have categorized the dataset for the dimension. As we can see from Table ","element":"span"},{"href":"#id-112","text":"10, ","element":"a"},{"text":"the majority of categories are agreed upon both by human annotators and GPT-4V, indicating reliability. There are only a few with (4), indicating that authors had to provide the final categorization due to dimensions that were hard to understand by non-experts in multimodal learning and by GPT-4V.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"A.2.4 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Details on annotation and participants","element":"span"}],[{"text":"The annotations in stages 1 (human annotation) and 2 (automatic inference with human verification) are all university students with some knowledge of machine learning. There were 10 sets of annotations each evaluated by two annotators for a total of 20 annotators. All participation in user studies was voluntary and done for pay at a level consistent with research participation at our university (15 dollars an hour). The annotations in stage 3 (final check) are done by 5 experts in the multimodal machine learning community for a final verification in case of misunderstandings in the first two stages.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"A.3 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Modeling categorizations and details ","element":"span"},{"text":"We also evaluate the performance of the models based on various modeling decisions. To achieve this, we categorize the models into various classes based on the following properties:","element":"span"}],[{"text":"1. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Interleaved modality training: ","element":"span"},{"text":"In the multi-modal setting, models are broadly trained/fine-tuned either by separately processing individual modalities using modality-specific encoders followed by fusion, or by interleaving the raw modalities first and then processing the interleaved input together.","element":"span"}],[{"text":"2. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Instruction Tuning: ","element":"span"},{"text":"Generative multimodal models can be trained/fine-tuned using objectives such as image-text matching, image-grounded text generation [","element":"span"},{"href":"#id-52","referenceIndex":62,"text":"62","element":"a"},{"text":"], etc., to generate relevant outputs. However, recently such generative models are also instruction instruction-tuned in order to generate outputs that resemble human responses. Therefore, we also categorise the models based on whether instruction tuning is employed or not.","element":"span"}],[{"text":"3. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Architecture: ","element":"span"},{"text":"For training multi-modal models, parameters can either be initialized using a pre-trained model and then are fine-tuned/kept frozen, or are initialized randomly and trained in an end-to-end fashion. Based on this choice, we categorize models into two classes -fine-tuned and trained from scratch.","element":"span"}],[{"text":"4. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Training Data Size: ","element":"span"},{"text":"The amount of data used for training the models, plays an important role in the performance and generalization of the model. Based on the size of the training data (in our work, the number of image-text or image-image samples), we categorize the models into three categories - Small, Medium, and Large.","element":"span"}],[{"text":"5. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Number of Parameters: ","element":"span"},{"text":"Model size is an important modeling decision as it affects the performance of the model, cost and efficiency of training, and the inference time. Hence, we also categorize the models based on both the total and trainable number of parameters, and compare the performance across these categories.","element":"span"}],[{"text":"6. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Diversity in Training Data: ","element":"span"},{"text":"Training multimodal models on data from different tasks, improves the diversity of the training data and may help the models to perform well on multiple tasks. By categorizing the models based on the diversity of the training data used, we evaluate the effect of using data from diverse tasks.","element":"span"}],[{"id":"id-102","style":{"width":"64%"},"width":1587,"height":506,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/32-0.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"License: ","element":"span"},{"text":"The ","element":"span"},{"text":"model ","element":"span"},{"text":"comes ","element":"span"},{"text":"with ","element":"span"},{"text":"BSD-3 ","element":"span"},{"text":"Clause ","element":"span"},{"href":"https://github.com/salesforce/LAVIS/blob/main/LICENSE.txt","text":"https://github.com/salesforce/LAVIS/blob/main/LICENSE.txt","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Access restrictions: ","element":"span"},{"text":"The model is available to use from the LAVIS repository ","element":"span"},{"href":"https://github.com/salesforce/LAVIS","text":"https://github.com/salesforce/LAVIS","element":"a"}],[{"style":{"width":"61%"},"width":1495,"height":338,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/32-1.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"License: ","element":"span"},{"text":"The ","element":"span"},{"text":"model ","element":"span"},{"text":"comes ","element":"span"},{"text":"with ","element":"span"},{"text":"BSD-3 ","element":"span"},{"text":"Clause ","element":"span"},{"href":"https://github.com/salesforce/LAVIS/blob/main/LICENSE.txt","text":"https://github.com/salesforce/LAVIS/blob/main/LICENSE.txt","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Access restrictions: ","element":"span"},{"text":"The model is available to use from the LAVIS repository ","element":"span"},{"href":"https://github.com/salesforce/LAVIS","text":"https://github.com/salesforce/LAVIS","element":"a"}],[{"style":{"width":"61%"},"width":1494,"height":253,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/32-2.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"License: ","element":"span"},{"text":"The model comes with BSD-3 Clause ","element":"span"},{"href":"https://github.com/Vision-CAIR/MiniGPT-4/blob/main/LICENSE.md","text":"https://github.com/Vision-CAIR/MiniGPT- ","element":"a"},{"href":"https://github.com/Vision-CAIR/MiniGPT-4/blob/main/LICENSE.md","text":"4/blob/main/LICENSE.md","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Access restrictions: ","element":"span"},{"text":"The model is available to use from ","element":"span"},{"href":"https://github.com/Vision-CAIR/MiniGPT-4/tree/main","text":"https://github.com/Vision- ","element":"a"},{"href":"https://github.com/Vision-CAIR/MiniGPT-4/tree/main","text":"CAIR/MiniGPT-4/tree/main","element":"a"}],[{"text":"4. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"O","element":"span"},{"style":{"fontWeight":"bold"},"text":"PEN","element":"span"},{"style":{"fontWeight":"bold"},"text":"F","element":"span"},{"style":{"fontWeight":"bold"},"text":"LAMINGO ","element":"span"},{"text":"is an open-source reproduction of the Flamingo [","element":"span"},{"href":"#id-113","referenceIndex":3,"text":"3","element":"a"},{"text":"] models. Unlike models that can only take one input image per sample (e.g., BLIP2, MiniGPT-4), OpenFlamingo can handle multiple images by interleaving images and texts. The architecture comprises of pre-trained Vision and Language encoder/decoder, where the layers of the pre-trained LLM are augmented with the vision encoder outputs which allows for cross-modal attention. All the pre-trained components are kept frozen except for the cross-modal attention component. For evaluation, we use the ","element":"span"},{"style":{"fontFamily":"monospace"},"text":"OpenFlamingo-3B-vitl-mpt1b ","element":"span"},{"text":"model from the OpenFlamingo Github Repository ","element":"span"},{"text":"*","element":"span"},{"text":". The chosen models has 1.4B trainable parameters and a total of 3.2B parameters. It is trained using 180M image-text samples.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"License: ","element":"span"},{"text":"Work ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"under ","element":"span"},{"text":"MIT ","element":"span"},{"text":"License ","element":"span"},{"href":"https://github.com/mlfoundations/open_flamingo/blob/main/LICENSE","text":"https://github.com/mlfoundations/open_flamingo/blob/main/LICENSE","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"model ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"use ","element":"span"},{"text":"from ","element":"span"},{"href":"https://github.com/mlfoundations/open_flamingo","text":"https://github.com/mlfoundations/open_flamingo","element":"a"}],[{"text":"5. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"LL","element":"span"},{"style":{"fontWeight":"bold"},"text":"A","element":"span"},{"style":{"fontWeight":"bold"},"text":"MA-A","element":"span"},{"style":{"fontWeight":"bold"},"text":"DAPTER ","element":"span"},{"text":"is based on the architecture of LLaMA Adapter [","element":"span"},{"href":"#id-114","referenceIndex":125,"text":"125","element":"a"},{"text":"] which augments the text tokens with learnable adaptation prompts. In addition to this, LLaMA Adapter V2 uses early fusion to add visual knowledge to the decoding LLM. The architecture uses both early fusion and late fusion, and while fine-tuning, all the pre-trained components are frozen except for the bias layers of the LLM, Visual Projection Layer and the zero-initialized cross attention module. We evaluate the ","element":"span"},{"style":{"fontFamily":"monospace"},"text":"BIAS-LORA-7B ","element":"span"},{"text":"model which uses LLaMA-7B as the decoder","element":"span"},{"text":"*","element":"span"},{"text":". The model is instruction tuned using 619K samples, and has 14M trainable parameters.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"License: ","element":"span"},{"text":"Work ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"under ","element":"span"},{"text":"GNU ","element":"span"},{"text":"General ","element":"span"},{"text":"public ","element":"span"},{"text":"license ","element":"span"},{"href":"https://github.com/OpenGVLab/LLaMA-Adapter/blob/main/LICENSE","text":"https://github.com/OpenGVLab/LLaMA-Adapter/blob/main/LICENSE","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"Model ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"use ","element":"span"},{"text":"from ","element":"span"},{"href":"https://github.com/OpenGVLab/LLaMA-Adapter","text":"https://github.com/OpenGVLab/LLaMA-Adapter","element":"a"}],[{"text":"6. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"E","element":"span"},{"style":{"fontWeight":"bold"},"text":"MU ","element":"span"},{"text":"is a large multimodal model trained using interleaved video, image and text data, trained in an autoregressive manner to predict the next token in the multimodal sequence. With the ability to produce the next visual token, Emu is also able to generate images and has been evaluated on the Magic Brush dataset in this work. The architecture uses pre-trained encoder and a decoding LLM such as LLaMA. EMU is first pre-trained using interleaved video, image, and text data, and all the parameters are updated during the pre-training. In the second stage, emu is further instruction-tuned. However, in this work we only evaluate the pre-trained version of Emu. We evaluate the Emu-14B model pre-trained using 82M samples.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"License: ","element":"span"},{"text":"Work ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"under ","element":"span"},{"text":"Apache ","element":"span"},{"text":"2.0 ","element":"span"},{"text":"license ","element":"span"},{"href":"https://github.com/baaivision/Emu/blob/main/LICENSE","text":"https://github.com/baaivision/Emu/blob/main/LICENSE","element":"a"}],[{"style":{"width":"90%"},"width":1441,"height":36,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/33-0.png","element":"img"}],[{"text":"7. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"F","element":"span"},{"style":{"fontWeight":"bold"},"text":"UYU","element":"span"},{"style":{"fontWeight":"bold"},"text":"-8B ","element":"span"},{"text":"is a decoder only architecture where the image patches are linearly projected into the first layer of the transformer architecture. Fuyu’s architecture is same as that of Persimmon-8B ","element":"span"},{"text":"*","element":"span"},{"text":", and we use the details of Persimmon-8B to categorise Fuyu into the model categories. Persimmon-8B has 9.3B parameters and is trained from scratch. In our work we evaluate the pre-trained model as the instruction tuned models aren’t available and the pre-training data sources and sizes are unknown. We evaluate the Fuyu-8B model available through HuggingFace ","element":"span"},{"text":"*","element":"span"},{"text":".","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"License: ","element":"span"},{"text":"Work is available under Creative Commons Attribution Non Commercial 4.0 International license ","element":"span"},{"href":"https://spdx.org/licenses/CC-BY-NC-4.0","text":"https://spdx.org/licenses/CC-BY-NC-4.0","element":"a"}],[{"style":{"width":"99%"},"width":1584,"height":61,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/33-1.png","element":"img"}],[{"href":"https://huggingface.co/adept/fuyu-8b","text":"https://huggingface.co/adept/fuyu-8b","element":"a"}],[{"style":{"width":"60%"},"width":1490,"height":295,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/34-0.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"License: ","element":"span"},{"text":"Work ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"under ","element":"span"},{"text":"MIT ","element":"span"},{"text":"License ","element":"span"},{"href":"https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/mit.md","text":"https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/mit.md","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"model ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"to ","element":"span"},{"text":"use ","element":"span"},{"text":"from ","element":"span"},{"href":"https://huggingface.co/microsoft/kosmos-2-patch14-224","text":"https://huggingface.co/microsoft/kosmos-2-patch14-224","element":"a"}],[{"style":{"width":"61%"},"width":1495,"height":383,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/34-1.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"License: ","element":"span"},{"text":"Work is available under MIT License ","element":"span"},{"href":"https://github.com/X-PLUG/mPLUG-Owl/blob/main/LICENSE","text":"https://github.com/X-PLUG/mPLUG- ","element":"a"},{"href":"https://github.com/X-PLUG/mPLUG-Owl/blob/main/LICENSE","text":"Owl/blob/main/LICENSE","element":"a"}],[{"style":{"fontWeight":"bold"},"text":"Access restrictions: ","element":"span"},{"text":"The model is available to use from ","element":"span"},{"href":"https://github.com/X-PLUG/mPLUG-Owl","text":"https://github.com/X- ","element":"a"},{"href":"https://github.com/X-PLUG/mPLUG-Owl","text":"PLUG/mPLUG-Owl","element":"a"}],[{"style":{"width":"61%"},"width":1507,"height":248,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/34-2.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"License: ","element":"span"},{"text":"None","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access ","element":"span"},{"style":{"fontWeight":"bold"},"text":"restrictions: ","element":"span"},{"text":"The ","element":"span"},{"text":"model ","element":"span"},{"text":"is ","element":"span"},{"text":"available ","element":"span"},{"text":"via ","element":"span"},{"text":"OpenAI’s ","element":"span"},{"text":"API ","element":"span"},{"href":"https://platform.openai.com/docs/guides/vision","text":"https://platform.openai.com/docs/guides/vision","element":"a"}],[{"style":{"width":"61%"},"width":1512,"height":339,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/34-3.png","element":"img"}],[{"text":"’HARM_CATEGORY_DANGEROUS_CONTENT’ ","element":"span"},{"text":"and ","element":"span"},{"text":"set ","element":"span"},{"text":"the ","element":"span"},{"text":"threshold ","element":"span"},{"text":"to ’BLOCK_NONE’ provided by the API ","element":"span"},{"text":"*","element":"span"},{"text":".","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"License: ","element":"span"},{"text":"None","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Access: ","element":"span"},{"text":"Available via API ","element":"span"},{"href":"https://ai.google.dev/gemini-api/docs/models/gemini","text":"https://ai.google.dev/gemini-api/docs/models/gemini","element":"a"}],[{"text":"Table 11: Evaluation metrics supported in HEMM ","element":"figcaption","subtype":"caption"},{"id":"id-115","text":"Metric ","element":"figcaption","subtype":"caption"},{"text":"Task ","element":"figcaption","subtype":"caption"},{"text":"Modalities","element":"figcaption","subtype":"caption"}],[{"style":{"width":"47%"},"width":760,"height":330,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/35-0.png","element":"img"}]]},{"heading":"B Experimental Details","paragraphs":[[{"style":{"fontWeight":"bold"},"text":"B.1 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Evaluation metrics","element":"span"}],[{"text":"We present our results on BARTScore [","element":"span"},{"href":"#id-44","referenceIndex":122,"text":"122","element":"a"},{"text":"] as models under our evaluation generate noisy free from text, however, we also support other text generation metrics under our evaluation suite listed in the Table ","element":"span"},{"href":"#id-115","text":"11 ","element":"a"},{"text":"below.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"B.2 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Evaluation protocol","element":"span"}],[{"text":"HEMM supports image generation tasks, models and metrics. However, currently there are only 2 image generation tasks (LNCOCO and M","element":"span"},{"text":"AGIC ","element":"span"},{"text":"B","element":"span"},{"text":"RUSH","element":"span"},{"text":") and 1 model (E","element":"span"},{"text":"MU","element":"span"},{"text":") that supports image generation. Hence, we perform all our evaluation on the remaining 28 text generation tasks and report the results on the image generation tasks in Appendix ","element":"span"},{"text":"C.","element":"span"}],[{"text":"Note: Since HEMM contains models that are unable to process multiple images in the same input, we modify the W","element":"span"},{"text":"INOGROUND ","element":"span"},{"text":"and IRFL tasks (as per ","element":"span"},{"href":"#id-101","text":"A.1) ","element":"a"},{"text":"in order to have a single image-text pair as input for each sample.","element":"span"}],[{"text":"For each dataset, we use the same prompts across all models as shown in Section ","element":"span"},{"href":"#id-104","text":"C, ","element":"a"},{"text":"for standardization, however, there can be a scenario where these models perform better with other prompts or scenarios and may perform poorly under our scenarios or prompts in our evaluation.","element":"span"}],[{"text":"For each dataset, the computed metrics for the models are normalized on a scale of 0 to 1, 0 corresponds to the model achieving the lowest score on that dataset, and 1 corresponds to the performance achieve by exactly generating the ground truth. For BERTScore [","element":"span"},{"href":"#id-116","referenceIndex":126,"text":"126","element":"a"},{"text":"], ROUGE [","element":"span"},{"href":"#id-117","referenceIndex":70,"text":"70","element":"a"},{"text":"], and RefCLIPScore [","element":"span"},{"href":"#id-118","referenceIndex":36,"text":"36","element":"a"},{"text":"] the maximum value is set to 1. BARTScore [","element":"span"},{"href":"#id-44","referenceIndex":122,"text":"122","element":"a"},{"text":"] uses the log of probabilities. Following [","element":"span"},{"href":"#id-45","referenceIndex":16,"text":"16","element":"a"},{"text":"], we calculate the maximum value for each dataset separately as BARTScore(r, r) where r is the ground truth sentence.","element":"span"}],[{"text":"Since details regarding training type for G","element":"span"},{"text":"EMINI ","element":"span"},{"text":"and GPT-4V, and modality processing for GPT-4V are not revealed, we do not use the scores from these models while evaluating the performance for the training type and modality processing dimensions. Further, for H","element":"span"},{"text":"ATEFUL ","element":"span"},{"text":"M","element":"span"},{"text":"EMES","element":"span"},{"text":", O","element":"span"},{"text":"PEN","element":"span"},{"text":"P","element":"span"},{"text":"ATH","element":"span"},{"text":", and M","element":"span"},{"text":"EMOTION ","element":"span"},{"text":"datasets, GPT-4V did not respond and generated ","element":"span"},{"style":{"fontStyle":"italic"},"text":"can’t provide assistance ","element":"span"},{"text":"and ","element":"span"},{"style":{"fontStyle":"italic"},"text":"\"indeterminate\" ","element":"span"},{"text":"for many samples. Hence, we exclude the results of GPT-4V on these datasets during evaluation.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"B.3 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Significance tests","element":"span"}],[{"text":"While comparing performance across categories in each dimension, we perform paired t-tests to determine the significance of the results. For datasets, specifically, for each category, we calculate the average performance of each of the 11 models on all the datasets in a category (","element":"span"},{"style":{"height":9.18},"width":28.28,"height":22.96,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/35-1.png","element":"img","alt":"ci","inline":true},{"text":") to create a vector ","element":"span"},{"style":{"height":16.08},"width":131.28,"height":40.2,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/35-2.png","element":"img","alt":"vi ∈ R11","inline":true},{"text":". Next, we performed pairwise t-tests between these vectors to determine the significance of the results. The p-values obtained through the t-tests are presented in Table ","element":"span"},{"href":"#id-119","text":"13. ","element":"a"},{"text":"We find that the difference between the performance on different categories is statistically significant (p-value < 0.05) for real-world use cases, multimodal interaction, external knowledge, and information flow dimensions, which explains that these are particularly difficult dimensions for today’s multimodal model.","element":"span"}],[{"text":"We also conducted t-tests for various categories in each of the modeling dimensions. For all models in a category (","element":"span"},{"style":{"height":9.18},"width":28.24,"height":22.96,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/35-3.png","element":"img","alt":"ci","inline":true},{"text":"), we use their average performance on each of the 28 datasets to construct a vector ","element":"span"},{"style":{"height":16.1},"width":139.36,"height":40.24,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/35-4.png","element":"img","alt":"wi ∈ R28","inline":true},{"text":". We then perform pair-wise t-tests across all the categories for all dimensions. As mentioned","element":"span"}],[{"text":"Table 12: Hyperparameters used for running inference for various models. Temperature for GPT-4V and Beam ","element":"figcaption","subtype":"caption"},{"text":"Size for GPT-4V and G","element":"figcaption","subtype":"caption"},{"text":"EMINI ","element":"figcaption","subtype":"caption"},{"text":"are unknown. We also report the average inference time in seconds for an image-text input. For each models, we take the average of inference times across all the datasets.","element":"figcaption","subtype":"caption"}],[{"id":"id-58","style":{"width":"64%"},"width":1588,"height":2347,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/36-0.png","element":"img"}],[{"id":"id-119","text":"Table 13: Standard deviation and p-values (from paired t-tests) across categories for each dataset dimension. On ","element":"figcaption","subtype":"caption"},{"text":"average, models achieve significantly higher scores on Multimedia and Affect as compared to other use cases. The p-values for Reasoning and Granularity dimensions are higher than 0.05, indicating that there is no category significantly more challenging than the rest.","element":"figcaption","subtype":"caption"}],[{"style":{"width":"84%"},"width":1347,"height":1408,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/37-0.png","element":"img"}]]},{"heading":"C All Results","paragraphs":[[{"text":"Due to query limits for GPT-4V and G","element":"span"},{"text":"EMINI","element":"span"},{"text":", we evaluated the two models only on 100 samples per dataset, and for a fair comparison, we performed our analysis using the outputs of all the models on those 100 samples. In this section, we present the results and analysis on the whole evaluation set using the outputs of all the models except GPT-4V and G","element":"span"},{"text":"EMINI","element":"span"},{"text":". Further, since our analysis was based on text-generation tasks, we present here the results on the image-generation tasks - M","element":"span"},{"text":"AGIC ","element":"span"},{"text":"B","element":"span"},{"text":"RUSH ","element":"span"},{"text":"and LNCOCO. Specifically, we evaluated E","element":"span"},{"text":"MU ","element":"span"},{"text":"(only model in HEMM that can generate images) on both tasks. We find the MSE and the CLIP-I score between the generated and the ground truth image for M","element":"span"},{"text":"AGIC ","element":"span"},{"text":"B","element":"span"},{"text":"RUSH ","element":"span"},{"text":"to be 0.17 and 0.54. For the LNCOCO dataset, the MSE and CLIP-I score are 0.18 and 0.50.","element":"span"}],[{"text":"Note: due to high inference time of some models (e.g., M","element":"span"},{"text":"INI","element":"span"},{"text":"-GPT-4, E","element":"span"},{"text":"MU","element":"span"},{"text":", O","element":"span"},{"text":"PEN","element":"span"},{"text":"F","element":"span"},{"text":"LAMINGO","element":"span"},{"text":"), missing image URLs in the N","element":"span"},{"text":"LVR","element":"span"},{"text":"2 dataset, and compute restrictions for larger evaluation sets like MM-IMD","element":"span"},{"text":"B","element":"span"},{"text":", V","element":"span"},{"text":"ISUAL ","element":"span"},{"text":"G","element":"span"},{"text":"ENOME","element":"span"},{"text":", and ","element":"span"},{"text":"I","element":"span"},{"text":"N","element":"span"},{"text":"ATURALIST","element":"span"},{"text":", we use the results from the same 100 samples used for evaluation in Section ","element":"span"},{"text":"4.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"C.1 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Dataset and model comparisons Dataset comparisons: ","element":"span"},{"text":"On average, the models achieve the highest scores on IRFL (0.53), W","element":"span"},{"text":"INOGROUND ","element":"span"},{"text":"(0.42), and N","element":"span"},{"text":"LVR ","element":"span"},{"text":"(0.40) datasets. Healthcare, Science, and HCI datasets are the most challenging use cases for the models with the average scores being the lowest for D","element":"span"},{"text":"ECIMER ","element":"span"},{"text":"(0.05),","element":"span"}],[{"text":"Table 14: Standard deviation and p-values for categories in various modeling dimensions. Models in the ","element":"figcaption","subtype":"caption"},{"text":"best-performing category in each dimension, receive significantly higher scores than the other categories.","element":"figcaption","subtype":"caption"}],[{"style":{"width":"64%"},"width":1589,"height":2430,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/38-0.png","element":"img"}],[{"text":"Table 15: Comparisons on different dataset categories. ","element":"figcaption","subtype":"caption"},{"text":"30 multimodal datasets are split into various groups based on their real-world use case, type of multimodal interaction, presence of reasoning and external knowledge, granularity of alignment, and types of information flow. Performance is measured via the mean BARTscore across 9 multimodal models.","element":"figcaption","subtype":"caption"}],[{"text":"Table 16: Comparisons on different modeling decisions. We group models based on the modeling and training decisions, including how they process modalities, their parameter counts, model architecture, training data size and diversity, and the presence of instruction tuning. Performance is measured via the mean BARTscore across all 30 tested multimodal datasets.","element":"figcaption","subtype":"caption"}],[{"style":{"width":"99%"},"width":1584,"height":908,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/39-0.png","element":"img"}],[{"style":{"fontWeight":"bold"},"text":"Multimodal Skills 4: External Knowledge ","element":"span"},{"text":"Average performance of models on tasks requiring external knowledge (0.20) is significantly lower than tasks not requiring knowledge (0.24). For example, on average, models perform better on N","element":"span"},{"text":"LVR","element":"span"},{"text":", FER-2013 and W","element":"span"},{"text":"INOGROUND ","element":"span"},{"text":"that do not require external knowledge as compared to tasks like ","element":"span"},{"text":"I","element":"span"},{"text":"N","element":"span"},{"text":"ATURALIST ","element":"span"},{"text":"and S","element":"span"},{"text":"LAKE ","element":"span"},{"text":"which require external knowledge to identify appropriate species or organs in the image.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Multimodal Skills 5: Information flow ","element":"span"},{"text":"Models achieve significantly lower average score on translation datasets (0.17) as compared to querying (0.20) and fusion (0.29) datasets. Lower scores on translation dataset is due to the presence of highly challenging datasets such as D","element":"span"},{"text":"ECIMER ","element":"span"},{"text":"which requires domain knowledge of molecules to generate the correct textual sequence.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"C.3 ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Modeling trends","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Model scale: ","element":"span"},{"text":"Since we do not consider GPT-4V and G","element":"span"},{"text":"EMINI ","element":"span"},{"text":"for analysis in this section, there are no models in the large category. Amongst small and medium models, we find no significant difference (p-value = 0.45) between the average performance of models from the two categories with small and medium models receiving 0.21 and 0.23 average scores respectively.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Pretraining data scale: ","element":"span"},{"text":"On average, models with medium pretraining data achieve the highest score (0.28) as compared to the models pretrained with small (0.16) or large (0.21) scale data. Although the average score of models trained with large pretraining data is lower as compared to models trained with medium pretraining data, we find that the former models perform better on tasks such as IRFL, W","element":"span"},{"text":"INOGROUND","element":"span"},{"text":", M","element":"span"},{"text":"EME","element":"span"},{"text":"C","element":"span"},{"text":"AP","element":"span"},{"text":", and D","element":"span"},{"text":"ECIMER ","element":"span"},{"text":"which require complex reasoning and external knowledge.","element":"span"}],[{"style":{"fontWeight":"bold"},"text":"Diversity of pretraining data: ","element":"span"},{"text":"Models trained with diverse pretraining data (0.24) perform better than models trained only on image-captioning datasets (0.20). The p-value for the paired t-test is 0.01 indicating that the difference is significant. On average, we find that models pretrained with diverse data achieve better scores on knowledge-intensive tasks such as ","element":"span"},{"text":"I","element":"span"},{"text":"N","element":"span"},{"text":"ATURALIST ","element":"span"},{"text":"and OK-VQA with improvements in average scores up to 0.21.","element":"span"}],[{"id":"id-49","style":{"width":"64%"},"width":1580,"height":701,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/40-0.png","element":"img"}],[{"text":"Figure 8: Model outputs on samples from E","element":"figcaption","subtype":"caption"},{"text":"NRICO","element":"figcaption","subtype":"caption"},{"text":", VQARAD, ","element":"figcaption","subtype":"caption"},{"text":"I","element":"figcaption","subtype":"caption"},{"text":"N","element":"figcaption","subtype":"caption"},{"text":"ATURALIST","element":"figcaption","subtype":"caption"},{"text":", and S","element":"figcaption","subtype":"caption"},{"text":"CIENCE","element":"figcaption","subtype":"caption"},{"text":"QA. In (a), all the models struggle to reason about the use of the ","element":"figcaption","subtype":"caption"},{"style":{"fontStyle":"italic"},"text":"zip code ","element":"figcaption","subtype":"caption"},{"text":"field in the UI, which will be used to ","element":"figcaption","subtype":"caption"},{"style":{"fontStyle":"italic"},"text":"search ","element":"figcaption","subtype":"caption"},{"text":"the TV provider. Example (b) underscores the complexity faced by models in interpreting medical images, particularly evident in their inability to recognize the absence of a kidney in the radiology image. As shown in (c), the highly fine-grained ","element":"figcaption","subtype":"caption"},{"text":"I","element":"figcaption","subtype":"caption"},{"text":"N","element":"figcaption","subtype":"caption"},{"text":"ATURALIST ","element":"figcaption","subtype":"caption"},{"text":"dataset is very challenging and none of the models can determine the species of the insect. In (d), all models provide incorrect responses when tasked with identifying the colony’s name, illustrating the challenges posed by tasks requiring external knowledge.","element":"figcaption","subtype":"caption"}],[{"style":{"width":"64%"},"width":1586,"height":782,"src":"https://cdn.bytez.com/mobilePapers/v2/neurips/97825/images/40-1.png","element":"img"}],[{"text":"1. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Challenging datasets","element":"span"},{"text":": Health, HCI, and Science are all relatively difficult use cases for today’s multimodal foundation models, which are statistically significantly harder than Multimedia and Affective Computing use cases. In particular, images of scientific diagrams, satellite images, medical images, memes, and rich social interactions pose challenges. It is therefore important to evaluate multimodal models on a diverse range of input modalities and output tasks to get a better measure of generalization performance.","element":"span"}],[{"text":"2. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Multimodal interactions","element":"span"},{"text":": Models perform better on redundant interactions but struggle when visual information is not directly referenced by text (i.e., uniqueness or synergy). Future benchmarks should contain richer multimodal interactions beyond redundancy, such as in analyzing sarcasm, humor, memes, science, environment, and education. These can serve as better test beds for multimodal models and enable their applications towards real-world multimodal interactions.","element":"span"}],[{"text":"3. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Reasoning, fine-grained, and knowledge","element":"span"},{"text":": We need better datasets that test for complex reasoning and fine-grained alignment - current ones do not pose enough challenges to today’s models, with no significant performance differences with or without reasoning and fine-grained alignment.","element":"span"}],[{"text":"We do find that tasks requiring external knowledge are significantly harder than no knowledge; bridging this gap can be a promising direction for multimodal research.","element":"span"}],[{"text":"4. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Model and data size","element":"span"},{"text":": Perhaps unsurprisingly, larger scales of data and models improve the average score across the board, with significant improvements of up to 75% as compared to medium-sized models. Training on diverse data sources also improves over models that only pretrain on images and captions. The tasks that show the most improvement are ","element":"span"},{"text":"I","element":"span"},{"text":"N","element":"span"},{"text":"ATURALIST ","element":"span"},{"text":"and M","element":"span"},{"text":"EME","element":"span"},{"text":"C","element":"span"},{"text":"AP ","element":"span"},{"text":"which are knowledge-intensive and require complex reasoning.","element":"span"}],[{"text":"5. ","element":"span"},{"style":{"fontWeight":"bold"},"text":"Model training","element":"span"},{"text":": Instruction-tuned models performed better than those with only supervised fine-tuning. Cross-modal translation (image-to-text) tasks show the most improvements (e.g., D","element":"span"},{"text":"ECIMER","element":"span"},{"text":", M","element":"span"},{"text":"EME","element":"span"},{"text":"C","element":"span"},{"text":"AP","element":"span"},{"text":", and S","element":"span"},{"text":"CREEN","element":"span"},{"text":"2W","element":"span"},{"text":"ORDS","element":"span"},{"text":"). However, some instruction-tuned models still struggle to follow the instructions (e.g., generating a caption when asked to classify an image, or generating long responses when asked to answer in a few words). Instruction tuning using larger datasets with diverse instructions can help alleviate this problem.","element":"span"}]]}],"_version":"3.3.4"},"paperNode":"$21:props:children:props:children:0:props:product"}]] </article></body></html>