Benchmarking Top Vision Vision Language Models (VLMS) for picture classification

presentation

In the rapid evolution of artificial intelligence, the ability to accurately interpret and analyze visual data is becoming increasingly important. From autonomous vehicles to medical images, image classification applications are large and influential. However, as the complexity of tasks increases, as well as the need for models that can smoothly integrate numerous modalities, such as vision and language, to achieve a stronger and nuanced meaning.

This is where vision language models (VLM) come into play, offering a powerful approach to learning multimodal by combining image and text inputs to generate significant results. But with so many models available, how can we determine which one performs best for a particular task? This is the problem we aim to address on this blog.

The main purpose of this blog is to define the high vision language patterns in an image classification task using a basic database and compare its performance with our model of general image recognition. Moreover, we will demonstrate how to use the model-standard tool to evaluate these models, providing knowledge of their strengths and weaknesses. In doing so, we hope to shed light on the current state of VLM and guidance practitioners in choosing the most suitable model for their specific needs.

What are the Language Language Models (VLM)

A vision language model (VLM) is a type of multimodal generator that can process both image and text inputs to generate text results. These models are very versatile and can be applied to a variety of tasks, including but not limited to:

• Answer the Question of the Visual Document (QA): answering questions based on visual documents.
• Chapter of Figure: Generation of Descriptive Text for Images.
• Image classification: Identifying and categorizing objects within images.
• Discover: condition of objects within an image.
  

The architecture of a typical VLM consists of two main ingredients:

1. Image feature extractor: This is usually a pre-trained vision model as vision transformer (year) or clip, which derives traits from the input image.
2. Text Decoder: This is usually a large linguistic model (LLM) such as Llama or QWEN, which generates text based on the features of extracted

These two ingredients are joined together using a layer of mode melting before they get into the language decipher, which produces the end result of the text.

VLM general architecture, image obtained from HF blog.

There are many vision language models available on the Clarifai platform, including GPT-4o, Claude 3.5 Sonnet, Florence-2, Gemini, QWEN2-VL-7B, Llava, and MinicPM-V. Try it here!

Current state of VLMS

The last rankings indicate that QWEN-VL-MAX-0809 has exceeded further GPT-4o In terms of average standards results. This is significant because the GPT-4o was previously considered the highest multimodal model. Increasing large -sourced -sourced models like QWEN2-VL-7B It suggests that open source models have begun to exceed their closed source counterparts, including GPT-4o. In particular, qwen2-vl-7b, despite its smallest size, achieves results that are close to those of trading models.

Setting the experiment

Equipment

Experiments were performed on the Lambda Labs device with the following specifications:

Patterns of interest

We focused on smaller models (less than 20b parameters) and included GPT-4o as a reference. Estimated models include:

results

Dataset: Caltech256

Caltech256 data data consists of 30,607 images in 256 classes, plus a clutter in the background. Each class contains between 80 and 827 images, with image size ranging from 80 to 800 pixels. A subset of 21 classes (including background) was randomly selected for evaluation.