How AI Learns to See and Read Together with Multimodal LLMs?

[Kübra Taşcı Kardaş]

In the next wave of AI innovation, models are no longer limited to words. They can now see, read, listen, and understand the world across multiple forms of data. In this article, we will discuss the multimodal models that can also see.

1. What Are Multimodal LLMs?​

Traditional Large Language Models (LLMs) like the old GPT you used or if you are technical Llama or Mistral models, process only one type of data: text. These models excel at reasoning, summarizing, and generating natural language but cannot directly interpret what’s inside an image, video, or audio clip.

Multimodal LLMs, on the other hand, can handle multiple modalities of information, where a modality refers to a specific type of input signal, such as text, images, sound, or video.

Let’s focus on the most common and practical combination today: text + images.

A multimodal model can both see (the image pixels) and read (the text in the image), a capability that enables entirely new categories of AI systems, from document understanding to visual question answering, and even medical image interpretation.

Figure 1: You can send an image and talk about it [1]

2. Real-World Use Cases​

Multimodal models are emerging in almost every domain where humans process both visual and textual cues. Let’s explore some representative examples.

• Image Captioning​

The most intuitive task is to give the model a picture, and it describes what it sees. You can send a picture of a cat on a laptop, and the response would be like describing it “A curious cat sitting on a laptop keyboard.”

• Visual Question Answering (VQA)​

You can ask questions about the image you sent: “How many people are in this photo?” or “What brand is the car?”. The model looks, reasons, and answers in natural language.

• Chart, Diagram, and Table Parsing​

Models can now extract numerical data from visual elements: reading charts, identifying anomalies, and even converting tables into LaTeX or Markdown automatically.

• Cross-Modal Reasoning​

You can ask questions that blend text and visuals: “Does the chart in this slide support the argument in the summary below?”. Such reasoning across modalities represents a step toward true multimodal cognition.

3. How Do Multimodal LLMs Work?​

Under the hood, these models combine language modeling and computer vision in a unified neural architecture. There are two dominant design families.

Figure 2: Two Methods of Multimodal Language Models [2]

Method A: Unified Embedding + Decoder Architecture​

This approach feeds both text and image inputs into a single decoder model, usually adapted from a pre-trained LLM such as GPT-2, Gemma, Llama3.2, or Phi-3.

• Step 1: Image Encoding​

An image encoder, typically a Vision Transformer (ViT) or a CLIP encoder, splits an image into small patches (like dividing text into tokens). Each patch is converted into an embedding vector representing visual features.

Figure 3: A classic setup of Vision Transformer (ViT) [3]

Figure 4 Linear projection layer that projects flattened image patches from a 256-dimensional into a 768-dimensional embedding space [2]

• Step 2: Linear Projection​

Since the image embeddings and text embeddings may have different vector dimensions, a linear projection layer (a single fully connected layer) is used to align them. For instance, a 256-dimensional image vector might be projected to match a 768-dimensional text embedding.

Bu bağlantıyı görüntüleyebilmek için kayıt olmalı zaten üyeyseniz üye girişi yapmalısınız.