A multimodal artificial intelligence (multimodal AI) refers to AI systems capable of simultaneously processing, interpreting and integrating several types of data (or terms and conditions), such as text, images, audio, video, or sensory data, to generate more complete and nuanced responses or decisions.
Unlike traditional (unimodal) AI models, which specialise in a single type of data (text/images/video/audio), multimodal AI mimics human cognition by combining heterogeneous sources for enriched contextual understanding.
Key features
- Integration of heterogeneous data
It merges modalities with different structures (sequential text, spatial images, temporal audio) using techniques such as the early, intermediate or late fusion to create a unified representation.
Example analyse a video by aligning the audio and visual tracks to detect emotions. - Advanced contextual understanding
By combining complementary data (e.g. an image and its text description), it reduces ambiguity and improves accuracy. For example, a model can generate an image caption or identify a bird using its song and a photo. - Robustness and resilience
If a modality is missing or noisy (e.g. poor quality audio), the system relies on other sources (e.g. visual or textual) to maintain its performance.
Practical applications
- Health Diagnostics combining MRI, patient records and genomic data.
- Autonomous vehicles fusion of LiDAR, camera and GPS data for safe navigation.
- Virtual assistants Interaction via voice, text and images (e.g. ChatGPT-o with ChatGPT Vision).
- Media video subtitle generation or image creation from text prompts (DALL-E).
Underlying technologies
- Natural Language Processing (NLP) to interpret text and speech.
- Computer vision image and video analysis using convolutional neural networks (CNN).
- Audio recognition Speech or sound event detection.
- Merging models : Architectures like transformers (e.g. GPT-4) combining modalities in a single model.
Challenges and limits
- Data alignment synchronise modalities temporally (e.g. audio and video) or spatially.
- Complexity of integration Represent heterogeneous data in a common space without loss of information.
- Calculation requirements Processing large volumes of multi-source data requires significant hardware resources.
Concrete examples
- GPT-4o (the "o" stands for "omni"): capable of generating text, interpreting images and processing audio.
- Tesla cars The new technologies: using data from cameras, radar and sensors for autonomous driving.
- IBM Watson multimodal analysis in oncology, combining medical images and text reports
