MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models

  Code: https://github.com/TIGER-AI-Lab/VLM2Vec

  Leaderboard: MMEB-V3

1 Introduction

Multimodal embeddings (Zhang et al., 2025a; Meng et al., 2025) are foundational to modern machine learning, mapping heterogeneous inputs—such as text, images, videos, and audio—into a unified, fixed-dimensional vector space. These representations power a vast ecosystem of applications, from semantic retrieval and recommendation systems to complex decision-making pipelines. As the field evolves, research has shifted from isolated unimodal encoders (Radford et al., 2021) toward unified multimodal embedding models that align diverse modalities. Such unified spaces are increasingly critical for emerging paradigms like multimodal Retrieval-Augmented Generation (RAG) and intelligent AI agents, which must retrieve and act upon information seamlessly across different sensory inputs.

Despite this progress, we identify a fundamental but underexplored challenge: modality as an explicit instruction constraint. Existing frameworks implicitly assume that modality is either perfectly aligned with semantics or acts as a passive attribute. In practice, however, modality often serves as an explicit requirement specified by a user or system. For example, a user may issue queries such as ”find an audio clip of a cat meowing” or ”retrieve a video showing a cat jumping”, where modality is a mandatory constraint rather than an optional attribute. This reveals a key limitation: current embedding models often fail to interpret modality as an explicit instruction constraint, instead treating it as a byproduct of semantic similarity, leading to semantically relevant but modality-mismatched retrieval results. This issue is particularly acute in agent-centric settings, where retrieval is a prerequisite for downstream actions such as tool invocation, GUI interaction, or memory access. Errors in modality understanding—such as retrieving semantically relevant but modality-mismatched results—can lead to catastrophic failures in the decision-making loop. However, current benchmarks such as MMEB (Meng et al., 2025) and UMR (Zhang et al., 2025a) provide limited support for evaluating this problem. They focus primarily on cross-modal alignment (e.g., text-to-image) and lack systematic evaluation of (1) full-modality coverage (especially audio and visual documents) and (2) instruction-following behavior in complex retrieval scenarios.

To systematically measure and diagnose limitations in modality-aware, instruction-conditioned embedding behavior, we introduce MMEB-V3, a comprehensive full-modality benchmark designed to evaluate embeddings under realistic, instruction-conditioned settings. MMEB-V3 extends prior work along four key dimensions: (1) Audio Tasks, expanding coverage to include audio classification, temporal grounding and cross-modal retrieval; (2) Text Tasks, incorporating instruction-following, reasoning, and multi-condition constraints; (3) Agent Tasks, evaluating embeddings in agent-centric scenarios such as tool retrieval, memory retrieval, and GUI control; and (4) OmniSET (Omni-modality Semantic Equivalence Tuples), which organize semantically equivalent content across modalities into unified tuples, enabling controlled analysis that disentangles semantic content from modality effects. Through experiments on MMEB-V3, we conduct a systematic analysis of embedding behavior under instruction-conditioned settings. Our results show that while existing models perform well on standard semantic alignment, they struggle to enforce modality constraints specified by instructions. In many cases, retrieval is dominated by the original query modality rather than the instructed target modality, leading to systematic modality mismatches. These findings highlight a key limitation in current multimodal embeddings: modality constraints are not reliably enforced as part of the instruction.

Our main contributions are as follows:

We introduce MMEB-V3, a comprehensive benchmark for evaluating full-modality embeddings under instruction-conditioned settings across diverse tasks.

We provide a systematic diagnostic analysis of modality-aware behavior, enabled by OmniSET (Omni-modality Semantic Equivalence Tuples), a controlled design that disentangles semantic content from modality effects.

We conduct a comprehensive analysis of modality-constrained retrieval, showing that (i) models often fail to retrieve the intended target modality, (ii) retrieval is asymmetric and biased toward the query modality, and (iii) instruction-induced shifts are insufficient or misaligned with the target modality.

2 Related Work

Multimodal Embedding Benchmarks.

The evolution of multimodal embedding benchmarks reflects a transition from coarse-grained cross-modal alignment to fine-grained, task-diverse evaluation. Early efforts primarily focused on image–text alignment, with benchmarks such as MSCOCO (Lin et al., 2015), Flickr30K (Plummer et al., 2015) and UMR (Zhang et al., 2025a) establishing standard protocols for retrieval and matching. While foundational, these benchmarks are largely constrained to static visual inputs and brief textual descriptions. Subsequent research has expanded this scope to specialized domains: ViDoRe-v2 (Macé et al., 2025) introduces document-level visual retrieval, while QVHighlights (Lei et al., 2021) and FineVideo (Farré et al., 2024) address temporal dynamics in video retrieval. Despite these advancements, a significant gap persists in evaluating instruction-driven, full-modality generalizability. While MTEB (Muennighoff et al., 2023) provides a comprehensive suite for unimodal text embeddings, it lacks multimodal support; conversely, MMEB-V2 (Meng et al., 2025) and M-BEIR (Wei et al., 2023) integrate images, videos, and documents but omit other critical modalities such as audio. Furthermore, emerging agentic tasks—including tool retrieval (Lu et al., 2026a), GUI control (Zhang et al., 2025b) and memory retrieval (Zhao et al., 2026)—increasingly rely on embedding-based solutions, yet they remain excluded from existing comprehensive benchmarks. Existing multimodal embedding benchmarks mainly focus on semantic alignment across modalities (e.g., text–image retrieval), while more complex scenarios—such as instruction-conditioned retrieval, multi-condition reasoning, and agent tasks (e.g., tool, memory, and GUI retrieval)—remain underexplored.

Multimodal Embedding Models.

Multimodal embedding models aim to learn shared representations for heterogeneous modalities (e.g., text, images, audio, and video) within a unified framework. Early approaches typically follow a dual-encoder paradigm, aligning modality-specific encoders in a common space, as exemplified by CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021), and their extensions to video and audio. Building on this foundation, recent work explores partial modality unification for embedding-based retrieval. Models such as GME (Zhang et al., 2025a), MM-Embed (Lin et al., 2025), VLM2Vec (Jiang et al., 2025), and VLM2Vec-V2 (Meng et al., 2025) unify subsets of modalities and achieve strong performance on their target benchmarks, often leveraging MLLM backbones or contrastive objectives. More recent efforts move toward broader modality coverage. Omni-Embed-Nemotron (Xu et al., 2025) supports retrieval across text, image, audio, and video, while OmniRet (Huynh et al., 2026) improves efficiency via multimodal token resampling and pooling strategies. WAVE (Tang et al., 2025) focuses on unified audio–visual embeddings, enabling cross-modal retrieval between audio and video. To complement these developments, we introduce MMEB-V3, which extends prior benchmarks with audio modalities, complex text retrieval, and agent tasks, enabling more comprehensive evaluation under realistic, instruction-conditioned settings.

3 MMEB-V3: A Unified Evaluation Framework for Omni-Modality Embeddings

3.1 Modality Coverage and Task Diversity

MMEB-V3 extends MMEB-V2 by adding 111 new tasks (see Table 1). It provides an evaluation framework covering four modalities—text, image, video, and audio—encompassing a total of 190 tasks. Beyond expanding modality coverage, MMEB-V3 organizes tasks across diverse cross-modal directions (e.g., A2I, T2A, A2V), allowing evaluation under a range of modality interactions. On top of this modality space, it includes multiple task types, such as audio classification, cross-modal retrieval, multi-condition text retrieval, and agent tasks. This design reflects both modality diversity and task diversity, with tasks evaluated under specific modality configurations rather than in isolation.

In addition to standard evaluation, MMEB-V3 includes a diagnostic component, OmniSET (Omni-modality Semantic Equivalence Tuples), which groups semantically equivalent instances across modalities to facilitate controlled comparisons. OmniSET is not used for leaderboard evaluation, but provides a setting to analyze the relationship between semantic content and modality effects. Figure 1 provides an overview of MMEB-V3. Figure 2(a) further illustrates the distribution of tasks across modalities and task categories.

Audio Tasks.

MMEB-V3 incorporates diverse audio tasks, including classification (e.g., ESC-50 (Piczak, 2015), UrbanSound8K (Salamon et al., 2014), NSynth (Engel et al., 2017)), cross-modal retrieval (e.g., Clotho (Drossos et al., 2020), SoundDescs (Koepke et al., 2023), AVE (Tian et al., 2018), SpeechCOCO (Havard et al., 2017)), and temporal grounding (TUT Sound Events (Mesaros et al., 2016)). For TUT Sound Events, we construct two tasks with different difficulty levels and use their average score as the final temporal grounding metric. The audio tasks evaluate both intra-modal understanding and alignment between audio and other modalities, providing a basis to assess acoustic-semantic representations.

Text Tasks.

Beyond standard semantic retrieval, MMEB-V3 includes text scenarios that involve instruction following (FollowIR (Weller et al., 2025), InfoSearch (Zhou et al., 2025)), reasoning (BRIGHT (SU et al., 2025), R2MD (Li et al., 2025)), long-context understanding (LongEmb (Zhu et al., 2024)), and multi-condition matching (MultiConIR (Lu et al., 2025)). A compact general retrieval benchmark (nanoBEIR (Thakur et al., 2021)) is also included to provide coverage of general retrieval settings. These datasets are composed of multiple tasks, resulting in a total of 53 tasks. The final Text score is computed as the average of NDCG@5 across these tasks.

Agent Tasks.

MMEB-V3 includes agent tasks, such as tool retrieval (Tool-REX (Lu et al., 2026a)), GUI trajectory retrieval (GAE-Bench (Zhang et al., 2025b)), and memory retrieval (KnowMeBench (Wu et al., 2026), REALTALK (Lee et al., 2025), PeerQA (Baumgärtner et al., 2025), and DeepPlanning (Zhang et al., 2026)). These tasks involve selecting tools or actions, or retrieving relevant information from past memory based on multimodal inputs and structured instructions. These datasets are composed of multiple tasks, resulting in a total of 47 tasks. The final Agent score is computed as the average of Hit@1 across these tasks.

Cross-Modal Equivalence Modeling.

We include a diagnostic component, OmniSET (Omni-modality Semantic Equivalence Tuples), which groups semantically equivalent instances across modalities into tuples $\{x^{T},x^{I},x^{V},x^{A}\}$. OmniSET is not used for leaderboard evaluation, but provides a setting for controlled comparisons between semantic content and modality effects. It enables analysis of how models handle modality as an explicit constraint under instruction-conditioned retrieval.

3.2 Instruction Diversity across Data Types and Tasks

We describe instruction diversity in MMEB-V3 from two perspectives: target-type constraints and task-type diversity. First, Data-type constraints refer to the expected type of returned data, including modalities such as text, image, audio, and video, as well as structured data such as tools and GUI elements. This dimension captures whether and how instructions constrain the modality or representation of the data. Second, Task-type diversity corresponds to the range of operations associated with different tasks. In MMEB-V3, these mainly include classification, retrieval, grounding, and QA, covering behaviors ranging from label prediction and candidate selection to localization and question answering.

Together, these perspectives offer a way to describe instruction diversity in MMEB-V3. This formulation shows that tasks vary in both what type of output is required and what operation is involved. Figure 2(b) visualizes the distribution of these patterns across MMEB-V3.

4 Experiments

4.1 Experimental Settings

Metrics.

MMEB-V3 uses task-appropriate metrics. We adopt Hit@1 for audio, image, video, and agent tasks, where the number of relevant targets is typically small and the evaluation focuses on identifying a highly relevant match. We use NDCG@5 for text and VisDoc tasks, which involve multiple relevant candidates and require fine-grained ranking. This design aligns evaluation metrics with task characteristics.

Baselines.

We evaluate against two groups of baselines, including omni-modal embedding models (Omni-Embed-Nemotron (Xu et al., 2025) and WAVE (Tang et al., 2025)) and vision-language embedding models (Qwen3-VL-Embedding (Li et al., 2026), VLM2Vec-V2.0 (Meng et al., 2025), VLM2Vec (Jiang et al., 2024), E5-Omni (Chen et al., 2026), LCO-Embedding-Omni (Xiao et al., 2026) and GME (Zhang et al., 2025a)).

4.2 Main Results

Table 3 presents a comparison of representative multimodal embedding models on MMEB-V3, covering text, image, video, audio, visual document, and agent tasks. Full results across all tasks are provided in Appendix A.3. A general pattern can be observed: no single model achieves optimal performance across all task categories. Models that perform well on certain modalities or tasks often exhibit weaker performance on others, suggesting trade-offs in current embedding approaches.

For example, models such as Qwen3-VL-Embedding achieve strong performance on text, image, and video tasks, but lack native audio capability, which may limit their applicability in settings involving audio. Fully multimodal models (e.g., E5-Omni) show relatively stable performance across modalities, yet do not consistently outperform specialized models within specific task categories. To further understand these trade-offs, Table 2 provides a fine-grained breakdown of newly introduced audio, text, and agent tasks. We observe that several task types are relatively challenging across models, including audio retrieval, as well as complex text and agent tasks involving reasoning, multi-condition constraints, and long-context understanding. This may indicate limitations in capturing fine-grained semantic structure and implicit constraints.

Overall, these findings suggest that the observed trade-offs are associated with both modality differences and task complexity, as well as instruction requirements, pointing to the importance of developing more unified and instruction-aware embedding models.

5 Analysis

We analyze three representative embedding models on the OmniSET dataset: Omni-Embed-Nemotron-3B, WAVE, and Qwen3-VL-Embedding-8B. OmniSET provides semantically aligned instances across modalities, enabling controlled analysis of modality effects and instruction-conditioned retrieval.

5.1 Explicit Modality Instructions Often Fail in Cross-Modal Retrieval

Key finding: Explicit modality instructions do not reliably lead to correct target-modality retrieval, and cross-modal behavior exhibits both asymmetry and modality bias. We first examine retrieval performance under explicit modality constraints. As shown in Table 4, most cross-modal directions remain challenging across all three models. In particular, Hit@1 is close to zero in many cases (e.g., T$\rightarrow$I, T$\rightarrow$A, V$\rightarrow$T), and even MRR remains low, indicating that the instructed target modality is often not retrieved. This suggests that modality-specific instructions alone are insufficient to consistently guide retrieval toward the desired modality. A small number of directions achieve relatively high performance, such as I$\rightarrow$V and A$\rightarrow$T. However, these cases should be interpreted with caution. In OmniSET, videos are generated from images and audio from text, which introduces stronger intrinsic similarity for certain modality pairs. As a result, performance in these directions may partially reflect dataset construction effects rather than purely instruction-following capability. We provide a detailed discussion of this potential bias in Appendix A.2.2. Beyond overall performance, two consistent patterns emerge.

(1) Cross-Modal Asymmetry. Cross-modal retrieval is highly directional. Strong performance in one direction does not imply comparable performance in the reverse. For example, I$\rightarrow$V achieves near-perfect scores (Hit@1 = 100.0 for Nemotron), while V$\rightarrow$I remains low (Hit@1 = 5.0). Similar asymmetry appears in other modality pairs (e.g., A$\rightarrow$T vs. T$\rightarrow$A), indicating that cross-modal relations are not bidirectionally aligned in the embedding space. (2) Modality bias. Retrieval results are dominated by modality proximity rather than target constraints. The Top-10 dominant modality (DM) is strongly correlated with the query modality across all models. For instance, Omni-Embed-Nemotron retrieves predominantly text for text queries (e.g., T2I: 84.9%), while WAVE consistently retrieves video regardless of the target modality (99.9% across all directions). Qwen3-VL shows a similar pattern, where retrieved modalities align more with the query modality than with the instructed target.

5.2 Model Sensitivity to Modality-Constrained Instructions

Key finding: Models vary substantially in their sensitivity to modality-constrained instructions, ranging from strong responsiveness to near invariance.

To better understand the retrieval failures, we examine how strongly each model responds to modality-constrained instructions at the representation level. We quantify this effect by measuring the cosine distance between the raw query embedding and its instruction-augmented counterpart, which captures the magnitude of the instruction-induced shift.

Figure 3 shows clear differences across models. Omni-Embed-Nemotron is highly sensitive to modality-constrained instructions, with an average shift of about 0.4 cosine distance; for most cross-modal directions, the shift exceeds 0.3. In contrast, WAVE is much less responsive, with an average change of only 0.08, indicating that modality-constrained instructions have limited effect on its query representations. Qwen3-VL-Embedding-8B falls between these extremes: across text–image–video tasks, its average shift is about 0.15, suggesting moderate responsiveness. These differences may be related to training objectives. Although Nemotron and WAVE both build on the Qwen2.5-Omni family, their responses to modality-constrained instructions differ substantially. While the instruction-related training details of Nemotron are not fully specified, its strong sensitivity and strong full-modality performance suggest that it likely benefited from more extensive instruction-related multimodal training. By contrast, WAVE focuses primarily on unified representation learning for visual and audio tasks, with training centered on multimodal retrieval and video QA rather than instruction-following. This difference in training emphasis may help explain its weaker response to modality-constrained instructions. Overall, sensitivity to modality instructions is clearly not uniform across models. Some models barely react to such constraints, whereas others exhibit substantial embedding shifts. However, as we show next, stronger sensitivity does not necessarily imply better alignment with the target modality.

5.3 Instruction-Induced Shifts Do Not Consistently Move Queries Toward the Target

Key finding: Instruction-induced shifts do not consistently reduce the distance to the target modality, and often move queries farther away.

We next analyze the direction of instruction-induced shifts, i.e., whether instruction augmentation moves queries closer to the intended target modality. We compare the distance from the raw query to the target with that from the instruction-augmented query to the same target. We focus on Omni-Embed-Nemotron, which exhibits the strongest sensitivity to modality-constrained instructions; visualizations for other models are provided in Appendix A.4. As shown in Figure 4(a), instruction-induced shifts do not consistently improve alignment. For Nemotron, only a few directions (e.g., T$\rightarrow$V, T$\rightarrow$A, A$\rightarrow$T, I$\rightarrow$A) show slight improvements, all below 0.09, while most directions instead increase the distance to the target modality. Moreover, this behavior is asymmetric across directions. For instance, T$\rightarrow$V exhibits a small improvement (+0.041), whereas the reverse direction V$\rightarrow$T shows a much larger degradation ($-0.158$). This asymmetry suggests that instruction-induced shifts do not establish a consistent bidirectional alignment between modality pairs, but instead interact unevenly with the underlying embedding geometry. The t-SNE visualizations in Figures 4(b) and 4(c) provide qualitative evidence. Although instruction augmentation induces noticeable movement in the embedding space, these shifts are not consistently oriented toward the target modality and can deviate toward other modality clusters. For example, image queries augmented with a text-target instruction are often observed to move closer to the video cluster rather than the text cluster. A similar trend is observed for Qwen3-VL-Embedding-8B (see Appendix A.4), where instruction-induced shifts yield only limited improvements in a small number of directions, while most changes do not reduce the query–target distance. Overall, instruction-induced shifts do not reliably translate into target-oriented movement. Even when models are sensitive to modality instructions, the induced changes are not consistently aligned with the target modality, limiting their effectiveness for cross-modal retrieval.

6 Conclusion

In this work, we advance the evaluation of multimodal embeddings toward the full-modality setting. We introduce MMEB-V3, a comprehensive benchmark that systematically evaluates embeddings across text, image, video, audio, and agent-centric scenarios. To enable controlled analysis, we further construct Omni-modality Semantic Equivalence Tuples (OmniSET), which isolate modality effects under consistent semantics and enable rigorous evaluation of cross-modal behavior under instruction constraints. Based on this framework, we conduct a systematic analysis of modality-constrained retrieval and identify several key limitations. We find that models often fail to retrieve the intended target modality, cross-modal retrieval is highly asymmetric and exhibits strong directional biases, and instruction-conditioned signals do not reliably guide retrieval toward the desired modality. These findings suggest that current multimodal embeddings remain limited in supporting reliable modality-aware retrieval under instruction constraints. We hope MMEB-V3 provides a useful benchmark for diagnosing these limitations and for guiding future research toward more controllable full-modality embedding models, particularly in emerging agent-centric applications.

Ethics Statement

We develop and evaluate general-purpose multimodal retrieval and embedding systems using publicly available datasets. No private or personally identifiable data is used. While our benchmarks may inherit biases from underlying corpora, we do not explicitly address fairness, and such biases may affect downstream applications. The proposed methods are intended for beneficial uses such as information access, but may have dual-use risks if misapplied. We encourage responsible use, careful dataset curation, and transparency through the release of code and evaluation protocols.

Appendix A Appendix

A.1 Details of Baseline Models

Omni-Embed-Nemotron (Xu et al., 2025) is a unified multimodal embedding model designed for retrieval over text, images, audio, and video. Built on the Thinker component of Qwen2.5-Omni-3B, it encodes different modalities into a shared embedding space and is trained with a bi-encoder contrastive objective for multimodal retrieval and RAG scenarios.

WAVE (Tang et al., 2025) is a unified audio-visual embedding model for cross-modal retrieval and multimodal understanding. It learns shared audio-visual representations through joint multimodal training and hierarchical feature fusion. Combined with unified embedding learning, it achieves strong performance on audio-visual retrieval and multimodal question answering tasks.

Qwen3-VL-Embedding (Li et al., 2026) is a multimodal embedding model built on the Qwen3-VL foundation model for cross-modal retrieval and understanding. It supports text, images, screenshots, videos, and mixed-modal inputs within a unified framework, learning unified representations across modalities and languages. Using dense embedding representations, it enables high-quality multimodal retrieval and clustering.

VLM2Vec-V2.0 (Meng et al., 2025) is a multimodal embedding model that extends unified representation learning to diverse visual modalities. It supports images, videos, and visual documents within a shared embedding space, and employs unified training across modalities to learn consistent representations. Using contrastive embedding learning, it enables effective retrieval in diverse multimodal settings.

VLM2Vec (Jiang et al., 2024) converts an instruction-tuned vision-language model into a unified multimodal embedding model for retrieval. It reformulates multimodal inputs into embedding-oriented representations under a unified framework, enabling consistent encoding across modalities. Using contrastive embedding learning, it achieves strong performance across diverse multimodal retrieval tasks.

GME (Zhang et al., 2025a) is a unified multimodal embedding model based on Qwen2-VL. It supports three input types—text, image, and image-text pairs—and maps both single-modal and combined-modal inputs into universal vector representations, enabling versatile Any2Any retrieval scenarios such as text-to-image, image-to-image, and multimodal search.

A.2 Details of Benchmark Construction

MMEB-V3 represents a significant advancement over its predecessor, MMEB-V2, by establishing a comprehensive unified embedding evaluation framework that encompasses text, image, video, and audio modalities. While previous iterations focused predominantly on static images and eventually expanded to videos and visual documents, MMEB-V3 fills critical gaps in the embedding landscape by integrating comprehensive audio support, complex text reasoning, and specialized agentic capabilities. In addition, we further enrich the image domain by introducing a multi-condition, multimodal retrieval dataset, MCMR (Lu et al., 2026c), which evaluates fine-grained cross-modal matching under multiple constraints. By consolidating 190 heterogeneous tasks into a standardized ranking-based evaluation, MMEB-V3 provides a rigorous testbed for developing general-purpose, omni-modality embeddings capable of instruction-following and cross-modal semantic alignment.

Audio Tasks.

To address the historical scarcity of audio-focused evaluation in unified models, MMEB-V3 introduces a suite of audio tasks covering classification, cross-modal retrieval, and temporal grounding. For several large-scale audio datasets, including NSynth, SpeechCommands, SoundDescs, and SpeechCOCO, we subsample up to 1,000 queries and limit the candidate pool to at most 10,000 instances. This strategy ensures computational tractability while maintaining a representative evaluation of model performance, consistent with common practices in large-scale retrieval benchmarks:

Audio Classification. Datasets include ESC-50 (Piczak, 2015), UrbanSound8K (Salamon et al., 2014), NSynth (Engel et al., 2017), Speech Commands (Warden, 2018), and CREMA-D (Cao et al., 2014), evaluating the ability to recognize discrete acoustic events and sound categories.

Cross-modal Audio Retrieval. This task evaluates alignment between audio and other modalities, including text-to-audio retrieval with Clotho (Drossos et al., 2020) and SoundDescs (Koepke et al., 2023), audio-video alignment with AVE (Tian et al., 2018), and audio-image retrieval with SpeechCOCO (Havard et al., 2017).

Audio Temporal Grounding. Using the TUT Sound Events 2017 dataset (Mesaros et al., 2016), models must localize specific acoustic events within continuous audio streams.

Text Tasks.

Recognizing that standard text retrieval tasks often fail to challenge modern information retrieval systems (Lu et al., 2026b), MMEB-V3 introduces more demanding scenarios involving instruction following, reasoning, and long-context understanding:

Instruction Following Retrieval. FollowIR (Weller et al., 2025) and InfoSearch (Zhou et al., 2025) evaluate whether models can retrieve documents satisfying complex instructions and constraints.

Reasoning Retrieval. BRIGHT (SU et al., 2025) and R2MD (Li et al., 2025) require logical inference and multi-hop reasoning beyond simple keyword matching.

Long-context Retrieval. LongEmb (Zhu et al., 2024) evaluates retrieval from long documents and extended contexts.

Multi-condition Retrieval. MultiConIR (Lu et al., 2025) measures the ability to satisfy multiple retrieval constraints simultaneously.

General Text Retrieval. nanoBEIR (Thakur et al., 2021), a curated subset of BEIR, provides a compact yet diverse benchmark for semantic retrieval.

Agent Tasks.

MMEB-V3 further evaluates agentic capabilities such as tool selection, GUI interaction and agent memory retrieval:

Tool Retrieval. Tool-REX (Lu et al., 2026a) contains over 43,000 tools (e.g., APIs and code functions) with structured metadata, evaluating the ability to retrieve appropriate tools for user intents.

GUI Agent Trajectory Retrieval. GAE-Bench (Zhang et al., 2025b) evaluates retrieval over multimodal GUI interaction trajectories represented by screenshots and structured actions.

Agent Memory Retrieval. LMEB (Zhao et al., 2026) is a large-scale benchmark suite for long-horizon memory modeling. We select four representative tasks covering diverse memory types: episodic memory (KnowMeBench (Wu et al., 2026)), dialogue memory (REALTALK (Lee et al., 2025)), semantic memory (PeerQA (Baumgärtner et al., 2025)), and procedural memory (DeepPlanning (Zhang et al., 2026)). These tasks are used to evaluate whether embedding models can retrieve and utilize relevant memory under diverse semantic structures and temporal dependencies.

Omni-modality Semantic Equivalence Tuples

Cross-modal retrieval is an information retrieval setting where a query in one modality is used to retrieve semantically aligned instances from a different modality. Despite recent progress in omni-embedding models that learn shared representation spaces across modalities, existing evaluations remain limited in measuring fine-grained, modality-aware alignment. To address this, MMEB-V3 introduces the concept of Omni-modality Semantic Equivalence Tuples, which is defined as a set of aligned instances $\{x^{T},x^{I},x^{V},x^{A}\}$ with each instance encoding the same semantic meaning in one of the four different modalities. Under this setting, cross-modal retrieval is defined as the task of retrieving the correct element from a shared (flattened) pool of OmniSET that (1) is semantically equivalent to the input instance and (2) belongs to the specified target modality in the instruction.

OmniSET contains 100 queries with human-verified hard negatives to increase discriminative difficulty. Each query is further expanded into 12 directional tasks, corresponding to the 12 ordered input-target modality pairs $(m_{in}\rightarrow m_{target})$ where $m_{in}\neq m_{target}$ and $m\in\{T,I,V,A\}$.

A.2.1 Construction of OmniSET

We introduce OmniSET (Omni-modality Semantic Equivalence Tuples), a dataset designed to construct semantically equivalent instances across different modalities. The goal is to create tuples that share the same underlying semantics while differing in modality, enabling controlled evaluation of cross-modal alignment and transfer. Built upon the widely used MSCOCO dataset (Lin et al., 2015), which provides diverse real-world images with human-annotated captions, we extend the original bimodal (image–text) setting into a unified four-modality framework. Specifically, for each selected instance, we construct an Omni-modality Semantic Equivalence Tuples (OmniSET) consisting of aligned representations in text, image, video, and audio modalities.

Our construction consists of two main components. First, we curate a set of approximately 200 high-quality query samples, each paired with human-verified hard negatives to ensure fine-grained semantic discrimination. Second, we augment the original modalities by synthesizing additional ones: motion videos are generated from images using Google Veo-3.1, and speech audio is generated from captions using Gemini-2.5-Flash-TTS. This results in semantically aligned multi-modal tuples with consistent content across modalities. The overall construction pipeline is illustrated in Figure 5, and the prompt templates used for video and audio generation are provided in Figure 6.

The hard negative construction process is performed as follows:

We filter out images with restrictive licenses (e.g., Attribution-NoDerivs and Attribution-NonCommercial-NoDerivs) that prohibit derivative content.

For each remaining image, we extract object annotations and compute the number of unique object categories.

Given a reference image, we rank all other images by the number of shared object categories and select the top 30 as candidate hard negatives.

We sample approximately 100 images as query instances.

We manually inspect the candidates to remove near-duplicates or overly similar samples, resulting in 15–20 hard negatives per query.

Finally, for each positive instance and its associated negatives, we generate videos and audio to form complete OmniSET across all four modalities.

We evaluate cross-modal retrieval by formulating queries and targets in different modalities. Each query is provided in one modality, and the model is required to retrieve a semantically matching instance from a specified target modality.

Given four modalities (text, image, video, and audio), each modality can query the other three, resulting in a total of 12 directed retrieval tasks (e.g., image $\rightarrow$ text, image $\rightarrow$ video, image $\rightarrow$ audio).

Shared candidate pool.

For each query, all 12 retrieval directions share the same candidate pool, which contains the corresponding hard negatives across all four modalities. Therefore, regardless of the query direction, the model must select the correct target from a unified mixed-modality pool. This design ensures that performance differences across directions are not due to variations in candidate sets.

Avoiding trivial matching.

A limitation of our dataset construction is that generated videos are highly consistent with their source images, which may simplify certain directions such as I$\rightarrow$V and V$\rightarrow$I. To mitigate this, the input instance is explicitly included in the candidate pool as a distractor, while same-modality retrieval (e.g., I$\rightarrow$I and V$\rightarrow$V) is excluded. This prevents trivial matching based on instance identity.

A.2.2 Impact of Synthetic Data and Potential Modality Bias

A key design choice in OmniSET is the use of synthetic data to complete modality coverage: videos are generated from images, and audio is generated from text captions. While this enables controlled construction of semantically aligned tuples across modalities, it may introduce systematic biases that affect certain cross-modal directions.

In particular, the generation process creates asymmetric dependencies between modalities. Video samples are directly derived from images, and audio samples are derived from text. As a result, modality pairs such as image–video and text–audio may exhibit higher intrinsic similarity than other cross-modal pairs. This effect can partially explain the unusually strong performance observed in directions such as I$\rightarrow$V and A$\rightarrow$T, where retrieval may benefit from shared generation artifacts rather than purely learned cross-modal alignment.

More broadly, this construction may introduce a form of modality preference, where certain modality pairs are more tightly coupled due to data generation rather than model capability. This could potentially amplify observed behaviors such as modality bias or directional asymmetry. For example, when a query modality is closely aligned with a generated modality, retrieval may favor that modality even when it is not explicitly specified as the target.

However, several aspects of our design mitigate the extent to which these biases affect our conclusions. First, all retrieval directions share a unified mixed-modality candidate pool, ensuring that performance differences are not caused by variations in candidate sets. Second, trivial matching is explicitly avoided by including the source instance as a distractor and excluding same-modality retrieval. Third, the key phenomena observed in our analysis—such as the failure of instruction-constrained retrieval and the misalignment of instruction-induced shifts—are consistent across multiple models and modality directions, including those not directly affected by synthetic generation.

We therefore view these biases as an inherent trade-off of controlled multi-modal construction: while synthetic data may strengthen certain modality pairs, it does not fully account for the broader patterns observed in our experiments. Nevertheless, future work could further investigate this issue by incorporating human-annotated multi-modal data or by disentangling synthetic and real modality pairs through targeted ablation studies.

A.3 Detailed Scores

We provide complete per-dataset evaluation results for each task .

Image tasks. Results are reported in Table 5.

Video tasks. Results are reported in Table 6.

VisDoc tasks. Results are reported in Table 7.

Audio tasks. Results are reported in Table 8.

Text tasks. Results are reported in Table 9.

Agent tasks. Results are reported in Table 10.

A.4 Visualization and Analysis

In this section, we provide additional visualizations to complement the analysis in the main text. These figures illustrate both the global structure of the embedding space and the behavior of instruction-induced shifts across different models.

Instruction-induced changes in query–target distance.

Figure 10 presents the change in cosine distance between queries and target modalities after instruction augmentation for WAVE and Qwen3-VL-Embedding-8B. Compared to omni-embed-nemotron-3b in the main text, WAVE exhibits consistently small changes across all directions, indicating that instruction augmentation has limited influence on the query representation. In contrast, Qwen3-VL-Embedding-8B shows moderate variations, but the improvements are sparse and inconsistent. In most directions, the distance to the target modality either remains unchanged or increases, suggesting that instruction signals do not reliably improve target alignment. These observations are consistent with the findings in Section 5, where we show that both insufficient sensitivity and misaligned shifts contribute to retrieval failures.

Embedding space geometry across models.

Figure 11 visualizes the embedding distributions of different modalities using t-SNE. For omni-embed-nemotron-3b, embeddings form relatively well-separated clusters corresponding to each modality, indicating a structured representation space. WAVE exhibits a more entangled distribution, particularly between audio and video modalities, reflecting its focus on audio–visual representation learning. In contrast, Qwen3-VL-Embedding-8B shows less clearly separated clusters, with significant overlap among modalities. This difference in geometry suggests that models vary in how modality information is encoded, which in turn influences their behavior under modality-constrained retrieval.

Instruction-induced shifts across query modalities.

Figure 12 further illustrates the effect of instruction augmentation on query embeddings for omni-embed-nemotron-3b, using video and audio queries as examples. Across different source modalities, instruction augmentation consistently perturbs the query representations. However, these shifts are not consistently directed toward the intended target modality. Instead, the updated query embeddings may move toward neighboring modality clusters or follow directions that are unrelated to the target. This observation provides additional qualitative evidence that, although the model is sensitive to instruction signals, the induced shifts do not reliably support target-oriented alignment.