NeurIPS 2026

EVIDENT Routing MLLM Adaptation through Entity-Grounded Visual Evidence for Cross-Domain Video Temporal Grounding

Geo Ahn1* Jiwook Han1* Youngrae Kim2* Joonseok Lee3 Jinwoo Choi1†

1 Kyung Hee University   ·   2 University of Southern California   ·   3 Seoul National University

* Equal contribution  ·  † Corresponding author

Paper arXiv Code BibTeX

TL;DR

Read the full abstract

Fine-tuning MLLMs for Video Temporal Grounding (VTG) often improves in-domain performance but degrades sharply under domain shift. We find that this failure is primarily driven not just by unseen query concepts, but by visual domain shift, which prevents the model from coupling its learned temporal localization knowledge with its inherent entity-attention capability. We introduce EVIDENT, a parameter-efficient adaptation framework that anchors temporal grounding in the inherent entity-attention of pre-trained MLLMs by routing VTG adaptation through explicit visual entity evidence: (i) an Entity Bottleneck Adapter that transforms dense visual tokens into compact entity-level slots, (ii) an Entity-Binding Distillation loss that instills objectness priors into the MLLM visual space, and (iii) an Entity-to-eVidence gating mechanism that steers the model to localize moments containing query-relevant entities. Experiments on cross-domain VTG benchmarks show that EVIDENT consistently improves out-of-domain robustness while preserving competitive in-domain performance with modest parameter overhead.

How EVIDENT works

Find the entities, then find the moment.

How EVIDENT works: for the query 'When does the person look at a book?', each frame's patches are grouped into entities, every frame gets a subject (person) and object (book) score, and only frames with both are kept, giving the evidence-grounded prediction [84s, 96s].

At a glance

Same query. New domain. EVIDENT still finds the person and the book.

On an unseen domain, the naïvely fine-tuned model doesn’t look at them anymore and misses the moment.

Teaser: on In-Domain and Out-of-Domain videos, zero-shot and naïvely fine-tuned MLLMs mislocalize the query 'During which time does the person look at a book?' under domain shift, while EVIDENT's entity-centric representation captures person and book and predicts the correct window.

Problem Analysis

Fine-tuning decouples attention from localization.

MLLMs already know where entities are. Fine-tuning teaches when, but on new visuals the two come apart: attention–localization decoupling. Qwen2.5-VL-3B, Cha. → QVH; same trend on InternVL3-2B.

Entity-attention transfers. Learned grounding doesn't.

(a) Zero-shot visual-text cosine similarity on Charades-STA and QVHighlights, (b) ID vs OOD R1@0.5 after fine-tuning, (c) G.T. attention increase after fine-tuning across layers 10–18 for ID and OOD.
Entity-attention transfers. Zero-shot, patch–word similarity (e.g. “book”, “baby”) is strong on both Cha. and QVH.
Fine-tuning breaks it. ID–OOD gap of 19.8 (Cha.→QVH) and 33.4 (QVH→Cha.) R1@0.5.
Learned grounding stays home. Attention on G.T. frames rises substantially on ID, but only marginally on OOD.

It's the visual gap, not the concept gap.

Concept gap: unseen query words. Visual gap: unfamiliar-looking videos.

(a) Seen vs unseen concepts: 44.0 vs 40.7 R1@0.5. (b) Visually similar vs dissimilar: 52.8 vs 39.1. (c) Perturbation analysis on ID and OOD.
Concept gap is marginal. Seen vs. unseen query concepts: 44.0 vs. 40.7 (3.3-pt gap, Cha.→QVH).
Visual gap dominates. Top-20% visually similar: 52.8; bottom-20%: 39.1 (13.7-pt gap).
It doesn’t look at the evidence anymore. On OOD, perturbing the G.T. interval hurts no more than a random one (−12.6 vs. −12.1); perturbing non-G.T. frames helps (+4.6).

Approach

EVIDence-grounded ENTity-attentive adaptation.

Slots separate entities; the MLLM's pre-trained attention decides which ones matter.

i EB Adapter

Compresses each frame's tokens into 4 entity slots in the early decoder layers, added back as a residual.

ii EB Distillation

Distills DINOv2 objectness into the slot maps, so each slot binds to one coherent entity.

iii E2V Gating

Scores each frame by whether the query's subject and object appear, and scales the adapter there.

    Qualitative Analysis

    What do the slots and the gate capture?

    Trained only on Charades-STA, the slots still split people, objects and background on unseen domains.

    EB Adapter attention visualization on Charades-STA (ID), QVHighlights (OOD), and DiDeMo (OOD): each frame and its four slot masks.

    Each frame colored by its top slot. Cha. (ID), QVH and DiDeMo (OOD).

    The gate lights up where the query's entities are.

    Each entity's score rises only while it is on screen: person and glass climb together, while the absent zebra stays flat. Because the gate multiplies the subject and object scores, the adapter acts only where both appear, on in-domain and out-of-domain videos alike.

    E2V gating scores per entity across frames on a Charades-STA sample (person, glass, phone, zebra) and a QVHighlights sample (person, dish, pot, computer).
    Cha. (ID, left), QVH (OOD, right). Arrows: when each entity is visible.

    Back to the diagnosis

    From decoupled to re-coupled.

    Out of domain, naïve fine-tuning keeps only 37.8% of the attention gain it learned in‑domain. EVIDENT keeps 87.6%.

    Cross-Domain Results

    Consistent OOD gains. Competitive ID performance.

    Train on one dataset, test on the others. Same Qwen2.5-VL-7B backbone as the LoRA baseline; EVIDENT trains only 2.42% of its parameters.

    Out-of-domain accuracy, same backbone

    +2.9avg. OOD gain (+10% rel.)
    Source → target. R1@0.7: top-1 window overlaps the ground truth at IoU ≥ 0.7. Full numbers below.
    Source
    Dataset    		 Method LLM Target Dataset
    Charades-STA QVHighlights DiDeMo
    R1@0.3R1@0.5R1@0.7mIoU R1@0.3R1@0.5R1@0.7mIoU R1@0.3R1@0.5R1@0.7mIoU
    Zero-ShotGPT-5– 39.718.36.228.4 –––– ––––
    Gemini-2.0-Flash– 55.629.09.535.1 72.158.141.954.9 ––––
    HawkEye7B 50.631.414.533.7 42.120.86.628.7 37.818.47.824.7
    TimeSuite7B 69.948.724.0– –12.39.221.3 ––––
    UniTime7B –59.131.952.2 –41.031.543.7 ––––
    VideoMind7B 73.559.131.250.2 –––– ––––
    Charades-STAEaTR– 67.755.233.147.7 31.717.06.421.5 32.720.17.821.2
    CG-DETR– 69.757.635.149.5 37.422.810.525.2 30.321.811.020.7
    Chrono-BLIP4B 79.168.948.758.6 67.244.523.744.3 56.043.223.137.7
    Qwen2.5-VL†7B 80.367.743.958.1 75.254.329.751.2 68.253.830.347.4
    EVIDENT (Ours)7B 80.369.149.259.9 77.559.836.955.2 69.255.432.348.5
    QVHighlightsEaTR– 40.827.213.028.0 70.359.640.353.1 36.017.86.524.1
    CG-DETR– 42.825.512.228.5 77.565.652.161.3 36.621.28.924.6
    Chrono-BLIP4B 63.339.320.442.7 86.176.662.770.7 43.024.411.833.7
    Qwen2.5-VL†7B 71.148.126.247.3 86.978.264.872.0 59.434.116.938.2
    EVIDENT (Ours)7B 73.350.228.049.0 89.582.269.175.3 62.436.017.440.6

    Gray: in-domain. Bold: best OOD. †: our LoRA baseline. “–”: seen in training or not reported.

    Not tied to one MLLM.

    Trained on Charades-STA, EVIDENT also improves OOD transfer on Qwen3-VL-8B and InternVL3.5-8B, by up to +10.3 R1@0.7 on QVHighlights for InternVL3.5.

    BackboneCha. (ID)QVH (OOD)DiDeMo (OOD)
    R1@0.5R1@0.7R1@0.5R1@0.7R1@0.5R1@0.7
    Qwen3-VL-8B†70.450.161.444.258.630.8
    + EVIDENT70.850.464.346.760.430.5
    InternVL3.5-8B†66.744.340.720.545.224.0
    + EVIDENT72.449.548.730.855.329.6

    †: fine-tuned baseline on the same backbone. Bold: best OOD result per backbone.

    Every component pulls its weight.

    Cha. → QVH, Qwen2.5-VL-7B. (b) and (c) use the adapter alone.

    (a) Components
    EB AdapterEB Distill.E2VID @0.5ID @0.7OOD @0.5OOD @0.7
    67.743.954.329.7
    ✓68.149.554.933.1
    ✓✓69.150.458.836.7
    ✓✓✓69.149.259.836.9

    EB Distillation is what makes the slots useful (+3.9 OOD R1@0.5); E2V gating adds +1.0 on top.

    (b) Where to put slot attention
    PlacementID @0.5ID @0.7OOD @0.5OOD @0.7
    After vision encoder34.618.819.512.7
    Adapter-style (ours)68.149.554.933.1

    Replacing tokens before the LLM collapses; a residual adapter keeps pre-trained alignment.

    (c) Entity-level decomposition
    BottleneckID @0.5ID @0.7OOD @0.5OOD @0.7
    LoRA67.743.954.329.7
    Adapter w/ self-attn.68.849.542.421.0
    EB Adapter68.149.554.933.1

    Self-attention overfits (−11.9 OOD R1@0.5); slots keep the ID gain and add +3.4 OOD R1@0.7.

    (d) Query-dependent E2V gating
    Entity sourceID @0.5ID @0.7OOD @0.5OOD @0.7
    Query-agnostic69.349.458.436.0
    Query-dependent69.149.259.836.9

    Per-query subject/object words beat fixed “person”/“object” by +1.4 OOD R1@0.5.

    More ablations from the appendix · number of slots, E2V feature source
    (e) Number of slots
    # Slots NsID @0.5ID @0.7OOD @0.5OOD @0.7
    469.149.259.836.9
    869.249.456.636.6
    1669.249.153.933.7

    More slots fragment entities: OOD R1@0.5 drops 5.9 from 4 to 16 slots.

    (f) Visual feature for E2V gating
    Visual featureID @0.5ID @0.7OOD @0.5OOD @0.7
    Before down proj.68.449.055.834.1
    After down proj.69.149.259.836.9
    After reconstruction69.250.351.929.7

    Gate on the LLM's visual stream; slot-reconstructed features lose the detail needed to score entities.

    If you find this useful

    Cite EVIDENT

    @inproceedings{ahn2026evident,
  title     = {{EVIDENT}: Routing {MLLM} Adaptation through Entity-Grounded Visual Evidence for Cross-Domain Video Temporal Grounding},
  author    = {Ahn, Geo and Han, Jiwook and Kim, Youngrae and Lee, Joonseok and Choi, Jinwoo},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}

    An earlier version of this work, SlotVTG, appeared at the CVPR 2026 GRAIL Workshop.