Fewer tokens than the compared token-level visual document representations.
Visual document retrieval · 2026
CausalEmbed
Auto-regressive multi-vector generation in latent space for compact visual-document embeddings.
1 The Hong Kong University of Science and Technology (Guangzhou) · 2 Alibaba Cloud Computing · 3 The Hong Kong University of Science and Technology · 4 University of Illinois Chicago
Project overview
CausalEmbed generates a sequence of compact latent vectors and compares documents and queries through late interaction.
Motivation
Visual-document retrieval needs compact representations without premature information collapse.
A single vector is efficient but restrictive, whereas retaining all visual tokens incurs substantial storage and retrieval costs.
Visual documents combine textual content, spatial layout, tables, figures, and typography. Dense retrieval systems must preserve these heterogeneous signals, yet existing representations typically choose between aggressive single-vector compression and token-level storage that scales poorly with corpus size.
CausalEmbed investigates an intermediate representation: a short, ordered sequence of latent vectors generated specifically for retrieval. Autoregressive generation allows each vector to condition on earlier vectors and encode complementary evidence, while the sequence length provides a direct mechanism for controlling the efficiency–accuracy trade-off.
Main findings
Efficiency and scalability
Generate multiple complementary vectors sequentially in latent space.
Use test-time generation length as a practical knob for the accuracy–efficiency tradeoff.
Methodology
Autoregressive multi-vector embedding
The model learns a compact latent sequence and performs fine-grained query–document matching through late interaction.
Multimodal document encoding
A visual encoder processes the rendered page and preserves textual, visual, and layout evidence in a shared hidden representation. A text encoder independently maps each query into the retrieval space.
Causal latent generation
An autoregressive generator produces document vectors sequentially. Each new vector attends to the encoded page and previously generated vectors, encouraging the sequence to capture complementary rather than duplicated information.
Late-interaction retrieval
Query and document vectors are compared with a late-interaction scoring function and trained using retrieval supervision. At test time, varying the number of generated vectors adjusts storage, latency, and retrieval accuracy without retraining the encoder.
Visual results
Architecture and retrieval evaluation
These views highlight the architecture, retrieval quality, and the effect of scaling the latent sequence.
Citation
BibTeX
@article{huo2026causalembed,
title = {CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embedding},
author = {Huo, Jiahao and Huang, Yu and Yan, Yibo and Pan, Ye and Cao, Yi and Ou, Mingdong and Yu, Philip S. and Hu, Xuming},
journal = {arXiv preprint arXiv:2601.21262},
year = {2026}
}