H Company Releases NeoMME: A Family of 260M and 800M Single-Tower Multimodal Encoders That Drop the Vision Tower and Causal Decoder

Most visual document retrievers in production are adapted generative VLMs: a separately pretrained vision tower plus a causal decoder that never emits a token. That is parameter and compute overhead for a job that only needs representations. H Company has released NeoMME, a family of 260M and 800M bidirectional encoders that drops both. One Transformer processes multilingual text tokens and raw 32×32 RGB patches through the same layers, trained from random initialization.

Every checkpoint ships under Apache 2.0 with day-zero Hugging Face Transformers support. The 260M retrieval fine-tune reaches 0.523 nDCG@10 on ViDoRe v3, indexes 51.3 pages per second on a single NVIDIA L40S, and encodes a query in 78.3 ms on a CPU-only host.

Text uses an ALBERT-style factorized embedding; images go through a from-scratch 2-layer MLP on non-overlapping 32×32 patches—no SigLIP-style tower, no patch merging. Both sizes support 16,384-token context (enough for two 4K UHD pages after patching), with sliding-window attention most layers and global attention every sixth layer plus the final one. Exact counts are ~263M and ~794M. A from-scratch BPE with a 131,072 vocabulary emits 44.4% fewer tokens than ModernBERT across 14 FLORES-200 languages.

Pretraining is discrete masked diffusion over text, optionally conditioned on visible patches. Multimodal segments draw corruption rates from 0.30 to 1 so the model cannot ignore the page. At 90% masking, visible patches raise masked-token accuracy by 38.4 points (260M) and 40.5 points (800M). Each run sees about 524 billion packed tokens on 16 or 32 H100s.

NeoMME-Retriever adds jointly trained dense (Matryoshka) and late-interaction (128-d) heads from one forward pass. On ViDoRe v3, 260M scores 0.523 and 800M 0.556—within 0.002 of ColQwen2.5-v0.2 at 3.75B, and 26.1 points above the best other sub-300M model MarkTechPost cites. Text-only BEIR-15 is weaker (late interaction 0.4881 / 0.5126 vs 0.5722 for a much smaller LateOn baseline), which the authors partly blame on ~430K text query examples versus hundreds of millions of contrastive pairs elsewhere. Hierarchical pooling plus asymmetric quantization can shrink a ~1.5 MB float32 late-interaction page index to 6.0 kB while retaining 95.19% of baseline nDCG@10.

**Signal Desk's read:** NeoMME is a clean argument that document retrieval did not need to inherit generative VLM baggage. Matching a multi-billion ColQwen-class score at 260M, with CPU query latency and aggressive index compression, is the deployability story enterprises actually buy. The weak spots—text-only retrieval and frozen natural-image transfer—are disclosed rather than papered over. The likelier near-term win is visual document search and RAG over PDFs and scans, not a general multimodal encoder crown. Watch whether the community treats the single-tower recipe as the new default for page retrieval, or whether ColPali-style adaptations keep winning on inertia.

ColPali and follow-ons established VLM-as-retriever as the default for visual documents. NeoMME is an explicit counter-design: bidirectional encoder, shared tower, masked diffusion pretraining, dual retrieval heads, Apache 2.0.