University of Houston
Self-supervised Vision Foundation Models (VFMs) have become essential backbones for downstream tasks due to their strong and transferable visual representations. However, their patch-token-level features are often too coarse for dense prediction tasks such as semantic segmentation and depth estimation when accurate fine-grained predictions are required. Feature upsampling methods have been developed to recover pixel-level detail but still face limitations. Learnable upsamplers are often designed for a specific encoder and must be retrained for different encoders. Image-guided methods that use shallow pixel encoders often introduce textural artifacts and lack the semantic guidance needed for accurate downstream predictions. We introduce PixelUp, a zero-shot VFM-agnostic upsampler achieving semantic awareness through a coarse-to-fine chain of windowed cross-attention architecture guided by multi-scale semantic features.
Average improvement across vision foundation models. Open-vocabulary and unsupervised segmentation are both training-free.
Outputs are shown for a frozen backbone at both resolutions. Inside the lens the map reverts to the raw 16×16 patch tokens and sweeps back to the PixelUp reconstruction; the Similarity view instead reports cosine similarity between the feature at the cursor and every other pixel.
PixelUp is a zero-shot, VFM-agnostic upsampler that lifts the coarse patch tokens of any frozen VFM to dense pixel-level features. A Pixel Encoder forms an initial query, and a Cross-Attention Chain refines the query with the multi-scale semantic features of a pre-trained frozen Semantic Encoder into the high-resolution query QHR and pooled key KLR, which combine the values VLR via neighborhood attention into the dense output VHR.
