We study positional encodings for multi-view transformers that process tokens from a set of posed input images, and seek a mechanism that encodes patches uniquely, allows SE(3)-invariant attention with multi-frequency similarity, and can be adaptive to the geometry of the underlying scene. We find that prior (absolute or relative) encoding schemes for multi-view attention do not meet the above desiderata, and present RayRoPE to address this gap. RayRoPE represents patch positions based on associated rays but leverages a predicted point along the ray instead of the direction for a geometry-aware encoding. To achieve SE(3) invariance, RayRoPE computes query-frame projective coordinates for computing multi-frequency similarity. Lastly, as the 'predicted' 3D point along a ray may not be precise, RayRoPE presents a mechanism to analytically compute the expected position encoding under uncertainty. We validate RayRoPE on the tasks of novel-view synthesis and stereo depth estimation and show that it consistently improves over alternate position encoding schemes (e.g. 15% relative improvement on LPIPS in Co3D). We also show that RayRoPE can seamlessly incorporate RGB-D input, resulting in even larger gains over alternatives that cannot positionally encode this information.
We live in a 3D world, and the positional encoding of transformers should be 3D-aware as well. In this project, we propose RayRoPE, a novel relative positional encoding mechanism designed for multi-view attention.
We argue that an ideal positional encoding for multi-view attention should satisfy four desirable properties illustrated below:
| Method | SE(3) Invariance | Uniqueness | Geometric Adaptiveness | Multi-frequency |
|---|---|---|---|---|
| Plücker Raymap | ✗ | ✓ | ✗ | ✗ |
| RoPE on Raymap | ✗ | ✓ | ✗ | ✓ |
| GTA, PRoPE | ✓ | ✗ | ✗ | ✗ (only for patch indices) |
| RayRoPE (Ours) | ✓ | ✓ | ✓ | ✓ |
We train the LVSM model with different positional encodings and compare their results below.
Left Video:
Right Video:
Select Scene (Input Views):
We evaluate the stereo depth estimation performance with the Unimatch model. The 3D point clouds below show the predicted depth for each method:
Ground Truth
UniMatch
PRoPE
RayRoPE (Ours)
Select Scene (Input Views):
RayRoPE predicts depth and uncertainties which are used to compute the positional encodings. Even without depth supervision during training, resonable depth predictions emerge, especially in the later layers. Depth prediction accuracy and the predicted uncertainties are inversely correlated.
Select Scene:
Explore how RayRoPE bias attention similarity as relative camera and predicted depth changes. The query camera (orange) is fixed at identity and the heatmap shows attention similarity between the center query token and all key tokens. We set the $Q$, $K$ features to contain constant value 1, so that the attention similarity is purely determined by the positional encodings. Multi-frequency is disabled.
This work was supported by Apple. We thank Zihan Wang and Qitao Zhao for insightful discussions throughout the project. Additionally, our implementation is built upon the open-source frameworks PRoPE and Unimatch; we are grateful to the authors for making their code available.
@article{wu2026rayrope,
title={RayRoPE: Projective Ray Positional Encoding for Multi-view Attention},
author={Wu, Yu and Jeon, Minsik and Chang, Jen-Hao Rick and Tuzel, Oncel and Tulsiani, Shubham},
journal={arXiv preprint arXiv:2601.15275},
year={2026}
}