Representation decomposition

The question

A vision model looking at a scene encodes two different kinds of knowledge in the same embedding: what each object is, and how the objects relate to each other in space. In DINOv3, these are entangled: you can read both out of the features, but you can't easily inspect or control either one alone.

Object identity alone doesn't specify a scene. The same two objects can appear in different spatial configurations, so a scene representation has to preserve information about both what is present and how it is arranged. I wanted to know whether those two kinds of information could be separated inside DINOv3's representation.

The decomposition

Using PCA, LDA, and learned linear projections, I separate DINOv3 patch embeddings into an object-identity subspace and a spatial-relation subspace. Identity shows up as clusters. The residual representation retains linearly decodable relational information even after the dominant object contribution is removed. The figure sketches the same split, and the interaction on the home page plays it back.

Experimental setup & results

• Setup · completed Controlled experiments across 2,000+ synthetic images, generated so that the target property and object identity vary independently, then 400 real-world objects and 4,800 paired scenes spanning eight spatial relationships.
• Against the baseline · completed Projecting out the object subspace leaves a residual that generalizes to held-out shapes at 93.3%, against 53.3% from the raw embeddings. Leave-one-shape-out, with permutation tests at p < .001.
• Transfer to real images · completed The signal survives the move from synthetic renders to photographs: 97.9% mean leave-one-object-out accuracy across the 400 real-world objects, with median accuracy of 100%.

Current direction: causal intervention

The decomposition is correlational evidence; the current experiments test it causally. I'm testing whether intervening on the relational component produces the predicted change in the model's spatial judgments while preserving object identity. If the split is real, intervening on one component should change one kind of behavior and leave the other alone.

Intervention experiments in progress · manuscript in preparation for ICML