Apple LiTo: Single Image to Realistic 3D

🔑 Enhanced Key Takeaways

• •LiTo uses a Perceiver-IO encoder to process up to one million surface light field samples into 8192 latent tokens of 32 dimensions each.[1][4]
• •The model employs higher-order spherical harmonics in its Gaussian decoder to capture specularities, highlights, and Fresnel effects.[1][5]
• •A latent flow matching model is trained on the representation to generate 3D objects conditioned on a single input image.[3][4]
• •LiTo outperforms baselines like TRELLIS in visual quality, input fidelity, and correct camera coordinate orientation.[6]

🛠️ Technical Deep Dive

• •Encoder: Perceiver-IO architecture with cross-attention and voxel-based self-attention; processes N surface light field samples (position, color, viewing direction) into k=8192, d=32-dimensional latent vectors.[1][4]
• •Decoders: Geometry decoder via flow matching for surface reconstruction; 3D Gaussian decoder predicts higher-order spherical harmonics for view-dependent radiance.[1][5]
• •Training: Uses multi-view RGB-D data; latent flow matching enables single-image conditioned generation.[3][4]
• •Input processing: Up to 1 million randomly subsampled observations per object, normalized for explicit direction-dependent supervision.[1]

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline