Transformer Struggles with 4-Day Forecast

🔑 Enhanced Key Takeaways

• •Theoretical analysis proves Linear Self-Attention (LSA) Transformers cannot achieve lower expected MSE than classical linear models for in-context time series forecasting under AR(p) data.[1][2]
• •Under Chain-of-Thought (CoT) inference, Transformer predictions exponentially collapse to the mean, exacerbating bias in iterative forecasting.[2][3]
• •Increasing context length or model depth provides diminishing returns for Transformers in time series tasks due to inherent representational limits.[3]

🛠️ Technical Deep Dive

• •Linear Self-Attention (LSA) models asymptotically recover optimal linear predictors only as context length approaches infinity with sufficient training.[1][2]
• •Under AR(p) processes, Transformers fail to extrapolate beyond linear regression performance in expected MSE for in-context learning scenarios.[4]
• •ProbSparse attention (e.g., Informer) and hierarchical pyramidal attention (e.g., Pyraformer) were early attempts to address quadratic complexity in Transformer-based TSF.[1]

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline