
High Dimensional, Dynamic Rotary Positional Embedding
HDD-RoPE is a new positional embedding technique that treats sequence position as multidimensional, allowing for more flexible rotation axes. It demonstrates faster convergence on the TinyStories dataset compared to standard xPos-based transformers.






