kernel-anvil: 2x AMD Decode Speedup

🔑 Enhanced Key Takeaways

• •Kernel-anvil leverages the ROCm HIP backend to bypass static kernel compilation limitations, allowing for dynamic tuning of occupancy parameters that are typically hardcoded in llama.cpp's upstream build.
• •The tool specifically addresses the 'wavefront occupancy' bottleneck on RDNA3 architectures, where default llama.cpp configurations often fail to saturate the GPU's compute units during memory-bound decode operations.
• •Integration requires a runtime hook that intercepts the GEMV (General Matrix-Vector multiplication) dispatch, enabling the injection of custom block/warp dimensions derived from the initial profiling phase.

🛠️ Technical Deep Dive

• •Targets the mmvq (Matrix-Matrix Vector Quantized) kernel in llama.cpp, which is the primary bottleneck for prompt processing and token generation on AMD hardware.
• •Profiling mechanism: Executes a micro-benchmark suite (193 iterations) that iterates through permutations of 'nwarps' (number of warps per block) and 'rows_per_block' to find the optimal occupancy for the specific GPU's memory bandwidth.
• •Implementation: Uses a JSON-based configuration file generated during the profiling step, which the patched llama.cpp reads at runtime to override default kernel launch parameters.
• •Architecture focus: Specifically optimizes for the 32-wide wavefront size characteristic of AMD RDNA3, which differs significantly from NVIDIA's 32-thread warp architecture, often causing suboptimal performance when using generic CUDA-ported kernels.

🔮 Future ImplicationsAI analysis grounded in cited sources