Parallelizing DL Hyperparameter Search on Single GPU

🔑 Enhanced Key Takeaways

• •Bayesian optimization tools like SigOpt enable hyperparameter tuning on single GPUs up to 10x faster than random search by efficiently sampling configurations and jointly optimizing metrics like accuracy and inference time.[2]
• •Batching heuristic evaluations on GPUs, as in batched A* and Weighted A*, delays computations until large state batches form, providing speedups for neural-guided searches applicable to DL hyperparameter sweeps.[1]
• •Single-GPU memory limits model sizes to around 70B parameters at FP16; techniques like data parallelism across multiple GPUs or quantization are needed for larger DL networks.[3]

🛠️ Technical Deep Dive

• •SigOpt's optimization loop on NVIDIA K80 GPUs: suggests hyperparameter configs (e.g., SGD params, architecture), trains models in MXNet/TensorFlow, observes accuracy/inference time, repeats until budget exhausted; outperforms random search with 480 vs 1800 evaluations for better Pareto frontier.[2]
• •GA* (GPU A*): uses multiple parallel priority queues for simultaneous node extraction/expansion across GPU threads, first parallel A* variant leveraging GPU compute.[1]
• •CB-DFS with Batch IDA/BTS: parallelizes on CPU/GPU for neural heuristics, gains significant speedups at large batch sizes by delaying evaluations.[1]

🔮 Future ImplicationsAI analysis grounded in cited sources