Apple ML introduces Cadmus, a small-scale system for autoregressive program synthesis. It features an integer virtual machine, a dataset of diverse true programs, and a transformer model trained for under $200 compute. This setup enables controlled experiments avoiding LLM pitfalls like OOD issues and high compute demands.
Key Points
- 1.Small models for true program completion
- 2.Includes VM, dataset, and low-cost transformer
- 3.Facilitates studies on fine-tuning and tokenization
Impact Analysis
Lowers barriers for program synthesis research with affordable, controllable models. Reduces reliance on resource-intensive LLMs, accelerating experimentation.
Technical Details
Autoregressive transformer trained on diverse program dataset. Custom integer VM supports true program execution.