Hard-to-Fake Coding Benchmark Exposes LLM Limits

🔑 Enhanced Key Takeaways

• •EsoLang-Bench evaluates five specific esoteric languages: Brainfuck, Befunge-98, Whitespace, Unlambda, and Shakespeare, selected for having 1,000-100,000x fewer GitHub repositories than Python to minimize training data contamination.[2][3][4]
• •The benchmark includes 80 problems per language (20 each in Easy, Medium, Hard, and Extra-Hard tiers), with evaluations using zero-shot, few-shot (3 ICL examples), and advanced prompting like self-reflection, all showing 0% on Medium/Hard/Extra-Hard tiers.[2]
• •EsoLang-Bench mimics human learning by requiring models to learn new languages via documentation, interpreter feedback, and iterative experimentation, rather than relying on pre-existing training data.[3][4]

🛠️ Technical Deep Dive

• •Each esoteric language has 80 HumanEval-style problems divided into 20 Easy, 20 Medium, 20 Hard, and 20 Extra-Hard tiers, testing equivalent computational primitives like loops and conditionals in unconventional syntax.[2]
• •Evaluated five frontier models (including GPT-5.2, O4-mini, Gemini) across five prompting strategies: zero-shot, three-shot few-shot, self-reflection, self-scaffolding, and agentic feedback loops.[2][3]
• •Success limited to Easy tier only, with best zero/few-shot accuracy of 11.2% on Befunge-98 using self-scaffolding; agentic systems improved 2-3x via feedback but showed no reasoning transfer to harder tiers.[2]

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline