Is Symbolic Regression Obsolete in the LLM Era?

Traditional Symbolic Regression (SR) Algorithms:

• Primarily based on Genetic Programming (GP), which evolves populations of expression trees representing mathematical functions.
• GP variants include Geometric Semantic GP (operators act in output space), Multiple Regression GP (pools subtrees), Cartesian GP (encodes expressions as acyclic graphs for reuse), and Linear/Polynomial/Sparse GP (constrains model class).
• Other methods encompass Bayesian methods, neural networks, and specialized approaches like Universal Functions Originator (UFO) and Exact Learning.
• Deep Symbolic Regression (DSR) utilizes reinforcement learning to train a generative Recurrent Neural Network (RNN) model of symbolic expressions, often employing a "risk-seeking policy gradient."
• AI Feynman employs a divide-and-conquer strategy, recursively applying solvers and problem decomposition heuristics, alongside dimensional analysis to reduce variables.

LLM-SR Hybrid Frameworks:

• LLM-SR: Combines LLMs' scientific knowledge and code generation with evolutionary search. LLMs propose equation skeleton hypotheses, which are then optimized against data.
• SR-LLM: Integrates Retrieval-Augmented Generation (RAG) with LLMs for incremental learning. It leverages external knowledge bases of human prior knowledge and historical discoveries to guide the search process and uses deep reinforcement learning to assemble symbolic primitives. It provides interpretability by outputting intermediate reasoning in natural language.
• LaSR (Symbolic Regression with a Learned Concept Library): Enhances genetic algorithms by using zero-shot LLM queries to discover and evolve a library of abstract textual concepts from high-performing hypotheses, guiding the evolutionary search.
• IGSR (Influence-Guided Symbolic Regression): Operates with an iterative "propose-and-prune" cycle embedded within a Monte Carlo Tree Search (MCTS). An LLM generates candidate basis functions, which are then evaluated using granular, per-term influence scores to quantify their marginal contribution to generalization accuracy, enabling systematic model refinement.
• COSINE (Co-Optimization of Symbolic Interactions and Network Edges): A differentiable framework that jointly discovers interaction graphs and sparse symbolic dynamics. It incorporates an outer-loop LLM to adaptively prune and expand the hypothesis space based on feedback from an inner optimization loop.

LLM Limitations in Symbolic Reasoning:

• LLMs often struggle with genuine logical reasoning, tending to replicate reasoning steps observed in their training data rather than performing true deduction.
• Performance can significantly decline when numerical values are slightly altered or irrelevant clauses are added to mathematical problems, indicating fragility.
• The concept of "computational split-brain syndrome" suggests LLMs can articulate correct principles but fail to reliably apply them, pointing to a dissociation between instruction and action pathways at an architectural level.
• They exhibit statistical pattern dependence, a lack of internal "world models," and difficulties with causal reasoning, primarily identifying correlations rather than true cause-and-effect relationships.
• LLMs can, however, circumvent some mathematical limitations by generating and executing Python code that leverages specialized libraries for computations.