The promise of AI-generated models
The promise is compelling. Describe a business challenge in plain language and let AI build the mathematical model that helps you make better decisions. For organisations working on supply chain optimisation, energy system planning, investment analysis, or operational decision-making, this sounds like a natural next step. Large Language Models (LLMs) can already write code, analyse documents, and generate surprisingly sophisticated outputs. It is tempting to assume they can also take over the development of quantitative models. A recent survey published in Transactions on Machine Learning Research offers an important reality check. The paper reviews hundreds of studies on reasoning in Large Language Models and arrives at a conclusion that is highly relevant for anyone relying on analytical models: there is an important difference between generating something that looks correct and reasoning through a problem correctly. That distinction is particularly important in quantitative modelling.
Modelling is not the same as coding
Much of the excitement around AI today is driven by its growing ability to generate software. Tools such as Claude Code can produce substantial amounts of working code, and Anthropic has stated that a large share of the code used internally is now generated by AI. It is easy to look at this progress and conclude that building quantitative models is simply the next step. However, generating code and formulating a model are fundamentally different activities. Code implements a design. A model creates the design.
Before a single line of optimisation code can be written, someone must decide which decisions matter, which constraints represent reality, which trade-offs should be considered, and which assumptions are acceptable. In a supply chain model, for example, there is no single “correct” formulation waiting to be discovered. The modeller must decide how inventory, uncertainty, costs, service levels, and operational realities should be represented mathematically.
This is closer to designing an algorithm than implementing one. A coding assistant can help write a sorting routine but deciding how a new algorithm should work requires a deeper understanding of the problem itself. In the same way, an LLM may be able to generate optimisation code from a specification, but creating the specification requires abstraction, judgement, and reasoning about the real-world system being modelled. That distinction is important because a model can be implemented perfectly and still represent the wrong problem. The challenge is not simply generating equations. The challenge is deciding which equations should exist in the first place.
When plausible is not the same as correct
The survey highlights several areas where current LLMs struggle. One of the most important is maintaining consistency across complex structures. In real-world planning models, decisions are rarely isolated. An inventory decision today influences tomorrow’s production plan. A battery charging decision affects future dispatch options. A transportation choice influences costs, emissions, and service levels elsewhere in the network. These relationships form a tightly connected system. An LLM can often generate each individual component convincingly. The challenge is ensuring that all components remain logically consistent when combined into a larger model. The research suggests that this remains a persistent weakness. The result is that a model may appear reasonable when viewed piece by piece while containing inconsistencies that undermine the overall formulation.
Reference: Song et al., “Large Language Model Reasoning Failures”, Transactions on Machine Learning Research, 2026.
The paper also identifies formal mathematical reasoning as an area where errors remain common. These are not necessarily dramatic mistakes. More often they are small inaccuracies: a constraint applied in the wrong direction, a variable linked to the wrong time period, or a business rule translated imperfectly into mathematics. What makes these errors particularly problematic is that they rarely cause obvious failures. The model still runs. The optimisation still produces an answer. The output may even look entirely plausible. Yet a model that solves the wrong problem can be more dangerous than a model that fails outright.
Perhaps the most revealing finding is that small changes in wording can sometimes produce different implementations of what is logically the same requirement. If descriptions of the same business rule result in inequivalent mathematical formulations, it raises an important question. Is the model genuinely reasoning about the underlying logic, or is it matching patterns it has seen before? The evidence suggests that, despite impressive performance, today’s LLMs still rely heavily on the latter. For organisations using quantitative models to support important decisions, this matters because optimisation models are not simply calculations. They are representations of reality. If that representation is flawed, the resulting recommendations may also be flawed, regardless of how sophisticated the mathematics appears.
The right role for AI
None of this means AI has no role in quantitative modelling. In fact, the opposite is true. It can help translate mathematical ideas into code, generate documentation, explore alternative formulations, and reduce the time spent on repetitive tasks. Used correctly, it allows experienced analysts to move much faster. What it cannot yet do reliably is replace the judgement required to decide what the model should represent in the first place.
At Doing The Math, that distinction is central to how we work. The structure of the model, the assumptions embedded within it, the validation of results, and the interpretation of outcomes are the responsibility of experienced quantitative specialists. AI helps us build faster. Human expertise ensures we are building the right thing. The cited research is a useful reminder that there is a significant difference between generating a model and understanding a system. As AI capabilities continue to improve, that gap might narrow. However, the value of quantitative analysis lies not only in the answer, but also in understanding what drives that answer, which assumptions matter, and how different choices affect the outcome.
Most quantitative models are ultimate decisions support tools. Their purpose is not simply to calculate an optimum, but to help decision makers understand trade-offs, risks and opportunities. That requires judgement, both when the model is formulated and when its results are interpreted.
The more relevant question is not whether AI is being used. It is where human judgement enters the process, and whether it remains present at the moments that matter most.