Human-in-the-Loop LLM Limitations Experimental Testing

Where the LLM Needs a Human: The Limits of AI-Driven Experimental Testing

An LLM wrote every line of code for a smoke-tunnel experiment, and still produced a string of confident, plausible and wrong results that only human judgement caught. The story of what went wrong, and why it matters for every laboratory.

⏱ 5 min read June 2026 Post-mortem Vision · CFD · Systems

In an earlier piece, an LLM ran a complete OpenFOAM aerodynamics study of a bear. In a follow-up session, the same model was asked to check that simulation against reality, filming smoke around a 3D-printed bear in a small wind tunnel. The simulation story read as a triumph. This is the other half. Along the way the model produced a run of confident, plausible and wrong results: a drag coefficient that was really a different quantity, an airflow “measured” as zero, a model badly mis-sized from a photograph, and a flow-direction “agreement” that was pure noise. Every one was caught by a person who knew roughly what the answer should be.

None of this means LLMs have no place in the laboratory; they are a force multiplier. It means that experiments have no safety net. No compiler rejects a wrong unit, and no test suite flags a misread column. Human attention is what holds the work up, and even the simulation used as the yardstick can be wrong. Only a careful, human-designed physical measurement is true ground truth.

On method: every error described here is real and taken from the live session; the figures reproduce the actual intermediate and corrected outputs.
01 The Other Half

Success stories hide the interventions

When an LLM-run project is written up, the story tends to flow straight from prompt to polished result. The corrections in between disappear, and the reader is left with the impression that the model can be trusted by default. In experimental work, that impression is dangerous. Software has guardrails; experiments do not. A type error refuses to compile, but a column read from the wrong position simply produces a number. A velocity field computed from a misunderstood image produces a velocity field, not a crash. The output looks exactly like a correct result, and the only thing standing between it and a published conclusion is a human who knows what to expect.

“In code, the machine tells you when you are wrong. In an experiment, only physics does, and physics does not raise exceptions. It quietly lets you publish the mistake.”

02 The Ledger

Six errors from one validation

The figure below is the actual ledger of corrections from the session: each output the model would have reported, why it was wrong, and what the human changed. The pattern is striking. Every failure was silent. Each wrong value rendered neatly into a plot or a table, and none looked odd on its own. Each was caught only because someone had outside knowledge for it to collide with.

Correction ledger: LLM outputs vs corrected values
Fig. 1 The correction ledger. Six independent stages produced confident, well-formatted, wrong output; none threw an exception.

03 Vision Is Fragile

When seeing is not measuring

The most stubborn failures were visual. Every measurement depended on finding the bear in the frame to set the scale from pixels to metres, and the model’s detection missed, repeatedly and by wide margins. Smoke hid the body, the polished floor threw back reflections, and background clutter confused the thresholds. One attempt merged the bear with its own reflection; another returned a wide, flat band no bear could ever be. Every downstream “measurement” inherited those errors without complaint. The fix was not a cleverer prompt. It was a person imposing a known reference scale by hand.

Front-view optical flow analysis panels
Fig. 2 The front-view analysis that finally worked, meaningful only after the model’s scale was fixed by hand against a gridded reference. The earlier automated attempts were not.

The deeper lesson is about fluency. An LLM can describe an image beautifully, and that eloquence invites trust. But a fluent description is not metric perception. Reliably picking out a hidden, reflected, low-contrast object and returning its true size is something the model simply cannot be relied on to do.

Where vision breaks

Smoke, reflections, poor contrast and clutter defeat automated detection, and they are exactly the conditions of real flow-visualisation footage. Treat any dimension or calibration an LLM derives from a photograph as a hypothesis to check against a physical reference, never as a measurement.


04 Confidently Wrong

Numbers that look right and aren’t

Consider the drag coefficient that wasn’t. The model read the wrong column of the solver’s force output and reported the pitching-moment coefficient as drag, the most ordinary off-by-one slip imaginable, producing a perfectly sensible-looking number for the wrong physical quantity. It was caught only because an earlier study had already established what the drag should be.

Then came a freestream of zero. The first attempt to extract the airspeed reported no upstream flow at all, in a tunnel visibly full of moving smoke. The cause turned out to be real physics: steady smoke streaklines form a standing pattern that barely changes from frame to frame even as the air races through it, so comparing frames correctly finds nothing. Recognising that “zero” as meaningful rather than a bug took knowledge of fluid dynamics that no code could supply.

Finally, the reproducible artefact. A comparison of flow direction against the simulation came back almost perfectly opposite: stable, repeatable and entirely spurious, because it was measuring the direction of a near-zero noise signal. Repeatability is not validity. The right response was to throw the metric away and use one suited to the physics, which then confirmed the position of the wake.

The reproducibility trap

An LLM will happily compute, plot and caption a statistic without asking whether it means anything. A stable wrong number is more dangerous than a noisy one, because it survives the usual sanity checks. Deciding whether a metric suits the data is a human responsibility.


05 Systems Integration

Complex systems need a human architect

Beyond the individual numbers, the validation stitched together a video pipeline, several optical-flow methods, simulation output and coordinate systems that disagreed with one another. Each piece worked; the whole kept failing. The image-processing code was correct but blind to how much memory it needed, and the operating system killed the run. The model had to be told that a top-down image of smoke sums the flow over every height, so comparing it with a single slice of the simulation is a modelling decision, not a coding one. And when the comparison came back inconclusive, the honest course, reporting the null result rather than tuning a threshold until it agreed, was a matter of scientific integrity the model does not enforce on its own. Someone has to hold the whole system in mind.

Wake comparison and resolution-limit analysis
Fig. 3 The analysis that should have come first: at this frame rate the smoke crosses the frame many times between captures, so its speed simply cannot be recovered. The model raised this only after being pushed.

06 The Simulation Isn’t Truth

The CFD can be wrong too

So far this reads as a story about catching the model’s mistakes by checking them against the simulation. That framing hides the deepest trap of all: the simulation is not ground truth either. Its turbulence model is known to over-predict separation around some bluff bodies, a steady solution cannot capture unsteady vortex shedding, and the predicted drag depends on choices of turbulence model, wall treatment, mesh and domain. Change any of them and the number moves. A simulation is a carefully reasoned hypothesis about the flow, not a measurement of it.

“When a crude experiment agrees with a simulation on the shape of the flow, it is tempting to call the simulation validated. But two methods can share the same wrong answer. Agreement is not correctness.”

The smoke tunnel could confirm where the wake sat, but it could not measure a single speed, so it never tested the simulation’s numbers. Breaking that circle takes a measurement that is both independent and quantitative: matched flow conditions, calibrated instruments with known uncertainty, enough speed to capture the physics, and controlled, repeated runs. Each of those is hands-on, judgement-heavy work that cannot be handed to a model that cannot feel a misaligned probe or notice a blocked pressure tap.

Neither the code nor the simulation is ground truth

The model’s analysis can be wrong. The simulation it is checked against can be wrong, and the two can even be wrong together. The only real arbiter is a careful, human-designed, precisely instrumented physical measurement. That experiment is not a formality at the end; it is the foundation.


07 A Working Division

Keeping the human in the loop

The answer is not to do less with LLMs. It is to put human attention where the risk of failure is greatest, and across this validation a natural division of labour emerged. In setting up, the model handled the boilerplate, the tool calls and the file wrangling, while the human decided what could be measured at all. In calibration, the model proposed detection code and the human imposed a known physical reference. During the runs, the model executed the tools and parsed their output while the human budgeted resources and sanity-checked the intermediate results. In analysis, the model implemented metrics quickly and the human judged whether they meant anything. And at the end, the model drafted a clear narrative while the human reconciled it with the physics and insisted that null results be reported as such.

  1. 1

    Anchor every number to an expectation before accepting it.

  2. 2

    Check calibration against a physical reference; never let a dimension taken from an image stand unchallenged.

  3. 3

    Establish what is measurable before measuring, rather than discovering the limits by surprise.

  4. 4

    Question the metric, not just the value; stability is not meaning.

  5. 5

    Let nulls be nulls; protect inconclusive results from being tuned into false agreement.


08 Conclusion

A powerful tool, not an autonomous scientist

The companion study stands, and its final results are sound. But they are sound because a person caught the misread coefficient, recognised the zero airspeed as physics, fixed the visual calibration, discarded the meaningless correlation, budgeted the memory and reported the inconclusive comparison honestly. Take that person away and you do not get a slightly weaker paper. You get a confident, beautifully formatted, wrong one.

The model drives the cost of implementation close to zero, and that is transformative. What it does not reduce is the cost of judgement, and experimental work is dense with judgement in a way that pure simulation is not, because it lives where abstractions meet a physical world that never checks your inputs. The most valuable skill is no longer writing the analysis code. It is knowing, at every step, roughly what the answer should be, and refusing to accept the one in front of you until it has earned your trust.

“Use the model for everything it is good at, which is a great deal. Just never confuse the fluency of its output with the correctness of its result. In the lab, those are different things, and only you can tell them apart.”

Companion study: How LLMs Can Run a Complete CFD Study, the simulation whose validation is dissected here.