Why LLMs Seem Random but Still Produce Reliable Results

Results from 700 runs across seven models show why direct answers stay inconsistent while executable methods can still produce reliable outcomes.

The criticism that LLMs are “too stochastic for production use” misunderstands their nature. I ran a systematic experiment to demonstrate this.

This experiment demonstrates the assistant-pattern mechanism on one task. For broader evidence across reasoning models, retrieval-augmented generation systems, and production deployments, see Update for February 2026.

The Experiment

Hypothesis: LLM “unreliability” is a usage problem, not a tool limitation.

Seven models tested:

• GPT-4, GPT-4.1, plus mini variants of each
• Claude 3 Opus, Claude 3 Sonnet, Claude 3 Haiku

100 iterations per model. 700 total runs.

Two approaches to the same task:

Test 1: Direct Question: “How many r’s are in ‘strawberry’?”

Test 2: Code Generation: “Write Python code that counts the incidence of a character in a string. Use it to count ‘r’ in ‘strawberry’.”

Both tests ask for the same count using different requests.

Results

Direct Question

Aggregate accuracy across all models: 25%

Three out of four answers were wrong. The dominant error was answering “2” instead of “3.”

Responses also varied in format: “2”, “two”, “There are two r’s in ‘strawberry’”, and others. The output was inaccurate and imprecise, requiring extra processing to parse.

The models have no counting mechanism; they predict probable tokens from training patterns. My likely explanation is that a model combines word length with English letter frequencies to produce a plausible count. Two r’s in a 10-character word is statistically reasonable. Occasional answers of “3”, “1”, and “4” probably reflect sampling of lower-probability tokens rather than counting.

We cannot inspect closed models to confirm the exact mechanism. The model is estimating rather than computing.

Code Generation

Aggregate accuracy: 100%. Aggregate precision: 100%.

For every model and every iteration, the executed code returned the integer 3 rather than a word or sentence. The output was accurate and precise, and deterministic given the same code and input. Production reproducibility requires recording the execution environment, including versioned dependencies and fixed inputs. The code artifact provides a path to reproducibility that direct answers do not. Answers to Critics gives the full qualification.

The Gap

Direct queries failed 75% of the time; generated code failed 0% of the time in this experiment.

The models and required answer were the same. Changing the request from a direct answer to a program produced reliable, deterministic results.

Code Variability

The code varied considerably, as expected from vague instructions and stochastic transformer outputs.

From the same model with the same prompt, I observed multiple distinct implementations:

# Approach 1: Loop with counter
def count_char(string, char):
    count = 0
    for c in string:
        if c == char:
            count += 1
    return count

# Approach 2: Built-in method
def count_char(s, c):
    return s.count(c)

# Approach 3: List comprehension
def count_char(text, target):
    return sum(1 for ch in text if ch == target)

# Approach 4: Filter and length
def count_char(string, char):
    return len([c for c in string if c == char])

Implementations varied in variable names, function structure, output formatting, and algorithm choice.

Yet every successful execution returned 3.

Variation in Form and Consistency in Function

The hypothesis was confirmed. LLMs are:

• Stochastic in form: They generate different implementations each time
• Deterministic in function: Those implementations produce consistent correct results

Variation in implementation can be mistaken for unreliability, even when the implementations produce the same outcome. The architecture explains why these results were predictable.

Token Prediction and Code Execution

When you ask an LLM to count letters directly, you’re asking it to do something it fundamentally cannot do. It has no counting mechanism. It can only predict the next probable token based on patterns in training data. The model generates a plausible-sounding answer, not a computed one.

When you ask it to write code, you’re using what it actually does well: translation between natural language and formal specification. The code then executes deterministically. The LLM’s stochasticity is confined to the representation; the computation itself is exact.

In neither case does the LLM perform the calculation. A direct response estimates an answer; generated code performs the calculation when executed.

The Architecture

The process is:

1. Human provides intent in natural language
2. LLM translates to executable code
3. Code executes deterministically
4. Human reviews results

The LLM translates between representations. Deterministic code performs the computation and can be inspected, tested, and verified.

Practical Implication

For tasks requiring precision, ask LLMs to generate code that performs the calculation.

Don’t ask: “What’s the NPV of this project?”

Ask: “Write code to calculate NPV given these cash flows and this discount rate.”

The first trusts the model’s pattern matching. The second produces auditable, testable, reproducible output. I develop this distinction further in Oracle vs. Assistant.

The 75-point accuracy gap is not a fluke. It demonstrates the difference between proper and improper use of the tool.

Test methodology: 100 iterations per model across seven models (GPT-4 family, Claude 3 family). Temperature=0 for reproducibility. Full results and code available on request.