When people refute the idea that "LLM is the new compiler from natural language to code" the common reason is that LLM is non-deterministic but compiler is deterministic.
But the determinism isn't the real distinction. LLM inference can be made deterministic (temparaute zero, etc.). And multi-threaded compiler can be non-dterministic.
And human developers are also non-deterministic. There is always the nondeterminism even before AI.
I think the real difference is how leaky the abstraction is. If you just need to understand language instead of compiler internal to write correct code, the compiler is non-leaky abstrction. If you rely on compiler to do auto SIMD optimization, and have to do hacks to trigger the optimization, compiler is leaky abstraction.
For AI, if you don't ever need to read the code and AI writes perfectly bug-free program that perfectly follows instruction, then AI is a non-leaky abstraction, but that's not the case for now. To make code maintainable and debuggable you still need to read the code if the project is complex enough.
> And human developers are also non-deterministic. There is always the nondeterminism even before AI.
Pointing out humans are not deterministic is irrelevant to the comparison. In the original example of LLMs replacing compilers, the deterministic compiler is being replaced, not the non-deterministic human. In the original example, we move from non-deterministic human -> sourced code -> deterministic compiler, to non-deterministic human -> sourced prompt(?) -> non-deterministic LLM. The important part is the sourced code into a deterministic final step so your build artifact doesn't change.
You can point out you can change temperature to zero or what ever, but then you don't actually get the build artifact you wanted in the first place.
Yeah, it's not the non-determinism; it's that they don't do what I say. I can't imagine writing a nontrivial program without, at a minimum, reading and editing the pieces as they're being produced. People aren't complaining about non-determinism; they're complaining about the results being non-deterministically sometimes wrong.
If the results are non-deterministic, unless I'm happy with literally any possible result, then yeah, I'm going to be unhappy with all of the other results. That's what non-determinism means. What would being right always with non-deterministic results even look like in terms of code output?
That actually would be significantly better, because then you could work out under what conditions they were wrong and correct for it. But being stochastically wrong means you have to check everything, because you never know when the errors will creep in.
If it were deterministic but chaotic, which AFAIK is the case for LLMs running at temperature 0, then you would have to likewise check every time you changed anything and couldn't meaningfully correct for it.
> I think the real difference is how leaky the abstraction is. If you just need to understand language instead of compiler internal to write correct code, the compiler is non-leaky abstrction. If you rely on compiler to do auto SIMD optimization, and have to do hacks to trigger the optimization, compiler is leaky abstraction.
Autovectorization is not a programming model!
I forgot who put me on that article - I think it might've been Horace He - but that is a pretty big idea in idea space for me.
> But the determinism isn't the real distinction. LLM inference can be made deterministic (temparaute zero, etc.). And multi-threaded compiler can be non-dterministic.
And yet, I've never used a compiler that had non-determinism I couldn't turn off, or an LLM product where Icould turn it off. I don't see the theoretical possibility for the opposite behavior as relevant if there are approximately zero examples in practice. In the real world, you can assume your compiler is deterministic and your LLM isn't, and you'll almost never be wrong. That's a real difference that abstraction is not going to fix.
I phrased your view differently in my earlier comments - we already allow "nondeterminism" (or better call it chaotic behavior) in compilers. But usually it is only limited to performance (CPU and memory consumption).
So the "LLMs are chaotic" (difficult to predict) argument is somewhat moot; the performance of GC language runtime or even modern CPU with hyperthreading is already chaotic too.
I think for many applications, people will treat correctness the same way.
This debate conflates two senses of the word deterministic:
1. Repeatable: the same inputs always lead to the same outputs
2. Algorithmic: the answer is arrived at by a set of steps that can clearly be shown to produce output with some desired property.
(2 is not a textbook definition of deterministic that I’ve ever heard, but I think it’s what people mean here)
LLMs don’t usually satisfy definition 1. They could, in principle, but it wouldn’t matter if they did. It wouldn’t tell you anything about whether the outputs are trustworthy.
What people seem to want is 2, but 2 is, I think, a contradiction. There cannot be an algorithmic process that can turns a natural language prompt into a working program, because natural language is too ambiguous to specify an output.
Nevertheless I agree with the main point of the article. Non-algorithmic processes (like intelligence) can produce algorithmic processes (like software). After all that’s what human programmers do!
Yeah, 1, while the correct meaning of "determinism", is just not an interesting property. It's unfortunate that people use the word to mean 2.
2 is an interesting property and I agree that it's not possible for LLMs to possess it because it's a property of natural language itself, not the model. Not possessing it doesn't mean that LLMs can't be used to write software (obviously), but it does mean that using them isn't like using a compiler. This is usually brought up in response to people claiming using LLMs is just another step in the process of increasing abstraction that led from punch cards to assembler to low level languages to high level languages. That point stands: LLM use is qualitatively different to those steps because LLMs aren't "deterministic" in the sense of 2.
There must be a better word for this, but I can't think of one right now.
Orthogonal to the blog, but fun fact: Determinism is crucial for large-scale training runs, and effective RL-runs as it avoids collapse of numerics, but it costs more(about 20% perf) because you need can't parallelize over the batch nor use some specific instructions that dont guarantee reduction order in the way way. Horace He from Thinky has a great blog about this:
Non-determinism doesn't matter, and in fact is desirable, for code generation. But as we scale non-deterministic code generation to truly ridiculous levels, testing that code reliably becomes a challenge. This is where achieving more deterministic code verification absolutely does matter.
It warms my heart to see so many people in the comments clarifying that LLMs can indeed be deterministic. This was not always the case!
I hope that we will soon see deterministic inference offered by frontier labs (presumably at a premium rate). In the meantime, you can get deterministic behavior from local models using the right vLLM flags. I've described some of the cool applications of determinism here: https://lukechampine.com/repligraphs
Is it really possible to get determinism while maintaining good GPU utilization? Parallelism alone introduces enough non determinism that I only see this really working on maybe a CPU?
The important factor is not determinism vs. non-determinism, that is just an implementation detail. The important factor is simply the rate of mistakes or bugs the LLM makes. If it is low enough, the problem is for all practical purposes gone. Is it low enough today, or not? That is the question..
This is what I was telling my programmer friends 5 years ago.
They were saying one cannot replace programmers because any such attempt has led to something as complicated as code, and thus needed programmers.
I was telling them that the same way managers give them vague instructions and they turn out with code, managers could give LLMs vague instructions and they output code.
> I was telling them that the same way managers give them vague instructions and they turn out with code, managers could give LLMs vague instructions and they output code.
I look at those kind of statements and wonder if there’s any kind of programming experience behind them. Managers give vague instructions because they have vague ideas and evaluate implementations vaguely. The people closest to the implementation are the users and the developers. The former needing the help of computers and the latter making the computer able to help the users. The manager is an accidental complexity to the situation.
How does one create a product like SolidWorks, Blender, or Excel? They didn’t came from vague instructions from managers. The users have very specific needs and the developers provide solutions for those needs. Nothing there is vague.
The problem isn't non determinism, but that changing a single token in the input can change the output completely. And you can't put that somewhere in production completely unattended.
It's the uncertainty of "who knows what the machine will output this time" if you have no idea of what the input is going to be.
Exactly. Doesn’t matter as much for coding, where there is a human in the loop at some
point. It does matter if you want to take the human out of the loop.
I'm passionate about and talk about this topic a lot. A common misconception is the idea that temperature or entropy must be 0 when seeded entropy enables both creativity and consistency. Attention mechanisms, layer dropout, initial weights, all of it. It can all be driven by seeded entropy. These systems can and have been served in a deterministic manner, but there are a litany of economic drivers against doing so.
> LLMs, and their respective harnesses, are actors in a system. Code is just an output.
This tracks well with my experiences integrating AI into a software dev pipeline. Sure the output of an LLM is not deterministic in the same sense as a compiler (at least when the LLM has the typical configuration). However, that just means the LLM should not be solely responsible for the final product of your pipeline.
I hate using LLMs as a tool, I never hated compilers, I’m very confident I wouldn’t have hated compilers if I was writing machine code. Some folks seem to like LLMs as a tool, some hate it.
I think the reason it is so polarizing, is whether you like using your brain to problem solve or not. Those who don’t want to use their brain to problem solve and would rather just spend all their time in meetings, talking to customers or whatever else, are ecstatic with LLMs. It lets them skip problem solving entirely.
Those who like to think hard and solve problems, hate it because there’s no problem solving anymore when you use LLMs, and it moves so fast that it is hard to not use LLMs at most tech jobs (outside of managers forcing it down your throat). I never cared at all for customers, I definitely don’t care for shareholders, I only joined tech because I found it interesting to tax my mind to solve real problems and it paid well. Sadly now it only pays well, I think I’ll leave soon, once I make my money.
The advantage of AI coding is that it's non-deterministic. In fact, as you code, your style tends to harden, but when you repeatedly run AI code toward the same goal, you come to think that the non-determinism is actually better.
The essay is conceptually muddled because it ignores all the basic computer science theory that is sorely needed to explain AI and LLMs, but underlying it is a valid issue of what it would take to "derandomize" an AI so it does what we would like it to do.
Unfortunately despite the author's indutrious optimism, this is equivalent to one or more open problems in theoretical CS. For example we don't know BPP = P or not. So are random algorithms tractable? We do not know. If you claim to know the answer, you could win a big math prize...
(It should be noted that despite the title, the author's main argument shows that output non-determinism does matter, since their entire effort is toward mitigating the stochastic behavior of LLMs.)
I'm so tired of the endless stream of articles that a.) don't know what "non-determinism" means in Computer Science and b.) don't really understand how LLMs work.
"Non-determinism" (i.e. the "N" in NP-Complete) does not mean "random", it means "following all paths at once" (there's a really elegant way to understand the List monad as representing non-deterministic computing). It's such a fundamental (and delightful) concept in computer science that refusing to understand what it means is a tragedy related to the loss of technical language itself.
But, even if we just sub "non-deterministic" for "inconsistent" or "random" or "stochastic" or whatever better term you come up with. Statements like the following just seem to not understand the basic architecture of LLMs:
> Given the same input prompt, a model's output is not guaranteed to be identical on every run, at least under normal running conditions.
"Normal running conditions"? The reproducibility of the output is 100% a property of how you choose to sample from an LLM. There are virtually endless ways you can define your sampler and whether or not the same result is reproducible is a fairly minor aspect of all the other choices you make when sampling.
At the end of the day, each step of the LLM gives you a distribution over next tokens, and you can do with that what you like.
In the end the article almost becomes a tautology: the output of an LLM represents a probability distribution over all the outputs it believes are valid with some variation around an expected output. Yes, that's literally what an LLM is.
The concept of determinism is useful in computer-science contexts other than automata theory. See, e.g., https://en.wikipedia.org/wiki/Deterministic_algorithm. It's not incorrect to use the term to refer to those things.
> "Non-determinism" (i.e. the "N" in NP-Complete) does not mean "random", it means "following all paths at once" (there's a really elegant way to understand the List monad as representing non-deterministic computing). It's such a fundamental (and delightful) concept in computer science that refusing to understand what it means is a tragedy related to the loss of technical language itself.
You opened by expressing disdain for articles that misuse the term "non-determinism", then proceeded to give a definition that also misuses the term "non-determinism" in the given context, or at best is a bit too vague for anyone to gain insight from.
Here's a better definition:
An algorithm that can produce different outcomes, for the same input, on different executions.
The "determining" needs to happen elsewhere. Hint, it's a human somewhere. At best it's someone under your employ, or at worst it's some poor sap who's about to have a very bad day.
> One of my fundamental objections to LLM-driven software development was that model output is non-deterministic.
Model outputs are deterministic. Your API provider may be injecting untunable amounts of randomness, but if you're not at the whim of an opaque cloud service, your model can be as deterministic as you please.
The unease that you describe isn't about models not being deterministic (because they are), it's that generating outputs from LLMs is an inherently inductive process (operating backwards by extrapolating from observations and examples) rather than a deductive process (operating forwards from careful logical reasoning founded on axiomatic principles). Deductive processes are more reliable, predictable, and analyzable than inductive processes, but they're also much harder to produce for many classes of problems.
The thing I think about most in this discussion of writing code and determinism is when I assign a ticket to a teammate. I don't expect a deterministic response from them. They're going to make decisions outside of what I would have potentially done, and ultimately what's important is whether the code they write meets the deliverable requirements or not.
And yet so many people expect deterministic responses from llms while they're entirely okay with non-deterministic responses from teammates. It's just not how it's been done in the past to always expect deterministic code generation from humans, and yet we expect that when we give work to agents.
But the determinism isn't the real distinction. LLM inference can be made deterministic (temparaute zero, etc.). And multi-threaded compiler can be non-dterministic.
And human developers are also non-deterministic. There is always the nondeterminism even before AI.
I think the real difference is how leaky the abstraction is. If you just need to understand language instead of compiler internal to write correct code, the compiler is non-leaky abstrction. If you rely on compiler to do auto SIMD optimization, and have to do hacks to trigger the optimization, compiler is leaky abstraction.
For AI, if you don't ever need to read the code and AI writes perfectly bug-free program that perfectly follows instruction, then AI is a non-leaky abstraction, but that's not the case for now. To make code maintainable and debuggable you still need to read the code if the project is complex enough.
Pointing out humans are not deterministic is irrelevant to the comparison. In the original example of LLMs replacing compilers, the deterministic compiler is being replaced, not the non-deterministic human. In the original example, we move from non-deterministic human -> sourced code -> deterministic compiler, to non-deterministic human -> sourced prompt(?) -> non-deterministic LLM. The important part is the sourced code into a deterministic final step so your build artifact doesn't change.
You can point out you can change temperature to zero or what ever, but then you don't actually get the build artifact you wanted in the first place.
Autovectorization is not a programming model!
I forgot who put me on that article - I think it might've been Horace He - but that is a pretty big idea in idea space for me.
And yet, I've never used a compiler that had non-determinism I couldn't turn off, or an LLM product where Icould turn it off. I don't see the theoretical possibility for the opposite behavior as relevant if there are approximately zero examples in practice. In the real world, you can assume your compiler is deterministic and your LLM isn't, and you'll almost never be wrong. That's a real difference that abstraction is not going to fix.
So the "LLMs are chaotic" (difficult to predict) argument is somewhat moot; the performance of GC language runtime or even modern CPU with hyperthreading is already chaotic too.
I think for many applications, people will treat correctness the same way.
https://mbrenndoerfer.com/writing/why-llms-are-not-determini...
1. Repeatable: the same inputs always lead to the same outputs 2. Algorithmic: the answer is arrived at by a set of steps that can clearly be shown to produce output with some desired property.
(2 is not a textbook definition of deterministic that I’ve ever heard, but I think it’s what people mean here)
LLMs don’t usually satisfy definition 1. They could, in principle, but it wouldn’t matter if they did. It wouldn’t tell you anything about whether the outputs are trustworthy.
What people seem to want is 2, but 2 is, I think, a contradiction. There cannot be an algorithmic process that can turns a natural language prompt into a working program, because natural language is too ambiguous to specify an output.
Nevertheless I agree with the main point of the article. Non-algorithmic processes (like intelligence) can produce algorithmic processes (like software). After all that’s what human programmers do!
2 is an interesting property and I agree that it's not possible for LLMs to possess it because it's a property of natural language itself, not the model. Not possessing it doesn't mean that LLMs can't be used to write software (obviously), but it does mean that using them isn't like using a compiler. This is usually brought up in response to people claiming using LLMs is just another step in the process of increasing abstraction that led from punch cards to assembler to low level languages to high level languages. That point stands: LLM use is qualitatively different to those steps because LLMs aren't "deterministic" in the sense of 2.
There must be a better word for this, but I can't think of one right now.
https://thinkingmachines.ai/blog/defeating-nondeterminism-in...
I hope that we will soon see deterministic inference offered by frontier labs (presumably at a premium rate). In the meantime, you can get deterministic behavior from local models using the right vLLM flags. I've described some of the cool applications of determinism here: https://lukechampine.com/repligraphs
https://thinkingmachines.ai/blog/defeating-nondeterminism-in...
(Shameless plug to my ad-free blog: https://www.lelanthran.com/chap15/content.html)
They were saying one cannot replace programmers because any such attempt has led to something as complicated as code, and thus needed programmers.
I was telling them that the same way managers give them vague instructions and they turn out with code, managers could give LLMs vague instructions and they output code.
I look at those kind of statements and wonder if there’s any kind of programming experience behind them. Managers give vague instructions because they have vague ideas and evaluate implementations vaguely. The people closest to the implementation are the users and the developers. The former needing the help of computers and the latter making the computer able to help the users. The manager is an accidental complexity to the situation.
How does one create a product like SolidWorks, Blender, or Excel? They didn’t came from vague instructions from managers. The users have very specific needs and the developers provide solutions for those needs. Nothing there is vague.
It's the uncertainty of "who knows what the machine will output this time" if you have no idea of what the input is going to be.
This tracks well with my experiences integrating AI into a software dev pipeline. Sure the output of an LLM is not deterministic in the same sense as a compiler (at least when the LLM has the typical configuration). However, that just means the LLM should not be solely responsible for the final product of your pipeline.
I think the reason it is so polarizing, is whether you like using your brain to problem solve or not. Those who don’t want to use their brain to problem solve and would rather just spend all their time in meetings, talking to customers or whatever else, are ecstatic with LLMs. It lets them skip problem solving entirely.
Those who like to think hard and solve problems, hate it because there’s no problem solving anymore when you use LLMs, and it moves so fast that it is hard to not use LLMs at most tech jobs (outside of managers forcing it down your throat). I never cared at all for customers, I definitely don’t care for shareholders, I only joined tech because I found it interesting to tax my mind to solve real problems and it paid well. Sadly now it only pays well, I think I’ll leave soon, once I make my money.
Unfortunately despite the author's indutrious optimism, this is equivalent to one or more open problems in theoretical CS. For example we don't know BPP = P or not. So are random algorithms tractable? We do not know. If you claim to know the answer, you could win a big math prize...
(It should be noted that despite the title, the author's main argument shows that output non-determinism does matter, since their entire effort is toward mitigating the stochastic behavior of LLMs.)
"Non-determinism" (i.e. the "N" in NP-Complete) does not mean "random", it means "following all paths at once" (there's a really elegant way to understand the List monad as representing non-deterministic computing). It's such a fundamental (and delightful) concept in computer science that refusing to understand what it means is a tragedy related to the loss of technical language itself.
But, even if we just sub "non-deterministic" for "inconsistent" or "random" or "stochastic" or whatever better term you come up with. Statements like the following just seem to not understand the basic architecture of LLMs:
> Given the same input prompt, a model's output is not guaranteed to be identical on every run, at least under normal running conditions.
"Normal running conditions"? The reproducibility of the output is 100% a property of how you choose to sample from an LLM. There are virtually endless ways you can define your sampler and whether or not the same result is reproducible is a fairly minor aspect of all the other choices you make when sampling.
At the end of the day, each step of the LLM gives you a distribution over next tokens, and you can do with that what you like.
In the end the article almost becomes a tautology: the output of an LLM represents a probability distribution over all the outputs it believes are valid with some variation around an expected output. Yes, that's literally what an LLM is.
You opened by expressing disdain for articles that misuse the term "non-determinism", then proceeded to give a definition that also misuses the term "non-determinism" in the given context, or at best is a bit too vague for anyone to gain insight from.
Here's a better definition:
An algorithm that can produce different outcomes, for the same input, on different executions.
Model outputs are deterministic. Your API provider may be injecting untunable amounts of randomness, but if you're not at the whim of an opaque cloud service, your model can be as deterministic as you please.
The unease that you describe isn't about models not being deterministic (because they are), it's that generating outputs from LLMs is an inherently inductive process (operating backwards by extrapolating from observations and examples) rather than a deductive process (operating forwards from careful logical reasoning founded on axiomatic principles). Deductive processes are more reliable, predictable, and analyzable than inductive processes, but they're also much harder to produce for many classes of problems.
And yet so many people expect deterministic responses from llms while they're entirely okay with non-deterministic responses from teammates. It's just not how it's been done in the past to always expect deterministic code generation from humans, and yet we expect that when we give work to agents.