Write a prompt, evaluate the prompt, understand that is succeeds 95% of the time.
Write a new prompt, evaluate, it now succeeds 99% of the time. Measure what changes between prompt #1 and prompt #2, understand what contributed to the performance jump.
Write a third prompt, this one succeeds 100% of the time. Increase the size of your evaluation set, find a 1/5000 error-class and a 1/10000 error-class, add some explicit code to correct for this cases.
Roll out to production, collecting usage metrics. You make some tweaks to your harness, your prompts. Eventually you have confidence that your system has fewer mistakes than 1 in 100k.
Now, multiply this iteration across all your different prompts and different ways that they might interact with one another.
There's a reason engineers are prissy about people coming along and saying "I write code, I'm an engineer" that people periodically try to sand-paper away.
Engineers don't just tie a sheet to a rock and throw it off a cliff and call themselves aerospace engineers.
They do full diligence on the theory, math, physics, material science, fluid dynamics, etc, and plan a controlled series of tests specifically designed to verify/challenge/disprove their concept and the theories behind it.
Sure, there's a team member ultimately responsible throwing half a dozen rocks off a cliff in the first test.
A technician.
The guy who throws the rock off the cliff is a technician.
The other glossed over part is that the above sounds like science.
Engineering often continues until the concepts and theories are developed into safe, practical methods. "If you stay within these parameters, you can confidently expect these results." The reliability can be codified and reproduced without going from first principles on every application of it.
It's not clear to me that the current AI fad is really developing such reproducible, safe methods. "If you stay within these parameters, you might get these results. Or a teapot. Or some subtly misleading fabrication."
You have to do full due diligence to validate every result. There is safe usage where the hard work was done up front so that day to day practice can skip to boring and reliable application.
To a software engineer, a (current) LLM is a stateless algorithm that performs an idempotent transformation on a large numeric input.
People who think it's a system that thinks and reasons have confused the agentic harness, perhaps forgotten(?) layer0[0] is a seed, the inference engine sets to a concrete value when the caller leaves it as 0.
They probably work on (current) AI software by repeatedly writing prompts like "DON'T READ THE FILES IN /tmp. SOME OF THE FILES IN /tmp ARE VERY LARGE. DUE TO THEIR SIZE, YOU ARE NOT TO READ THE FILES IN /tmp." and wondering why the model becomes obsessed with files in /tmp 100k tokens into every conversation.
"Most classical engineering fields deal with probabilistic system components all of the time. In fact I'd go as far as to say that inability to deal with probabilistic components is disqualifying from many engineering endeavors."
> It's not engineering if you're just guessing as to what is degrading the performance and what might improve it.
Engineering is literally the art of making educated guesses and then testing/proving/disproving/improving upon them. Nothing is exact. Everything is approximate. Iterate until the result is good enough.
If you can identify gradient (what direction your change will impact the ultimate goal), then just repeating the process (or reverse-process) can find local maximum.
Still it can be a software engineering if the gradient candidate / measuring gradient / repeat process can be done at scale.
Things like Voice to Text and biometric unlocks (fingerprint scanners, face ID) have worse success rates and they're used every day by billions of people.
The fundamental issue is, that "we" somehow decided it would be a good idea to throw all the fundamental ideas of computing (determinism, context, separation between data and execution,...) away and try to solve the issues by running a probabilistic/stochastic word generator on top of deterministic circuits instead at 10 magnitude worse efficiency.
It's not a fundamental issue. Determinism and "separation between data and execution" are artificial constructs, make-believe universe in which we design classical code, and a whole lot of hardware engineering goes into allowing us to briefly forget it's all fake.
Real world is probabilistic in practical / metrological, if not fundamental sense, and separation between data and execution does not exist. Our reality does not support such separation.
> a probabilistic/stochastic word generator on top of deterministic circuits instead at 10 magnitude worse efficiency
It's 10 magnitude better efficiency end-to-end, if you factor in design time you'd have to spend to get your "deterministic circuits" (which really aren't, we just paper over it) into shape so they deterministically solve a specific problem, for each problem you want to solve - where with the "stochastic word generator", you just need to change the text prompt.
v = setup()
if v == 1:
side_effect_1()
elif v > 1:
side_effect_1()
side_effect_2(v)
else:
raise Exception()
then we can "refactor"
v = setup()
if v < 1:
raise Exception()
side_effect_1()
if v > 1:
side_effect_2(v)
i know that this might seem "dumb" that the code was ever setup the first way but code can grow into that shape pretty easily. this refactor "removes" the v==1 branch. this new code also follows the "early return" pattern, which improves readability.
to add some context from "the other side": last year, i listened to 2930 distinct albums of which 50% were albums I had not listened to before. Even on a 30 year timeframe and a good discount factor for album repetition I don't think I can fit into the budget you propose without making significant concessions in my discovery process.
Of those 2930 albums, how many can you write down from memory and to how many songs could you sing along? Getting flooded with too much information in a short time isn't necessarily a good thing for your brain. It hurts attention span and memory formation.
The calculation might look like you derive value, but it could also accidentally impact your brain negatively long term.
My response is going to be about Gemini generally and less about NotebookLM.
Google's last frontier model release was Gemini 3.1 Pro, which was in February of this year[1]. At the time, it was ahead of the (at the time) flagship models of Opus 4.6 and GPT 5.2/5.3. From my recollection of the time, it was the best model in the world.
Anthropic released Opus 4.5 Nov '25, 4.6 in Feb '26, 4.7 in April, 4.8 in (late) May. Then Fable in June. 4.7 beat 3.1 Pro on multiple metrics. Fable eats it for breakfast. However, I want to note the 3 month gap between those first two Opus versions.
OpenAI released 5.2 Dec '25, 5.3 Codex Feb '26, 5.3 Instant Mar, 5.4 Mar, 5.5 (late) May, 5.6 July. 5.4 beats 3.1 Pro on agentic benchmarks[2], seems to be similar/losing on non-agentic. 5.5 seems stronger than 3.1 Pro[3].
Gemini 3.5 Pro is alleged to be launching within the week. Why do I type this all out? Because I think Google is getting a bad rap. They are delayed on a frontier release by a month or two and are being regarded as if they cannot release frontier models. I think their last release demonstrates strength and we need to see a weak release before we call them "behind" (in any reasonable sense). These companies swap back and forth constantly. I recall a multi-month span where 2.5 Pro was just the best thing out there by a large margin (in my opinion).
In my experience, Gemini 3.x wasn’t just getting a bad rap, it was significantly worse in practice. It could analyze codebases and report back from a one-shot prompt as good as Claude or Codex but any slightly complex task that carried on for more than a few minutes led to hanging, seemingly infinite loops, and bizarre and nonsensical hallucinations, to the point of being unusable for serious work. The Claude and Codex counterpart models at the time rarely had such issues for the same type and duration of complex work, if at all. To be fair, later Claude especially started having hanging issues as many people noticed but that’s been better recently.
The author explain this in more detail in the article; I also was initially confused. The key details come from their broken down pandoc command. Specifically, these two flags:
--pdf-engine=typst
-V template=article.typ
Despite the input content being a .md file, we see that the .md contents populate the .typ file. Pandoc then understands how to convert the populated .typ file into a .pdf. The author also notes in their hyperlinked .typ document where the body content is placed:
// THIS IS THE ACTUAL BODY:
What is not clear to me is the actual mechanism that tells pandoc where to place the body contents in the template. I presume it's some "magic" from the pandoc docs. The pandoc templates documentation[1] does reveal to us that `-V` is setting a variable called `template`. I don't have pandoc locally, but from another post[2] from the same author and a chatgpt query, it seems that the `body` symbol in the template is the chosen substitution symbol.
Pandoc templates use $...$ or ${...} for variable substitution, yes. body is one of the special default variables: the rest are documented in the manual. If you scroll to the bottom of the template linked from the article, you'll notice a $body$, along with a number of $if(...)$ $endif$ conditionals.
(This actually interferes with Typst's math mode. But you can manually construct math blocks, so no real problem. Pandoc variables are only valid within templates anyway.)
I looked at Luma, but the website shows that for the part of Western Europe I'm in, it only supports Amsterdam or Brussels. Not even Cologne. Or am I using their website wrong?
I can't smell much at all. one time when i was 17 my friend told me i was kinda smelly (i had just exercised). I've been stressed about it since. the human brain can latch on to the oddest things
there is something "wrong" with the ux that is hard to pin down. these things generate even text summaries more rapidly than i can read them. i need a better method for dumping info into my brain + dynamic control (if necessary)
When I take time to read all of the output, I often find that it's mostly noise. I don't like noise so I usually don't bother.
But a person can use subagents, if they want, to filter that down. This burns tokens in a big hurry, but I think subagents can be arbitrary local commands (eg, a local LLM).
Or, you know: Just slow down. :) It doesn't always have to be a race, does it?
Write a new prompt, evaluate, it now succeeds 99% of the time. Measure what changes between prompt #1 and prompt #2, understand what contributed to the performance jump.
Write a third prompt, this one succeeds 100% of the time. Increase the size of your evaluation set, find a 1/5000 error-class and a 1/10000 error-class, add some explicit code to correct for this cases.
Roll out to production, collecting usage metrics. You make some tweaks to your harness, your prompts. Eventually you have confidence that your system has fewer mistakes than 1 in 100k.
Now, multiply this iteration across all your different prompts and different ways that they might interact with one another.
reply