There's a good amount of literature about this (check the other comments), but you can vastly simplify this into two things you need to do:
1. Your service that retries should have some retry budget. This is a good place to be "smart", because you can reason entirely locally instead of turning it into a distributed systems problem. The best library I've seen for this was doing Exponential Moving Average of requests per second sent down that pipe (not counting retries) and only allowing 20% more requests per second as retries, total. Each individual request could be retried 3 times. This was critical as it bounds the additional load from retries.
2. Whenever a service retries but has to give up, the error it sends to its callers should never be retried. There has to be some agreement that that HTTP code will never be retried. This prevents the multiplicative factor of retry on top of retry, which is why those storms can generate so much load.
Everything else is nice-to-have, but those two alone should bound the total requests you get in a retry storm.
> 2. Whenever a service retries but has to give up, the error it sends to its callers should never be retried. There has to be some agreement that that HTTP code will never be retried. This prevents the multiplicative factor of retry on top of retry, which is why those storms can generate so much load.
This can't really be tolerated in practice, though, because it means that one bad component somewhere in your stack, one that is able to accept and respond to requests but for whatever reason isn't able to make requests to its backends, poisons the whole stack. You can't take one backend's word for it that the failure is not localized and therefore retryable.
> 2. Whenever a service retries but has to give up, the error it sends to its callers should never be retried. There has to be some agreement that that HTTP code will never be retried. This prevents the multiplicative factor of retry on top of retry, which is why those storms can generate so much load.
Ooh, I like the idea of propagating "no retries" hints in the responses back upstream. Have you seen it implemented in the wild, or in public discussions about the practice?
In gRPC, the statuses it returns in trailers can include arbitrary details, and Google has a well-known proto for common ones in `google/rpc/error_details.proto`. One such detail is RetryInfo [1].
When we implement retries where I work, the general rule is that if a request is suitable for retry, it should include the RetryInfo in the error status and use it as the base delay for the exponential backoff. The absence of that detail means don’t retry, and we have a client interceptor that parses the response status and retries according to that logic.
I get a feeling of dejavu for this one. Most of my experience has been in Java and in most places I worked in the past we had this hierarchy of exception classification which gets reflected into the http status codes as well. On high level the HTTP status codes in case of errors are already classified as re-tryable or not, the convention varies globally but can be adopted in a standard manner within a company.
The reason why I brought up the exception propagation is cause within a large enough service with multiple layers of depth the exception hierarchy provides with similar context.
The hard part is not implementing something like this, its about maintaining it consistently across every new change. With small product teams this architecture concept/convention/constraint can easily get lost/forgotten and what you are left with is a theoretical system which does not works as desired when the storm comes
Not GP, but IME it's not fixable by model selection, but being zealous about guiding output and vision, and pushing back on all the bad habits LLM in general has (eg verbose output as a band-aid for emergent intelligence). As soon as something is introduced into your codebase, it will continue being picked up into context until you remove it and any reference to it from any potential context entrypoint. If you don't any model will keep venturing down wrong/bad paths.
I also read that comment as an adversarial situation at work.
It used to be that when someone else at your company was asking for something that wasn't a priority, you would erect bureaucratic roadblocks to protect your time. Now, the new normal is to just forward their questions to AI and sling the slop back over to them.
I was bothered by the writing and fought through it, and carefully went through everything in the article... and you didn't miss anything. It's a tweet expanded to article length.
If you can actually time the market, sure. I cannot, so I don't pull money out of my stock indexes; I just send a larger fraction of my new investments into bonds.
I've been telling people the same thing. The tech industry has been cargo-culting "scaling the organization" where people are rewarded for org size. That zeitgeist has ended, and we've not even overcorrected much the other way.
> the CPUID example in the article [...] forgot to bind ECX as an input.
I'm not really familiar with this stuff, but the example uses what it calls a "pin" (which in their docs is a type of "binding") on ECX before calling CPUID.
You don't really need to be familiar with either "this stuff" or Odin to spot that this clearly takes a single parameter named "leaf" and that's the input, which goes in EAX. However CPUID may care about ECX as input and that's only used as an output in this uh, "template".
Rust provides this for both x86-64 and the original 32-bit x86 and this is a function, not an Odin-style "template" but hopefully this helps show what you're supposed to do.
I'm not sure the example template was supposed to be canonical, vs demonstrating multiple output destructuring. Certainly the actual instruction definition in the checker library seems to understand that there could be two possible inputs:
But I'm also not entirely sure why the example should not have compiled. It seems to me that the idea here is to be able to define a typed set of something equivalent to a function that inlines some assembly, but nothing about that inherently requires that the number of input or output parameters to the template match the parameters in the underlying assembly calls. There's no reason (in my mind anyway) why this shouldn't be a perfectly valid template:
// Returns the extended feature flags obtained by calling CPUID
// with EAX=7 and ECX=1
cpu_extended_feature_flags :: asm() -> (a, b, c, d: u32) [
a = %eax,
b = %ebx,
c = %ecx,
d = %edx,
] {
mov %eax 0x7
mov %ecx 0x1
cpuid
}
> But I'm also not entirely sure why the example should not have compiled
The source you presented seems fine - it's explicitly setting the register. The trouble with the cpuid definition in the article is that it just doesn't set ECX at all
It does actually seem as though Odin is intended to notice this problem but maybe is fooled that the register is pinned (because we want its result value) and so the diagnostic doesn't trigger.
Reading this code reminded me that my annual summer leave ends this weekend because it would get so much review feedback if he worked with me. "Commenting out" blocks of code is NOT OK and neither are "if (false)" blocks.
Sure, I agree that if you're making a CPUID inline assembly template that isn't setting the register value, and it accepts any values for EAX that would also cause it to read ECX then the behavior is going to be undefined and likely unexpected, but that doesn't seem to be a reason for this not to compile.
For one, requiring an input parameter when it isn't mandatory would mean that you're spending cycles setting a register with a value that you don't need and is just going to be overwritten anyway. A good number of the CPUID calls never read from ECX, and if you're going to call any of them, then the value in ECX is irrelevant. And sure, it's only an extra instruction or two, but presumably if you're dropping down to inline assembly, you kind of care about every wasted instruction.
// Returns the maximum input value for basic CPUID information
cpu_extended_feature_flags :: asm() -> (a, b, c, d: u32) [
a = %eax,
b = %ebx,
c = %ecx,
d = %edx,
] {
mov %eax 0x0
cpuid
}
I admit I never touch inline assembly or really assembly at all save for the occasional microcontroller project, so maybe I'm missing something that's obvious to people more familiar with this. But to me the article seems to be sayin that the Odin templates will type check and validate that IF you have inputs from Odin types that are being put into registers or used as operands to your assembly, or are mapping registers and outputs from your assembly back to Odin types, that those mappings will be type compatible, and when you get those mappings wrong, you'll get a more useful error. But I didn't read it as saying that it will prevent you from writing assembly that does something completely unrelated to those inputs or outputs.
For CPUID specifically this isn't going to be on a fast path. As its name might suggest it's a way to have the runtime CPU tell us what features are available so that we can adopt different implementations of perf-sensitive routines for that CPU.
So e.g. maybe the CAD software executes a dozen CPUID instructions during startup, and then based on those it uses AVX512 vectorized operations later on some hardware but uses SSE instead on other hardware.
I haven't read Intel's manual for a modern x86-64 CPU, and I certainly haven't read the community notes about this stuff which would tell you if, despite the documentation you need to behave differently, but my assumption would be that everybody clears ECX if they don't want a non-zero subleaf and that this is either known to be necessary or an obvious way to avoid nasty surprises in code that is never perf critical.
I think we might be talking past each other here. Whether the assembly op is `cpuid` or something else, isn’t really my driving concern. The comment that started this thread (which I’m pretty sure wasn’t yours, so I don’t necessarily expect you to have an answer) was that they didn’t think this example should have compiled because `cpuid` may read from a second register as part of its operations, depending on the value passed to the first register.
But that seems to be saying that the inline assembly template should have the same shape as the underlying assembly calls that are made. If the underlying call might read 3 registers, then the complaint appears to be that the templating system MUST require a template that calls that assembly to also have 3 inputs, even if it won’t use all 3.
My thinking on this is the templating is a syntax for function declarations where the function is assembly code and not more higher level language code. We don’t require that functions have the same number of inputs and outputs that the code called within the function has in order for it to compile, I don’t see why it would be necessary for the assembly templates to work any differently. I get that `cpuid` specifically isn’t likely to run in the hot path of any code, but the underlying principle is the same. If you’re dropping to assembly, you likely have some performance tuning you’re trying to do. A templating system that requires your template inputs to 1:1 map to all the possible inputs of all the assembly calls you use in the template, regardless of wether you need or use them is adding extra instructions and waste for (to me) no obvious benefits.
> A templating system that requires your template inputs to 1:1 map to all the possible inputs of all the assembly calls you use in the template, regardless of wether you need or use them is adding extra instructions and waste for (to me) no obvious benefits.
But we're not suggesting this ridiculous and arbitrary restriction. What we're suggesting, and in fact I think what Bill actually intended in Odin, is something much more useful.
A templating system that requires all possible inputs of all the assembly calls you use in the template are well defined.
What you seem to want is exactly the thing Bill doesn't like about existing functionality. You can cheerfully ADD two registers together and then use the result but without ever determining what's in those registers. What is the result? -shrug- might be anything.
What Odin seems to intend (but this CPUID example seems to suggest is buggy) is that it will check you've actually written code which means something. You do not need to make those registers inputs of your template, you just need to make sure they're well defined. For example you could set them directly in the template, as with your leaf 7 sub-leaf 1 example.
That (modulo bugs) is a big improvement for this particular corner of the language.
I'm starting to get a better handle on what you're getting at, but I still think I disagree that the code shouldn't compile. To really settle this I guess we would need Bill to weigh in on his intent, because I read this article about being all about types and type checking. He says:
However every instruction has a set of valid forms. Each form dictates the
kind of each operand (register, memory, immediate, label), the class of
each register (general-purpose, vector, mask), the width of each operand,
the range each immediate may take, and what the instruction clobbers
(flags, memory, particular registers).
...
That is not the absence of a type system: that is a type system; a rather
rich, dependent, per-instruction one.
...
The params are your inputs, as plain names with Odin types. The results are
your outputs, again plain names with types, sharing the same signature
syntax as an Odin procedure.
...
The params and results are optional, like with a normal procedure type, and
the bindings are completely optional if they are not necessary.
That last part in particular is important here, bindings are optional if they're not necessary. Well if the "types" for any given ASM instruction come from their valid definitions, then we'd have to look at the intel instruction definitions[1].
Some instructions like ADD[2] (Vol 2A 3-14) are defined to have multiple forms and those forms each take 2 operands. So the type of the instruction would be `ADD T-OP1, T-OP2` for each of the possible combinations of operand types for the ADD forms. And the templating system and compiler would validate that if you pass a given parameter of a given type to one of those operands, that your types match up.
Some like ANDPD (Vol 2A 3-63) have two forms, one that takes 2 operands and one that takes 3. The 2 operand one says that the first operand is a read/write register, where as the 3 operand forms say the first operand is write only. Presumably if you took an input and bound it to the first operand in a 3 operand form, the compiler would at least emit a warning about this if not an error.
But now if we look at the definition of CPUID (Vol 2A 3-203) the only valid form it has is a form with no operands. So if we were defining a type for the type system to compare against, the only possible correct answer (to me) is `cpuid` with no operands. That is the type system should happily allow inline assembly that calls `cpuid` but not inline assembly that calls `cpuid %eax %ecx` because that's not a valid form of the instruction.
Further the definition explicitly says that for some values of EAX, the value in ECX would be ignored entirely, and for invalid values in either field, the output values are all "Reserved". That tells me that as far as a type checker is concerned, any invocation of `cpuid` as long as there are no operands is a valid invocation. A type checker doesn't check the values of the fields being used, and IMO any argument that since it could read ECX, then the type system should enforce you set a value it to it can be equally countered by an argument that since cpuid will explicitly ignore ECX with certain values, it should never require you to set ECX because that might cause you to set it with a value and get an unexpected result because the value you provided to EAX was one of the values that causes it to be ignored. In either case we're asking the type checker to help us prevent a logic bug, not a type bug.
You might say that the "always require both fields to be set" is at least an easy check that could be applied universally, then the question would become how would it intersect with multiple invocations of CPUID? If you invoke:
mov eax, 0x0
mov ecx, 0x0
cpuid
cpuid
that is valid and the outcome of that should be (assuming I'm reading the documentation correctly) that after the first invocation, EAX would contain the maximum valid value for EAX when invoking cpuid, and after the second invocation it should contain the results of invoking cpuid with whatever that value was (and obviously whatever was in ECX after the first invocation). If the purpose of the check is to prevent invocations where ECX might contain an arbitrary value that makes no sense, we'd have to mandate that there's some additional steps in between the two cpuid invocations to reset EAX and ECX. Otherwise all we've done is make it possibly even more confusing when ECX changes after the first invocation, but the code strongly implies it should be 0x0.
Given the stated goals in the article seem to include not requiring the explicit statement of implicit behavior, requiring setting ECX for instruction that will ignore it would seem counter to the goals.
As a side note, I'm curious why (based on your comment about ADD), it would be desirable for a type checker to want to prevent you from doing something like this:
It's perfectly valid to not always care about the starting value in a register even if you're going to use it. Plenty of pseudo-rng type code has read arbitrary registers or addresses as a source for some of their calculation without ever caring what the starting value was. I feel like in some way this gets to the heart of what we're disagreeing about. I read the article as saying "within the bounds of the shape of assembly code as defined by the ISA, odin templates can help enforce types for those shapes and help make wiring normal types to registers for input and output easy. I feel like what you're saying that in addition to that, it's also supposed to help stop you from doing things that are likely to give you non-sensical results.
> It's perfectly valid to not always care about the starting value in a register even if you're going to use it. Plenty of pseudo-rng type code has read arbitrary registers or addresses as a source for some of their calculation without ever caring what the starting value was.
No. Never do this. If you don't care and just want random numbers use for example RDRAND, if you do care, design a proper PRNG with your preferred characteristics and seed management. YOLO programming is a bad idea always but it's especially bad when applied to assembly.
To underscore what this software is for let me quote you a diagnostic it emits:
{instruction} implicitly reads {register}, but nothing in this template produces a value for it; pin an input parameter to {register}, or write {register} before this instruction
I agree you shouldn't do it, that doesn't make it invalid to do.
> To underscore what this software is for let me quote you a diagnostic it emits
Well, then I stand corrected on the intent. That pretty clearly spells out that the template checker will validate that (at least for non-branching assembly) all implicitly read registers are given an explicit value and error if not. So that should mean the example in the article actually would not compile, since the rexcode definition lists both RAX and RCX as `implicit_rd`.[1] I appreciate you finding that bit of validation code and persisting in helping me learn something new, this isn't an area of coding I spend a lot of time in so it was instructive.
[1]: Does it compile? I don't have any odin dev stuff set up, but I might just try it in the next little bit just to see.
I don't run nightly Odin. I do happen to a have a non-nightly Odin installed because I was wondering if some of the trash in their kitchen sink library is in fact trash (it is) or whether it's clever in a way I didn't understand ‡
However my reading of the implementation is that:
1. Bill over-sells the value of rexcode. rexcode is Odin code, so the actual "value" derived in Odin's own compiler is just that it scrapes the data out of rexcode. That's not nothing but it's not much.
2. Bill's new template feature doesn't remember that we can pin an output, so it notices that ECX is pinned and concluded it is safe for CPUID to read it. That's a very small bug, and it's even possible I've misunderstood it, but that's my reading.
‡ Odin provides a lot of sorting algorithms. I wanted to measure how fast they are, which I did by comparing against a Rust install on the same toy machine, all of them are much slower than Rust's built-in sorts, but interestingly the provided "slice sort" which is most analogous to Rust's [T]::sort and [T]::sort_unstable is a "Smooth sort" which is relatively obscure and actually is only as slow for pre-sorted input as my Rust sorts are for unsorted input whereas most of what's provided is way slower even for pre-sorted input.
1. I don't think I am overselling rexcode, rather there isn't anything like it in the first place as a single library with that many ISAs and IRs. That's what I love about it, since I can now trivially make (non-optimizing) compilers without needing to use LLVM or use another external tool (including outputting C or assembly). Also all of those tables become binary blobs, so you could just use them in your favourite language any way.
2. That was a bug and it has now been fixed, along with many other analysis bugs.
3. The sorting algorithms are getting an overhauls soon to be a lot faster.
It is an easy to overlook this, but even for someone in position of power to change, creating different code with the same hash is borderline impossible.
Non-sequitor? They're not providing a (sha-1) hash, they're providing source code to integration partners using their business channels, not public git providers. Those business channels include contracts etc to "secure their supply chain".
You and I aren't in those business channels, and we're not being given anything with a hash. There's simply no hash to collide with?
Interestingly, this was tackled in this blog post[0] a month ago. They claim that plan files aren't token-efficient, because after reading the plan the workhorse model then reads all the relevant files anyways.
That link just says the planning stage should vet the idea concretely so that the plan focuses on a solution that won’t immediately have to pivot.
And I think plan files should focus on general ideas and invariants, not do “implementation as prose”. That way they perform as mini-ADRs that are useful historically, especially to mine why the system is the way it is.
My plan and implementation files are task specific (so specific workhorse reads only its own slice), and the workhorse itself is spawned from an orchestrator with a very specific small prompt.
I managed even the orchestrator to NOT read the plan whole, at once, but in sections.
The most useful thing is the task ledger the task agent leaves behind, which alongside its structured status message makes a very resilient handoff between all stages.
1. Your service that retries should have some retry budget. This is a good place to be "smart", because you can reason entirely locally instead of turning it into a distributed systems problem. The best library I've seen for this was doing Exponential Moving Average of requests per second sent down that pipe (not counting retries) and only allowing 20% more requests per second as retries, total. Each individual request could be retried 3 times. This was critical as it bounds the additional load from retries.
2. Whenever a service retries but has to give up, the error it sends to its callers should never be retried. There has to be some agreement that that HTTP code will never be retried. This prevents the multiplicative factor of retry on top of retry, which is why those storms can generate so much load.
Everything else is nice-to-have, but those two alone should bound the total requests you get in a retry storm.
reply