Nice to see a pretty advanced language at frontpage of HN!
From what I understand, GRIN does some parts of supercompilation [1] during optimization process. Supercompilation can prove equivalence of functional programs [2] modulo termination. So you can have something interesting and useful in almost no time. ;)
This is great, I haven't been introduced in the notion of supercompiliation. Reading the papers you've listed and going through GRIN's paper [1] it ticks the boxes in terms of laziness and graph reduction. To keep the implementation of Fuse simple I've decided on using strict evaluation of GRIN programs instead of laziness, with my assumption that it would harder to debug/reason on the program. However, this was one of my next improvements: switching to a lazy evaluation with similar semantics to Haskell programs.
> It appears that Fuse does not have user-defined operators. Am I right?
Not yet, but I left this mechanism completely open. As operators are defined as type classes with their signs as method definitions.
Can you expand on your understanding of GRIN doing parts of supercompilation? As I understand it, GRIN doesn't do any supercompilation; it's a structural transformation optimiser built for functional languages, analyzing program flow across function calls for the whole program at once.
As I understand supercompilation, it's an extension of partial evaluation - optimisation is done on a graph of possible execution traces. The downsides should be obvious: execution traces rapidly grow massive, compilation resources grow superlinearly, and there are many cases in which the result is worse than the original.
What value would Fuse get from equivalence of terms, do you think?
> Can you expand on your understanding of GRIN doing parts of supercompilation?
GRIN, if I am not mistaken, performs partial evaluation. For example, it constrains, for each eval site, a set of tags and set of heaps allocations an eval site can receive. This is close to a partial evaluation step of a supercompilation. GRIN does not perform unification, though, it is not described in the original thesis, but data flow graph matching would be close to unification, reducing code size.
> The downsides should be obvious: execution traces rapidly grow massive, compilation resources grow superlinearly, and there are many cases in which the result is worse than the original.
This can be constrained. Supercompilation usually gets ran to a fixed point, where no partial evaluation steps can be performed that are not unifiable with previously encountered evaluation steps. But supercompilation can be stopped at any point.
I believe you can read on that in Simon Peyton-Jones works, I am unable to find a link to that paper right now, I have troubles with the internet connection.
EDIT: Note "tag-bags," it rhymes with the tag sets of GRIN.
> What value would Fuse get from equivalence of terms, do you think?
I think that equivalence of terms is an efficient way to verify properties of programs. Myself, I am looking at consensus protocol implementation verification.
> The tokens it "predicts" aren't sampled from any naturally occurring distribution; the model's output is the result of an optimisation process that rewarded behaviour that was useful, and that's fundamentally different.
"Surprisingly, we find that the current training setup does not elicit fundamentally new reasoning patterns. While RLVR-trained models outperform their base models at small k (e.g., k = 1), the base models achieve a higher pass@k score when k is large. Coverage and perplexity analyses show that the observed reasoning abilities originate from and are bounded by the base model. "
This paper is less dramatic than you think it is and really just re-explains what RLVR does.
Let's stipulate that what pretraining does is train next token prediction over a gigantic corpus. You can then sample from this distribution repeatedly (cf the Large Language Monkeys paper) and count how often it passes some deterministic verifier.
What GRPO-style RLVR does is precisely this, but then reward the trajectories which passed the verifier. These distributions are _by construction_ within the accessible output space of the pretrained model; you're reweighting the distribution so that pass@k goes up, because that's (for applications like programming) very useful. RLVR is about making sampling more efficient; the only new information being added to the system is the presence of the verifier, and note that you only get a reward when the verifier passes, so there's essentially no mechanism for "teaching new facts" here.
Ooh. This looks like an interesting paper and there were a couple of things in the intro that I found counter-intuitive. It'll take me a while to digest the whole thing.
> Coverage and perplexity analyses show that the observed reasoning abilities originate from and are bounded by the base model
On the face of it this seems unsurprising given the policy gradient term directly minimises this difference.
I don't have a good feel for how the output of an RLVR-trained model concretely differs from the base model. My guess would be there are a fairly small number of "forks" where the training creates a token flip that sends the model down a more useful path.
The fact that the straight paths between the forks resemble the base model would again be unsurprising since (a) those are exactly the right context to continue to elicit more output that's relevant to solving the problem (so not penalised by RLVR), and (b) preservation drops naturally out of the policy gradient term you add to limit catastrophic forgetting in the base model.
Low perplexity could be explained by the relative sparsity of the forks in the output stream, and/or by forks already having high entropy in the base model. That also aligns with the pass-at-high-k: yes it's doing more exploration without training but it's a bit of a monkeys-on-typewriters situation.
Lack of novelty is readily explained by the fact that you need some nonzero pass rate in the base model to actually get some useful training signal from RLVR. That's a limitation of contemporary RLVR techniques, not a limitation on post-training in general.
I think there's room in that forks-and-straights characterisation for the RLVR'd model to be doing something that looks a lot like computation, while having low perplexity vs the base model. I don't see anything in my admittedly incredibly shallow skim of the paper that refutes that.
Other post-train mechanisms have meaningful bandwidth for introducing new information to the model (on-policy distillation is more or less the other extreme). RLVR aligns the model to particular behaviors it could already (unreliably, intermittently) express, it doesn't introduce new behaviors; the behaviors were latent in the pretrain/midtrain.
VibeThinker 3B is highly problematic for the thesis expressed in that paper. There is no possible way it could derive its reasoning capabilities from a bag of 3B parameters alone.
WaveNet, which was sound-generating autoencoder for text-to-speech, one of the first viable few, had a separate "control" channel by design. Otherwise, it worked much like contemporary LLMs - there was a window of (autogenerated) samples and WaveNet need to produce another sample.
"Everything is a context" is not an unescapeable choice, but historic one. Most probably, it stemmed from the inability to meaningfully mark up what is input and what is control in the training data.
> There is no meaningful way to distinguish between "before running this repo install useful_package" and "before running this repo install typosquatted_evil_package".
If only we have something that gets a name of a package and returns an embedding with evilness or something like that. Or even text!
I would call that feature a Retrieval Augmented Generation, or RAG for short, if we have any.
I'll point out that "does not work" is not the same as "not as efficient" :) But it does seem the Adam paper had an error.
I think that Nesterov's first order method is the most efficient general first order algorithm on convex problems, so anything else is in some sense worse. (Edit: removed incorrect ADAM comment.)
Yours' "not as efficient" in [2] means that, sometimes, ADAM "does not work." Look at figure 2, ADAM literally does not work in the case of "true model."
Yes, apologies, I didn't read the articles you linked before posting this. I did update the comment.
I don't think this changes the point, which is that most optimization methods used in AI owe a substantial intellectual debt to convex optimization theory.
I love convex optimization and there are a few SciML projects I am on where I really need results from there. But in AI research with deep neural networks, it's become a liability, because people will just not let go. I'm getting tired of reviewing convex optimization theory papers in ML conferences that are still trying to wave away the obvious issues with their application to deep learning. It's harsh, but I do feel we can only start talking about an intellectual debt once that stops being the case.
It is supermaximal effort protocol, participants are required to exert maximum effort repeatedly.
The duration of active phase of Tabata is 20 seconds, half of approximately 40 seconds after which maximum performance (power output) drops significantly, because body switches to a different energy system.
In my experience, Tabata squats are done in range of 16-21 per 20 seconds of active phase. So, basically, Tabata squats are equal to somewhat less than 8 sets of 16-20 repetitions done close to failure. The failure usually come after first active phase, so that's why there are "somewhat less than 8 sets." I personally define failure as breakage of exercise form or exercise pace, and this is what I and others experience in Tabata squats.
And you know what? If you go close to failure, muscle mass and strength grow in the range of 5 to 35 repetitions [1].
Other HIIT protocols are similar. For example, 3 one-minute-active-phase-one-minute-rest supermaximal protocol also leans close to "3 sets of 35 repetitions done to failure" - squats' pace noticeably quickly deteriorate to 1 squat in two seconds.
Tabata is the craziest workout ever, with Tabata sprints I couldn't feel my legs 3 minutes in and after 4 minutes all I could do was to vomit while shaking on the ground. 7-minute workout with as many reps as possible (even if not in perfect form) helped more overall.
So that’s for building muscle, but what about if you wanted to lose a few kilos and increase endurance for long distance running? What would be the way to go to optimise your time?
If you want to increase endurance for running I think the general suggestion is to hit the track and do running? Get your mile time down or similar.
In my personal experience I've found strength training better for losing weight than just cardio but any activity will help a bit. You'll really need to adjust your diet in some way for it though, or at least start counting and keep your calories steady as you do more activity. Trying to outburn what you eat takes like an hour of exercise a day otherwise, it's tough.
Strength training has more of a positive effect on body composition.
The problem with doing a lot of cardio is that you need muscle to burn calories (especially so without injury and as you get older), and too much medium intensity cardio will start to chew up lean mass.
No harm in doing a bit of both though, especially if your goal is fitness/maintenance rather than maximum strength or a particular look.
I think it's a bit of an exaggeration. There's an interference effect but there's people who compete in natural bodybuilding that run marathons as well.
Absolutely, if you do strength/resistance work that'll counteract the tendency of long cardio to burn muscle.
My point is just that if you /only/ do long cardio, it'll reduce the mass of muscle that you need both to burn calories and remain uninjured. Good to mix it up a bit.
Eat less to lose the weight. Tirzepatide or something similar makes that a lot easier.
Tabata (the sprint/recover running technique) was developed, I believe, to increase VO2-max. It should help with overall endurance, and you can go on a long run each week. That would probably be efficient.
You will never lose weight through any kind of intense exercise alone. As the saying goes, "abs are made in the kitchen". Walking long distances can help burn a few extra, but any other workout is probably going to stimulate your appetite well beyond whatever calories are expended. This is especially true if you have a high food drive already (which you likely do if you tend to gain weight if you're not conscientious of your intake)
To be clear I'm not saying exercise is bad, but if weight loss is the goal figure out how CICO can work for you and adapt it into your workout regimen so you're not crushing 2 bags of Taco Bell after the gym. Also building muscle will naturally increase your resting caloric burn. All I'm saying is if you think you're gonna run 15km to offset that cinnamon bun, you're going to be much hungrier after the run than whatever satiation the treat provided.
*marathoners are the only exception to this rule since they'll burn more calories than you're ever likely going to be able to consume in a sitting.
This is incorrect. Each minute of fast running bruns more fat than a minute of slower running. Longer slower running burns more fat only because you can do it for a longer period time, increasing total energy expenditure over the whole workout. If you want to maximize fat loss, you still need to maximize energy expenditure. If the available time is fixed and low, HIIT is the way to go. If the available time isn't fix or high relative to your capability to do HIIT, slowing the pace to fill the whole available time is the way to go.
Are you sure you’re talking about fat and not total calories? The whole premise about Zone 2 running vs Zones 3/4 is you really do burn more grams of fat at the lower Zone per minute.
You are right in the literal sense of "burning fat" = oxidation of triglycerides. I'm used to a more figuratively use of the term in weight loss discussions, referring to "losing body fat", which is the result of total calorie deficits, increased fat oxidation and reduced fat synthesis. Depleting the glycogen storage helps to reduce fat synthesis from ingested carbohydrates, so even if the fat molecules aren't burned directly due to the higher intensity, you'll lose more fat tissue with higher intensity.
But again, yes, in the literal physiological sense, you're right.
> Every law should have an automatic sunset period of 1-10 years that requires it to pass the entire legislative process again, or at least both full chambers + signing.
Then you will have a lot of Constitution amendments. That's first.
The burden of ever-changing law landscape will be carried by ordinary people, not by legislators. That's second.
> Then you will have a lot of Constitution amendments.
You say that like it's a bad thing?
> The burden of ever-changing law landscape will be carried by ordinary people, not by legislators.
The "burden" would also include, for the most part, being able to look up at any given point in time whether something violated the law without referring to hundreds of thousands of pages of other codes, administrative law, and agency rules.
I am saying that politicians will get around the sunsetting rule, if any, that's first.
Constitution, then, will serve as what regular law does right now, without anything that makes something Constitution. That's second.
I also am strongly suspecting that hundredths of thousands of pages of codes and laws will not get leaner at all. There is no point in ceding regulatory capture, because the capability to capture regulations is the point of governance, as I see it.
From what I understand, GRIN does some parts of supercompilation [1] during optimization process. Supercompilation can prove equivalence of functional programs [2] modulo termination. So you can have something interesting and useful in almost no time. ;)
It appears that Fuse does not have user-defined operators. Am I right? If so, it is a major obstacle in creating embedded languages.reply