X posts — 2025

I've long believed that AI image generation has much bigger societal costs (loss of trust in the image media, very effective scamming) than benefits (AI images in blog posts, indie movies...?). I admit I didn't anticipate the generation of diagrams, which is a nice benefit (not nice enough to make the whole net-positive though).

For instance, here's a modest one I generated after learning about the Shapley value. I had this diagram in mind and I wanted it to exist, so I could add it to my Anki card. Gemini 3 Pro created it for me

@datagenproc link??

@TutorVals for now, just regular LLMs that tend to remove the parts they find unusual, or add well-meaning "fixes" that create bugs. But yeah in AI control setups one could also consider misaligned models

LLMs are increasingly used over copy-paste for refactoring tasks like moving a block of code. They aren't as reliable as copy-paste, and the git diffs for moved code blocks are very difficult to read, so this may become a fruitful area of vulnerability research.

as a toy example, the new code in this diff is slightly different (in this case, an additional comment), but it's not highlighted in the git diff

LLMs creating technical debt while solving the current problem seems underrated. Are there coding benchmarks taking this into account? Could be implemented e.g. by not asking for a single change, but repeated changes on a codebase as it grows in complexity.

Thanks for the clarification. If expert choice / token choice with buffers aren't used at inference (i.e. there are no possible buffer overflows), then I agree that your methods should make sparse MoEs fully deterministic too, which is great.

Thinking about it, the hypothesis that "GPT-4 has higher logprob variance because it's a sparse MoE" could be right, even if the sub-hypothesis "and it's because of buffer overflows" is wrong, due to the gating networks acting as variance amplifiers (a small activation difference can result in a different set of experts). I've been tracking logprobs from a number of models and I was under the impression that sparse MoE models indeed have higher variance, but I haven't looked into it that much

So by the same token, is "Buffer Overflow in Mixture of Experts" (https://arxiv.org/abs/2402.05526v1) wrong too (as in, nobody does inference that way)? I don't think they say the experts are on different GPUs, but they do say "An uneven assignment of experts is generally an undesirable property as this results in under utilization of some experts. To partially mitigate this issue, a buffer capacity limit is usually set for each expert"

@cHHillee @thinkymachines Interesting, how is the blog post incorrect? To me its conclusion seems intuitive if you consider a sparse MoE model with fixed-size buffers in front of experts, to distribute the load evenly across the experts (which will be on different GPUs for large models)...?

@thinkymachines Great post, however I think there should be a disclaimer about sparse MoE models: they result in higher non-determinism than batch size, and your solutions don't apply to them. See: https://152334h.github.io/blog/non-determinism-in-gpt-4/

I'm impressed by the results, but I don't think AlphaEvolve is a big deal, because it's only an alternative to RL in very narrow settings: the range of tasks that can be expressed as "solution of a standard program" and "cheaply and reliably graded" seems small. For instance I tried to think of ways it could be useful in cyber, and couldn't come up with anything. I think it's telling that many examples where it surpassed SOTA are complex and boring math problems, like the packing results.

Great post. The flip coin of "more deliberate effort is needed than in the pretraining scaling era to improve cyber(defense) capabilities" is that we don't need to worry as much about models acquiring concerning cyberoffense capabilities simply as a side effect of getting smarter (though we still need to worry about it because there is still generalization, especially from coding and agentic training)