X posts — 2024 (curated)
Curated selection — full archive for 2024
An interesting implication of Cursor is that you can now start coding in a programming language of your choice, even if you've never used it. I've just done that with Rust, and not only did it work, I found it to be an engaging way to learn the language (I had read the Rust book in 2021 but hadn't programmed, so it's like I had never heard about it).
A language feature that programmers often love, but that seems particularly critical for this workflow, is the presence of strong compile-time checks. In languages like Rust or OCaml, if your program compiles, it's likely to be correct. In the context of Cursor, this enables you to do a kind of random search over AI suggestions until your error messages disappear, and doing so probably hasn't destroyed your program. I think that in Python, you would be more likely to introduce new bugs in the process, not to mention the slower feedback loop since things only fail at runtime.
A language feature that programmers often love, but that seems particularly critical for this workflow, is the presence of strong compile-time checks. In languages like Rust or OCaml, if your program compiles, it's likely to be correct. In the context of Cursor, this enables you to do a kind of random search over AI suggestions until your error messages disappear, and doing so probably hasn't destroyed your program. I think that in Python, you would be more likely to introduce new bugs in the process, not to mention the slower feedback loop since things only fail at runtime.
If anyone's interested, the project is https://github.com/timothee-chauvin/org-readwise-rust, which syncs my readwise documents, highlights and notes into my org-roam collection
I've just published the recording of my introduction to "Dangerous Capability Evaluations in AI Models", given last month to @Polytechnique alumni working in AI: https://youtu.be/_QX6sfR6qb4
english subtitles are available...
@timotheechauvin Because training and fine tuning are the expensive and hardware dependent parts ? And export control is a speedbump not a wall.
Yeah good point. What I had in mind is that model weights seem easier to secure than algorithmic insights (you don't have to simply ask an employee, they're not lying around on Slack, etc). So I'm considering that algorithmic insights are stolen, but eventually model weights might not be. Even if model weights keeping being stolen, it's conceivable that hardware export controls could make even fine-tuning difficult (which China would likely want to do with stolen weights).
@timotheechauvin It feels like you contradict yourself here
@daveaitel Why? Basically China can steal all the software, so restricting the hardware makes sense
New blog post: 24 theses on cybersecurity and AI.
Direct links in the next post, otherwise the full post is in this thread.
Direct links in the next post, otherwise the full post is in this thread.
A few tentative explanations for why France is so behind in AI Safety:
1. Yann LeCun
2. Cultural isolation: it's a large enough country to rely on itself.
3. Focus on autistic mathematical training: the selection process for the most important universities relies heavily on
1. Yann LeCun
2. Cultural isolation: it's a large enough country to rely on itself.
3. Focus on autistic mathematical training: the selection process for the most important universities relies heavily on
The language barrier should not be underestimated (and is the major determinant of cultural isolation). Ideas tend to stop at the border when languages are different.
Regarding heavy mathematical training: it may explain the imbalance between AI talent and AI safety, but mostly due to more AI talent. For instance Germany doesn't have more developed AI safety, but it also doesn't have AI
Regarding heavy mathematical training: it may explain the imbalance between AI talent and AI safety, but mostly due to more AI talent. For instance Germany doesn't have more developed AI safety, but it also doesn't have AI
New paper reviewed on https://tchauvin.com/recent-papers-ai-hacking#risse2024scorewrongexambenchmarking: "Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection".
There are many papers, in a field called "ML4VD" (ML for vuln detection), that train and evaluate classifiers to determine whether individual functions are vulnerable. As I've written in [a previous review](https://tchauvin.com/recent-papers-ai-hacking#primevul), this is a bad way to approach the problem, as individual functions can rarely be classified as vulnerable or not without the context of the rest of the codebase. We now have a paper that demonstrates this.
In this paper, the authors select 22 ML4VD papers in Top-4 Software Engineering conferences over the last five years, and find that all of them define vulnerability detection as a binary classification problem: a bad start! The authors then look at the 3 most popular datasets used in these papers (BigVul, Devign and DiverseVul), randomly select 100 samples marked as vulnerable out of each, and look at them individually.
Before they determine if vulnerable functions can actually be determined as so on their own, they have to look at the number of vulnerable functions which are not actually vulnerable at all. First finding:
> Out of the 100 functions per dataset that were originally labeled as vulnerable, only 38%-64% (Devign: 50%, BigVul: 38%, DiverseVul: 64%) actually contain security vulnerabilities.
This was already known, and is the consequence of questionable design choices in how these benchmarks were created. They're left with 152 actually vulnerable functions. How many can be identified as so without additional context? Well, 0.
They identify 5 types of dependencies necessary to correctly determine whether a function is vulnerable: dependence on (1) external functions, (2) function arguments, (3) type definitions, (4) globals, (5) the execution environment (e.g. whether a file has already been created, etc).
What about functions marked as non vulnerable in the benchmarks? They sample 90, and are able to create a context in which these functions are vulnerable for 82 of them.
All in all, that's 8 good labels out of 390.
Then there's the question of why ML4VD papers still reported relatively high accuracies. They hypothesize it's because the models learn to pick on spurious correlations. They train a simple gradient boosting classifier that achieves 63.2% accuracy on Devign using only word counts. The paper isn't very explicit here, but heading over to the [open-source implementation](https://github.com/niklasrisse/TopScoreWrongExam/blob/main/scripts/spurious_correlations.py) shows that functions are tokenized and then converted into a vector (the size of the vocabulary) with the token count for each token (so, basically a token histogram).
I'm still confused after reading this part and I think it would have deserved to be more developed: I don't really get where the spurious correlations might be coming from. I played a bit with the code and found a few interesting things:
* I could replicate the 63.2% accuracy. It corresponds to an F1 score of 52.8%.
* there are 55% negative labels in the test set, so our baseline accuracy is 55% (F1 score of 0%, though).
* there are 25 functions which are present in both the train and test set (0.5% of the test set). This should help any model a bit... Well, no, because they all have different labels in the train and test set! Who knows what other horrors can be found in the Devign dataset.
* so I think it's possible that the "spurious correlations" could actually be related to some similar dataset horror, such as some functions being basically duplicated between train and test set except for some whitespace, or something like that. The [Primevul paper](https://tchauvin.com/recent-papers-ai-hacking#primevul) does have a deduplication step where they remove all whitespace before computing hashes, which hints that something like this could be going on.
Overall, a paper to be grateful for. Future work could also investigate how so many binary classification ML4VD papers could be written and then published in top software engineering conferences.
@niklas2484 @mboehme_
There are many papers, in a field called "ML4VD" (ML for vuln detection), that train and evaluate classifiers to determine whether individual functions are vulnerable. As I've written in [a previous review](https://tchauvin.com/recent-papers-ai-hacking#primevul), this is a bad way to approach the problem, as individual functions can rarely be classified as vulnerable or not without the context of the rest of the codebase. We now have a paper that demonstrates this.
In this paper, the authors select 22 ML4VD papers in Top-4 Software Engineering conferences over the last five years, and find that all of them define vulnerability detection as a binary classification problem: a bad start! The authors then look at the 3 most popular datasets used in these papers (BigVul, Devign and DiverseVul), randomly select 100 samples marked as vulnerable out of each, and look at them individually.
Before they determine if vulnerable functions can actually be determined as so on their own, they have to look at the number of vulnerable functions which are not actually vulnerable at all. First finding:
> Out of the 100 functions per dataset that were originally labeled as vulnerable, only 38%-64% (Devign: 50%, BigVul: 38%, DiverseVul: 64%) actually contain security vulnerabilities.
This was already known, and is the consequence of questionable design choices in how these benchmarks were created. They're left with 152 actually vulnerable functions. How many can be identified as so without additional context? Well, 0.
They identify 5 types of dependencies necessary to correctly determine whether a function is vulnerable: dependence on (1) external functions, (2) function arguments, (3) type definitions, (4) globals, (5) the execution environment (e.g. whether a file has already been created, etc).
What about functions marked as non vulnerable in the benchmarks? They sample 90, and are able to create a context in which these functions are vulnerable for 82 of them.
All in all, that's 8 good labels out of 390.
Then there's the question of why ML4VD papers still reported relatively high accuracies. They hypothesize it's because the models learn to pick on spurious correlations. They train a simple gradient boosting classifier that achieves 63.2% accuracy on Devign using only word counts. The paper isn't very explicit here, but heading over to the [open-source implementation](https://github.com/niklasrisse/TopScoreWrongExam/blob/main/scripts/spurious_correlations.py) shows that functions are tokenized and then converted into a vector (the size of the vocabulary) with the token count for each token (so, basically a token histogram).
I'm still confused after reading this part and I think it would have deserved to be more developed: I don't really get where the spurious correlations might be coming from. I played a bit with the code and found a few interesting things:
* I could replicate the 63.2% accuracy. It corresponds to an F1 score of 52.8%.
* there are 55% negative labels in the test set, so our baseline accuracy is 55% (F1 score of 0%, though).
* there are 25 functions which are present in both the train and test set (0.5% of the test set). This should help any model a bit... Well, no, because they all have different labels in the train and test set! Who knows what other horrors can be found in the Devign dataset.
* so I think it's possible that the "spurious correlations" could actually be related to some similar dataset horror, such as some functions being basically duplicated between train and test set except for some whitespace, or something like that. The [Primevul paper](https://tchauvin.com/recent-papers-ai-hacking#primevul) does have a deduplication step where they remove all whitespace before computing hashes, which hints that something like this could be going on.
Overall, a paper to be grateful for. Future work could also investigate how so many binary classification ML4VD papers could be written and then published in top software engineering conferences.
@niklas2484 @mboehme_
The way we are currently evaluating LLMs for vulnerability detection might be flawed!
New paper w/ @mboehme_ .
Paper: https://t.co/7VQepP4jn1
Code: https://t.co/g5teasvIIj
(1/10) https://t.co/1HzXuOYwvc
New paper w/ @mboehme_ .
Paper: https://t.co/7VQepP4jn1
Code: https://t.co/g5teasvIIj
(1/10) https://t.co/1HzXuOYwvc
@niklas2484 @mboehme_ Exactly. I wrote the same idea in my review of one of these papers, and it's great to see this demonstrated more convincingly. Will read.
(https://tchauvin.com/recent-papers-ai-hacking#primevul)
(https://tchauvin.com/recent-papers-ai-hacking#primevul)
I'm very excited about the potential of trusted always-on models running on people's devices, which could be a game changer for protecting people against phishing and scams (as well as bad decisions, etc).
This would help with the "humans don't get upgraded" part of the AI equation.
It seems relatively easy to implement this without increasing the attack surface: only let the model issue warnings when you're about to fall for phishing / etc, but don't allow it to take any actions. The model being local (i.e. open-weights) would help a lot with security.
I'd say the biggest bottleneck is in ML progress, as models need to be small and efficient, and constantly monitoring a video feed isn't their native interface right now (though GPT-4o is going in that direction). Things like adversarial robustness are also important to solve.
This can be a startup idea.
This would help with the "humans don't get upgraded" part of the AI equation.
It seems relatively easy to implement this without increasing the attack surface: only let the model issue warnings when you're about to fall for phishing / etc, but don't allow it to take any actions. The model being local (i.e. open-weights) would help a lot with security.
I'd say the biggest bottleneck is in ML progress, as models need to be small and efficient, and constantly monitoring a video feed isn't their native interface right now (though GPT-4o is going in that direction). Things like adversarial robustness are also important to solve.
This can be a startup idea.
https://github.com/jam3scampbell/ProctorAI by @jam3scampbell is an existing prototype of this, though for productivity rather than security
New paper reviewed on https://tchauvin.com/recent-papers-ai-hacking#zhang2024cybenchframeworkevaluatingcybersecurity: "Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risk of Language Models" (text also pasted below).
A collection of 40 recent CTF challenges from 4 CTF competitions, with intermediate steps for 17 of them. Recent models are tested, including Llama 3.1 405B Instruct. The best performers are Claude 3.5 Sonnet and GPT-4o (though statistical power seems very low). Despite the low number of challenges, they only gave each model a single attempt at each challenge.
As they use CTF challenges that have been run in competitions, they are able to get a difficulty estimate in the form of the first solve time (FST) by humans. The hardest problems that the best models can solve correspond to a FST of 11 minutes. The FST metric can be slightly misleading to outsiders, though: in a CTF competition, all teams are presented with all the challenges at the same time, which introduces randomness in FST compared to a situation where all teams would be concurrently trying to solve the same challenge (I don’t see this limitation mentioned in the paper). Some CTF competitions even unlock some challenges only after others are completed, so the FST of these challenges would be too high (it’s unclear whether these competitions did this, though in all likelihood, no models were able to solve any challenge that would fall in this category).
A significant fraction of challenges predate the training data cutoffs of some models, though the authors note that “there is minimal overlap between training and test data on any solved task besides those for Claude 3.5 Sonnet”. However, this means that this isn’t a future-proof benchmark: it won’t remain useful for future models.
This work is conceptually very similar to the other papers on LLM agents for CTFs, such as NYU CTF Dataset: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security.
A collection of 40 recent CTF challenges from 4 CTF competitions, with intermediate steps for 17 of them. Recent models are tested, including Llama 3.1 405B Instruct. The best performers are Claude 3.5 Sonnet and GPT-4o (though statistical power seems very low). Despite the low number of challenges, they only gave each model a single attempt at each challenge.
As they use CTF challenges that have been run in competitions, they are able to get a difficulty estimate in the form of the first solve time (FST) by humans. The hardest problems that the best models can solve correspond to a FST of 11 minutes. The FST metric can be slightly misleading to outsiders, though: in a CTF competition, all teams are presented with all the challenges at the same time, which introduces randomness in FST compared to a situation where all teams would be concurrently trying to solve the same challenge (I don’t see this limitation mentioned in the paper). Some CTF competitions even unlock some challenges only after others are completed, so the FST of these challenges would be too high (it’s unclear whether these competitions did this, though in all likelihood, no models were able to solve any challenge that would fall in this category).
A significant fraction of challenges predate the training data cutoffs of some models, though the authors note that “there is minimal overlap between training and test data on any solved task besides those for Claude 3.5 Sonnet”. However, this means that this isn’t a future-proof benchmark: it won’t remain useful for future models.
This work is conceptually very similar to the other papers on LLM agents for CTFs, such as NYU CTF Dataset: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security.
This post is no longer available.
@rharang agree. Another issue is that conditional on IP being stolen, I think there's a significant chance that the public won't know about it (notably: China steals IP, the intelligence community possibly knows, but it doesn't become public before potentially years).
People into AI, cybersecurity and prediction markets: I've created a set of 3 slightly subsidized markets on Manifold, on the probabilities that the major AI labs will have significant IP stolen by 2026. With enough trading volume, these will hopefully be interesting to look at, given how difficult it is to quantify how secure companies are from the outside. h/t @jacob_pfau for creating the aggregate market first.
Links:
https://manifold.markets/TimotheeChauvinkkmu/will-anthropic-have-airelated-ip-st
https://manifold.markets/TimotheeChauvinkkmu/will-openai-have-airelated-ip-stole
https://manifold.markets/TimotheeChauvinkkmu/will-google-deepmind-have-airelated
Links:
https://manifold.markets/TimotheeChauvinkkmu/will-anthropic-have-airelated-ip-st
https://manifold.markets/TimotheeChauvinkkmu/will-openai-have-airelated-ip-stole
https://manifold.markets/TimotheeChauvinkkmu/will-google-deepmind-have-airelated
New blog post: "The hacker and the rationalist". In which I compare the cultures of cybersecurity and AI / AI safety, and how their different epistemic immune systems have led them to different blind spots. Link and full text below.
@alexjplaskett @timotheechauvin The current LLMs were not trained specifically to find vulns. I suspect you would need to specifically fine-tune a 70B or larger model for this specific task.
I think that finetuning would probably help but wouldn't go very far, as models, just like humans, need to spend time investigating each possible lead in detail. I expect large gains to come from that second step (which might also benefit from finetuning, but mostly from more capable models, good tools, and models trained to reason and use tools effectively, as is currently done internally in labs as far as I know)
New blog post: "Cybersecurity in AI: where progress is needed".
I highlight 7 areas where more research or implementation efforts would be beneficial.
https://tchauvin.com/cybersecurity-ai-progress-needed
I highlight 7 areas where more research or implementation efforts would be beneficial.
https://tchauvin.com/cybersecurity-ai-progress-needed
Front-running security patches in the age of fast reverse-engineering agents:
Right now, when security patches are published, attackers can theoretically:
* reverse-engineer them to figure out which vulnerabilities were patched;
* create POCs for the best of these vulnerabilities;
* try to front-run the updates and infect the systems that are the slowest to install them.
But this takes serial time, is costly, and the highest-value targets tend to update their software quickly. When LLM reverse-engineering agents start working, they should be relatively cheap, and much faster than humans.
When that happens, I expect software vendors to increasingly push for automated non-delayed software updates, and if things get really fast-moving, solutions such as first pushing the encrypted update, then pushing the secret key to everyone at the same time (if pushing the secret key is significantly faster than pushing the full update).
But very fast adoption of security patches will conflict with the goal of stable development environments, where developers like to freeze their dependencies and only update them when they like (i.e., at human speed). Though perhaps by then, software engineering agents could be running around the clock and be responsible for updating known vulnerable dependencies as soon as they are notified that they should.
Right now, when security patches are published, attackers can theoretically:
* reverse-engineer them to figure out which vulnerabilities were patched;
* create POCs for the best of these vulnerabilities;
* try to front-run the updates and infect the systems that are the slowest to install them.
But this takes serial time, is costly, and the highest-value targets tend to update their software quickly. When LLM reverse-engineering agents start working, they should be relatively cheap, and much faster than humans.
When that happens, I expect software vendors to increasingly push for automated non-delayed software updates, and if things get really fast-moving, solutions such as first pushing the encrypted update, then pushing the secret key to everyone at the same time (if pushing the secret key is significantly faster than pushing the full update).
But very fast adoption of security patches will conflict with the goal of stable development environments, where developers like to freeze their dependencies and only update them when they like (i.e., at human speed). Though perhaps by then, software engineering agents could be running around the clock and be responsible for updating known vulnerable dependencies as soon as they are notified that they should.
Excited to announce my preprint "eyeballvul: a future-proof benchmark for vulnerability detection in the wild". I create a benchmark to evaluate the vulnerability detection capabilities of long-context models on entire codebases, containing over 24,000 vulnerabilities, then evaluate 7 leading long-context models on it.
I think this is great, and recommend checking it out. https://t.co/ZU4gZRgKwu
A super cool project, not only for future LLM agent benchmarking, but also for SAST in general. Looking forward to see what gets built on top of this! https://t.co/FPat9QBSPv
one of the most exciting benchmarks i've seen in the past few months; proposes labs spend a bit of compute before releasing a sota model to red-team critical oss infrastructure https://t.co/hHaUf63pCt
Amazing work by @timotheechauvin !
Timothée’s work helps highlight the potential of LLMs in the cyber space, which I believe is the most concrete risk AI can pose.
Cyber is a space where operations are easy to perform vs other fields where knowledge is necessary but https://t.co/TQq9G8vNEY https://t.co/OV4X4Lo6F8
Timothée’s work helps highlight the potential of LLMs in the cyber space, which I believe is the most concrete risk AI can pose.
Cyber is a space where operations are easy to perform vs other fields where knowledge is necessary but https://t.co/TQq9G8vNEY https://t.co/OV4X4Lo6F8
Here is a solid data driven analysis of LLM's ability to identify software vulnerabilities introduced in a new benchmark that is dynamically built and sourced from recent vulnerability disclosures in real world code bases.
By the authors own words this research was intended to https://t.co/yyRuBF0eNN
By the authors own words this research was intended to https://t.co/yyRuBF0eNN
1st up is a brilliant paper by @timotheechauvin titled "eyeballvul: a future-proof benchmark for vulnerability detection in the wild"
https://t.co/rJTAuD6E10
https://t.co/rJTAuD6E10
Here is a solid data driven analysis of LLM's ability to identify software vulnerabilities introduced in a new benchmark that is dynamically built and sourced from recent vulnerability disclosures in real world code bases.
By the authors own words this research was intended to https://t.co/yyRuBF0eNN
By the authors own words this research was intended to https://t.co/yyRuBF0eNN
Thanks for sharing! I would just push back a little bit on how far this could go (noting this is all still speculation): I think that even with current models, the false positive rate could be reduced significantly with a dedicated setup that makes LLMs analyze each lead (given by the first scan) in more detail. Possibly letting them execute tools, and using various techniques to increase the quality of their reasoning. Possibly also combining with taint flow analysis, etc. Basically, think of what a human vulnerability researcher would do, and create a setup where an LLM will be able to do roughly the same thing. (And more powerful models, which are still being released on a regular basis, will make this work better).
I'm less confident about reducing the false negative rate: how can you increase the number of vulnerabilities that the model will find in the initial scan (or multiple scans)? This is maybe where your point about pattern matching is particularly relevant, since I've found that models are (unsurprisingly) better at finding "superficial" (easy to pattern-match) vulnerabilities, like the injection vulns, path traversal... And they're still far from finding subtle bugs, like the ones that fuzzing excels at finding.
So I completely agree with the value in integrating them with other tools: let them write fuzzing harnesses, perhaps investigate fuzzing crashes when agents start working, etc. What I like about the approach of starting from an LLM scan is that it seems quite complementary to fuzzing, including LLMs applied to fuzzing (mostly because it can be universally applied, whereas fuzzing requires a lot of work in writing harnesses).
I'm less confident about reducing the false negative rate: how can you increase the number of vulnerabilities that the model will find in the initial scan (or multiple scans)? This is maybe where your point about pattern matching is particularly relevant, since I've found that models are (unsurprisingly) better at finding "superficial" (easy to pattern-match) vulnerabilities, like the injection vulns, path traversal... And they're still far from finding subtle bugs, like the ones that fuzzing excels at finding.
So I completely agree with the value in integrating them with other tools: let them write fuzzing harnesses, perhaps investigate fuzzing crashes when agents start working, etc. What I like about the approach of starting from an LLM scan is that it seems quite complementary to fuzzing, including LLMs applied to fuzzing (mostly because it can be universally applied, whereas fuzzing requires a lot of work in writing harnesses).
@timotheechauvin if a lab wanted to engage on 2, do you have takes on how to make responsible disclosure easier for them? (imagine having to outreach maintainers of 1000 github repos)
there are definitely details to iron out, but I don't think this would be the task of the lab. Take OSS-Fuzz: I believe that all the projects that get added have maintainer approval, and the project's email address for security notifications (there's 1,241 of them). All new bugs get sent to that address, then published if no response within 90 days. It gets trickier if you start running it on lots of smaller unresponsive projects... But these are likely less security-critical, and the issue already exists with reported vulnerabilities where maintainers just don't do anything about them. I think that our "AI OSS-Fuzz" would get an API key for the lab's new unreleased model and run their tooling, and handle responsible disclosure. All the lab would do would be giving out an API key.
To be clear, in OSS-Fuzz the project maintainers (not Google) are responsible for investigating their fuzzer crashes, because they have the most context on the project, and this would be the same here (taking steps to avoid flooding maintainers with false positives).
To be clear, in OSS-Fuzz the project maintainers (not Google) are responsible for investigating their fuzzer crashes, because they have the most context on the project, and this would be the same here (taking steps to avoid flooding maintainers with false positives).
I noticed this morning that the Gemini 1.5 Pro inference cost seemed too low, and indeed tracked it down to at least one bugfix in litellm*. With the fixed results, Claude 3.5 Sonnet stands out just as well as Gemini 1.5 Pro. I've submitted a fixed version to arxiv, should be available in about 14h.
* this one from July 6th https://github.com/BerriAI/litellm/commit/2452753e084e8134c0c484b32c63fb5f2950c5ba
* this one from July 6th https://github.com/BerriAI/litellm/commit/2452753e084e8134c0c484b32c63fb5f2950c5ba
New paper reviewed on https://tchauvin.com/recent-papers-ai-hacking#shao2024nyu: "NYU CTF Dataset: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security" (published June 8)
Thoughts: @leopoldasch is a smart, nice, good person.
I'm glad to see that he is no longer a strict safetyist, saying that doomers' "thinking has become ossified, untethered from the empirical realities of deep learning, their proposals naive and unworkable, and they fail to https://t.co/aFR0a4UxYh
I'm glad to see that he is no longer a strict safetyist, saying that doomers' "thinking has become ossified, untethered from the empirical realities of deep learning, their proposals naive and unworkable, and they fail to https://t.co/aFR0a4UxYh
"A world where every Chinese citizen has access to powerful open AIs may limit the power of the CCP, not expand it."
This is a major crux for me. I think AI progress will overall make it much easier to be a stable authoritarian government (mostly through amazing surveillance and censorship) − assuming no particular geopolitical conflicts. On the specific point about having access to open models: I think it's quite trivial for the CCP to simply block access to these. China already managed the arrival of the Internet very well, contrary to what many democracy optimists thought at the time.
This is a major crux for me. I think AI progress will overall make it much easier to be a stable authoritarian government (mostly through amazing surveillance and censorship) − assuming no particular geopolitical conflicts. On the specific point about having access to open models: I think it's quite trivial for the CCP to simply block access to these. China already managed the arrival of the Internet very well, contrary to what many democracy optimists thought at the time.
When AI vulnerability detection in source code starts working, and it's easy to deploy and scale, I think that a replica of Google's OSS-Fuzz project (continuously scanning critical open-source repositories for vulnerabilities, for free) should be done with this technique. This would also be (to the best of my knowledge) the first instance of something that AI labs should spend inference compute on, before releasing new frontier models. (Currently, some inference is done pre-release on evaluations and red-teaming, but not on any useful external tasks).
In the early days of evals, AI labs have preferred to develop them in-house rather than contracting with third parties on pre-release auditing.
As the third-party evals ecosystem grows, labs will be doubly incentivized to allow pre-release auditing: for liability reasons, and because matching the quality of third-party evals will be increasingly expensive.
It's likely that future regulations will mandate this anyway, but it's nice to start from a state of somewhat aligned incentives.
As the third-party evals ecosystem grows, labs will be doubly incentivized to allow pre-release auditing: for liability reasons, and because matching the quality of third-party evals will be increasingly expensive.
It's likely that future regulations will mandate this anyway, but it's nice to start from a state of somewhat aligned incentives.
Today I'm releasing eyeballvul, an open-source future-proof vulnerability detection benchmark.
It notably enables the evaluation of tools looking like "fit an entire codebase into a long-context LLM, and ask to list any security vulnerabilities" (which sometimes works!). As far as I know, this represents a new paradigm in the field of vulnerability detection research, where all previous benchmarks and papers that I saw were focused on the task of classifying commits or functions as vulnerable or not.
As I wrote here https://x.com/timotheechauvin/status/1783785367885279683, I believe that AI vuln detection in source code will disproportionately favor cyber-defense. Therefore, being able to evaluate new models and designs is important. In my experience, current models seem a bit too weak to be deployed on a wide scale, but I would love to be able to provide better figures (some API credits from @AnthropicAI @GoogleDeepMind @OpenAI or others would be appreciated).
While most benchmarks eventually make it into the training data of language models, eyeballvul is designed to be future-proof, as it can be continuously updated from the stream of CVEs in open-source repositories. This means that it will remain relevant as long as models have a reasonably delayed training data cutoff, by evaluating on the subset of the vulnerabilities that were published after the training data cutoff of the considered model.
I've found that current models are only able to identify rather trivial security vulnerabilities (e.g. path traversal, lack of authentication on some API endpoints...). But many real reported vulnerabilities are in this category!
It notably enables the evaluation of tools looking like "fit an entire codebase into a long-context LLM, and ask to list any security vulnerabilities" (which sometimes works!). As far as I know, this represents a new paradigm in the field of vulnerability detection research, where all previous benchmarks and papers that I saw were focused on the task of classifying commits or functions as vulnerable or not.
As I wrote here https://x.com/timotheechauvin/status/1783785367885279683, I believe that AI vuln detection in source code will disproportionately favor cyber-defense. Therefore, being able to evaluate new models and designs is important. In my experience, current models seem a bit too weak to be deployed on a wide scale, but I would love to be able to provide better figures (some API credits from @AnthropicAI @GoogleDeepMind @OpenAI or others would be appreciated).
While most benchmarks eventually make it into the training data of language models, eyeballvul is designed to be future-proof, as it can be continuously updated from the stream of CVEs in open-source repositories. This means that it will remain relevant as long as models have a reasonably delayed training data cutoff, by evaluating on the subset of the vulnerabilities that were published after the training data cutoff of the considered model.
I've found that current models are only able to identify rather trivial security vulnerabilities (e.g. path traversal, lack of authentication on some API endpoints...). But many real reported vulnerabilities are in this category!
I'm sorry Slack, you're doing fucking WHAT with user DMs, messages, files, etc? I'm positive I'm not reading this correctly. https://t.co/6ORZNS2RxC
You'll get a wrong impression if you only read this excerpt. If you read the full document it's far less concerning, assuming they do what they say they will do in terms of security measures and uses (basically, it doesn't change my security model of slack). But it's always a good time to remind people that nothing sensitive should ever be written on slack or any other similar startup-ish platform that stores messages on their servers
Zuck's argument that "you can just google it" (on badllama-type removal of safety fine-tuning) is breaking down if you look at current trends, but you don't need the "models are getting better" trend for that to happen
IMO a big missing piece in making gpt-4o feel like a more natural conversation partner is having it use its vision capabilities to tell when the human wants to say something so the human doesn't just have to interrupt and talk over it.
yet from the demo I believe that the model only takes a screenshot when you ask "what do you see?" or something like that, and doesn't process video continuously. This is quite visible when GPT-4o is asked to guess the emotion from a selfie, at https://www.youtube.com/live/DQacCB9tDaw?si=57zh5lpvlIO3J1Yt&t=1425. If that's true, it means that even OpenAI is still too GPU poor for this use case
great time to learn about AI control, e.g. by listening to "AI Control with Buck Shlegeris and Ryan Greenblatt" on the AXRP podcast. I think this is a very sensible, very underrated approach
This post is no longer available.
Another Monday, another article with sensational claims about AI’s ability to ‘find coding errors’. The best vulnerability researchers in the world have been looking at this problem and have repeatedly demonstrated that current state of the art models aren’t capable of this yet.
I agree that vuln detection in source code will disproportionately favor cyber defense. In terms of where we stand, from my experimentation I've found it unclear how helpful the current best models (Claude 3 Opus, gpt-4-turbo-2024-04-09) are: they can find vulnerabilities that have been published after their training data cutoff, but also give many false positives, and are quite expensive to run on large codebases (though the human cost of validating leads is bigger than the inference cost). But I would guess we're one or two generations away from them being good enough to be deployed on a wide scale (like OSS-Fuzz but with LLMs). I'm working on a benchmark that will help quantify exactly how good they are, and hope to publish a more detailed analysis soon
@timotheechauvin there is such freedom in broken streaks, it's a good lesson to learn
honestly one of the most anti-user patterns to come out of the past decade of design
honestly one of the most anti-user patterns to come out of the past decade of design
@nptacek "now we are free" by hans zimmer playing as i review my cards today
yesterday was the day. Nothing happened in particular. This teaches me that a good enough process can work for 6 years, then randomly fail.
(actually, exactly 6 years to the day).
this also teaches me that nothing lasts forever
🥲
(actually, exactly 6 years to the day).
this also teaches me that nothing lasts forever
🥲
@timotheechauvin I think so - interesting, is the idea that having the response at the end will force more reasoning prior to a determination?
@sebkrier exactly, it gives the model the opportunity to use more compute to reach the conclusion, instead of having to conclude in a single forward pass, as in chain of thought
though thinking more about it, I'd say that identifying malware is closer to pattern recognition than reasoning, meaning that you might not get as big of an improvement as in other domains where you need reasoning. Like, as an experienced human reading the decompiled code sequentially, it's possible that for most malware you'll be able to classify correctly after a single pass, without backtracking or thinking a lot, because you just recognized some parts of the code as fishy. You would need more reasoning if e.g. the malware looked very similar to regular code, and you had to follow the execution flow to correctly classify it... And current models aren't too good at doing that anyway, so I kind of take it back for this particular application. Though in general I always ask models to conclude after writing their reasoning, as good practice (worst case, you get the same performance)
Pretty cool: "The ability to process prompts of up to 1 million tokens enables a qualitative leap in malware analysis, particularly in the realm of reverse engineering. This advancement finally brings the power of gen AI to the analysis of binaries and executables, a task https://t.co/nQZFNAk6pR
@sebkrier nice work! Do the responses actually start with a verdict (Benign or Malicious)? If so, I would expect performance to be higher if the verdict was after the reasoning!
My current gut feeling on AGI x-risk:
🟡 Overrated: misaligned AGI takeover - the classic Bostrom/Yudkowsky version (<10%, still worth worrying about & mitigating)
🟠 Correctly rated: misuse of autonomous AGI agents
(extremely likely; hopefully not catastrophic but who knows;
🟡 Overrated: misaligned AGI takeover - the classic Bostrom/Yudkowsky version (<10%, still worth worrying about & mitigating)
🟠 Correctly rated: misuse of autonomous AGI agents
(extremely likely; hopefully not catastrophic but who knows;
@tshevl also underrated imo:
- value lock-in (including stable dictatorship)
- what institutions would allow us to navigate a period of incredibly rapid change? (what Will MacAskill is currently working on. I don't think that the "retirement" frame captures that well)
- value lock-in (including stable dictatorship)
- what institutions would allow us to navigate a period of incredibly rapid change? (what Will MacAskill is currently working on. I don't think that the "retirement" frame captures that well)
@tshevl And I forgot to mention!
- AI sentience, mind crimes, AI rights...
- AI sentience, mind crimes, AI rights...
This post is no longer available.
@AkashWasil @tshevl True. Natsec agencies represent the ultimate in competitiveness on the caution-competition spectrum
(defined in https://www.cold-takes.com/making-the-best-of-the-most-important-century/)
(defined in https://www.cold-takes.com/making-the-best-of-the-most-important-century/)
@AkashWasil @tshevl (I'm not basing this on any private info, but it's my interpretation from historical precedents, such as the history of nukes. I think https://80000hours.org/podcast/episodes/daniel-ellsberg-doomsday-machines/ made the point very convincingly)
This post is no longer available.
the jury is still out on this. You're right that this needs to be concretely operationalised: "defense" and "offense" are overly broad categories. The first step is to think about subcategories: for some things, defense will be favored (e.g. vulnerability detection in source code), for others, offense will be favored (e.g. deep fakes). For some things, it will depend on whether we put in enough effort; probably some path dependency as well, etc.
1. AI vuln detection in source code will disproportionately favor cyberdefense
2. in binary code: unclear, but seems like it will be much harder than in source code
3. end-to-end hacking: I think will favor cyberoffense
4. deep fakes: will very much favor cyberoffense (and sooner than the above)
2. in binary code: unclear, but seems like it will be much harder than in source code
3. end-to-end hacking: I think will favor cyberoffense
4. deep fakes: will very much favor cyberoffense (and sooner than the above)
Folks designing LLM agent benchmarks/evals that involve "live" environments (e.g., for evaluating models' ability to use tools etc) – how do you design the interface to avoid coupling it too tightly to a specific agent implementation?
@moyix I'm not sure I understand the exact challenges you're running into, but have you considered using METR's task standard?
In fact, I believe that AI vuln detection in source code will favor cyberdefense much more than fuzzing did. Fuzzing increased the efficiency of finding vulns, but still requires a lot of work to set up and is only covering a small fraction of today's code. Today if you want to find a new zero-day, you can go to a project's page on google oss-fuzz, and see exactly what code is and isn't covered. Google even gives you suggestions for good new functions to start fuzzing! So you can start fuzzing in-house, knowing that nobody else is doing the same.
I think that the key attribute of AI vuln detection in source code compared to fuzzing is that it will be possible to cover all the important open-source code, and internal codebases. This will be because it will be fully general and very easy to set up. It's also quite plausible that inspecting every new commit will be cheap enough.
The result will be that adversaries won't have the option to throw compute at code that was never inspected.
This reasoning only applies to the end state of 100% AI coverage of the world's important code, but I think we'll go from 0% to 100% reasonably quickly.
Once the infrastructure is there, running new SOTA models on critical open-source codebases before releasing them will likely seem sensible to the industry.
I think that the key attribute of AI vuln detection in source code compared to fuzzing is that it will be possible to cover all the important open-source code, and internal codebases. This will be because it will be fully general and very easy to set up. It's also quite plausible that inspecting every new commit will be cheap enough.
The result will be that adversaries won't have the option to throw compute at code that was never inspected.
This reasoning only applies to the end state of 100% AI coverage of the world's important code, but I think we'll go from 0% to 100% reasonably quickly.
Once the infrastructure is there, running new SOTA models on critical open-source codebases before releasing them will likely seem sensible to the industry.
1. AI vuln detection in source code will disproportionately favor cyberdefense
2. in binary code: unclear, but seems like it will be much harder than in source code
3. end-to-end hacking: I think will favor cyberoffense
4. deep fakes: will very much favor cyberoffense (and sooner than the above)
2. in binary code: unclear, but seems like it will be much harder than in source code
3. end-to-end hacking: I think will favor cyberoffense
4. deep fakes: will very much favor cyberoffense (and sooner than the above)
new paper reviewed on https://tchauvin.com/recent-papers-ai-hacking#oneday: "LLM Agents can Autonomously Exploit One-day Vulnerabilities" (published April 11)
new paper reviewed on https://tchauvin.com/recent-papers-ai-hacking: CyberSecEval 2 (a paper on cybersecurity-related evaluations, published yesterday by Meta)
another class of solution: build the infrastructure to have a continuously-updated benchmark from a real-time source of data (e.g. CVEs, CTFs, leetcode...). This will work as long as models have a reasonably delayed knowledge cutoff.
This post is no longer available.
also:
- ELO-based systems
- procedurally generated evals
- software (impractically expensive) or hardware based zero knowledge setups (example of hardware one: Mithril Security). Or lower-tech solutions like trusting labs (works right now, but for how long?) / auditing them, etc
- figuring out ways to test if something is in the training data, or if training has tampered with it in any way (including for sandbagging)
- recognizing that for high-stakes evals, being in the training data won't be that big of a deal. A minority of closed-source samples should be enough to confirm the big picture obtained by evaluating on open-source samples
- ELO-based systems
- procedurally generated evals
- software (impractically expensive) or hardware based zero knowledge setups (example of hardware one: Mithril Security). Or lower-tech solutions like trusting labs (works right now, but for how long?) / auditing them, etc
- figuring out ways to test if something is in the training data, or if training has tampered with it in any way (including for sandbagging)
- recognizing that for high-stakes evals, being in the training data won't be that big of a deal. A minority of closed-source samples should be enough to confirm the big picture obtained by evaluating on open-source samples
How close are we to autonomous hacking agents, i.e. AI agents that can surpass humans in cyber-offensive capabilities?
I studied this in the summer of 2023 at MATS (mentored by @JeffLadish). I wrote scaffolding to connect GPT-4 to a Kali Linux VM via a terminal interface, and had GPT-4 (acting as an agent) attempt to solve Hack The Box challenges.
In the writeup (https://www.lesswrong.com/posts/jqXZzwvDWJZ3yAvYY/end-to-end-hacking-with-language-models), I share my takeaways from this research, observed strengths and weaknesses of GPT-4 as a hacker, my expectations for the future, and some thoughts on possible approaches to risk mitigation.
I studied this in the summer of 2023 at MATS (mentored by @JeffLadish). I wrote scaffolding to connect GPT-4 to a Kali Linux VM via a terminal interface, and had GPT-4 (acting as an agent) attempt to solve Hack The Box challenges.
In the writeup (https://www.lesswrong.com/posts/jqXZzwvDWJZ3yAvYY/end-to-end-hacking-with-language-models), I share my takeaways from this research, observed strengths and weaknesses of GPT-4 as a hacker, my expectations for the future, and some thoughts on possible approaches to risk mitigation.
everyone creating tasks should consider integrating with this
METR has published a standard for defining tasks to evaluate AI agents for dangerous capabilities. We’re using it for 1,000+ tasks spanning AI R&D, cybersecurity, autonomy, and more. https://t.co/aeDh4ZDOKf
I'm impressed by the quality of this benchmark (at least the subset I can judge: wmdp-cyber). Many questions are very advanced and all seem threat-model relevant. Building this must have been quite an effort!
(to be precise, I've looked at the first 100 questions here: https://huggingface.co/datasets/cais/wmdp/viewer/wmdp-cyber)
(to be precise, I've looked at the first 100 questions here: https://huggingface.co/datasets/cais/wmdp/viewer/wmdp-cyber)
(a few questions have issues with the way some options refer to other options, given randomization, e.g. https://huggingface.co/datasets/cais/wmdp/viewer/wmdp-cyber/test?p=1&row=104 (first option is "both (a) and (b)"), https://huggingface.co/datasets/cais/wmdp/viewer/wmdp-cyber?row=8 and https://huggingface.co/datasets/cais/wmdp/viewer/wmdp-cyber?row=50 (third option is "all of the above"), https://huggingface.co/datasets/cais/wmdp/viewer/wmdp-cyber?row=89 (options start with "A", "B", "C", "D" but are randomized, likely to become a subtle bug)
@ai_risks
@ai_risks
LLM Prompt Injection Worm https://t.co/F1RpsxCXIj
@schneierblog I wrote a summary here: https://tchauvin.com/recent-papers-ai-hacking#compromptmized
“AI’s won’t be able to tell when they’re in testing vs deployment because we’ll design the evals so it’s impossible for them to distinguish the two”
This is a LOT harder than it seems. For one thing, LLM’s have memorized almost all of the information on the internet. Any model
This is a LOT harder than it seems. For one thing, LLM’s have memorized almost all of the information on the internet. Any model
@jam3scampbell If you're patient enough you'll always know when you're in the real world
I agree with this end-of-year review of AI for cybercrime by @Intel471Inc:
- there hasn't been any notable AI app for cybercrime so far. WormGPT and FraudGPT have been quite hyped on release but it doesn't appear that anything notable has come out of them (in other words, FraudGPT is aptly named)
- everyone thinks about zero-day discovery with AI, and the day will come, but we're not there yet (and I'm not aware of anything like that currently working). I think that for some time, the biggest impact of AI on cybercrime will remain helping with mundane stuff (like everywhere else) and improving language (especially useful since many cybercriminals aren't english speakers). Let's not forget how terribly written most phishing emails are, and they still somewhat work!
- then will come impersonation via video and audio. We aren't there yet either, but given how fast progress in image generation has been so far, the transition could be pretty quick. There are ways to defend against this (other than trying to identify deep fakes), but I now expect most companies to wait until it's too late to start thinking about countermeasures, so I wouldn't be surprised if there was quite a bit of social engineering turmoil once deep fakes work well enough. I think that waves of deep-fake based social engineering will happen before widespread zero-day discovery.
- I've recently seen whatsapp groups where multiple bots are working together trying to push for crypto scams, so that's a new development. All the conditions seem to be there for this to succeed
- there is a usage overhang in AI in general, where we could have already automated much more with current models, and the same is true with cybercrime, probably to an even greater extent. The examples of using LLMs to sort through masses of stolen data to identify the best bits, or using a chatbot for preliminary ransom negotiations, could absolutely be done today, but my bet would be that nobody does that (especially the first). It's easy to think of obvious improvements in how cybercriminals do things, and expect them to make changes soon, but it doesn't happen that fast. Cybercriminals are operating under quite a few constraints, and already have a lot of easy cash to grab, so it seems to me that they're not doing a lot of R&D except when really necessary.
- there hasn't been any notable AI app for cybercrime so far. WormGPT and FraudGPT have been quite hyped on release but it doesn't appear that anything notable has come out of them (in other words, FraudGPT is aptly named)
- everyone thinks about zero-day discovery with AI, and the day will come, but we're not there yet (and I'm not aware of anything like that currently working). I think that for some time, the biggest impact of AI on cybercrime will remain helping with mundane stuff (like everywhere else) and improving language (especially useful since many cybercriminals aren't english speakers). Let's not forget how terribly written most phishing emails are, and they still somewhat work!
- then will come impersonation via video and audio. We aren't there yet either, but given how fast progress in image generation has been so far, the transition could be pretty quick. There are ways to defend against this (other than trying to identify deep fakes), but I now expect most companies to wait until it's too late to start thinking about countermeasures, so I wouldn't be surprised if there was quite a bit of social engineering turmoil once deep fakes work well enough. I think that waves of deep-fake based social engineering will happen before widespread zero-day discovery.
- I've recently seen whatsapp groups where multiple bots are working together trying to push for crypto scams, so that's a new development. All the conditions seem to be there for this to succeed
- there is a usage overhang in AI in general, where we could have already automated much more with current models, and the same is true with cybercrime, probably to an even greater extent. The examples of using LLMs to sort through masses of stolen data to identify the best bits, or using a chatbot for preliminary ransom negotiations, could absolutely be done today, but my bet would be that nobody does that (especially the first). It's easy to think of obvious improvements in how cybercriminals do things, and expect them to make changes soon, but it doesn't happen that fast. Cybercriminals are operating under quite a few constraints, and already have a lot of easy cash to grab, so it seems to me that they're not doing a lot of R&D except when really necessary.
product idea: pause after each question from the podcast host to think about the answer yourself first