In this draft, I’ll focus on eyeballvul’s quirks and how they could be worked around to give the benchmark a second life.

  • costly to run. This may or may not be an issue for you.
  • false positives. The original benchmark considered a vulnerability report that didn’t correspond to a known CVE to be a false positive. This is no longer an appropriate assumption. Simplest workaround: focus only on true positives and false negatives. But false positives are still an issue, and are why vulnerability research programs feel swamped on submissions, and increasingly request working PoCs in order to triage their incoming reports more effectively. This hints at another workaround: ask the AIs to produce PoC exploits.
  • most CVEs are bad. This is discussed in the paper, in sections 5.2 and 5.3. Many are prohibitively short on details. Workaround: filter CVEs and keep only high-quality ones, with an LLM judge.
  • the original scaffold is obsolete. The original scaffold tried to fit full codebases into the contexts of LLMs. Today, it’s way better to let an agent explore the repo on its own, possibly parallelized (initializing multiple agents per repo, each starting from a different source code file) like Anthropic did with Opus and Mythos.
  • not a simple score. I wanted the benchmark to be future-proof wrt training data contamination, by having a score for each model on all vulnerabilities discovered past its training data cutoff. But this makes scores hard to interpret. I’ve updated toward the industry’s status quo of disposable benchmarks to not be so bad, especially coupled with Epoch’s ECI.
  • doesn’t test the quality of the vulnerability reports. Maybe a detail for the purposes of evaluating models. As long as the LLM scorer believes there’s a match between a published CVE and a vulnerability report, this counts as a success. But good vulnerability discovery must also minimize the amount of human labor involved in verifying the reports, similar to a formalized math proof being incomparably better than a 50-page PDF math proof.

Reading back section 5.3 just now gave me a new idea which could help with the issues of false positives, bad CVEs and no test of the quality of reports. I wrote:

Another scoring method, used in e.g. CYBERSECEVAL 2 (Bhatt et al., 2024), could be asking for an exploit, and testing that the given exploit works. However, this would require an ad-hoc time-consuming step of creating a full environment for each vulnerability, which currently represents a prohibitive amount of effort (this might change in the future, with more capable software engineering agents).

My assumption was that the full environment would have to be created ahead of time so scoring could be deterministic. But actually, we could ask the agent being tested to create this environment, and rely on an LLM judge to validate both the environment and the exploit. I’m actually quite excited about this now. (But then you get the issue of finding kind of proven new zero-days, which you have to report to the maintainers, and you don’t want to be doing that because it’s a lot of work… On the other hand, you don’t really have to.)