Max on Nostr: Yes, a good harness is very important, seems like its all about context management, ...

Yes, a good harness is very important, seems like its all about context management, even tiny models find critical bugs if you scope their context to the right part of the codebase.

Anyhow, I've found a surprising amount of vulnerabilities with a trivial prompt like "do a security audit for this project, research best practices online first".

Sure, professional pen testers with advanced harnesses will find even more bugs. But the fact that a non-dev with a one shot prompt can find criticale is wild.

Anyhow, more eyes on the code is always a good thing.