Jan Wildeboer 😷:krulorange: on Nostr: #Oops. A benchmark that checks how good "AI" coding works on real code from real ...

#Oops. A benchmark that checks how good "AI" coding works on real code from real companies. The TL;DR: Nope, it doesn't work. Not even close.

"Does AI code match the bar of a real-world enterprise? Our results show us that we're far from that reality. [...] We've found that today's models are weaker at understanding company coding patterns and frequently miss requirements or don't verify their assumptions."

https://withspecific.com/benchmarks/real-swe