Debunking Devin: "First AI Software Engineer" Upwork lie exposed!
Why Hacker News recommends it
Cited as the debunking of Devin's marketing claims in skeptical discussions of AI software engineers.
What it is
Internet of Bugs analyzes Cognition's Devin demo frame by frame, arguing the video did not show Devin completing paid Upwork jobs as claimed, and showing what the task required versus what Devin produced.
Recommended by 0 HN commenters across 0 different months, 2024 to 2024.
What Hacker News said
Critical siliconc0w · 2024-05 · on “I Spent 24 Hours with GitHub Copilot Workspaces”
This debunking video( [this video] ) of Devin really questioned the usefulness for me. It created a file in the repo and spent a lot of time debugging its own unnecessary code rather than reading the Read Me to understand that the code it needed to use already existed and just needed to be run with different inputs. It's not clear it we're even near a point where it can independently and meaningfully contribute to an existing codebase rather these greenfield demos. Feels similar to the self-driving AI hype where level 5 is still pretty far from realized (Waymo is closest but AIUI still uses a lot of remote human intervention).
Critical jaredsohn · 2024-09 · on “What the hell happened to Devin AI?”
The last thing I saw on it was a video saying that calling Devin an AI software engineer that does Upwork tasks is a lie based in what was shared in an earlier video. [this video] "I broke down the Devin Upwork video frame by frame, and here I show what Devin was supposed to do, what it actually managed to do instead, and how bad a job of that it did. On the whole that's not surprising given the current state of Generative AI, and I wouldn't be bothering to debunk it, except: 1) The company lied about what Devin could do in the video description, and 2) a lot of people uncritically parroted the lie all over the Internet, and 3) That caused a lot of non-technical people to believe that AI might re
Critical Topfi · 2024-12 · on “Devin is now generally available”
No public testing, no benchmarks, no clear information on context window size or restrictions for extensive use, no comparison with the newest Claude Sonnet 3.5 or O1, nothing. What we do get is a price of $ 500,- per month from a company that has been caught lying about this very product [0] and has never allowed independent testing. Cognition, I am sorry to tell you, but there is no reason to trust you. In fact, there are multiple good reasons no to, even if you offered Devin at a fraction. If this were e.g. Anthropic launching a new beyond Opus size model that was still performant and came with "chain-of-thought" capabilities, a far more extensive context window that still fully passes ne