I'm not considering the cost of inference, so assuming an unlimited inference budget, will GPT-5 be able to replace the average web developer?
1,000
1.00@ErikBjareholt If you can think of better criteria you can create a market and I'll bet on it and post a link in the comments for the people on this market.
I have no doubt that GPT-5 will be able to do this. Claude 3 Opus can already do it.
As with all these similar questions about "AGI," people have such ridiculously unrealistic expectations about what models are supposed to be able to do that the bar keps getting raised.
I also think that Manifold, which generally consists of above "average" people, continually overestimates what it takes to be successful in the world. The "average" person in this world is extremely incompetent, as is an "average" web developer.
The most likely way this question ends up at NO is that the model produces amazing code that far exceeds the web developer's code quality 99% of the time, and it gets stuck on one line it never saw on its training data. Thus, the question will resolve NO, even if a developer needs to spend a day fixing the problem and then ending up with superintelligent code that no human could have produced.
@SteveSokolowski Yes I basically agree that a very obvious failure case would be that the model does as well or better 99% of the time but fails 1% of the time, which would make it impractical to replace the web developer's job because he would have to monitor and correct GPT-5's work constantly.
@ersatz But the problem is that web developers also fail 1% of the time, too. My stock trading models didn't work for 4 months. I tried all sorts of things and it turned out that one slot on the motherboard had failed.
Neither I nor GPT-4 figured that out. But as soon as I bought a new motherboard, GPT-4 was able to do what I've found is the thing it's best at - designing models.
There's always going to be something that any entity can't fix. The expectations that people have of godlike perfect machines are ridiculous.
@SteveSokolowski You're confused. I'm just interested from an economic point of view, i.e. could the GPT-5 replace the job of an average web developer? My self-driving car analogy again: if you have to pay a driver to be behind the wheel then you haven't replaced his job! Even if the car drives itself just fine 99% of the time. If you have to pay a web developer to check everything GPT-5 does, then you haven't replaced their work either.
@ersatz Why do these machines have to be like humans?
Perhaps a junior web developer makes mistakes related to his inability to understand advanced concepts and he needs to be taught by the senior web developers. Perhaps GPT-5 makes unhumanlike mistakes that are related to forgetting things in its context window, rather than not understanding the concepts, and senior web developers are needed to fix those bugs.
GPT-5 would still be better than a junior web developer; it's just that the type of work that needs to be done with it would be different. We need to consider that the bot doesn't have to be human for it to be superior to a human.
@SteveSokolowski I think the problem GPT-5 would run into are the "human" tasks required of a typical web developer. Asking project leads/clients clarifying questions of their vague requirements, identifying flaws in dependencies you don't own and communicating with those teams to implement fixes, understanding what changes would break downstream projects and when that is preferable to not making the change in the first place etc. Actions that require forecasting and understanding of business/end-user needs.
I think a lot of people have this view that your typical "web developer" is just some code monkey that reads an issue with clear requirements and spits out some React component, or Express middleware (Claude Opus is quite good at this), but there is so much more to it.
Guys, if you think it's ambiguous you probably don't appreciate the huge leap from replacing 99% of a job to replacing 100%. If you need someone to check the code, fix problems etc. then you haven't actually replaced their job. If you want an analogy, it's like having a self-driving car that's as good as an average human 99% of the time vs. one that's as good as an average human 99.9999% of the time: huge leap. More than a decade of hard work and tens of billions of investment were needed to progress from the former to the latter.
@ersatz Even great developers write code with problems and have other developers check their code (code review). It's more likely you don't really understand web development.
@pzet To make things crystal clear: if GPT-5 can produce code for all the tasks of an average developer requiring as little code review as an average developer then this market will be resolved to YES.
@ersatz So you've decided you will almost certainly resolve it no already? That does indeed make it unambiguous, but one has to wonder what the point of the market was then.
Nothing about this is "crystal clear".
@ErikBjareholt I haven't decided to resolve it NO! I'm not betting on this market, so I have no interest one way or the other. I'm just interested in the odds of GPT-5 being able to replace the average web developer, because it seems like low-hanging fruit to me.
@ersatz As you wrote above:
Yes I basically agree that a very obvious failure case would be that the model does as well or better 99% of the time but fails 1% of the time, which would make it impractical to replace the web developer's job because he would have to monitor and correct GPT-5's work constantly.
Since its absolutely certain that the model would have a failure rate at some percent of developer tasks, and you only think perfection is passable, then that says otherwise to me.
If you think a 1% failure rate isn't enough to replace most what a web developer does today (to the point of "web developer" not being the job description of the operator anymore), then I submit to you that you've already made up your mind.
I don't see how a bar of ~perfection is "low hanging fruit"? (although I do agree the task is low hanging fruit, which I think we've learned building gptengineer.app, but the perfect fruit you're asking for isn't low hanging)
I'm building this at gptengineer.app. GPT-4 can arguably already do this (to an extent). If you consider that the "average web developer" isn't a high bar, then it becomes even more muddy.
I think this needs a clear resolution condition. Since we are only evaluating the model, I assume any tool use is allowed (as long as those tools don't use better models).
@pzet I'm just going to go by the consensus, I doubt the resolution will be ambiguous, it'll most likely be clear yes or no.
@ersatz I disagree. GPT4 can already implement convincing websites that could fool non-developers into thinking that it can replace the average web developer. I think unless more strict resolution criteria is set, it will undoubtedly be ambiguous.