Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (Joel Becker, Nate Rush, Beth Barnes, David Rein) released with the observation that completion time of PRs is 19% longer when using AI, but that developers think that it reduced completion time by 20%.
A few notes from me:
- This is on large-scale, mature repositories and conducted with maintainers intricately familiar. It is mentioned in the paper (C.1.2) that ‘developers note that AI is much less helpful on issues where they are expert.’ It is also mentioned that ‘LLM tooling performs worse in more complex environments.’ (C.1.3).
- They were provided with web interfaces or Cursor Pro but usually opted for the latter. In some cases this differed from their usual tooling, and I personally find this an annoying and unproductive way to code.
- Being in a study, developers may have felt pressure to use AI in situations that would otherwise be unnecessary.
- I would be interested in a similar study where developers are put in smaller repositories they aren’t familiar with.
Super interesting paper, and I look forward to future studies and whatever further findings come from it. I don’t look forward to seeing the discourse as AI advocates dismiss these results and AI haters take them at face value, despite the paper’s cautioning against overgeneralising.