Developers using Copilot are statistically better in GitHub's latest study
In a recent study by GitHub, GitHub Copilot was tested to determine whether the AI coding assistant is helping developers write code that is objectively better or worse than code written without AI. To do this, GitHub worked with 202 Python developers, 104 coding with Copilot and 98 without. The developers were tasked with creating a web server of restaurant reviews and assessing functionality with 10 unit tests.
For each of the completed submissions, at least ten developers went line by line through the code, not knowing if it was written with or without the help of AI. All 1,293 of the resulting reviews evaluated the code samples' readability, reliability, maintainability, and conciseness. Additionally, reviewers evaluated if the code should ultimately be approved.
The study's results bode well for the use of AI in coding, with GitHub highlighting four key findings.
- Copilot helped developers pass all ten unit tests 56 percent more often than non-AI developers, leading to increased functionality.
- The Copilot-assisted code is more readable and helps developers write an average of 13.6 percent more lines without encountering readability issues.
- Readability, reliability, maintainability, and conciseness improved by an average of 3.29 percent, with conciseness receiving the most significant boost of 4.16 percent.
- Copilot-assisted code was approved 5 percent more often than non-AI code. This means less time is needed before the code is ready for production.
For more detailed information on the study's methodology and results, visit the official GitHub blog post linked below.
Source(s)
