Research: How GitHub Copilot helps improve developer productivity

Software developers know that an estimated 90% of all computer code has already been written for some application somewhere in the world, so it’s counterproductive to have to manually enter it into a job all over again. Besides, the artificial intelligence (AI) tools we now have in 2022 already know where all that code is in the libraries and can fetch basically anything in a matter of microseconds.

Companies such as startup Tabnine and open-source tool shop GitHub are plugging directly into this trend of development tools that stay one or more steps ahead of code that’s being written. They read the beginning of the code line and then make intelligent assumptions – then offer suggestions – in real time on how to complete it. 

GitHub’s Copilot goes GA 

A year ago, GitHub launched a technical preview of Copilot, a new AI pair programmer that plugs into any source editor and offers coding suggestions. Despite there being a limited number of tester seats, those who started using GitHub Copilot told GitHub that the tool immediately became an indispensable part of their daily workflows. 

And why not? As soon as someone begins typing a particular line, the application recognizes it and offers suggestions on how to finish; it even closes the brackets correctly, and in order, each time.

As of June 21, GitHub Copilot became generally available to all developers. 

“The feedback we have heard and continue to hear substantiates our core thesis: AI can help make developers more productive and happier while coding,” GitHub principal researcher Eddie Aftandilian said in a media advisory. “Even still, we wanted to test our theory and see if GitHub Copilot itself actually leads to higher productivity rates among developers.” 

Productivity survey: Do Copilot’s suggestions help?

To find this out, GitHub’s research and engineering teams partnered to combine qualitative survey data from more than 2,000 U.S.-based developers with anonymized data to determine if developers believe GitHub Copilot is making them more productive — and if the data proves they actually are, in fact, more productive when using GitHub Copilot.

The idea behind GitHub Copilot is to help make developers happier and more productive by keeping them focused on what matters most: building great software, Aftandilian said. 

But “productivity” is a word that contains a wide range of possible practical meanings. Ideally, do developers want to save keyboard strokes, or do they really want to avoid searches on Google and StackOverflow? Should GitHub Copilot help them stay in the flow by giving them highly accurate solutions for mechanical, calculator-like tasks? Or should it inspire them with speculative stubs that might help unblock them when they’re stuck?

The answer is all of the above, to be sure. But GitHub’s research offered more specific answers. The study focused on answering three questions:

• Do people feel like GitHub Copilot makes them more productive?
• Is that feeling reflected in any objective usage measurements?
• Which usage measurement best reflects that feeling?

“We found developers didn’t care that much if they needed to rework the suggestion, as long as GitHub Copilot gave them a suitable starting point,” Aftandilian said. “And this makes sense: GitHub Copilot isn’t designed to build software by itself — it’s designed to offer helpful suggestions that make it easier to stay in the flow. Copilot offers developers the parts but leaves it up to them to assemble and design the finished product.”

One keystroke, multiple options

Does Copilot offer multiple options depending upon keystrokes entered?

“Copilot can return suggestions with as little as one keystroke,” Aftandilian told VentureBeat. “However, the quality of suggestions tends to improve with the amount of context available via both code and comments. Copilot frequently offers multiple suggestions via a ‘cycling’ menu, as well as the ‘Open Copilot’ side panel that displays 10 suggestions.” 

Developers tell GitHub they like Copilot, Aftandilian said, because it helps them stay in the flow longer and focus on application architecture rather than the syntax of specific APIs and other methods, etc.

“We’ve heard feedback from developers that even when Copilot isn’t exactly right, users appreciate its suggestions as a helpful starting point,” he said. “For example, Copilot will often suggest the right control structure (e.g., the structure of a loop) even if it doesn’t get the details perfectly right. The user can keep the structure but edit the details, and they find that helpful.”

You can find the final research here in a published academic paper that GitHub teams presented last month at PLDI’s MAPS ‘22 conference in San Diego.