GitHub Copilot Claims to make Developers 1.55x more Productive 👩🏽‍💻💻🚀

Is it creating more time and space for developers to focus on solving bigger problems and building even better software?

Hey Everyone,

You can listen to the VP of GitHub on Lenny’s podcast here. So GitHub Copilot is really trying hard! But Microsoft’s Github Copilot didn’t just want to measure developer productivity, but developer happiness as well.

It is just me, or is there an underlying desperation in their marketing?

Their event is also happening in November:

🎉 #GitHubUniverse 2022 👇
🐦 20% off early bird tickets https://github.co/3edd84R
🗓 Nov. 9-10 (with a virtual component as well).

Microsoft as you know bills GitHub Copilot as your A.I. pair programmer. In theory I really like the idea!

GitHub Copilot uses the OpenAI Codex to suggest code and entire functions in real-time, right from your editor. It is generally available to all developers for $10 USD/month or $100 USD/year. It will also be free to use for verified students and maintainers of popular open source projects.

But does it really make you more productive and happy? Those are big claims.

Some of the claims are sort of creepy, somewhere between sketch and cringe, but that’s sort of normal for Microsoft’s branding. Still if there claims are true, it’s sort of a big deal. This is not a sponsored post, I really wanted to understand their claims and so-called study.

👨🏻‍💻 What Exactly is GitHub Copilot?

GitHub Copilot is an AI-based system that will take notice of what you are coding and make suggestions for what it is you are about to type. You can think of it as IntelliSense on steroids, but users are claiming that it is a big step up. The idea is all based on the sort of AI that lead to the GP-3 language system. Take a very big network and train it on a huge corpus of natural language data and it will extract the regularities.

The GitHub Next team conducted research using a combination of surveys and experiments designed to try to operationalize the benefits in an academic sense.

When GitHub first launched a technical preview of GitHub Copilot in 2021, their hypothesis was that it would improve developer productivity and, in fact, according to the company, early users shared reports that it did. In the months following its release, they wanted to better understand and measure its effects with quantitative and qualitative research. To do that, they first had to grapple with the question: what does it mean to be productive?

👓 Microsoft has Big Dreams when they Acquired GitHub

Microsoft acquired GitHub for $7.5 Billion in 2018, just four years ago.

Because AI-assisted development is a relatively new field, as researchers they have little prior research to draw upon. Obviously if a company measures its own product, they will want to paint it in a favorable light, but let’s try to pretend that they are objective academics.

They wanted to measure GitHub Copilot’s effects, but what are they? After early observations and interviews with users, they surveyed more than 2,000 developers to learn at scale about their experience using GitHub Copilot. Then, they designed their research approach with three points in mind:

1. Holistic

2. First-hand perspectives

3. Everyday development scenarios.

• Look at productivity holistically. At GitHub we like to think broadly and sustainably about developer productivity and the many factors that influence it. We used the SPACE productivity framework to pick which aspects to investigate.

• Include developers’ first-hand perspective. We conducted multiple rounds of research including qualitative (perceptual) and quantitative (observed) data to assemble the full picture. We wanted to verify: (a) Do users’ actual experiences confirm what we infer from telemetry? (b) Does our qualitative feedback generalize to our large user base?

• Assess GitHub Copilot’s effects in everyday development scenarios. When setting up our studies, we took extra care to recruit professional developers, and to design tests around typical tasks a developer might work through in a given day.

Do you want to start using GitHub Copilot today? Get started with a 60-day free trial, and check out our pricing plans. It’s free to use for verified students and maintainers of popular open source software.

🖥️ Benefits of Using GitHub Copilot

I’m not a developer so I’ll just state what the company says here:

GitHub Copilot enables developers to

• Get AI-based coding suggestions: Get code suggestions that match a project’s context and style conventions, and cycle through different options to decide what to accept, reject, or edit.

• Use your preferred environment: Integrate GitHub Copilot with popular editors, including Neovim, JetBrains IDEs, Visual Studio, and Visual Studio Code as an unobtrusive extension.

• Code confidently in unfamiliar territory: Code in new languages or try something new, and let GitHub Copilot suggest syntax and code in dozens of languages—so you can spend more time learning by doing.

💾 Results of Surveys

• Being more productive with repetitive tasks seems to be a featured benefit.

Curiously Microsoft wanted to portray how it could make “coding more fun”.

(With Copilot) I have to think less, and when I have to think it’s the fun stuff. It sets off a little spark that makes coding more fun and more efficient.

- Senior Software Engineer

I found this particularly amusing. Being augmented by A.I. in our job it turns out makes developers and software engineers happier.

GitHub even conducted a test to show how much faster and speedier GitHub Copilot made software engineers. Apparently, developers who used GitHub Copilot completed the task significantly faster–55% faster than the developers who didn’t use GitHub Copilot.

So then the PR is to say that Copilot is like your coding buddy, or new BFF. The new study by GitHub shows that Copilot, its AI code programming assistant, results in a significant increase in developer productivity and happiness. Copilot uses Codex, a specialized version of GPT-3 trained on gigabytes of software code, to autocomplete instructions, generate entire functions, and automate other parts of writing source code.

Listen guys whatever makes developers and Microsoft happy, I’m all for A.I. tools augmenting us in our repetitive tasks. I totally understand Microsoft wants ROI and revenue from its acquisitions.

GitHub Copilot wouldn’t be possible without GitHub’s vibrant community of students and creators. While AI-assisted coding is still a new field and needs more research, GitHub’s study provides a good look at what to expect from tools such as Copilot. At at time when rage and awe are concerning artists and designers regarding text-to-image generation, I wonder what other sorts of tasks upgrades will do to us in other professions.

If you’re a student and want to participate in the program, apply for the GitHub Student Pack to get started. And if you’re an open source maintainer, check out our FAQ to see if you qualify to start using GitHub Copilot at no cost.

As for happiness and job satisfaction that’s going pretty far to claim. According to the GitHub’s findings, 60–75% of developers feel “more fulfilled with their job, feel less frustrated when coding, and can focus on more satisfying work” when using its Copilot tool. My internal devil’s advocate alarm bells are raging, but maybe you are more chill. Since I’m not a developer I can’t test it myself to see if I’m pacified into a new coding zone with Microsoft’s AI brains beside me.

Most academics who have looked at the study closely say that it's probably premature to say that Copilot has had an effect on students, because the software has only been publicly available for a few months.

Still I found it pretty interesting that Microsoft and GitHub are trying to quantify the benefits like this. I wonder if this means their uptake of GitHub Pilot is going well or going poorly. According to GitHub, more than 1.2 million developers used Copilot's technical preview in the past 12 months. Given how many actually use GitHub, that doesn’t sound to me like a huge number.

Join 33 other paying subscribers for additional content and locked archive posts.

I’ve started a new Newsletter where I will try to post content for software engineers and technology workers called Data Analytics Future, which might evolve into something interesting.

If I don’t see you again, have a good weekend.