GitHub makes Copilot generally available despite weak effectiveness

GitHub Copilot has been officially launched to the wider developer community after passing a 12-month technical preview trial conducted by 1.2 million developers.

The Microsoft-owned company said the AI pair programmer works across “dozens” of development languages and can offer suggestions to auto-complete blocks of code.

Validating concerns from developers, GitHub also said that a recent evaluation showed just 26% of all suggested completions were accepted by developers, meaning that around three in four suggestions are rejected.

Some concerns raised by developers relate to doubts over the accuracy of suggestions, the difficulty required in debugging suggested code, and how much time the feature will actually save.

GitHub said that more than 27% of users’ code files were generated by GitHub Copilot, on average, and the figure rose depending on the language used. The figure jumped to 40% for Python projects, for example.

“GitHub Copilot does not write perfect code,” the company said. “It is designed to generate the best code possible given the context it has access to, but it doesn’t test the code it suggests so the code may not always work, or even make sense.

“GitHub Copilot can only hold a very limited context, so it may not make use of helpful functions defined elsewhere in your project or even in the same file.”

Copilot may also suggest outdated uses of libraries or languages, due to it being trained on publicly available code, and that suggested code for some languages may be more accurate than others.

Python, JavaScript, TypeScript, and Go might perform better compared to other programming languages, it said.

The functionality of GitHub Copilot is powered by OpenAI’s Codex generative pre-trained AI model. The model was trained using natural language text, and publicly available code, including some from GitHub projects.

The downside to training such a model on publicly available code is that code samples are becoming increasingly difficult to come by, given the rising use of APIs or new libraries and frameworks.

Copilot will receive ongoing training from new public samples that become available, GitHub said, and the company will provide ways to highlight newer APIs and samples to raise their relevance in GitHub Copilot’s suggestions.

The new product is available now and costs $10 per month or $100 for an annual subscription (£8.17/£81.70) and be used in major integrated developer environments (IDEs) such as Visual Studio and Visual Studio Code, Neovim, JetBrains IDEs through an extension.

The product is free for verified students and maintainers of ‘popular’ open-source projects - whether the project meets the ‘popular’ criteria will be decided on the subscription page.

All users can redeem a 60-day free trial of GitHub Copilot, too, and the tool will be offered to companies later this year, the company said.