Skip to main content

IBM’s CodeNet dataset aims to train AI to tackle programming challenges

IBM Logo
IBM logo is seen on Gae Aulenti square in Milano, Italy, on December 23 2019
Image Credit: Mairo Cinquetti/NurPhoto via Getty Images

Join us in Atlanta on April 10th and explore the landscape of security workforce. We will explore the vision, benefits, and use cases of AI for security teams. Request an invite here.


At its Think conference this week, IBM introduced Project CodeNet, which the company claims is the largest open source dataset for benchmarking around AI for code. Consisting of 14 million code examples, 500 million lines of code, and 55 programming languages including C++, Java, Python, Go, COBOL, Pascal, and FORTRAN, CodeNet is approximately 10 times larger than the next most similar dataset, which has 52,000 samples.

According to a study from the University of Cambridge’s Judge Business School, programmers spend 50.1% of their work time not programming; half of the rest of their time is spent debugging. And the total estimated cost of debugging is $312 billion per year. AI-powered code suggestion and review tools, then, promise to cut development costs substantially while enabling coders to focus on more creative, less repetitive tasks.

CodeNet focuses specifically on the problems of code translation, code similarity, and code constraints. The goal is to advance the development of AI systems that can automatically translate code into another programming language, identify overlaps and similarities between different sets of code, and customize constraints based on a developer’s specific needs and parameters.

Programming language translation could be especially useful, given that migrating an existing codebase to a modern or more efficient language like Java or C++ requires expertise in both the source and target languages. For example, the Commonwealth Bank of Australia spent around $750 million over the course of five years to convert its platform from COBOL to Java. Transcompilers could help in theory — they eliminate the need to rewrite code from scratch — but they’re difficult to build in practice because different languages can have a different syntax and rely on distinctive platform APIs, standard-library functions, and variable types.

VB Event

The AI Impact Tour – Atlanta

Continuing our tour, we’re headed to Atlanta for the AI Impact Tour stop on April 10th. This exclusive, invite-only event, in partnership with Microsoft, will feature discussions on how generative AI is transforming the security workforce. Space is limited, so request an invite today.
Request an invite

The CodeNet dataset

CodeNet contains samples designed to train AI to complete a range of programming tasks, including code search and clone detection. Beyond this, the dataset has metadata and annotations with a rich set of information spanning code size, memory footprint, CPU run time, and status, which helps to distinguish correct code from problematic code.

Over 90% of the sample problems in CodeNet come with descriptions that contain a problem statement and specifications of the input and output format. For over half of the problems and seven million examples, IBM also curated sample inputs and outputs from the problem description.

Using CodeNet, data scientists can execute code samples to extract additional metadata and verify outputs from generative AI models for correctness. IBM says that this will enable researchers to program “intent equivalence” when translating one programming language into another.

“Given its wealth of programs written in a multitude of languages, we believe Project CodeNet can serve as a benchmark dataset for source-to-source translation and do for AI and code what the ImageNet dataset did years ago for computer vision,” Ruchir Puri, IBM fellow and chief scientist at IBM Research, wrote in a blog post.

IBM isn’t the only company pursuing AI-driven code completion and auditing. Codota is developing a platform that suggests and autocompletes scripts in Python, C, HTML, Java, Scala, Kotlin, and JavaScript. Ponicode taps AI to check the accuracy of code, and DeepCode is developing an AI-powered system for whole-app code reviews (as are Amazon and Intel). Perhaps one of the most impressive projects to date is TransCoder, an AI transcompiler Facebook researchers developed to convert code from one programming language into another. Another contender is a model from OpenAI that was trained on GitHub repositories to generate entire functions from English-language comments.

VB Daily - get the latest in your inbox

Thanks for subscribing. Check out more VB newsletters here.

An error occured.