OpenAI is developing a research program to evaluate the economic impacts of code generation models and is inviting collaboration with external researchers. Rapid advances within the capabilities of enormous language models (LLMs) trained on code have made it increasingly vital to check their economic impacts on individuals, firms, and society. Codex – an LLM developed by OpenAI by fine-tuning GPT-3 on billions of lines of publicly available code from GitHub – has been shown to generate functionally correct code 28.8% of the time on a sample of evaluation problems (Chen et al. 2021). This may occasionally have vital implications for the long run of coding and the economics of the industries that depend upon it. On this document, we lay out a research agenda to evaluate the consequences of Codex on economic aspects of interest to policymakers, firms, and the general public. We make a case for this research agenda by highlighting the doubtless broad applicability of code generation models to software development, the potential for other LLMs to create significant social and economic impact as model capabilities advance, and the worth of using Codex to generate evidence and establish methodologies that could be applicable to research on the economic impacts of future models. We propose that academic and policy research deal with studying code generation models and other LLMs in order that evidence on their economic impacts might be used to tell decision-making in three key areas: Deployment policy, AI system design, and public policy. To assist guide this research, we outline six priority consequence areas inside the realm of economic impacts that we intend to make use of Codex to check: Productivity, Employment, Skill Development, Inter-firm Competition, Consumer Prices, and Economic Inequality. For every area, we briefly discuss previous literature on the impacts of artificial intelligence on each of those outcomes, describe questions that we imagine to be key inputs to the three decision-making areas mentioned above, and supply examples of research that might be conducted with Codex. To catalyze work that builds off of this initial research agenda, we’re announcing a Call for Expressions of Interest from external researchers to collaborate with OpenAI researchers and customers to raised measure the economic impacts of code generation models and other LLMs.
Home Artificial Intelligence A research agenda for assessing the economic impacts of code generation models