How businesses should actually measure AI success

Sarah Friar, the finance chief at OpenAI, recently proposed a new way for businesses to measure the return on their AI investment. Rather than focusing on how much data a model can process, she suggests companies switch to practical metrics that measure real-world output. This includes tracking the cost of completing specific, successful tasks, checking how dependable the results are, and calculating the return on the expensive computing power required to run the systems.

To understand these measurements, think of an AI model as an incredibly well-read assistant that has memorized the structure of language from billions of pages of text. When you give it a prompt, the system isn't really thinking; it is playing a complex game of completion. It looks at your text and checks its internal patterns to guess what word, or fragment of a word, should come next. It repeats this process thousands of times per second until a response is finished. Traditionally, companies have measured success by the size of this internal knowledge base, which is like judging an employee by the size of their library. These new metrics act as an audit: by tracking the cost per successful task, the company calculates the actual price of getting a correct, finished job, rather than just the price of paying for the assistant's time. Dependability acts as a filter for reliability, measuring how often the work holds up without needing a human to double-check or correct it.

This shift represents a move toward maturity. When a new technology first arrives, businesses often treat it like a magic trick and ignore the usual rules of accounting. Now that the initial excitement is fading, the same logic used to measure any other business expense is finally being applied to AI. If a company cannot define what a successful task looks like or how much it costs, they are likely just experimenting rather than building. This is the difference between treating AI as a science project and treating it as a standard tool in the company toolbox. If a tool costs more to run than the value of the work it produces, even the most advanced AI is eventually going to be cut from the budget.