Beware the Sirens of AI Speed

There’s some good news, some less good news, and a healthy dose of realism in this latest dispatch. AI is expensive, that much is clear – but worth it if it achieves things faster or at greater scale (than the competition), right? Let’s unpick this.

First on the costs, the problem is that AI is open-ended. In principle, the more you spend, the more you get done. There is no ceiling to what can be achieved, however, rocket fuel still has a cost per litre. A fair illustration comes via OpenAI’s own research – costs per researcher have increased approximately ten-fold since the start of this year. Yes, they’re doing ten times as much but this (running at $600-7000 in tokens per researcher per day currently) is not infinitely scalable.

The ceiling is literally the size of the corporate piggy bank, so better spend wisely. Which brings to – are organisations doing that? Certainly, the brightest and best have been coming up with ways to reduce your bills, from response caching to better initialisation. Hyperscalers are in for a pound releasing FinOps agents, comfortable in the knowledge that right now the biggest job is to keep people on their platforms (this is not the time to worry about undercutting one’s business). Cost management is also driving broader trends, not least how US companies are outsourcing 60% of token traffic to China, says OpenRouter.

Moving away from AI efficiency and cost cutting, is it being effective and delivering on its potential bang per buck? Looking at the software development world, substantial evidence points towards the opposite. For example, the killer finding of a report from Undo is that release cycles are the same as without AI: developer time may be faster, but effort has moved to debug. No wonder that cyber tools company Codenotary’s recent informal survey of developers suggests a lack of trust among teams.

Meanwhile, GitClear’s Maintainability report suggests more code is not necessarily better code, nor does it lead to improved business outcomes (I’m not a huge fan of the word ‘outcomes’ but the point stands). In his focus on ROI, (AI dev startup) Qodo’s CEO Itamar Friedman channels DevOps’ The Phoenix Project (itself an homage to one of my faves, Eli Goldratt’s ‘novel’ The Goal which covers manufacturing efficiency). AI benefits/AI costs = AI ROI, says Itamar.

Good ROI accuracy is a question of context, however. Reduced pull requests and software development velocity could be seen as ROI for the director of software delivery, but (as Phoenix Project authors make clear) these criteria don’t matter a jot if they’re not linked to business benefit. Here’s the good news: the team behind UC Berkeley’s Agent’s Last Exam have been tackling just this. They’ve mapped out tasks based on 55 business scenarios, and have ranked a fair few frontier models against them.

Ultimately, organisations will keep on top of spend by being smart about what it is for. Bluntly, if you don’t know what your targets are for building new systems (or doing anything else), chances are you won’t hit them. Don’t be drawn by sirens screaming about speed, or you may find yourself thrown on the rocks of despair when it comes to delivery.