Buying AI tools is the easy part: AI benefits depend on redesigning how you operate

It has never been easier to buy artificial intelligence. Effective tools can be procured in weeks, their use piloted in days, and their benefits demonstrated to an executive committee by COB on Friday. The results, however, are not following the spending. S&P Global Market Intelligence found the share of companies abandoning most of their AI initiatives jumped from 17 per cent in 2024 to 42 per cent in 2025, with organisations scrapping nearly half their proofs-of-concept before production and implementation.

The pattern holds true in Australia. When we evaluated the Australian Government’s whole-of-government trial of Microsoft 365 Copilot for the Digital Transformation Agency, covering more than 60 government bodies, we found clear benefits but moderate use, with benefits realisation hinging on training, use cases tailored to each agency’s work, and adaptive governance. Much of this is about organisational conditions, rather than the capability of the AI tools

When an AI initiative underperforms, the instinct is to blame the product: the tool was immature, the vendor oversold, the model made things up. Sometimes one or more of these things is true, but the pattern in the evidence, and in our work with public and private sector organisations, points at something less comfortable. The tool can perform exactly as it was meant to, with the initiative still failing due to everything that was supposed to change around it but didn’t.

Buying an AI tool is the easy part. Benefits realisation requires organisations to do the harder work of redesigning how they operate.

Much of the initial enthusiasm around AI stems from its capacity to tackle discrete technical tasks. The benefit materialises, or doesn’t, in the system around it: the workflows it slots into, the people who must trust and verify its outputs, the data it draws on, and the governance that decides what it is allowed to do. The tool is one component of that system, and increasingly the cheapest and most replaceable one.

We saw this in our recent work with a government department that was exploring ways to apply AI to the analysis of legislative and policy documents across a broad portfolio. In this case, a specific use case and demonstration of a vendor tool provided tangible results.

However, broader discussions with the client began to uncover deeper, equally pressing considerations. How should the organisation approach AI adoption as an ongoing, iterative process, rather than as a one-off technological implementation? What would this implementation mean for the culture, capabilities, and operating model of the organisation?

This distinction is critical. While vendors are invaluable for building and fine-tuning AI tools, organisations face challenges that no generic AI product can solve. Identifying high-effort, high-impact areas for AI use, ensuring the ability to evaluate multiple tools over time, and avoiding vendor lock-in require a holistic strategy. Decision-makers need visibility into how AI will impact (and be impacted by) their culture, workflows, and readiness for transformation. This emphasis on the human side is key.

Being able to answer the question, “Can the tool do the task?” tells us nothing about whether the organisation can convert that capability into benefit.

The fastest way to build judgement about AI is direct contact with it, on the organisation’s own problems. We recently designed a three-day program with a state government central agency, in partnership with a government-backed AI collaboration initiative, that put senior leaders in front of advanced AI models and real problems drawn from their own functions. The point was to build first-hand judgement about what the technology can and cannot do, judgement that changes how those leaders frame business cases, set expectations, and sponsor change. Experimentation done this way generates buy-in and counters resistance precisely because it is grounded in the organisation’s own work rather than a vendor’s demonstration data.

As impactful as these exercises are, however, in terms of skills development and showcasing technical potential, experimentation holds broader implications for organisational culture. It can help foster buy-in, counter initial resistance, and make the case for transformational capabilities, whether by testing new workflows or exposing leaders to the role of AI in reshaping traditional hierarchies.

For many employees, AI remains opaque and quietly threatening, and unaddressed fear of automation will undermine even the best technology. In another engagement, an evaluation of a prototype supporting market regulation expanded from the tool’s efficacy to harder questions: what practitioners needed to understand about the tool’s limits, and how literacy would be built across teams so that buy-in was earned rather than assumed. That shift in scope helped the project to find the real problems that needed to be addressed. 

The least valuable use of AI is to make the existing process marginally faster. The benefit case strengthens as organisations move from automating tasks within current roles to re-imagining the roles, processes and decision rights themselves. Changing what work still needs a person, where judgement concentrates, and which steps exist only because the old constraints did.

NSW’s Smart Planning Approvals pilot shows the difference: rather than helping assessors process the existing queue faster, it uses AI to give applicants a “first pass” compliance check on complying development applications before lodgement, moving the checking step earlier in the process and freeing council planners to concentrate on complex proposals where their judgement matters. A useful test for any AI initiative: if the “after” picture is the “before” picture with fewer hours attached, the thinking is not finished.

The strongest objection runs like this: the technology is improving so quickly that organisational redesign is wasted effort. Whatever friction exists today, a more capable model will soon dissolve it, so the rational move is to wait.

The objection misreads where the constraint sits. Each model generation raises the ceiling of what is technically possible, but the technical ceiling was never the binding constraint. An organisation that could not realise benefits from this generation of tools is unlikely to realise them from the next, because the things that blocked it – unredesigned processes, unprepared people, unclear decision rights – are untouched by a model upgrade. IBM’s analysis of why AI returns disappoint reaches the same conclusion: culture, governance, workflow design and data strategy are the main constraints on value, not the technology.

Properly understood, the rate of improvement is an argument for more organisational work, not less. If capability shifts every six months, redesign cannot be a one-off retooling around a single product. Part of the required shift is building resilience, agility and adaptability into the workforce itself: people who are confident experimenting with each new generation of tools, processes designed to be revised rather than re-engineered from scratch, and structures that let the organisation absorb the rate of technological change rather than be periodically disrupted by it. Organisations that build this adaptive capacity are positioned to compound their advantage with every model release. Organisations that buy tools without it repeat the same struggle each time.

For a senior leader accountable for a function where AI is expected to lift performance, this argument lands at three moments.

If you are preparing the case for adoption, weight it toward the organisational work. A case that itemises licences and integration costs but treats process redesign, capability-building and governance as implementation detail is a case for buying a tool, not for realising a benefit. The organisational line items belong in the case and must be costed, scheduled and owned.

If you have already selected the tool, the work has not finished. Procurement answered the easier questions. The ones that decide whether benefits arrive, such as whose role changes, which process gets redesigned, who is accountable for adoption, are still open. For as long as they stay open, the organisation will pay for capability it cannot yet convert into benefits.

And if you are considering shutting down an AI initiative because the benefits have not been realised, diagnose before you decide. Establish whether the tool failed or whether the organisation never changed around it. A capable tool retired because the redesign never happened is a failure postponed, waiting to repeat itself with the next tool.

The honest post-mortem question is not: “Did the technology work?” 

It’s: “Did we?”

Get in touch to discuss how to turn AI adoption into measurable organisational benefit.

Connect with Will Prothero on LinkedIn.