The AI Gap Isn’t Technical: It’s What You’re Afraid to Lose

Last semester, a group of University of Sydney master’s students (myself among them) had a main assignment: to hack (find vulnerabilities) apps and websites operated by relevant organizations and report their findings to the relevant stakeholders. The more critical the vulnerability and the stronger the evidence, the higher the score. What made the Introduction to Cybersecurity unit (INFO5995) different was that students could use AI (responsibly) to build automated pipelines that would help identify security-related bugs 24/7.

After a three-month competition, the results were incredible. 71 vulnerabilities found in the digital assets of companies across diverse industries and even government organizations. The protagonists were students with an average age of 25 who, for the most part, had no background in Cybersecurity.

In mid-August I attended an applied AI talk organized by the PMI Sydney Chapter, where several industry professionals gathered to discuss how companies across the city were implementing these new technologies. When the speaker asked how many people had the “agents” option enabled on their team’s Copilot, the vast majority just looked at each other. “I know how to use it, but my company hasn’t enabled that option yet,” someone said.

The contrast is obvious. The speed at which we are witnessing the technological advance of LLMs is clearly becoming an adaptation challenge for many incumbent companies. The reasons are completely valid: concern for customer data security, the real capabilities of teams, and the real impact on P&L. What is certain is that nothing is going to stop this race, and it is worth analyzing what could be limiting, at the source, the adoption of these new technologies.

Same technology, two interpretations

In a world where anyone has free 24/7 access to the same LLMs, our teams still face various limitations in using agents day-to-day to automate routine work. As Ben Thompson explains in his latest Stratechery piece, the technological progress driven by AI is the same for every organization, but only some are actually prepared to turn it into a transformational capability.

Thompson revisits Clayton Christensen’s old distinction between sustaining and disruptive innovation and argues AI is set up to be both at once, depending on who is holding it. For an incumbent, adopting AI is evaluated against everything already built: the existing customer base, the existing revenue, the existing reputation. Any mistake can potentially bring a huge loss, so the rational move is to keep a human in the loop, move carefully, and treat AI as a productivity layer on top of what already works. As Thompson puts it, that risk calculus “will make AI sustaining, but nothing more.”

For a startup, the calculation flips entirely. Their base case is already failure: there is no existing revenue to protect, no reputation to defend. That means anything that raises the odds of survival is pure upside, with nothing to lose. So startups lean into AI without the hesitation an incumbent can’t afford to skip: for them AI is effective disruption. Same technology, same models, same access, but one side is protecting a fortress and the other has nothing behind them but the horizon. Clearly different incentives, which in the long run will bring very different outcomes.

GenAI pilots are failing

Exactly one year ago, MIT’s State of AI in Business report revealed that the billions of dollars invested in GenAI pilots are not delivering results: 95% of pilots in this space fail. Behind the brilliant demos and slides shared at launch, many of these pilots have no transformational impact and run out of steam the moment they collide with real organizational fabric: data quality, human judgment, compliance, politics, etc. What does the 5% of pilots that survive do differently? They design for friction and confront it head-on to force adaptation. They embed GenAI into high-value workflows, redesign them instead of layering another tool on top, and learn from user feedback by establishing improvement loops.

Retrieval-Augmented Generation (RAG) systems are a clear example of the incumbent instinct: reach for the tool, skip the redesign. Most companies assume a failing RAG pipeline needs a better, more expensive model, when the real fault typically sits earlier, in retrieval itself. As PM NorthStar put it in this year’s debate, the naive RAG pipeline “everyone copy-pasted in 2023” is on its way out, not because retrieval doesn’t work, but because nobody bothered to notice it was broken. The RAG systems that actually survive contact with production don’t retrieve blindly on autopilot: through reasoning loops, the model decides whether it even needs to search, forms its own query, and judges whether what came back is enough. The teams that succeed are the ones that took the time to match the retrieval strategy to the actual problem, instead of shipping the first pipeline that technically worked. Friction, once again, is what separates the pilots that survive from the ones that don’t.

Will the tool save us, or will we save ourselves?

Technologies change, the behavior repeats. Almost 10 years ago I faced the same problem when purely digital banking sales were the new trend and, as a Product Lead, I was responsible for bringing them to life. Everybody wanted to get their products live as soon as possible, so you could feel the pressure in the air.

Previously, three online forms had been built to enable customers to open bank accounts, apply for credit cards and take out loans online. Quick deployment with no architecture discussion. None of them delivered results. It wasn’t until we stepped back, reviewed the scope of the entire project, challenged the internal workflows, set the right metrics and defined how the solution would scale that we finally started seeing results. Of course, it took sweat and tears. Friction is there to be embraced, not skipped.

So while AI keeps advancing at an accelerated pace, the clearer it becomes that the main obstacle to its maximum impact is us: our incentives, our tolerance for risk and our willingness to actually do the hard work.