Why your Data/AI Transformation is doomed to fail

and what you can do to make it succeed!!

It's 2015, and I'm in our monthly leadership team meeting when the CEO announces the launch of a transformation project to make the company "data-driven" by setting up a data lake. Fast forward to 2023 and I'm in our monthly leadership team meeting with a different executive team, and the CEO making the exact same announcement — only now it's an "AI-first” transformation.

Look, I've been in this data analytics game for 20 years now, and honestly? It's just one buzzword rollercoaster after another. I've watched the industry go data warehouses -> In memory databases -> Big Data and Data Lakes where all you had to do was dump data and data modelling was suddenly ‘legacy’ and unnecessary -> then you needed to model data again and the go-to was Data Vault -> then suddenly everyone needed a Data Mesh -> and now its all about GenAI and prompt engineering. The corporate amnesia is very impressive.

I've had front-row seats to some epic fails – like watching a Fortune 500 company flush hundreds of millions down the toilet on a data lake that turned into a “data swamp." Or my personal favourite: a finance company that announced a massive investment on an AI strategy with a straight face while 95% of their reports were still being cobbled together in Excel by Bob in finance who was "pretty good with spreadsheets."

Signal from Noise

• Data transformations are bloody hard, and most fail.

• Transformations often fail as they focus on hot trends and ignore core foundations, and face change resistance.

• Simply implementing new technology or tools does not deliver business transformation.

Behind the Dashboard

Every CEO wants to be AI first. Every company wants to be ‘data-driven’. Everyone wants to transform their business by doing <insert buzzword of the week>. And yet, many crash and burn spectacularly.

The movie script is pretty much predictable at this stage. It’s early Tuesday morning, you’re caffeinated beyond what's medically advisable, and CEO kicks off the monthly leadership team meeting. Here's how it typically goes down:

CEO: "We need to be 'AI-First'! The board is breathing down my neck about this!"

CDO: "Great, we should start by addressing our legacy systems and fixing our data quality issues. Remember that 20% of our critical customer data attributes are inconsistent or incomplete, not to mention this data is spread across 3 CRM’s and 2 marketing systems, none of which have any common identifiers. We need to start by cleaning and consolidating our Customer data in one spot - we need a Customer360.”

CEO (looking annoyed): "Yeah yeah, but I need quick wins. McKinsey says early AI adopters gain an 'unfair advantage.' The board wants us to be an AI-First company by Q4."

CMO (jumping in): "Absolutely. We have recently announced our ‘AI Driven personalisation’ strategy - we need an AI model that tells us what Bob in Tasmania will eat for breakfast this Sunday. Can I have an MVP deployed for next month's board presentation."

Me (buckling under pressure): "Sure!… Let me mobilise a transformation project.”

Sound familiar? I've buckled under this pressure more times than I'd like to admit. My personal low point: green-lighting a $2M project to build a mass personalisation model knowing full well our customer data was scattered across 17 different systems, none of which shared common identifiers. The result: six months later, our consultants had built us a beautiful dashboard, sitting on an array of spreadsheets that were guesstimating connecting incomplete information about customers. Yikes! We tried recovering the situation by hastily setting up a skunkworks BAU project in the background to fix the our data in the spreadsheets which obviously didn’t work! What worked was intentionally focussing on building scalable data foundations - investing in a Customer 360, defining how data from different systems connect and identifying and resolving data quality issues. Something that I had suggested in the first place!

Here's the raw truth: Executives want quick results with minimal investments. They don’t understand why data foundations matter, and they absolutely hate how long building them takes. Can you blame them? They're being bombarded with articles screaming that "AI laggards will die!" while their board members return from Davos asking why competitors are doing cool AI things and they aren't.

The second ugly reality? Most "data transformations" are just glorified IT projects. This usually happens when technology leaders drive the agenda. With vendors constantly coming out with shiny new tools and selling FOMO, it's easy to mistake buying tech for actual transformation.

I've seen companies drop millions on data lakes, visualisation tools and data mesh platforms only to end up with data swamps, low platform expansion and expanding their spreadsheet farm by adding Tableau/PowerBI reports. Tech & tools matters, but they are just enablers.

Let's be brutally honest: transformations are hard. McKinsey says 70% fail. For data transformations? I reckon it's closer to 80-85%.

What separates winners from losers? The successful data transformations I've witnessed weren't about technology at all—they were fundamentally change initiatives. While AI is genuinely disruptive (and I'm not downplaying its potential), the real challenge isn't implementing AI models or turning on Microsoft Co-pilot. It's rewiring how people make decisions, redesigning core business processes, and sometimes completely transforming business models.

Playbook || Data -> Insights -> Action

After watching a few data/analytics/AI transformation projects implode spectacularly (and cleaning up the wreckage of many more), I've put together this playbook that actually works. Nothing fancy – just an approach and practical tips that I have learnt from my experience. While many of these may seem obvious, it’s surprising how many companies ignore the obvious and chase the exotic.

Building the case - Clearly define the “Why”

Your CEO doesn’t really care about a data lake or an AI model. What they really care about is to make more money, stop bleeding money, and keep customers & stakeholders happy. Everything in your company should be in service of those goals, including your data transformation.

As CDO, your first job is to figure out what problem you're really solving. Your problem statement can’t be a motherhood statement like "become data-driven”; you need to find real business problems that keep executives up at night (am sure there are plenty around).

The formula is pretty simple

1. Pick the biggest specific business problems in your company (revenue drop, customer churn, operational bottlenecks, regulatory scrutiny)

2. Quantify the pain ($$ lost, time wasted, opportunities missed, fines incurred, reputational damage)

3. Get specific about how data/analytics/AI will actually fix it, inc. defining clear success metrics

Bonus points if the metrics in your business case should be tied to your CEO’s bonus.

You also need to find a way to frame and articulate the business case in very simple terms and create an elevator pitch. And you need to communicate this to everyone involved again and again and again. When someone asks "why are we doing this?”, everyone from your data analyst to your CEO should be able to answer this question, hopefully with the same answer.

Side note on technology disruptions: It’s perfectly fine to want to jump on the bandwagon and position your company as an early adopter, even when the strategy and benefits case is not 100% clear. To be successful, the trick is to clearly define how these technology disruptions can be harnessed towards business value.

As an example, frame your GenAI initiative as ‘To address service performance issues, use GenAI to automate sentiment analysis & service quality analysis in real-time from voice calls made to the call centre, and implement a workflow to provide real-time feedback to the contact centre agent and team leader. Expected improvements include 15% increase in first time resolution, resulting in reduction of $1.75m agent time, and 20% improvement in NPS’.

The strongest business cases aren't about the technology at all – they're all about connecting data analytics capabilities to tangible business outcomes in ways even your most technically ignorant executive can understand and care about.

Executive sponsorship - Get authority from the right person

Now that you've built a solid business case that connects data & analytics to actual business outcomes, you need someone with enough organisational firepower to make it happen. Even the most brilliant business case if useless if you don’t have the right executive sponsoring it.

Directors of Analytics or mid-level management sponsoring a project driving change across customer, marketing and sales is like a Stockman on an Australian cattle station trying to muster cattle with a feather. Sounds harsh, but it’s true. Weak sponsors kills projects. Weak sponsorship can take form of absent sponsors, sponsors without enough power to influence the organisation, or sponsors who aren’t bought into the need for the project.

I learned this lesson the expensive way. Five years ago, I led a data governance initiative with the enthusiastic backing of our Head of Marketing. We had quantified the business case perfectly (as I outlined above)—poor data quality was costing us $3.2M annually in wasted marketing spend. Everything was great until we needed IT to change their data collection processes. The CTO, who hadn't been brought in early, simply said "not a priority”. The Sponsor, our Head of Marketing, tried his best but he was too organisationally weak to make any difference. And that was that. Project flatlined. Three months and $400K down the drain because I had the wrong sponsor.

The ideal Sponsor is one who has influence over all impacted parts of your company. For many data analytics projects, this in theory translates to the CEO being the sponsor. But is the CEO really going to care about your piddly customer segmentation project when they have a large company to run and shareholders to satisfy - realistically, no. In most cases, your CEO would be an absent sponsor which you don’t want.

Your best option is a C-level executive with both political capital and staying power. This needs to be someone who benefits directly from your project so they have a vested interest in the success of your project. Also, don’t pick a shared services Chief (e.g. Chief Strategy Officer, Chief Operations Officer etc.) to deliver a project that delivers benefits across business units. In my experience, this is the worst thing you can do and this often results in the change project being executed (or at least perceived) as an ivory tower, and lack of buy-in from the business units.

Meet with your sponsor and make sure they understand what they are signing up for. Your sponsor needs to act as your cheerleader and must be willing to defend you and your project in times of crisis, make sure you explain this and they are willing to sign up for this. Explain what changes your project may deliver, especially if it involves people changes and get their support. Set clear expectations and outline and the level of commitment you require, and agree on a cadence, roles and responsibilities and ways of working between you and your sponsor.

In case you can’t find a single executive to sponsor your project, have two executives to act as joint sponsors. While am not a huge fan of multiple sponsors and I believe strongly having only one decision maker, having joint sponsors can sometimes be the only way to secure executive commitment to your project. You just need to make sure you set expectations from the two sponsors clearly and define a RACI if required. Be prepared to negotiate outcomes between the sponsors in cases of conflict. A co-sponsorship model can be used effectively to drive transformation, but it often means more work for you, so keep this in mind.

Push back hard against any notion of having more than one or two sponsors. I once led a transformation project that was co-sponsored by 4 out of the 7 Chief’s in our company - needless to say the project didn’t go well and my mental and physical health suffered greatly during this period.

Remember: weak sponsors = expensive failure + poor mental wellbeing. Strong and supportive sponsors = probable success + a good night’s sleep.

Get the right leadership in place

Your next job is to assemble your leadership team. You need a balanced group of specialists who can tackle different aspects of this complex challenge. Don’t be tempted to set up roles that wear multiple hats.

I've usually start with the following key roles while initiating a project:

1. Data Transformation Lead

• What they do: Overall accountability for delivery, stakeholder management, roadmap ownership

• Key traits: Political savvy, executive presence, communication skills, decision-making ability

• Background: Often an external hire, previous experience as an executive or Senior Director. I usually do not recruit generalist project managers.

• Red flags: Avoids conflict, gets lost in technical details, can't communicate with executives

• Reality check: This person should spend 60% of their time managing up and across, not down

2. Business Lead

• What they do: Ensures solutions actually solve business problems, manages expectations

• Key traits: Domain expertise, credibility with business users, analytical thinking

• Background: experience in frontline and senior operational roles in the business, passion and experience in using data and analytics in business operations. This person has to come from within the business and cannot be an external hire.

• Red flags: Can't challenge what the business think they want vs what they really need (everyone wants a faster horse), doesn't understand data analytics technicalities and limitations

• Reality check: This person will save you from building technically brilliant solutions nobody uses

3. Data & Analytics Product Manager

• What they do: Define products strategy and roadmap, design reusable and fit-for-purpose products and manage lifecycle from development to launch to ongoing maintenance

• Key traits: Deep data analytics acumen, product mindset, domain expertise

• Background: This is an upcoming role in the industry. I usually look for people who can demonstrate experience delivering business outcomes with data analytics. This is typically not a data engineer, but a business aligned data/business analyst, business lead, or a business operations leader. Prior product management experience is not essential as this can be taught.

• Red flags: Analysis paralysis, emphasis on technical features as opposed to business features

• Reality check: Balancing innovation vs data governance/risk is a constant challenge.

3. Data Architect

• What they do: Technical strategy, architecture decisions, platform selection

• Key traits: Systems thinking, pragmatism, ability to simplify technical complexity

• Background: previous background in hands-on engineering roles and experience working at a program level. I usually avoid hiring candidates who resemble ‘enterprise architects’ as they operate at a very high level and aren’t willing/able to deign practical solutions and clear blockers

• Red flags: overcomplicated solutions, "perfect" being the enemy of "good”, inability to communicate plainly

• Reality check: You want someone who's built and scrapped several architectures, not a theorist

4. Change & Adoption Manager

• What they do: User adoption, changing behaviours and mindsets, implementing new ways of working

• Key traits: Empathy, high EQ, enthusiastic and high energy, ability to deal with people (harder than you think)

• Background: Change managers stump me! I haven’t yet been able come up with a consistent way to find great change managers. All I know is finding great change mangers is more challenging than finding any other role in my teams. So my approach these days is to cast a wide net and be open to candidates from any background.

• Red flags: over focussing on delivery aspects of change (ticking checkboxes) and not enough on people

• Reality check: Without this role, you'll build things nobody uses or understands.

Did you notice a pattern in how I recruit key leaders - I tend to avoid people who are just capability experts. Instead I look for domain experts who have experience in applying capability to deliver business outcomes. As an example in interviews, most change managers will focus on developing change management strategies, change plans, stakeholder maps, organisational change load and change resistence heat maps. All these are no doubt very important but I look for change managers who will talk about how they improved their company’s ‘near miss incidents reporting rate’ by 20% in their first 3 months. Guess which one you want.

Realistically assess your company for change readiness

On my first day in my new company, 3 people make the following comment - ‘we have tried running a data change project 3 times in the past 5 years, all led by accomplished leaders, which have failed. The state of our data is still abysmal. Why do you think it’s going to be different this time?’. I quipped - ‘it’s going to be different this time because I am leading the project’. Needless to say I didn’t inspire much confidence.

The reality was my company was burnt by previous unsuccessful attempts to transform the way they use data. They were very cynical and had really low expectations. In its current state, my company wasn’t change ready!

You need to remove your rose tinted glasses and critically assess the following aspects of your company, and factor limitations into delivery of your project -

1. Readiness to Change

• Past transformation scars: If your company is littered with failed transformation attempts, expect resistance.

• Incentive alignment & decision making culture: Do performance metrics support data-driven decision making? If sales leaders get bonuses based on gut feelings, your fancy prediction models won't matter.

Change fatigue: Is your company already juggling multiple transformation initiatives? Adding another might just result in organisational exhaustion.

2. People Capability

• Are your data experts just Excel jockeys? Do they still hoard data and are resistant to a self-service model?

• Do people have basic data literacy skills and understanding of basic data concepts?

• Do your data teams have enough domain expertise or do they just have technical skills?

• Do you have enough people with skills to use modern tools e.g. cloud data platforms, pyspark, PowerBI

• Do you have people who are strong in data governance, risk and regulations? Not just on the policy side but also on the operational side

3. Technical Reality

• Data quality truth: Run actual data profiling on critical systems and understand how quality of data may impact your outcomes. If this is a real issue, you are better off starting with a data quality remediation program as the first step

• Integration complexity: Map data flows between systems to understand the real spaghetti mess you're dealing with. What looks like three systems on PowerPoint is usually fifteen with manual handoffs between them.

• Technical debt: Some legacy systems are duct-taped together so precariously that touching them causes cascading failures. Identify these landmines early.

4. Cultural assessment

Should you adapt your transformation to fit your culture, or push to change the culture? In my experience, you need to do both. The trick is finding the right balance.

Where culture is strong and positive: Design your approach to work within existing strengths. If your company excels at cross-functional collaboration, leverage that for data governance.

Where culture is toxic or counterproductive: Be explicit about the behaviours that need to change and build them into your transformation plan. If hoarding information is rewarded, you'll need specific interventions to enable data sharing.

Your assessment doesn't need to be perfect, but it needs to be honest. False optimism is your enemy, and in particular plagues people who have been at a company for a long time and hence are institutionalised - avoid this at all costs.

5. Define a Realistic implementation roadmap

The glossy roadmap your consulting vendor created might have helped you sell your business case, but you can be assured this is worthless for implementation purposes. The other risk you need to watch out for is overly positive assumptions in your roadmap - most things in enterprise companies take much longer than you expect, so you need to make sure you don’t commit to unrealistic timelines that ignore operational realities.

A viable roadmap acknowledges that Rome wasn't built in a day—and your data transformation project won't be either. The organisations that succeed understand that transformation is a marathon disguised as a series of program increments and sprints.

Here are some practical tips based on my experience -

• Run a 2 speed delivery process: one that focusses on delivering quick wins and the other that delivers foundational, complex and medium-long term objectives based on rigorous planning and project management. Delivering value quickly and consistently keeps the wolves at bay while you properly mobilise and plan your project for delivery.

• Align your roadmap to business outcomes and not technical features. Do this even in cases where extended time is required for building foundations.

• Always have a ‘plan on a page (PoaP)’ that shows all moving parts and dependencies at a high level, and tie detailed implementation plans to this PoaP.

• Use dates and milestones: dates and milestones can be used very effectively to drive a sense of urgency and galvanise teams to come together. However use these sparingly, as you risk burning people out.

• Make work and outcomes visible: Use tools like Jira to make work and priorities transparent to everyone, and implement simple & consistent reporting that shows progress against business outcomes. This should be consumable by both technical and non-technical stakeholders to maintain support

• Build shock absorption into your roadmap: keep some buffer up your sleeve to protect against delays. The risk is every person and team will likely build fat in their estimates. Make sure detailed plans are as fat free as possible, and bubble up as much fat as possible to the program level.

• Build in explicit recovery periods: After each major deployment, schedule 2-3 weeks for stabilization before starting the next phase. Teams that rush from milestone to milestone burn out by Phase 2.

• Allow time for transition to BAU and hypercard: a 2 week hypercard and transition plan is not sufficient for even the most simple of projects. A key part of your transformation project is transitioning capability into BAU, and making sure this is well embedded. Incorporate time for training, parallel runs, vendor transitions etc.

6. Change management strategy

You can create the best analytics models and you can execute the best transformation project, but they mean absolutely nothing if Elaine from finance is still exporting all data to her Excel models because "that's how she’s always done it."

Here are few practical tips to make sure you don’t end up building expensive shelf-ware -

• Map your organisational antibodies: Every organisation has "antibodies" that attack change. Identify who stands to lose power, budget, or comfort if your project succeeds.

• Speak stakeholder’s languages: Your data engineers care about CI/CD. Your CFO cares about cost reduction. Your sales team cares about closing deals faster. Translate your project benefits into whatever currency matters to each group. Lesson I learnt the hard way - don’t explain the benefits of data modelling to the marketing team!

• Create a coalition of the willing: Find and nurture your early adopters. Give them extra attention, training, and resources. Their success stories will create FOMO and convert the skeptics better than any slick slide deck.

• Build a communication drumbeat: Silence breeds rumours, and rumours breed resistance. Establish regular touch points—weekly emails, monthly town halls, whatever—even when you think there's nothing to report. "We're still on track" is valuable information to people whose work lives will be disrupted.

• Anticipate the Valley of Despair: You start your transformation at the top and there’s excitement and optimism. As the reality of the challenges sets in, progress seems slow, and frustration increases, leading to a "valley" of low morale. Plan for this dip and keep morale up by providing your team with required support, consciously celebrate small wins, introducing small team events, reinforce your vision and make sure key leaders are visible and motivating people.

• Don't mistake training for adoption: Just because people attended your training doesn't mean they'll use your new system. Build adoption metrics into your roadmap and track them religiously. I've seen beautiful dashboards with precisely zero unique visitors after the launch meeting.

Change management is by far the most important factor that determines benefits realisation (or not). As the CDO, you need to pay special attention to this aspect of your transformation. Because when your project ultimately succeeds or fails, it'll rarely be because of your perfect data platform. It'll be because you either won or lost the battle with human behaviour.

Key Takeaways

Tie Data Initiatives to Real Business Value

Don’t start with tech. Build your case around tangible business problems, quantify the pain ($$, risk, reputation), and explain in plain language how data/AI will fix it. Your transformation pitch should resonate with both your CEO and frontline teams.

Secure Strong, Vested Executive Sponsorship

The right sponsor can make or break your project. Pick a C-level executive with authority, skin in the game, and staying power. Clarify expectations, commitment, and ways of working early.

Treat Change Management as a core activity

The biggest risk isn’t your tech stack—it’s human behaviour. Map resistance, build a “coalition of the willing,” align messaging to what stakeholders care about, and track adoption relentlessly. Change doesn’t fail because of tools—it fails because of people.