Crossing the GenAI Chasm

From 95% Failure to Repeatable ROI

November 25, 2024 • 5:20pm Start • Microsoft Vancouver

The Chai Moment at 6:20pm 🫖

Tonight's Journey

5:20 Welcome & Introductions

5:30 MIT + Cleanlab Research

5:55 Segment 1: The Selection Problem

6:20 🫖 The Chai Moment

6:25 Segment 2: The Human Factor

6:45 Segment 3: The Solution Blueprint

7:05 Q&A and Networking

Tonight's Output: A playbook on patterns that work

We are gathered on the unceded traditional territories of the xʷməθkʷəy̓əm (Musqueam), Sḵwx̱wú7mesh (Squamish), and səlilwətaɬ (Tsleil-Waututh) Nations.

Exploring With

Panelists

Dr. Curtis Northcutt

CEO, Cleanlab

Aman Sidhu

City of Vancouver

Hubert Duan

Microsoft

Moderator

Girish Limaye

GovAI.fm

In partnership with:

Vancouver Tech Journal

Revolution Data Platforms

Mohamed Sharaf

Founder

The $40 Billion Reality Check

$30-40B

Enterprise GenAI Investment

Massive capital allocation with minimal returns

95%

Zero Return Rate

Organizations seeing no meaningful ROI

5%

Production Success

Custom enterprise tools reaching production

2 of 8

Sectors Transformed

Only two sectors showing structural changes

Debunking the Myths

1

AI Will Replace Most Jobs

Reality: Limited layoffs from GenAI, only in already AI-affected industries. No consensus on hiring levels next 3-5 years

2

GenAI is Transforming Business

Reality: Adoption is high, but transformation is rare. Only 5% have tools at scale, 2 of 8 sectors show structural change

3

Enterprises Are Slow Adopters

Reality: Extremely eager to adopt - 90% have seriously explored buying an AI solution

4

Model Quality, Legal, Data, Risk Hold Back AI

Reality: What's really holding it back - tools don't learn and don't integrate into workflows

5

Best Enterprises Build Their Own

Reality: Internal builds fail twice as often as external partnerships

Where the Divide Shows Up

"Would you assign this tool to AI or a junior colleague?"

Quick Tasks

(emails, summaries, basic analysis)

AI

70%

Human

30%

Complex Projects

(multi-week work, client management)

AI

10%

Human

90%

Key Finding

For simple work, AI wins (70% preference). For complex work, humans dominate by 9-to-1 margins. The dividing line: memory, adaptability, and learning capability.

The 95% vs 5% Divide

95%

FAILURE PATTERNS

Generic tools • Central AI labs • Model benchmarks • Horizontal solutions • Static systems • Pilot purgatory

5%

SUCCESS PATTERNS

Deep integration • Frontline leadership • Business metrics • Narrow workflows • Adaptive systems • Production value

We explore these patterns across three dimensions...

1

SEGMENT 1

The Selection Problem

Building Project Selection Framework

MIT research reveals what separates the 5% winners from the 95% who fail

Making the Right GenAI Bets

Investment Bias Reality Check

50% of budgets go to sales & marketing, but back-office yields better ROI

The Measurement Trap

Easily measured, highly visible outcomes get unnecessary attention

Deep Integration Wins

BPO-style embedded solutions succeed; light-touch SaaS tools fail

Narrow Focus with Domain Fluency

Winners target narrow but high-value use cases. Domain expertise and continuous learning beat broad features and flashy UX

Edge-to-Core Strategy

Successful implementations start with non-critical processes, prove value, then scale to core workflows

2

SEGMENT 2

The Human Factor

Discovering Organizational Success Patterns

MIT's analysis of organizational patterns in the successful 5%

Organizational Success Patterns

Scale Paradox

Big firms lead in pilots but lag in scale-up execution

Buy Beats Build (2:1)

Strategic partnerships twice as likely to succeed as internal builds

Deep vs Light Touch

BPO-style embedded partners win; SaaS vendors fail

Frontline-Led Innovation

Source from frontline managers, not central AI labs. Distributed experimentation, no approval bottlenecks

Business Metrics Focus

Winners measure business impact, not model benchmarks. Hold vendors to real outcomes

Prosumer + Executive

Bottom-up prosumers paired with top-down executive accountability

3

SEGMENT 3

The Solution Blueprint

Defining Design Principles

MIT findings on why adaptive systems beat static tools

The Shadow AI Paradox

Personal AI Tools

90%

Workers using shadow AI

ChatGPT, Claude, Copilot

Flexible interfaces
User-controlled iteration
Immediate responsiveness

Enterprise AI Tools

5%

Tools reaching production

95% fail to scale

Static systems
No learning capability
Limited customization

The Fundamental Learning Gap

This feedback points to the fundamental learning gap that keeps organizations on the wrong side of the GenAI Divide. Users appreciate the flexibility and responsiveness of consumer LLM interfaces but require the persistence and contextual awareness that current tools cannot provide.

The Learning & Memory Gap

Why Generic Tools Win, and Lose

Enterprise Tool Problems:

"Our purchased AI tool provided rigid summaries with limited customization. With ChatGPT, I can guide the conversation and iterate until I get exactly what I need."

But ChatGPT Also Falls Short:

"It doesn't retain knowledge of client preferences or learn from previous edits. For high-stakes work, I need a system that accumulates knowledge and improves over time."

— Corporate Lawyer

The Learning Gap That Defines the Divide

Barriers to Core Workflow Integration

"It doesn't learn from our feedback" 68%

"Too much manual context required each time" 63%

"Can't customize it to our specific workflows" 58%

"Breaks in edge cases and doesn't adapt" 50%

The Gap:

Users need tools with consumer LLM flexibility AND enterprise memory/learning capabilities. Neither current option delivers both.

Workforce Transformation

✓ No Mass Layoffs

Most winners don't see workforce reduction

Selective Impact

Customer support, software engineering, admin functions

Cost Optimization

Reduced BPO spending and external agency use

Revenue Growth

Improved retention via intelligent follow-up

"Our hiring strategy prioritizes candidates who demonstrate AI tool proficiency. Recent graduates often exceed experienced professionals in this capability."

— VP of Operations, MidMarket Manufacturing

Let's Build Our Playbook

Three Discussions to Bridge the Divide

1

The Selection Problem

Building our project selection framework

2

The Human Factor

Discovering organizational success patterns

3

The Solution Blueprint

Defining our design principles

Our Goal Tonight

Create a practical playbook combining MIT findings, expert insights, and your real-world experiences

Get the Full Playbook

Sign up to receive the complete Enterprise GenAI Playbook and notifications for upcoming sessions.

govai.fm/#signup ?