Design for Digital Takeover: Part 1

Episode Transcript

Jerry Kane: Welcome to the Design for Digital Takeover of the Filene Fill-In Podcast, presented by the Filene Research Institute as a part of the Design for Digital Center of Excellence. I’m your host, Jerry Kane, and each episode is crafted especially for credit union executives who are leading through digital change.

We explore the insights, innovations, and operational playbooks that matter now, from member experience and analytics to AI strategy and sustainable growth. You’ll hear from researchers, operators, and visionaries who are turning technology into real business value. Today, I’m excited to welcome Sam Ransbotham, professor of analytics at Boston College’s Carroll School of Management and Mastrangelo Dean’s Faculty Fellow.

Sam teaches analytics in practice and machine learning and artificial intelligence courses, bringing a pragmatic, outcomes-first approach to data and AI in organizations. Many of you will also know Sam as the host of MIT Sloan Management Review’s Me, Myself, and AI podcast, a series that separates AI success from AI hype by talking with leaders on the front lines. The show is produced in collaboration with the Boston Consulting Group, providing listeners with an inside look at how top companies design for scale, govern risk, and measure impact.

Beyond the classroom and the studio, Sam has shaped the field as a senior editor at Information Systems Research, an associate editor at Management Science, and an academic contributing editor at MIT Sloan Management Review, roles that put him at the intersection of rigorous research and executive practice. He has also co-authored widely read MIT Sloan Management Review research on analytics and AI adoption, including reports on closing the gap between AI ambition and action and on winning at AI at scale. Sam has earned his Bachelor of Science in Chemical Engineering, his MBA, and his PhD from the Georgia Institute of Technology, a rare “triple-Jacket” combination that helps him translate between technical depth and business strategy, exactly what AI leaders need as AI moves from pilots to production.

In our conversation, we’ll dig into what distinguishes organizations that realize measurable value from those that stall, how to set the right defaults, where to put the guardrails, which use cases to scale first, and how to build AI systems your teams can trust. Powered by Filene’s commitment to research and innovation, let’s get started. Sam, welcome to the podcast.

Sam Ransbotham: Great to be here. Always good to talk to you, Jerry.

JK: Well, thanks, Sam. You set me up for that. So you’re the AI guy these days, and you’ve been doing a lot with AI. AI’s all we hear about these days, AI this, AI that. How much of AI is hype, how much of it is real, and what should executives be paying attention to?

SR: I think that’s a fair assessment. It seems to be coming at you from every direction right now. But I think I’ll do the good student thing and say that the answer is all of the above.

There’s a ton of hype, and it gets exhausting to hear yet another breathless discussion about AI. At the same time, there’s some real stuff going on. And, you know, that’s the problem, figuring that out. That’s the material problem.

JK: What’s some of the cool, real, or impactful stuff that you’ve seen in your work, both from an organizational perspective and from your own personal use?

SR: Everybody likes to talk about their own personal use. I’m afraid it’s slipping into like talking about your own dreams, talking about how I use AI. There are so many things going on, and really part of the problem here is a labeling problem. We draw a giant circle around anything and call it AI, especially if it helps the marketing. But there’s still truth to it.

And some of the stuff that I like is boring. What I mean by that is you don’t see it. The fact that your credit card transaction is less likely to be flagged as fraud, that’s not interesting to anybody. That’s boring. That’s not cool. It doesn’t have a robot. It doesn’t jump on stage. I like these videos, like the Boston Dynamics robots jumping around. I mean, I’ll watch those like anybody else, but they don’t affect my life. What affects my life is a lot of this boring stuff that is just getting one incremental improvement a bazillion times. It’s a long tail.

JK: Do you think the average person knows that AI is doing these things, or do they just stop getting annoyed and they’re just, whew, thanks? Sort of like spam. When spam stops showing up, it’s like, oh, great, no more spam, no thought about how that happened.

SR: One of the things that we do, you mentioned the AI guy here, and part of that is that I’m the editor for the AI Initiative at MIT Sloan Management Review. And one of the things that we do is an annual survey, an annual report. Jerry, I know you’re familiar because you’ve done those for years.

JK: Quite familiar. And that’s actually what led to my work with Filene here that’s leading this podcast. It’s basically, I’m taking the work I did with MIT Sloan Management Review and Deloitte, and now I’m doing it for credit unions.

SR: So yeah, I mean, you clearly know, I mean, our model is to talk to a bunch of people and figure out what they’re doing. And one of the fun things related to your question, I’m going to it, don’t worry, I’m getting to it.

JK: We got an hour. You can get to it when you want to.

SR: We asked people, in one of our surveys, how much you’re using artificial intelligence, and about half those people said, what are you talking about? Actually, it was close to two-thirds who were saying, I’m not using AI, you know, that’s newfangled AI stuff. And then the next questions in our survey started asking about specific technologies, and they said, well, yeah, I use that, of course. And so we had about half the people, in the course of just one or two questions, change their impression about whether or not they were using artificial intelligence. Because, as you point out, it’s everywhere. It’s the fact that your Zoom background is blurry. Is that AI? Well, yeah, I mean, it’s got elements of it, but it certainly doesn’t feel like a gee-whiz robot. It’s not autonomous driving, therefore it cannot be AI. So I think that’s a big problem.

It’s not Her, it’s not Terminator. I was doing a workshop yesterday where I asked people what AI was, and they went straight to that. And as a good professor, I don’t like saying, no, you’re wrong, student. But I did, because I needed to make a point. This is wrong. We’re taught never to say, hey, you’re wrong, in class. You always try to redirect. But in this case, I made an exception. There’s so much misinformation out there that I do think it needs to be corrected. People are expecting Rosie the Robot or people expecting Her or Terminator. And that’s not what we’re talking about here necessarily. Most organizations are not using anything remotely similar to anything you’ve ever seen in a Hollywood movie. But that doesn’t mean they’re not using artificial intelligence. That’s a very different question.

JK: So a lot of my experience now, and Sam, this will be a successful podcast if I don’t get to question two that I’ve prepared for you, that’s sort of my goal. It’s just like an academic presentation, right?

SR: Right.

JK: We should have invited George, and then it would have been complete, and we would have just devolved into chaos. So what I’m seeing is, in my executive class, almost everybody says they’ve used generative AI. Almost nobody is using it on a daily basis. And almost no organization has a systematic way for sharing best practices, for sharing with one another these individual uses across the organization, or even putting guardrails around it. Is that true of your experience, or are you seeing something different?

SR: Not that you’re wrong, but man, I think you’re right and wrong.

JK: Okay, great.

SR: I say that just to be difficult, of course. But no, I think there are organizations that are much further along in this. So let’s kind of break down where you’re talking about. You actually specifically mentioned generative. And I think that’s actually one of the truly phenomenal things that’s happened here lately.

Even I, when I do presentations about academic papers, if I put a table up there that has a regression, that has a machine learning model, nobody has ever run a machine learning model themselves in that room. So you were automatically at a disconnect when you were trying to explain the results of some statistical model or some regression model. Actually, our friend George Westerman and I did a presentation one time where I presented some regression results, and then he came up afterwards and said, who understood what the heck Sam was talking about? And practically, nobody had ever run those models themselves.

But when you talk about generative, people have. Everybody has done that, from your kids to my kids, to even younger, to even my parents. Just everybody has done that. And that has been a massive shift in society where suddenly AI was this abstract thing, maybe it’s autonomous driving, maybe it’s a robot from the movies, to now it’s this interface that I can type something in and get an answer that seems like a human.

JK: No, I agree. What I see is everybody has at least experimented with it. What I’m not seeing is the systematic integration into workflows necessarily, where organizations are having best practices shared within their organization, where they can all learn. Everybody’s dabbling in their own way. They haven’t put a business discipline around it yet.

SR: I think there’s a couple of things going on there. One is, again, we’re talking about generative, but if we talk about other aspects, like any other ML model that learns, there’s a ton of that with best practice in organizations. So let’s put that aside and focus on generative.

What we’re seeing, I think, is a combination of lots of things going on. One of those things is this constant, there’s a new model today, it’s different, it’s better problem. So everyone’s experimenting because no one can ever get to a settled state where they understand what’s going on enough to put it into a process, because by the time they do that, it has changed. And that’s, I think, a product of our sort of hyper-competition in this industry right now.

But the other thing, too, let me give you a specific example. One of the people we had on our podcast was Miqdad Jaffer from Shopify. Shopify is a company that helps people put their listings onto Amazon or other retail sites. Their model is pretty simple in terms of there’s a product, you have a picture of it, you describe the picture in text, and that’s what shows up on the retail website. Within three months of ChatGPT 3.5 coming out, they had integrated self-generated prompts into their workflow so people could take a picture and it could generate five or six different options about how to describe their product. And that was in their workflow. So it’s not that nobody is. And I’m just amazed at how quickly they were doing it. But it’s tough when all these things change constantly too.

JK: That makes sense. So with change, I know I’m dating us. Have you—well, not dating, the podcast becoming dated. GPT-5, what are your thoughts on it?

SR: I love these things that are coming out all the time. I find it particularly overwhelming. A lot of people, I was just at this business group down the street and they were saying, which one should we use? And I’d say, you know, any of them are better than none of them. And so everyone has their particular flavor that they like. I’m a little wary about making some sort of vendor endorsement here, but I think they’re all pretty amazing. They all have different strengths and weaknesses. Some of them are better for one particular application than another. And I think that’s what’s really difficult about the entirety of AI right now. It depends.

People hate “it depends” as an answer.

JK: Yes, of course.

SR: Except for academics, because that’s our business. That’s how we make our living. People want, “Here’s the right answer, here’s the one to choose. Don’t think about this, just use this.” What I’m seeing is this “it depends.” It depends on what you’re using it for, what context, what data.

And increasingly, I’m seeing people internalizing large language models in some way and layering their own data on top so that it’s custom for their application. A couple of things are going on there. One, it helps make sure that the answers are custom for an organization. But two, it insulates people from the underlying model. If the IT group swaps out from one model to another this week, that can be insulated from the end users of the process, which I think is pretty important as we develop the infrastructure around these things.

JK: Except I would listen to a podcast about this very issue. Most of the people that are actually doing it are power users, and many know the difference between the models and know how to do it. And those are the ones you risk upsetting by not giving them the choice of their preferred model.

SR: There’s a difference between what you and I use for our individual stuff and what a company uses as a process. That’s important. Companies standardize on tools all the time, and I may not agree with them, but I use them.

One thing I learned from you was about Microsoft Windows, how there may be a bug in Microsoft Windows, but a whole suite of businesses has built whole businesses on that bug. And if they take out that bug, you collapse those businesses. As GPT becomes widespread, there become adoption issues, and there’s a whole suite of complexities that come along with it.

You mentioned adoption. You and I have studied many technology adoptions over the years and hope to study many more. It bothers me when people think about these as something crazy and completely new. We’ve seen new technologies before. Whenever Everett Rogers was describing how farmers adopted seed corn, we had fundamental principles of adoption. Those are still holding strong here. Nothing new under the sun.

JK: You know, it feels to me a lot like 1995 all over again, you know, with the web. Do you have that same feeling, and what lesson should we learn? What lesson should we take from that feeling?

SR: Yeah, I mean, I think that’s a great analogy. I hadn’t really thought about it. Yeah, 1995. When we were super excited about the internet and it was going to change everything and then suddenly everything crashed and, oh my gosh, it was a dot-com bust. It did change everything. And how is Amazon possibly going to beat Walmart? Because Walmart’s not going anywhere.

Right? You know, we use a lot of that internet thing now. It’s pretty established. And so, when you start off kind of thinking about hype and hype cycles, that’s how these things play out. We get too excited, we get under-excited. We eventually figure out where these things are going to land. And that process is going to happen here too. It may happen faster here though, if you think about how long it took us to get digital.

JK: I’ve been thinking about this a little bit lately, and argue with me if you’d like. You know, as I think back to dot-com 1.0, I remember them burying all that fiber optic cable and spending billions of dollars. And then, oh crap, what do we do with it now? And we figured out stuff to do with it. Did we think it was going to be TikTok or Facebook? No. But we did figure it out.

Now we’re building out these massive compute facilities, these massive data centers. It feels like we’re investing trillions. Are we possibly going to need all of that? Are we going to get to another cycle where we’ve built these fusion reactors or increased our energy production and computing capacity, and AI doesn’t live up to the promises? Then we’re stuck with that massive overinvestment that takes us ten years to figure out what to do with.

SR: I don’t see it. I mean, okay, that certainly is a scenario. And actually, one of the points I like that you just said there was “nobody knows.” We are literally the first human species to ever go through this, the possibility of having tools able to do the things that current tools are able to do. You can’t fault us for trying some things that don’t work out and some that work out much better than we thought.

Maybe, to your energy point, I’d like to hope we wouldn’t use as much energy. But what we’ve seen with anything in terms of capacity or CPU or bandwidth or storage, we fill it. We find a way to use that infrastructure once it’s there, good or bad.

JK: Oh yeah, and that was my point. We figured out what to do with the internet. It just wasn’t, you know, pets.com necessarily. It just wasn’t only delivering dog food over the internet. And we figured out new applications, but it took ten years.

And I wonder if the real impact, I mean, yes, we’re seeing impact now, and these tools are remarkable right now. But I’m wondering, from an organizational perspective, if it’s really going to be a ten-year timeframe, because it’s going to take that long to revamp workflows, to revamp reporting structures, to revamp all of the things that organizations do. That moves much more slowly than the technology does.

SR: Exactly. Somebody should write a book on that. That’s exactly the point. And we see these things over and over again. That’s what we’ve seen with prior technologies, that processes, people, and cultures are much slower to change than technology. And there’s a reason for that, because technology will have fits and starts and things that don’t work out like we thought, and organizations are trying to smooth that off.

You can take these tools and have them do the same process you’ve always done, better and faster. If you want to date us, that’s the Steve Austin, better, stronger, faster, Six Million Dollar Man. Still man, better, stronger, faster. And we can do that with lots of processes. That’s fine. There’s nothing wrong with making stuff better, stronger, faster. But what’s more complicated, and what you just alluded to in your book, is these other changes are much harder.

I think there’s going to be a lot of opportunity for those changes here. A specific example is Stanley Black & Decker came on our podcast, and I thought, they make tools, how could that be related to AI? And one thing Mark was telling me was that once the ability to do quality control got a lot cheaper, they could do it at different points in the manufacturing process. They had designed a process around doing quality control at a certain point because of the combination of expensive materials, rework, and cost of inspection.

But when that dramatically changed, when they were able to do QA at a much different cost structure, it changed their process for making stuff. And plants don’t change overnight. You can’t change the tooling for a plant overnight, so those are going to take time to change. And that’s exactly your point.

JK: And that’s even just dealing with stuff, not people. Brynjolfsson has the example of, from steam power to electric power. It really didn’t change productivity until you redesigned the factory floor to accommodate these new tools. That’s when things happened.

So we have this new steam engine, or we have this new electric power, and it’s going to take us a while to figure out how organizations should do it. One thing I recommend companies do is start to do some disciplined experimentation. Try some stuff, figure out what works. That which works, share it widely across the organization. That which doesn’t, kill it quickly. But also, don’t kill the messenger that tried something. If nothing’s breaking, then you’re not trying enough new stuff.

SR: That risk tolerance, I think, is what a lot of companies wrestle with. And the lack of time for experimentation—it’s costly.

JK: Or, I just don’t think the cultures are built for it, particularly in financial services. Let’s remember, this is a financial services podcast. You want to get risk out of the picture. You don’t want variance, and experimentation is antithetical to that because you get sued if you do it wrong.

So I think it’s even more important to have discipline and an organizational process to do this. Because people are going to do it anyway. You can tell them all you want, but they will. I could not do my job as department head without generative AI right now. There is not enough time in the day. My predecessor tried; she burned herself out. Now, I haven’t written a memo to the provost, to my boss, without using generative AI ever. It doesn’t write it for me, but it saves me about 70% of the time.

SR: And as a follow-up question, are you going home 30% earlier?

JK: Me? I’m an academic, of course not.

SR: No, you just do more.

JK: Exactly. I do podcasts with it, Sam. This is what I do.

SR: You do more. And I think that’s our current story. You hear so much about the concerns about jobs and labor, and I completely understand many of those points. But certainly, for most people I’ve ever talked to, none of them are going home any earlier. They’re doing more stuff.

JK: And I think that’s true. Now, I would argue I’m doing more interesting stuff. Writing memos is not what I signed up for in this job, but having great podcast conversations with interesting guests is a fun part of the job. And that’s what I can do now.

I can experiment with generative AI in the classroom. This semester, I’ve used deep research to generate readings for my course, and it’s worked out really well. My experience with AI readings is that they’re either really good but old, like 2016, or they’re cutting edge and only deal with a little sliver of what’s going on. It’s hard to find the right reading for an up-to-date introduction to AI. So I built it myself, and it’s worked out pretty well.

Actually, we should compare notes afterward because we’re doing some of that same stuff here, and I imagine that we can learn from each other about how that process is working. What we’re doing here is we’ve started teaching with AI brown-bag lunches, so at least we’re sharing within the faculty. Because what I found myself doing is I was going to companies saying, “You should have brown-bag sessions on how to use AI and share best practices.” And I wasn’t taking my own advice. So I’m trying that. You’re welcome anytime.

SR: We’ve had AI days here to share practices, and we’ve actually got a pretty good group on campus trying to promote that and trying to institutionalize some of it in ways that are difficult with the speed of the technology. There’s always somebody doing something weird, which is good, but also bad.

And we’re in weird cultures. I mean, state universities are very different than financial services. Academics are different. State schools are different. Catholic schools are different. There are all sorts of cultures.

JK: I want to shift to the “get off my lawn” portion of the program, which I know you enjoy. I’m actually getting to question one that I sent you. So I’m hearing a lot about agentic AI, and I know Filene has asked me, should we do stuff on agentic AI? What is agentic AI, and where do you put it on the hype cycle?

SR: Let me start easy to hardest. Agentic AI is more about these tools working independently. Like anything, there’s a continuum. At the extreme, an AI agent would be something that acted alone and acted on behalf of the company to do something. The other extreme would be something that you used almost as a tool. So you can think of it as a continuum from being a tool to being an independent worker.

You and I have written about agency and the importance of agency as a term. These tools are getting more agency, and that’s a big deal right now. Our next report, coming up in November, is very focused on companies using agentic AI.

And I should check with you when you’re going to release this podcast so I don’t steal my own thunder here. But there’s a lot going on. I went in thinking we were too early, that we’d be asking questions about things people hadn’t yet done. But there’s more going on than I expected. There’s still plenty of hype—this is AI, after all—but some companies are doing real things.

A lot of it goes back to Everett Rogers and the diffusion of innovations. More and more of the vendors of the tools that companies have standardized on are including agentic options. Your trialability and ease of adoption are super high because of that.

JK: So like Salesforce, those types of things?

SR: Salesforce, and even productivity suites are getting more of these tools in there, like Copilot. These vendors want to capitalize on the land grab.

JK: See, that’s why I’d be interested. I think agentic AI is all hype. I think two years from now we’ll look back on it differently.

SR: I disagree.

JK: What are the things that you’re seeing, the successful use cases?

SR: A good example, we were talking to a legal firm. They are increasingly having more and more of their content, like your memo to the provost, with less and less human involvement. Now, I don’t think anybody is at the point of just saying, “Fire off the letter to the provost without review.”

I did an experiment with my email to see if Copilot could do better, and within three emails, I stopped because it was just generating nonsense. I also tried a scheduling app to let an AI take over, and I had to stop that too. Some of that was because its priority was scheduling the meeting, and as you know, often my priority is not to schedule a meeting—just to push it off.

JK: Then I should be honored that you accepted my invitation to be on the podcast.

SR: You slipped me a twenty. I’ll do anything.

One interesting thing about agentic AI, and tying it back to your point about 1995 and the internet—we designed the internet for human interaction. Tim Berners-Lee said he invented it as a social tool. We’re on the cusp of something that’s going to let us switch entirely to where it’s more machine-to-machine.

We’ve had API development over the years, more and more companies putting in REST APIs for most of their services. But increasingly, I think we’re going to have an internet where the humans are the small portion of it. And if that’s the case, then agentic AI has to be a big part of it. Now, there are some hurdles before we get there, and they’re not small. There are a lot of issues with control.

JK: I like the idea of somebody doing something for me until they do it wrong.

SR: Exactly.

JK: But see, my point is, AI struggles in the wild. It does really well in controlled environments where the data is clear, where the conditions are clear. It can struggle when it gets out beyond its training data. Isn’t that what we’re asking agentic AI to do? Go deal with the big wide world for me and just do what I want, not what I tell you to do, but what I actually mean?

We know AI, and we’ve both failed at some of these attempts. Why are others going to do any better?

SR: Oh no, I think you’ve got to back off your extreme there. There’s a range of this that’s going to happen, and it’s not going to happen all at once.

Let’s make an analogy to driving. Everyone’s ready for automated cars, and that was the use case for AI five years ago. I’m not hearing as much about it anymore. But if you think about what happened with driving, we had lane-departure assistance, adaptive cruise control, steering assist, parking assist. We were expecting Rosie the Robot to sit in the front seat and drive the car.

We can get a lot of value out of incrementally chipping away at these little cases. Then we’ll look down and realize that when these things all come together, we’re pretty close to having automation. It doesn’t have to happen right out of the gate.

And we’re holding these tools to a standard that’s too high. There are about 40,000 people a year killed by human drivers in the U.S. every year. When we have one accident involving AI, it’s top of the fold. And it should be, because it’s life and death. But when we’re doing emails, people don’t die.

I think we have to think about our standard. You’re saying, “I’m going to turn this AI loose on the internet to go figure out everything.” You wouldn’t have any more success doing that with an undergraduate research assistant. You’d have the same sort of supervision, monitoring, and control.

Organizations need to get around the idea of managing these tools more like we manage human resources than like we manage technology. That’s going to require a big shift.

JK: That’s really interesting. And maybe it’s not fair that I’m getting you to level five agentic AI out of the gate, but that’s what I hear the consulting companies selling. I think part of the reason for the hype cycle is that you have an underlying successful technology, and then you have all these people trying to capitalize on it, selling stories they can’t deliver on, or that will take ten years to deliver.

I remember, back to the car analogy, I think it was you that sent out the email to the department. All the smart tech people said, “How long until autonomous vehicles?” Half of the really smart people said two years, and half said twenty years. It’s looking closer to the twenty-year group being right.

SR: You know, I don’t know if you’ve been to Phoenix, but they have Waymos all over the place. I’ve heard people say they prefer them to human drivers because they’re much more careful.

JK: Have you seen some of these drivers?

SR: Oh yeah. I lived in Boston, and now I live in Atlanta. I’ve seen both sets of drivers. So we have to be really careful about the standards that we hold these things to.

One of the great quotes from our podcast that I keep coming back to is from Gina Chung at DHL. Her quote was, “The first day is the worst day for AI.” The point is that we’re going to put these things in place, and they’re going to perform poorly. What we’re hoping is that tomorrow, they perform a little bit better.

That’s the same as when you hire someone at a company. Day one, you don’t throw them out because they didn’t know everything off the bat.

JK: No, but by year one, if you were sold that they were an expert and they’re not, that’s where the manager questions come in.

SR: Exactly. These are the classic “it depends” issues. But I guess what I’m seeing, and into the hype, is that the level of hype around agentic AI, to me, far exceeds the current capabilities and where we’re likely to be in the next three to five years.

JK: I think that can be true. But that doesn’t mean, at the same time, that there isn’t something there. It’s the classic ups and downs. There are people overpromising it, there are people dismissing it, and the truth is somewhere in the middle.

This is my podcast, so I get the last word. I still think there are theoretical reasons why agentic AI is going to be harder than people are making it sound. It’s the wildness, the uncertainty of the context. That’s why I think it’s going to be a much harder problem. I’m discouraging companies from jumping there too quickly. Let’s figure out how to use generative AI better at the organizational level first. We know those tools work really well. We know they deliver value now. Why are we chasing the next big thing before we’ve figured out how to use this one?

SR: I think that makes a lot of sense. I can agree with that.

Maybe to frame it, what we deal with here is extrapolation. We look at the way things are today and the way things were yesterday, and we draw a line between those. As humans, we assume that line is going to keep going at the same rate or even increase. But there’s a lot of reason to think that much of artificial intelligence will follow diminishing returns.

We’re going to attack the easier problems first because they’re easier. We can show success. That means as we try to extrapolate, our model should probably assume diminishing returns, not endless exponential growth.

JK: Agreed. But what I’m hearing a lot right now is that AI isn’t following those diminishing returns yet. The more compute and data we pour into it, the better it seems to get. That’s why we’ve got some of this hype right now. Even with the stumble of GPT-5, or at least it not being as magical as people were expecting, Ethan Mollick had a great post today showing that the models are still improving steadily. It’s just the expectations that keep outpacing reality.

When are we going to have Her? When are we going to have HAL? When are we going to have Terminator? Which leads me to my next question.

SR: Oh no, Terminator?

JK: Yeah, well, not AGI yet—artificial general intelligence. When I teach this, I talk about the difference between weak and strong AI, or narrow and general. You’re the guest—tell me the difference.

SR: Most of the applications we see are narrow. These tools work really well in well-defined contexts. Even if you think about something like the Turing Test—ask a question, get an answer—it’s still a narrow form of success.

But you can ask a human all kinds of stuff that’s very general, and they’ll do a pretty good job. In general, artificial intelligence struggles with that generality—with knowing a lot about lots of different things. That’s the weak versus strong, specialized versus general distinction.

JK: I’ve seen predictions saying we’ll have artificial general intelligence in 18 months. One of the more conservative ones said 2030. Do you buy that?

SR: No. I’m shortening that right away. I don’t buy it.

JK: I’m 100% with you. Tell me why.

SR: Back to extrapolation again. These problems get harder and harder. It’s not like we’re just cleaning up the last few easy parts. These are big, complex problems when we start talking about generalization.

One example I use a lot: some financial companies use AI for customer service. A bad use of AI is the phone tree, where you have to say “reservations,” and it doesn’t understand your accent. I’ve never successfully used a customer service chatbot, and I defy anyone who says they’ve gotten real value out of one.

But the better companies are using AI to monitor latency on the phone line—that tiny delay in speech that can indicate where someone is calling from. We can’t hear the difference, but the machines can. They can pick up on stress in someone’s voice. You and I don’t talk enough for me to know when your voice sounds stressed, but the machine does.

My point is, machines are really good at narrow things—detecting stress, latency, patterns. But a human agent is great at taking a general, messy problem and narrowing it down. We’re still far ahead in that area, and I think we will be for quite some time.

JK: Yeah, and I agree. There was an article I read that said, “Artificial general intelligence has been 20 years away for the last 50 years.” It’s always just over the next hill.

I think some of the hype around AGI is being used to justify massive spending on these models. It’s like, “We’re right around the corner. If Google beats us or OpenAI beats us, we’re doomed.”

SR: Yes, exactly. And it gets political very quickly. When you’re talking about worldwide spending, there’s some saber-rattling involved.

I’ll push back slightly, though. Some of the issue here is that our definitions keep changing. I’d argue that ChatGPT with its voice functions could probably pass what Alan Turing had in mind in the 1950s. It passes the Turing Test now.

If you haven’t tried the voice versions, or NotebookLM, you should. We could have uploaded our readings and told it to “have a conversation with Sam and Jerry.” It could make fun of me—and it wouldn’t be wrong.

JK: That doesn’t go into print. That’s a live thing.

SR: Exactly. We change what we expect. And that’s okay. Part of the problem is that we define these tools by our expectations. One of my favorite definitions of artificial intelligence is “the use of computers to do what people normally think of as requiring intelligence.”

That definition has so many flaws. You can’t have a definition with “normally,” because what’s normal keeps changing. And we’ve defined artificial, but we’ve never really defined intelligence.

JK: I think what happens is, once AI does something, we stop thinking of it as intelligence. We move the goalposts.

SR: Exactly. You can have a conversation, but it can’t drive a car, or hug a child, or walk down the street. It’s still narrow.

I was listening to a podcast with Demis Hassabis from Google DeepMind. He said we’ll have AGI by 2030, which is one of the more conservative estimates. But then elsewhere in the same podcast, he says, “We don’t know how to get computers to ask good questions or make intuitive inferences.” That’s part of intelligence! You can’t just say, “We’ll have AGI,” but leave out those core components.

JK: Exactly. It’s like, “We’ll have it, except for the important parts.”

SR: Right. The problem is that when AI doesn’t live up to those expectations, people will say, “It was all hype,” and throw the baby out with the bathwater. Like in 2001, people said, “The internet doesn’t matter,” or “It’ll never take off.” It takes time.

JK: It feels like we’re promising magic right now.

SR: We are. And we have to be more mature about it. If I invented a technology tomorrow, would it be overhyped? Yes. Because you want to sell it. Would it be overpromised? Yes. And there will always be people saying it won’t do anything.

We just need to accept that technologies go through a figuring-out phase. The truth is somewhere between the overpromising and the underestimating.

JK: So, as we’re coming to the bottom of our hour, let’s talk about something practical. What’s an executive supposed to do? What’s the right way to approach this?

SR: You mentioned earlier the idea of experimentation. That’s really important right now. There’s so much going on. I get that you don’t want to throw your budget at every new experiment. On the other hand, you don’t want to be caught unaware of what others are doing.

From an executive perspective, you’re going to have to make some experiments that don’t work out. It’s going to be a little painful to say, “We tried that, and it didn’t work.” But that’s necessary.

Also, not every company needs to build giant models. Many will be consumers of models that others create. It’s tempting for organizations to have people fired up about the tech and say, “We’ll build our own LLM.” In some cases, that might make sense. But you have to figure out within your company what your strategy is and how AI—whether agentic or not—supports that strategy. That’s a very different question than, “How can I use this tool?”

JK: I agree 100%. I’d just add one thing. Once you’ve run those experiments and found what works, roll it out across the organization. I see a lot of companies experiment, pat themselves on the back, and say, “Good job, we innovated.” But they don’t scale it.

SR: Exactly. The framing I use is production versus consumption. Most of the time we’re thinking about production—building the tool, developing the output. That involves a relatively small group of people. But the hard part is consumption: getting the whole organization to understand when and how to use these tools. That’s culture change, and that’s harder than the tech.

JK: I think that’s where the value is going to be over the next ten years—figuring out how to rebuild or start organizations that are AI-empowered.

SR: I don’t think we have that answer yet. No one’s ever dealt with this before. There’s not a right answer. There will be some wrong answers.

JK: You wrote your book five or six years ago, before this generative AI wave, but most of it still holds true.

SR: I’m shocked at how well it’s held up. I’ve talked with my co-authors—we didn’t intend it to be this evergreen, but it has turned out that way.

JK: Just do a search and replace on the word “digital” with “AI,” and resell it.

SR: Don’t tempt me.

JK: We’re running to the bottom of our time, but I want to ask one last fun question. You host your own podcast on AI, Me, Myself, and AI with MIT Sloan Management Review. Who are one or two of your favorite guests and why?

SR: Oh gosh, that’s like asking which of my kids are my favorite. What’s been fun is that every story is interesting and different. People are doing all kinds of things.

I mentioned Gina Chung earlier—her “first day is the worst day for AI” line. Through the podcast, we’ve talked to Vandi Verma, who drives the rover on Mars and flies the helicopter on Mars. That’s cool stuff.

But even the World Wildlife Fund is using AI in surprising ways. It’s everywhere. One guest, Dave Thiel, even has an ant named after him. It’s just fascinating.

Most of the people on Me, Myself, and AI are doing interesting things with these tools. The theme is that these are tools, and how we use them matters. The people making those choices are just like you and me—figuring it out as they go.

JK: I 100% agree. And I’d add, even for credit unions that aren’t the most technologically advanced, these tools seem like rocket science at first. But it doesn’t take long to learn them. Once you do, you start discovering new things, sharing ideas. For the credit union executives listening—start experimenting. Try these tools in your spare time and get better at them. You can do this too.

I tell my kids, managers won’t be replaced by AI, but managers who use AI will replace managers who don’t. And that extends to organizations. The ones that learn to use these tools will outperform those that don’t.

Back in 1995, people asked, “What if my employees don’t want to use email?” Can you imagine that question now? In ten years, that’ll be the same for AI. Start now, and it’ll be less painful later.

SR: That’s a great way to put it.

JK: Well, Sam, this has been even more entertaining than I expected. If nobody ever listens to this podcast, I’ve still learned a ton. This has been one of the more interesting AI conversations I’ve been part of. It’s been great to reconnect.

SR: Always fun. Thanks for having me, Jerry.

Announcing!

Quinn Kinzer: And before we wrap up, a quick update for you all. In the new year, we’ll be relaunching the Filene Fill-In with a renewed focus on our Centers of Excellence. We’ll be featuring important conversations with our fellows and deep dives with guests who bring fresh insights into Filene’s core research questions.

To get you ready, we’ll be dropping a few bonus episodes after the Design for Digital takeover, so keep an ear out for those and be sure to hit subscribe so the latest and greatest is always at the top of your feed every time we release. We can’t wait to share what’s ahead, and thanks so much for listening.

We’ll see you soon on the Filene Fill-In.