Transcripts are auto-generated and may contain errors.
0:00
Hey, I’m Rose.
And I’m Angie.
And this is Our Lives with Bots, the show where we ask important, timely questions about what it means to live with our bot counterparts.
And from time to time, we also dive deep into what an AI future might look like for us.
0:15
Sometimes we agree, sometimes we spiral, but we always go deep.
So far in AI, there are very strong similarities to have things played out in the past.
It’s not going to be this thing that’s a separate species that puts everybody out of work.
Air Canada had a customer service chat bot that made-up some fake refund policy.
0:34
If there’s going to be even a 1% failure rate, but that’s going to lead to legal liability for the company, it’s not a good automation tool.
People are looking at these capability benchmarks.
They don’t tell you about all of these downstream innovations that are necessary.
Our goal is to move benchmarks from completely useless to mostly useless.
0:49
Folks who say they’re the worried about super intelligence are the ones pushing for ways of approaching AI that are most likely to bring about the threats from super intelligence.
By delegating so much to AI, we’re almost ensuring that when things do break, they’re going to break catastrophically.
1:05
We should choose to ensure that humans stay in the loop.
Welcome viewers to our 4th and final episode of our Series 3 on AI and work and a is impact on our relationship to work.
We have a very special guest here recording with us today, Arvind Narayanan from Princeton University, a professor of computer science and director of the Center for Information Technology Policy.
1:30
So Arvind studies the societal impact of digital technologies, especially AI.
You might have seen his book AI Snake Oil, which I actually have or right here, believe it or not.
But in any case, Arvind will be talking about a is impact on work.
1:46
Really interesting stuff from computer science perspective, thinking about benchmarks and will AI really take all of our jobs.
And it’s interesting because Arvind has quite an optimistic perspective, not a skeptical perspective, which I find quite intriguing and a little bit refreshing.
And maybe our viewers will too, Angie, since we are sometimes, I guess, skeptical.
2:09
A little skeptical, I would say Arvind has a realistic perspective, which I really appreciate.
I think that’s a good way to say it.
Yes, Agree.
Yes.
I think that, yeah, he has a lot of expertise in this space.
And it’s also given that he is a computer scientist, he has a unique perspective that’s very grounded.
2:26
So really excited to have him on.
And viewers were really excited to show you this episode and, you know, wrap up the series with a bow.
So with that said, let’s get onto it, Arvind, and welcome to our Series 3 on AI and work with our Lives with bots.
2:41
Thank you so much for being on with us today.
Hi Rose and Angie.
Thanks for having me.
It’s fun to be here.
I’m a computer scientist.
I’m a professor at Princeton University.
I’m also active in tech policy.
I’m the director of a center here that works on tech policy.
2:57
The work that I do has 2 flavors. 1 is empirical work.
So for instance, I lead a team of researchers, especially in collaboration with Syesh Kapoor, who’s doing doing his PhD with me, looking at AI agents, What can they do?
What can’t they do?
How can we evaluate these claims rigorously as opposed to having to take company’s claims at face value?
3:18
And the other part of my work that is perhaps better known is combining my computer science expertise with a little bit of social science, economics and so forth to think about frameworks for how we should think about, you know, AI for the medium term.
Not so much concerned with the immediate harms here in today, although I’ve worked on that as well.
3:39
My passbook, what Ziyash is called AI snake oil that it gets at how do we think about AI as impacts today?
But my more recent work, in particular of the essay called AI as Normal Technology, looks at how should we think about the next 20 years or so.
So let’s take a little bit deeper dive into this AI as normal technology perspective.
3:57
So it’s kind of in contrast to this AI hype and AI doom and gloom that’s going on in the news and media and public discourse.
Your vision is kind of distinct about AI impact on work in society, which is what you call AI as normal technology.
So what is this perspective, and broadly, what does it suggest for the workplace and people’s relationship to their work?
4:19
There are a few key points that underpin our perspective.
One is we’re not AI capabilities deniers.
The capabilities have been advancing rapidly.
These tools are already useful.
In fact, we’ve been ourselves using agentic AI tools for our own coding software engineering purposes for for the last two to three years.
4:39
And I’d like to think we’re enthusiastic early adopters of these tools, right?
So I just want to put that out there.
This is not coming from a place of skepticism.
What we are skeptical of is claims coming from both people who are hyping these tools as well as afraid of these tools that there’s going to be very rapidly A progression towards super intelligence and then, you know, human labor will be superfluous, but at the same time AI might be able to take over the world, etcetera.
5:04
We go into a lot of detail on how these claims come from from a misunderstanding of both how AI works as well as, perhaps more importantly, human intelligence, and why we don’t think there’s a straight line from the kind of improvement we’re seeing now to superhuman intelligence.
5:19
And I think that there’s just a definitional incoherence at the heart of a lot of these claims.
People aren’t actually very clear about what they mean for AI to be super intelligent.
Happy to get into more detail on that if that’s of interest.
But starting from those perspectives, what we’re saying is we acknowledge that this is a very powerful and general purpose technology and it’s not going to quickly become superhuman.
5:40
Then we need to think carefully about what kinds of effects is going to happen because it is going to have very powerful effects, good and bad effects.
And it’s not going to be this thing that’s a separate species that puts everybody out of work.
And then so we go back to past general purpose technologies like electricity, like the Industrial revolution.
5:56
And we think at least so far in AI, there are very strong similarities to how things played out in the past.
So one thing that’s very important to us is that even though there are rapid capability improvements, those don’t translate immediately to massive labor impacts because in our view, the barriers are downstream of the capability improvements, kind of the smartness of the models, if you will, needs to be translated into useful products.
6:19
People need to learn to use those products.
In many cases, businesses need to restructure to take advantage of these new possibilities.
So those are some of the core principles.
And we think that this is going to be very powerful, but the impacts are going to unfold over a period of roughly, you know, 20 years or so.
And we have a lot of agency down stream of the models themselves to shape how that impact is going to unfold.
6:41
I’m curious what you think about this discrepancy between kind of all of the push toward AI use versus your perspective that it’s not necessarily at the stage that it will take over people’s jobs.
Because you see in the corporate world that a lot of executives are telling their workers to use AI and then it seems that there’s a little bit of a replacement there.
7:04
So what’s that balance there for you in terms of your perspective on on what’s happening?
Yeah.
I think there are, there are a lot of misconceptions I think on both sides, people who are pushing to put AI and everything as well as people who are very skeptical about AI.
So let me go through what I think are some of those misconceptions.
7:23
So far, these generative AI and agentic AI tools have been particularly useful as kind of collaborators, as ways to assist people to be more effective at their jobs.
They have been far from effective at complete automation and there are a lot of reasons for it.
7:39
One big thing is reliability.
Our team has a lot of research on the capability of reliability gap.
Even if AI is going to do a good job 99% of the time, you know, if you’re going to put something out there in a high stakes domain like law or medicine or even customer service, we’ve seen if there’s going to be even a 1% failure rate, but that’s going to lead to legal liability for the company.
7:58
It’s not a good automation tool, right?
So there was this famous case where Air Canada had a customer service chatbot that made-up some fake refund policy and the case when all the way to the Canadian Supreme Court and the court actually forced the company to abide by this non existent policy.
So that’s what I mean by a good tool for assisting human workers, not a good tool for replacing people.
8:18
So that’s kind of the first misconception.
The second misconception is jumping directly from demonstrated capabilities to the idea that these tools are not going to be effective at whatever it is replacing or assisting people.
I think that’s not the case.
8:34
There’s so much that needs to happen downstream.
A good example of this is in software engineering where these models have been actually really capable for the last couple of years or so.
But we’ve seen tools particularly like Anthropics clawed Code really take off in the last few months.
8:50
And the reason that’s happened is not so much the capabilities, it’s doing all of the what’s sometimes called scaffolding around those model capabilities to turn those kind of raw models, if you will, into effective assistants and collaborators.
And I think this partly explains a lot of the tensioners between people who, you know, might look at some benchmark score or might briefly try it out for a few minutes as an executive might, and then decide that this is something that every worker should use.
9:18
But then the workers start using it.
It doesn’t really, you know, fit into their workflow.
And they’re noticing all of these missing gaps between capability and product usefulness, right?
So I think that’s another reason why there’s a lot of polarization around this issue.
And then the final thing I’ll say is that we’ve been very clear for the last three years or so, and we set this in our book AI snake oil, which is primarily around the limits of AI, but we were very clear that this is a tool that’s useful to every knowledge worker.
9:44
I think people saying it’s it’s useless our being in my view, you know, haven’t tried it for long enough.
So for any any purpose in my case, I gente coding, but in a lot of other things, it takes a few months of use before you get to a point where you can fully integrate it into your workflow.
And maybe you’re someone who doesn’t want to put in that work.
10:01
You have other priorities totally fine or you have other principled reasons to avoid using AI tools.
But unless you’ve put in that level of work into trying to adapt these tools to be useful for you, I don’t think it’s credible to say I’ve tried these tools and I think they’re useless.
I mean, that makes a lot of sense.
10:18
And you know, one of the things that you talked a lot about is just how slow the adoption might be and that it would be likely much slower than what a lot of people are talking about.
And I think you use the term diffusion and it being much, much longer and perhaps and people might expect, I’m curious to what extent that might relate to the benchmarks of people.
10:37
People use aren’t necessarily the benchmarks that they think that they are and whether if they got the right benchmarks, would that change the speed of diffusion at all?
You know, what do you think about that?
Yeah, definitely.
So let’s get into why diffusion tends to be so slow.
A lot of people will say AI is this powerful general purpose technology.
10:55
It can be used for almost any task.
And that’s why diffusion has been unprecedentedly fast.
I’ve looked into those numbers, some of the claims around how many people are using Chanchi PT or any other AI tool.
They don’t make a distinction between someone, you know, using it once a week to write a Limerick or something versus actually using it in their work in a way that enhances what they’re doing.
11:18
And so in my view, diffusion of this technology, at least so far has been just as slow as with past technologies.
Again, it takes a period of decades.
And I would say that especially with general purpose technologies, we should expect diffusion to be slow because the supporting infrastructure just isn’t there.
11:36
My favorite historical case study of this was done by economic historian Paul A David, and he looked at when electricity first became a thing, what did that do for productivity in factories?
So initially people expected that you can replace these big steam boilers with electricity and it’s going to make everything so much cheaper, more efficient, etcetera.
11:54
That didn’t really happen.
It turned out it wasn’t really driving cost benefits.
It took them a while to figure out that the way to actually take advantage of electricity is not as a drop in replacement for steam.
And when I think about that, I think about all the AI people saying it’s going to be a drop in replacement for human workers.
12:10
In my view, absolutely the wrong way to go about it.
It turned out that the right way was to take advantage of the fact that electricity was much more portable, and so you can move it, generate it wherever you want.
So instead of 1 big steam boiler producing power that’s then mechanically transmitted everywhere around the factory, you can move the power generation to where it needs to be.
12:29
So that led to the logic of the assembly line.
That led to breaking tasks down into much smaller components.
It led to workers being trained differently, changes in the relationship between the worker and the firm.
And all of these adaptations cumulatively took a period of something like 40 years.
12:45
And yeah, I think that’s what what we should expect with AI as well.
And to your point, Angie, exactly, people are looking at these capability benchmarks.
They don’t tell you about all of these downstream innovations that are necessary, right?
So, So what the equivalent of having to change the whole layout of your factory, right?
13:00
There’s no benchmark for that.
And I think it’s going to be hard to build benchmarks for that.
So we have a big team working on benchmarks.
We’re trying to build better benchmarks, but our goal is to move benchmarks from completely useless to mostly useless.
I think there’s only so much that benchmarks are going to be able to tell you.
13:16
A lot of the most useful kind of information is going to be qualitative, you know, with early adopters experiences, what are the pain points, how can those pain points be addressed, etcetera.
I think one thing that might be helpful for our viewers in terms of benchmarks is looking at the benchmarks, for example, of the Turing tests and kind of breaking that down, since I think that will be fairly familiar to our viewers.
13:35
So can you talk about the Turing tests and how that was a failure of benchmarking in a sense and that has somewhat been done away with?
Curious in your perspective on that.
Yeah.
Yeah, definitely.
So there are various things we could try to get AT with benchmarks.
One thing we could try to get AT is whether AI is going to be useful for some particular task like software engineering.
13:57
And we have benchmarks like sweep bench that get it that the Turing test is a very different kind of thing.
The goal there was not to see if it can be useful for some particular purpose.
And as yours might already know, the point of the Turing test was can an AI fool a person at the other end of a text chat basically into thinking that it’s a human?
14:17
Turing had some complicated set up in terms of gender, but that’s generally not considered very relevant to what the test was trying to get at.
So, you know, can a model pretends to be a human is basically what the test was.
And that’s not, it’s not trying to get at usefulness for any particular task or any set of tasks, right?
14:32
It’s for the most part, we don’t want to deploy AI in a way that it’s pretending to be a human.
But what Turing’s underlying assumption was that it’s going to be hard to probe the internals of AI so we can use AIDS behavior as a proxy for its internals.
And he thought, and a lot of people thought, and you know, as when I was in college learning about AI, I completely believed this, that the only way to pass the Turing test to fluently behave like a human, to simulate human behavior is to have the entire suite of underlying cognitive capabilities.
15:03
And if a model didn’t have human equivalent underlying cognitive capabilities, at some point it’s going to trip up and it’s going to be busted.
But as we know with large language models, that’s generally not the case.
We can look inside these models and we know that the way they quote, UN quote, think is very dissimilar to how people do.
15:22
And their understanding of the world in many ways fall short of even a toddler.
While in other cases they have very impressive capabilities.
So what’s called the jagged frontier, and despite having this very jagged frontier, despite not satisfying the spirit of the Turing test, right, having equivalent capabilities across the board, they are able to very convincingly mimic being a human.
15:42
So that’s one reason why I think the Turing test has fallen out of favor.
It was quite interesting for me, Ivan, when I was looking at the concept of, you know, AI as normal tech and you spoke about it as being 3 things, as being a description, almost a prediction, and then as a prescription.
15:58
So something about how we should see AI and how we should be interacting with AI.
And I thought that was quite interesting in terms of like a normative kind of discussion.
I was curious as to what approaches you were taking around trying to help people see that differently and what steps you were taking in terms of almost like as a change agent or a theory of change.
16:19
I was curious if you could share some ideas about that.
Yeah, definitely.
One ironic thing about AI discourse right now is that the folks who say they’re the most worried about super intelligence are the ones pushing for policies or ways of approaching AI that, in our view, are most likely to bring about the threats from super intelligence.
16:39
So for people for whom super intelligence is an existential threat, it’s so dangerous that it must be tightly controlled.
And so, for instance, it would be dangerous to have openly available AI models that are very powerful that anyone can download and use.
And so they imagine a world where only one or two companies are building this super powerful kind of AI.
16:58
And because they imagine AI to be so capable, not just at technical things at which it is undoubtedly very capable, but also in judgments and all of the other harder to define aspects of human cognitive abilities.
Because they think AI is or soon will be super intelligent, they want to delegate as many decisions as possible to AI itself.
17:18
And the hope is to essentially quote, UN quote, build God, but to do it in a quote, UN quote aligned way so that this machine God is going to act in our best interests.
And we think that’s an incredibly brittle approach.
They’re not going to stop this technology from proliferating very widely because the knowledge to build it is very, very widely available.
17:37
Engineers move very freely between AI companies.
There’s something like, you know, I would say on the order of something like 100,000 people in the world who have some part of the relevant skill set in order to build highly powerful AI.
And some folks will say, oh, it requires these huge data centers.
17:53
And so if only we can exercise control, if governments, that is, can exercise control of the level of data centers, then we can stop this technology from proliferating.
But if you look at the exponential curve at which the costs have been coming down to train and run these AI models, I don’t think that approach is going to buy you too much time either.
18:11
Maybe a couple of years.
But if if the whole point of your approach is to put these existential threats back by a couple of years, I don’t think that’s that’s a very solid approach.
But perhaps most importantly, by delegating so much to AI, we’re almost ensuring that when things do break, they’re going to break catastrophically.
18:28
So our approach is very different.
AI want to go back to what I said earlier, that while super intelligence like capabilities are possible in narrow technical domains like chess, which involve a lot of calculation, we don’t think super intelligent judgment is really a thing.
18:43
Because human judgment, you know, in complex tasks is not bottlenecks at all by our biology.
It’s just bottleneck by how much information is out there in the world that we can draw upon in order to make better judgments.
And so whatever bottlenecks exist for humans, equally well exist for AI as well.
19:00
And we can and should keep it that way.
If we set aside this worry about AI, it’s going to be super intelligent at things like persuasion and you know, break out of this box, etcetera and start seeing it as a very, very powerful tool, but nonetheless a tool that leads to a very different approach.
19:17
I think proliferation of AI capabilities is good because whatever risks there are that from AI, and we acknowledge that there are many, we’ve recently seen the mental health harms from chat bots, for instance.
We can ensure that these threats arise gradually as this technology becomes more widely proliferated, instead of suddenly being unleashed upon the world because of one company being in control of AI, developing a ton of new stuff, and then deciding to release it or not release it and keep those capabilities internal.
19:45
So that’s a way in which our prescriptive approach actually differs radically from a lot of the writing on super intelligence.
So you mentioned that AI, I cannot do these tasks necessarily as autonomously, that it requires human judgment in the loop.
And one of the things that you say is that you predict that an increasing percentage of human jobs will be related to AI control.
20:07
So what does that mean from a definitional sense?
What is AI control?
And then what does it mean from perhaps a normative sense?
So what it means for work more broadly?
Sure.
Yeah.
So I think this entire point is kind of inherently normative.
It’s not so much that AI can’t do these judgment tasks.
20:24
We can allow AI to, you know, kind of run amok and take on increasing levels of autonomy.
I think we will very quickly discover that failures are unpredictable.
And when they do happen, they’re very costly.
So normatively, we’re saying this is not the world we should head towards, right?
We should choose to ensure that humans stay in the loop.
20:41
And again, in areas like software engineering, I think, which is an early indicator of what I think many other domains are going to look like, a lot of AI assisted software engineering already looks like this, where what the engineer is doing is primarily doing things to stay in control of the AI essentially.
20:57
Because you can already today choose to simply, you know, just type what app it wants to build and go ahead and build it.
I mean, it’s going to be, you know, shitty code, but it’s going to work.
But the problem is at some point it’s probably going to fail.
And when it fails, nobody understands the code.
Nobody’s going to know how to fix it.
21:13
Right now, software engineering is a very mature industry.
People have a very good understanding that this kind of failure case can arise even before AI.
We’ve had so much past experience with building things that are too complex for anyone to understand.
And then, you know, companies just completely fail because nobody can understand their own code base anymore.
21:32
So software engineering is an industry where we have a lot of prior experience with recognizing that when you build something you can’t control, and this was even before AI with people creating large code bases without the kind of discipline and structure to actually stay in control of that code base, you get to a point where nobody can understand the code anymore and you can’t fix bugs in the code and companies just fail because they get into situations like this.
21:59
And so from that perspective, you know, the industry is pretty risk averse.
And at least in this narrow sense, there’s also a move fast and break things culture.
I don’t want to minimize that, but in terms of recognizing the necessity of software engineers to understand their code.
So the vast majority of AI use that I’m seeing in software engineering, and I’m not talking about, you know, two people in a garage putting something out there, right?
22:20
Companies that have a business to actually protect, they’re, in my view, doing it in a pretty responsible way.
They’re not just letting AI loose.
They’re having their engineers be the ones figuring out what the model needs to do, reviewing the code, creating and running tests and various other kinds of quality control measures.
22:37
So I see progress continuing in this direction where most of the actual typing of the code, if you will, will be done by AI and the work of the human is all of the other stuff that goes into software engineering.
And it turns out there’s so much other stuff that goes into software engineering, not the typing of the code, that even when you almost completely delegate writing the code itself to large language models, it turns out that was not even the majority of how software engineers were spending our time.
23:03
We always kind of knew this in the back of our minds, but now we’re really directly recognizing this.
And I think, yeah, it’s going to be similar in a lot of different fields.
Like you said earlier, you know, people would be foolish to think that AI is not going to make a difference in in the workplace.
23:19
Like you said, you know, it might take a few months to actually engage with it, but you can see the real benefits to some extent.
One of the things that we’ve discussed over the last couple of episodes, though, is that people that are explicitly encouraged to use it in the workplace, you know, encounter some real roadblocks, whether it’s the individuals or even some organizations.
23:38
And we’re not obviously talking about the big four that are designing it and building these platforms, but you know, just in your general corporate sort of sized business.
So what are some of the roadblocks that you see and how would you recommend that individuals can overcome individuals and maybe medium sized organizations can overcome these roadblocks so they can get these benefits?
23:57
Yeah, definitely.
So I would say one big one is if you’re delegating a task to AI, how do you verify the correctness?
And you know, I don’t have any general prescription for this.
It really, really depends on what someones job is and what tasks they’re using it for.
24:13
But I can reflect on my own practices.
I try to find tasks that fits one of two patterns.
One, it’s either low stakes and you know, it’s it’s OK if AI gets it wrong once in a while.
Maybe when I’m prototyping a piece of software, there’s going to be a whole separate process after prototyping where you evaluate whether the software is something you even want to build and then go ahead with the actual building of the software.
24:39
So there are many stages of this process where you have a high tolerance for errors.
So that’s one and the other, if there is some reason to expect that verification is going to be faster than doing the whole thing yourself.
And again, it depends on what the task is, but in many cases, you can check a sample of the output or you can have automated ways to have another AI system check the output.
25:01
You know, it doesn’t work in a very general way, but it depends on what the task is.
So that’s the first roadblock somebody might experience.
How do you ensure the reliability of the system’s output?
Maybe the second roadblock is how can you delegate things in a way that in terms of the judgment that needs to go into it, in terms of a lot the normative decisions that need to be made and whatever task you’re doing, how do you ensure that you as a human stay in control?
25:25
What is the appropriate division of Labor between people and AI?
And something that can be really useful is for people in a company to have some kind of community of practice, if you will, some way of exchanging information.
What has worked well, what has worked less well, 1/3 barrier that people might experience is the worry about deskilling.
25:46
So if you’re letting AI handle too much of the work upload, you know, six months down the line, are you still going to have enough of those capabilities to do things yourself, to even be able to supervise the output of AI?
And in my own work, what I try to do is I take a lot of the time I’ve saved due to the productivity gains of using AI and actually use a lot of that time for improving and sharpening my own skills.
26:10
And, you know, even going to a higher skill level than I would have previously thought possible.
And I’ve built a bunch of my own AI tools to help myself learn more effectively, actively.
It’s actually it takes just an hour or two of work to build a whole space repetition learning system, for instance, AI coding tools are very powerful at this kind of thing.
26:28
So I found that I’ve actually I’m spending a lot more time learning, but it takes conscious effort to change your workflow so that you make that time for learning.
You’re always going to be under a deadline for the next thing.
And if you know, we don’t have first of all, we should have the freedom from our employers, you know, to be able to carve out that time, but also we should have the self-discipline to be able to carve at that time.
26:48
And if if we don’t do that, I do agree that descaling is going to be a big threat.
And then maybe the 4th and last one, another barrier that people often experience is the integration between AI tools and their existing data stores, their workflows, you know, the other tools that they use.
27:05
And you know, this is getting better.
It’s still very early days.
And this is exactly the kind of thing that we mean by capabilities are increasing rapidly, but the products are not necessarily getting better quite as quickly.
So you can be either an early adopter or a middle or a late adopter.
27:20
So you can either give it some time for these tools to get better, or you can try to figure out your own integrations between highly capable models and the existing tools and data sets that you use.
Yeah, that makes a lot of sense for the individual worker and the sort of roadblocks that they may experience.
And more broadly, do you have a sense of what particular sectors or types of jobs might be most impacted by AI thinking?
27:43
Maybe in the immediate term, but also in the medium and long term?
Yeah, for sure.
So we’ve already talked about software.
I think the second biggest one is media.
So I’m talking things like video production and editing, marketing, various other things, to some extent news media as well.
28:01
So I can tell you in my own experience, I’ve started making YouTube videos now and I’ve been pretty shocked by how well AI is able to do the work of, you know, I’m mediocre, but for my purposes, adequate kind of video editor and producer.
And again, this is an area where the products are quickly getting better.
28:20
In addition to capabilities.
It really comes down to whether the use of AI is considered appropriate or not by the community, right?
So AI could be used today for churning out news articles, for instance.
But you know, journalists have strong norms about what is considered the core of journalistic work and integrity and accuracy and so on.
28:39
Yes, there have been many examples where those norms have been violated, but still some core of that profession, I do think by and large maintains those norms.
So I think those are going to be the factors dictating the speed of diffusion.
I think the capabilities are in fact going to continue advancing pretty quickly.
28:54
Let’s see, maybe let me give one more example.
Law is another big one.
We have a paper coming out pretty soon on AI and law and we do think that the capabilities in many ways are already there slash are going to get better quickly.
What we worry about, though, is that the adoption of AI and law is going to lead to an escalating arms race of more and more AI use, more and more documents produced between plaintiffs and defendants, because that’s exactly what happened with past waves of adoption of information technology.
29:24
So the big thing was ediscovery.
Discovery is the phase of litigation where documents are produced in order to identify what is going to be relevant in terms of the evidence at trial.
And when this process started to go from paper, which is slow and cumbersome, like you can’t even do a control F rights to an electronic format, people thought this is going to save time dramatically.
29:47
But what actually happened was the number of documents produced went up by three orders of magnitude to the point that the total human time spent in discovery has actually substantially increased, not decreased.
And there are strong reasons to expect that you might have similar things with AI, which is, by the way, great from a, you know, job security perspective.
30:07
Our conclusion is that AI is not going to replace lawyers.
If anything, it’s going to lead to even more work.
But there’s also reasons we might worry about it.
You know, what does this mean for us if we’re worried that we’re already a very litigious society?
Are we, you know, just creating make work for lawyers or charging, you know, 1500 an hour for work that could have been done much more effectively if you had de escalated the arms race?
30:28
So what we run into again and again are normative questions more than technical questions.
To that point, you’ve spoken a bit about deskaling, we’ve spoken about the risk of an arms race, but can we just hear any other kind of concerns you’ve got in terms of how AI might impact workers in terms of their experience of having AI in the workforce?
30:50
And I’m thinking here about things like diminishing share of income, maybe disruption of training pipelines, perhaps even impacting how they experience their work in terms of job satisfaction.
But maybe the things at play on your mind would be great to get your perspective on.
The job satisfaction 1 is is huge.
31:08
It’s going to really vary from job to job whether AI is automating the fun parts or the boring parts.
In my experience, again, going back to software engineering, the reason this has been so fun is because a lot of the more boring parts can be automated.
31:23
And frankly, going back to a time before AI assistance for coding feels like, you know, going back to punch cards instead of keyboards or something like that, right?
It’s, it’s just almost feels primitive now and something that I don’t even want to contemplate.
But I can very much imagine, you know, when when it comes to artists, for instance, I mean, we can have debates about whether AI art is actually art, but to the extent artists want to incorporate it, it feels like it’s doing a lot of the actually parts that are creative and fun.
31:50
So the experience can vary dramatically between different professions.
And so I think, you know, we’re going to have to figure out what is it about a particular profession that’s worth preserving?
And so even if AI can do something, should we let it do that?
And if the answer in some cases is no, what are the opportunities to exercise coordination and collective action so that we don’t have a race to the bottom where everybody feels compelled to adopt this tool in order to derive productivity benefits?
32:18
And yeah, you mentioned a couple of other things.
One big one that I worry about is disruption of training pipelines.
So what I mean by that is going back to the example of AI in law, If AI is doing a lot of the things that paralegals can do or junior lawyers can do, and less of the complex judgement oriented things that senior lawyers, senior associates and partners can do, what that might mean is there’s much less demand for a junior folks.
32:43
And so how do people climb up the ladder to where is the next generation of senior folks going to come from?
And tech companies are worried about that as well for software engineering.
Yeah, I don’t know the answers to those questions.
We have some tentative suggestions.
There needs to be, I think, more investment in these training programs.
33:00
So, for instance, one way in which hospitals have historically solved this problem for medical residents, right?
So doctors need to undergo, you know, well over a decade of training.
But while they’re in that training process, the hospital is investing them.
But after they’re trained, they might go somewhere else.
33:15
So the investment that the hospital has put into training those residents, they’re not able to capture the benefits of that.
So how do you how do you deal with this?
One answer in that community has been that residency programs are located in large, prestigious teaching hospitals.
And So what the hospital gets out of it is status.
33:34
And so when you’re a high status hospital, it allows you to attract, you know, top surgeons or whatever, and that you might be able to do much more complex cases than a lot of other hospitals are able to handle, right.
So they’ve had these weird ways.
You know, when you consider from a point of view of economic theory, you might think that it does not rationally make sense to train doctors for so long when you don’t get to keep them at the same hospital.
33:56
But they’ve had ways to to overcome this problem.
And I think many other professions in the future are we’re going to have to get creative if it’s going to be the case that it takes, you know, a decade of someone learning various skills before they can actually outperform some of these AI tools.
34:12
And again, it might mean that we see a whole host of new occupations coming out in the next decade that we just can’t even begin to imagine what they might be.
Exactly.
Exactly.
And I think it’s really useful to think about all of the ways that technology has impacted us in the past to kind of inform what it might look like now.
34:32
But I’m, I want to touch on that point where you talked about, hopefully it’s not just a race to the bottom.
And it requires also a lot of intention and, you know, effort to make sure that when you use AI, you’re actually using it to upskill, you’re using intentionally, etcetera, and kind of monitoring your own use.
34:51
But there’s also this trend toward offloading tasks on 2 agents that not just one person is doing, but other people are doing.
And so now we have an agent to agent sort of interactive environment and to kind of pivot a little bit.
35:07
I’m sure you’ve seen the Moult book and the Claude bot plethora of agent to agent interaction that’s happening.
And granted that that’s a little bit of a fad happening, but it also is somewhat reflective of potentially what the future looks like with agent to agent interaction.
35:24
So curious what your thoughts are on that and whether that is reflective of what we’re moving toward generally with work as well.
Curious to hear your thoughts.
Yeah, I I don’t want to speculate too much on what Claude Bot and Moulds book represented.
Still too early.
35:39
I feel like I haven’t had enough time to really study it.
It’s it’s too quickly evolving.
I feel confident enough to say, you know, people who say this is some kind of take off or singularity or agents are going to go off and build their own world.
That stuff is way, way too premature.
Just like when ChatGPT first came out, people were not used to this idea of a bot that could talk to you.
36:00
And so, you know, initially they read too much into it.
They assumed these machines must be much more human like than they actually are.
And, you know, there was a lot of initial hype that then gradually died down.
I think maybe we’re seeing the first time instead of AI mimicking a single human, they’re mimicking the social organization of humans online.
36:21
And maybe that’s, you know, you’re seeing that initial shock for a lot of people.
So I do think a lot of that is going to gradually wear off.
But at the same time, I think even in some of our research, we’re seeing that a collection of agents can do things that a single agent can’t.
So that’s definitely a trend.
36:37
Whether the kind of experimental kind of uncontrolled agent conversation is going to become anything more than a fad, I can’t say.
But multi agent systems, yes, have been a thing.
They are going to be a thing.
Thank you Arvind, I think it’ll be interesting to watch this space.
36:53
Agree it’s very, very early days and who knows what they’re going to do.
They’ll probably burn themselves out sooner than turn themselves into a long term trend.
Let’s hope.
It’s been really, really great having you on the show with us.
I know our listeners will be very grateful for having heard everything from you today.
37:08
Arvind, is there any last thing that you want to share with our listeners before we say goodbye to you?
Maybe I can share some last words of comfort on the AI will take everybody’s jobs issue.
Yeah.
37:24
So as listeners might know, with every wave of technology, there have been claims that it’s going to massively replace human jobs.
And historically, that has basically never been borne out.
And there are even famous examples of the opposite.
For example, when ATMs first became a thing, over the next decade or two, the number of human tellers actually increased, not decreased.
37:47
And the reason for that is that once you had an ATM, banks found it much cheaper to open a regional branch because most of the time you don’t need a person there, you just need an ATM.
But once you do have a regional branch, at least some of the time you need a person to be employed there for the things that ATMs can’t handle, right?
38:05
And so this is called Jeben’s paradox sometimes.
But the broader lesson is that a lot of our worry about AI taking jobs comes from a presumption that there is a fixed amount of a certain kind of work to be done, whatever that is.
38:21
And I’ve talked about how in the example of law, for instance, what happened when you had productivity gains was actually the amount of legal work that lawyers do dramatically went up.
And I strongly suspect that this is going to happen in most areas.
Software engineering is another great example.
38:37
The amount of code that we produce now, it’s more than a million times more than what we used to produce in the early days of software engineering and even before AI.
It got to a point where every car now has 100 million lines of code on its onboard computers.
I’m not talking about self driving cars, right?
Just regular cars, which we don’t think of as software as 100 million lines of code.
38:57
And at least in white collar work, what we think is going to happen in almost every job is that when it becomes cheaper to produce a unit of work, we’re going to realize that the demand can actually go up by several orders of magnitude.
And so, yes, there might be job losses in some areas.
39:13
They have much less to do with the technology itself and the preexisting labor arrangements between employers and workers.
The areas where we’re seeing job losses are things that are already kind of gig work where you don’t have a good relationship, you’re not strongly integrated with the company you’re working in.
39:31
Let’s say if you’re a translator who, who does gig work online, it’s easy to for that to be replaced with AI.
But if you’re a translator whose work actually involves understanding the cultural context, for instance, right, and working closely with whoever it is you’re translating things for, that’s much harder to replace with AI because the work you’re producing, it’s not just the words you’re translating, it’s all of that knowledge that you’re bringing to the organization.
39:57
So it’s those factors, I think, that are really going to determine whether AI replaces jobs or not.
And so now more than ever, I think it’s important to ensure that we have the right labor rights and labor arrangements in place.
And if we have those in place, we don’t have to worry too much about increases in technical capabilities.
40:15
That was a fantastic wrap and so reassuring for our listeners.
It’s a wonderful call to have AI as a normal technology, empirically grounded and very reassuring for our workers out there, for our industry drivers, and to our researchers.
And just such a wonderful wrap of AI in the workplace for our Series 3.
40:35
So thank you very much from myself and from ours.
Yes, and I like that There will always be more work.
That’s the optimistic part of things, but maybe more than you could ever hope or what for.
And I just hope that it’s all meaningful, right?
So hopefully there’s a good shift in that We have good labor laws that the policy side of things will be able to protect us and as we move forward, but that also that, you know, people can rest assured that there will be work and it’s a matter of finding that meaningful work in this time.
41:05
So thank you.
Appreciate it so much.
Thank you.
Thank you Angie, this has been such a fun conversation.
Likewise agreed.
Thank you so much for being on.
Of course it’s my pleasure.
That was a great episode, Angie.
That was a great episode.
41:21
I really enjoyed talking to Arvind.
I think what I appreciated the most is that he brought quite a different perspective with AI as normal, well, AI as a normal technology rather.
And I appreciated that as a contrast to the doom and the gloom sort of cycles that are out there at the moment or boom rather kind of, you know, what’s going on.
41:42
So I appreciated that as a different way of seeing it and kind of a more historical basis for looking at what’s what’s happening with AI, which I think is quite useful for people in industry.
Agreed.
And it was really helpful to get all of those concrete examples of what it looked like from past technological advances and what diffusion really looked like in terms of the speed of uptake and what the bottlenecks were within organizations to actually get to the level of adoption that they aspired to.
42:12
And kind of all of the things and steps along the way that that they had to rectify in order to get it done.
So like what was it the going from steam boilers to electricity, etcetera.
And I also really liked the point about lawyers and the fact that, you know, progressing and optimizing things actually just created more work for lawyers.
42:34
Like, yes, oh, all of a sudden you can have so many more documents because you can generate them more, but then the lawyers have to look at more documents.
And I don’t know, I think it’s interesting that maybe that will be the case with AII think AI will make it much easier to search for different cases, for example, to use as leverage, but also to, you know, sort through materials.
42:57
But then we’ve seen so many instances recently of, you know, AI being used in court cases as in generative AI to create deep fakes or digital duplicates of people to give testimony.
For example, there was a man who actually passed away and there was a video of him, a digital duplicate.
43:15
It was an AI generated video of him speaking to the judge about his experience, having his own life taken.
And the judge actually responded to it in a very emotionally, you know, sympathetic act, which I was expect I was, I was surprised by.
43:33
So that wasn’t the one that you sent me, where the judge dismissed it.
No, that was a different 1.
So there was a more recent one where the judge dismissed some sort of AI generated content that a lawyer had put together.
It was something of the sort where the lawyer was trying to, you know, cut, not cut corners, but trying to make something even better.
43:53
But really just offloaded the whole process onto AI and it turned out to be like a terrible product, just misleading.
It was like a video or something, maybe of his own voice overlaid on the video and was showing it to the judge.
And the judge was like what the fuck is this?
44:10
Yeah.
And then she just missed it, right?
Yes, yeah, yeah, yeah.
But going back to Arvind’s interview today, because, but that was hectic.
We should put that in our hack video so we can share that with people.
Because that was.
Oh yeah.
We should, we should.
And to our listeners, we are recording a hype video soon.
44:27
So stay tuned for that.
But I, I did think the, the empirical work was good.
I found the one data point that I thought was really interesting is like when Arvind talked about how people say, well, lots of people are using ChatGPT, for instance.
44:43
And I think he cited a number from 2024 was like 40% of workers are using it.
And I wrote this down to hold on a moment. 40 percent, 40% of adults are using generative AI.
Okay, so not chatty generative AI.
This is from August 2024.
45:00
But the fact is that we’re doing it infrequently.
And as a result, it actually only accounted for naughty .5 to 3.5% of their work hours.
And as a result, it only created an impact of on the productivity of naughty .125 to naughty .875% productivity improvement.
45:20
And it’s like those kinds of stats that I thought like that’s the, that’s the real data that we need to try and understand what’s really going on.
Because you see, like in some of the other research we looked at earlier in the series where they’re like, well, why isn’t it improving productivity when everyone do the 800 million weekly users?
45:41
So why aren’t we getting the productivity gains?
So it’s like as there’s usage and then there’s usage.
So I thought that was that kind of information is really useful.
Agreed.
And I thought so much of what Arvind said underscored the things that we’ve talked about in the past, like you said.
And I was thinking about the Goldman Sachs research that said that companies are operating at under 10% or under 15% level of AI adoption for using AI in production and how Goldman Sachs said that number was lower than expected.
46:11
And then also Marisa, in our education episode, so on impact of AI and children and young people, Marissa said that given the reports that you’re seeing about the adoption of AI and education, you would expect more people to know within education about AI tools, how to use them.
But that AI actually really isn’t being integrated into education because of there are these sort of institutional bottlenecks that are being talked about, maybe infrastructure, maybe funding, maybe know how of how to use these technologies.
46:41
So I think we definitely got a more practical picture of what’s going on today.
And I hope that people can walk away from it optimistic, but also cautiously optimistic, recognizing that here are kind of the pieces that you can use to put together the puzzle of what an optimistic future could look like.
47:00
So I don’t know, I thought it was very interesting the different bottlenecks because that’s not something I necessarily think about a lot when I think about AI adoption.
I think more about AI adoption on the individual level and how that thinking more like individuals who have access to the Internet can use AI.
But then for it to actually influence the organization at a a wider level, a greater level, it requires so many more institutional reforms that it’s, it’s too early for for us to see any of that, I guess.
47:31
What I also appreciated hearing from Arvind is just the kinds of use cases that he’s employed to try and help himself like, well, any sort of productivity gains he gets, he kind of reinvested time to upskill himself in further AI tooling or further skill development.
47:50
And again, I thought that was a great use case for us to see again on the optimistic side.
But you know, we’ve spent a lot of time talking and worrying about deskilling.
And so that was a nice alternate perspective again to hear from somebody that that can actually use it.
So I thought that was quite nice.
I thought that was too.
48:05
And I really liked also what Arvind talked about when we asked them what sectors or jobs might be most impacted.
And he mentioned how that requires the sector to be kind of on board with using AI and it to be accepted as used.
So like for example, creative industries, there are perhaps groups of creatives who don’t want to use AI and don’t prescribe using AI.
48:29
And so that might look different than for example, let’s say someone who is in graphic design or advertising who maybe doesn’t want to do all of kind of the back end creative stuff or hire people to do it.
Usually that’s a different story.
But again, it all requires intentional use.
48:45
And I do still worry that with the push for productivity, a lot of kind of the meaningful labor and meaningful work that people find value in is going to be offloaded.
And then it’ll be the AI control jobs that Arvind talked about where people are kind of just monitoring AI use.
49:06
And maybe that’s most of their time is spent doing that and they don’t actually have the time to upskill in more interesting domains or in more interesting tasks.
What do you think?
Well, I think that’s the risk that’s kind of where we need to pay attention to making sure that we are aware of why do we do what we do and, and kind of paying a lot of attention to keeping that as the focus of it.
49:31
Because I think that can slip whether it’s to AI or to kind of inertia and our jobs or for all sorts of reasons that aren’t technological.
And so I think it’s always a good idea to kind of keep that front of front of mind of like, why am I doing this?
49:47
What actually brings me joy and keep kind of returning to that throughout our careers to make sure that we’re doing the things that give us the most joy and where we get the most fulfillment from.
And so I think that’s important and we can remind people of that during this series.
50:02
And I think we’ve done a good job.
Yeah, well, I hope we did a good job.
And listeners, we always want to hear from you for feedback about our series.
But this is a wrap for Series 3.
And as we promised, however, we will be talking talking about the survey on AI ethics in the workforce and your use of AI at work.
50:21
We’ll be talking about that in Hype series later on.
But thank you so much for those who have answered our survey.
And if you haven’t, feel free to go back to the notes on episode 1 of series 3 and take a look.
But we’ll be talking about that at a later date.
But for now, that’s it from Angie and I for our lives with bots with Series 3.
50:38
And thank you so much for being a listener.
Bye from Rose.
Bye from Angie.
See you next time.