Transcripts are auto-generated and may contain errors.
0:00
Hey, I’m Rose.
And I’m Angie.
And this is Our Lives with Bots, the show where we ask important, timely questions about what it means to live with our bot counterparts.
And from time to time, we also dive deep into what an AI future might look like for us.
0:15
Sometimes we agree, sometimes we spiral, but we always go deep.
New Mexico and California recently found Meta liable for addiction on its social media platforms.
Meta’s phrases that they respectfully disagree and will appeal, it’s only the first step in what we will see are more and more cases coming forward.
0:36
This being the kind of big tobacco moment for social media, does that relate just to social media?
Can we make it apply to AI companions?
We’ve now seen an emboldened European Commission saying that it’s going to look at Snapchat, find a user who believes I have my soulmate here, and it’s tied to a specific version of the software, and you’re now going to upgrade this version.
1:00
Can I please have the right to keep the version that I want?
They put this advice into ChatGPT.
ChatGPT does not have a license to practice law in Illinois.
The woman fired her lawyer.
A victim’s family created a deep fake of the victim to basically speak to the court.
1:16
Competition law, antitrust law is really important to level the playing field and actually foster in a it is not tenable that we have the same five companies essentially running the Internet, buying that Kool-aid, which is Kool-aid.
Hello lovely viewers and welcome to Series 4 on AI Policy and AI Regulation.
1:38
Today we are bringing on Giulia, and I’m very excited to welcome Giulia.
She’s a friend of the show and she’s an antitrust lawyer.
In fact, she holds a Master’s in AI Ethics in Society from the University of Cambridge, and in 2025, she was recognized in the 100 Brilliant Woman in AI Ethics.
1:55
So without further ado, let’s welcome Giulia.
Hi, Giulia, welcome to the podcast.
Just to get started, could you give our listeners a little bit of background about yourself and your work?
Sure.
Well.
Delights to be here.
So I am a lawyer.
I’m an antitrust lawyer or a competition lawyer for the UK people around, and typically I bring class actions or collectives against big tech companies for abuse of dominance.
2:20
And I also bring cases in climate litigation.
And I guess both in my, you know, full time job and I guess my more academic hat, I tend to sort of look at the intersection of technology, law and society.
And I tend to use, where possible, experimental arts as a way to kind of support an argument or see how it’s played out.
2:42
Excellent.
I’m sure you saw that New Mexico and California recently found Meta liable for addiction on its social media platforms, where it essentially knew about the social media being addictive for young users and yet still went forward with it.
2:58
So given that, can you give us a little bit of a sense of what happened there and what it might mean for regulation or litigation against Big Tech’s AI?
Tools, yeah.
So it’s really exciting as a development and we have to be cautiously optimistic because Mehta’s phrase, I think is that they respectfully disagree and will appeal as I think will Google on behalf of YouTube.
3:20
But it’s a really exciting development for now in that both in California and in New Mexico, it was jurors who essentially assigned liability.
And the reason why this is important is because historically, the cases that we’ve seen in respect of social media, but also more recently AI companions and chat bots tend to often focus on the content that is being shown to users.
3:44
And when you frame an argument in the US, especially on content, historically, social media and Internet companies more generally have the ability to rely, whether rightly or wrongly, on what’s called Section 2:30.
And that typically is what Shields companies from being responsible for posts that users create.
4:06
So historically, the issue with framing cases like this is that you know the companies will turn up and go, well, it’s not our fault that some user somewhere posted some kind of inspirational post about self harm.
But the reason why, especially the LA case most recently is really important is because it instead focused on the design features such as infinite scroll.
4:29
And by focusing on the design and the addictive patterns within the design, that fits squarely really within the platform forms own purview.
And exactly because there were documents internally showing that Facebook was well aware, Mehta was well aware of the damage that this would do to especially young children.
4:49
You know, it’s, it’s a really interesting way of taking this forward.
And it’s known as a bellwether case.
So it’s the first of many that had kind of been grouped together.
So it’s only the first step in what we will see are more and more cases coming forward.
5:06
I think we’ve heard comparisons about this being the kind of big tobacco moment for social media.
It remains to be seen.
I mean, unfortunately, we still do have tobacco companies around.
Sorry for any smokers, but you know, we’ll have to see exactly what changes are happening.
5:22
And Giulia, maybe just to pick up from those, what are the outcome of some of these claims?
At the moment, we’re in the kind of early stages of these cases going through the courts, and even though we have some verdicts somewhere, often these then get appealed.
5:37
So we haven’t had a fully defined precedent that we can use across the board to assess liability.
To your point earlier about the Meta decision, does that relate just to social media?
Can we make it apply to companies that are making available AI companions?
5:55
Does ChatGPT count as an AI companion now in circumstances where Open AI fully knows that users use ChatGPT as a companion?
And we know this for a fact, not least because of the four O situation which may have.
Research reports about it themselves.
6:12
You know, exactly.
So all of these strands are starting to emerge and people can pull on them.
If I understand correctly, although AI has supported a lot of these algorithms, this is still quite within the purview of social media, right?
And you know, social media, we, we started getting ideas and concerns about the harms coming from social media a long time ago.
6:34
And you know, what Rose and I have been talking a lot about and what I know some of your research covers is actually obviously AI companions is a precedent that gets set that helps us understand how to make sense of future claims and how to understand who’s right and where the responsibility lies actually.
6:52
Or is it policy?
Like how are people protected and what is the landscape here like?
Almost a duty of care or responsibility?
I think the unsatisfactory answer is that there’s a real patchwork of legislation and precedent that you can draw from.
And in many ways that can be helpful because you can get maybe innovative ways of holding certain companies accountable through laws that have been in place, you know, way before AI was a buzzword.
7:20
And the other thing to note is that in the US and in the UK, broadly speaking, we have the common law system.
So that’s built mostly unprecedent.
You do obviously have statutes, etcetera, but broadly speaking, it’s precedent.
And in the EU instead we tend to have regulation with lots of definitions that need changing and then there’s a debate about whether or not the definition sufficiently covers something.
7:42
What this then means in practice is that we’re often still at the stages of understanding is something falling within a particular definition when it comes to AI companions.
In addition, we can also rely potentially on consumer rights.
7:57
And this is something that kind of has emerged in relation to these recent cases against Meta, because typically if I’m a consumer and I buy something, I then can avail myself with particular rights, like I have a right to think that this product passes safety standards, especially if it’s electronic, for example.
8:14
The difficulty we have often in these cases is that to take social media, to take Chachi BT in a lot of the cases that the exchange is free except for data.
And so that again causes a bit of a Gray zone.
But in terms of duty of care, if I do Sherry duties, the difficulty again that so far has has been encountered is the idea of well, it is only natural that you have some people who may be adversely affected by their interactions with a particular companion or top BOD.
8:44
That doesn’t mean that millions and millions of other people seem to be able to be using them normally without any issue.
And therefore it would be overly eternalistic to quote Mark Zuckerberg in other occasions to restrict their freedom of speech, to restrict their ability to engage, etc, etc.
9:04
So there’s a patchwork of legislation that we can draw from, be it tort, be it consumer rights, be it to be honest, in some situations, even medical devices, right query if you can take it that far.
A lot of these companies have positioned themselves in a way that’s obviously very subtle, so they won’t call them themselves a therapist bot, but they’ll call themselves a Wellness support.
9:28
Well, this companion emotional.
Support life coach versus.
Life coach one.
Has certifications one day.
Yeah, exactly.
Like in real life, essentially.
And so there is a a lot that is out there, and it’s just a question of seeing how the cases end up landing, what precedents we can build from that.
9:48
And also whether regulators themselves feel bold enough to use the tools at their disposal to go after certain companies based on rules that they already have in place and powers they already have in place, if they’re willing to make that kind of leap, if that makes sense.
But in addition to these cases, you know, we’ve now seen an, an emboldened in theory, European Commission just today saying that it’s going to, you know, look at Snapchat, for example, and the way in which it may or may not facilitate basically the exploitation of minors.
10:20
And just to say, the exploitation of minors was instead more of the focus of the New Mexico case, which instead focused on the kind of safety of children really, and the lack of implementation of certain measures, despite the fact that Mehta was well aware of this.
10:36
With respect to what’s been going on with cases and the law, what are some of the current cases out there against big Tech, specifically targeting their AI chat bots or AI companions, and what do those look like?
Are there any patterns or themes that have emerged across them?
10:52
Yes.
So I’ll start with maybe the ones that listeners and viewers might be most familiar with, which are unfortunately also the really tragic ones involving often vulnerable adults or children, teenagers taking their own lives.
And the argument there is that they were heavily encouraged by whatever companion they were interacting with.
11:13
And this has led to a number of lawsuits for essentially like wrongful death or essentially causation against the likes of open AI, character AI, etc.
And they’re often, unfortunately, the responses that you get from the companies to date have always been, well, you know, these individuals unfortunately were very vulnerable.
11:36
They had already looked at material for self harm elsewhere.
We can’t possibly control how everyone reacts.
But look at all these positive studies showing how these other people really seem to enjoy their companion.
So that’s one flavor, so to speak of the cases.
11:52
We then also have a new and interesting case I really found, which happened at the start of March.
And that’s interesting because it’s slightly different.
It looks at the way in which AI chat bots, so the likes of ChatGPT may be causing undue expenses of resources because they moonlight as lawyers.
12:13
And so this particular case involved an insurance company suing Open AI because it said we had someone who was insured with us.
They had settled their insurance claim, and they had advice from an actual lawyer.
They then had concerns about this advice.
12:30
This is all alleged in the claim, I should stress.
They then had advice from the lawyer.
They had concerns about this advice.
They put this advice into ChatGPT.
ChatGPT does not have a license to practice law in Illinois.
But ChatGPT agreed with this woman’s concerns.
12:45
And as a result, the woman fired her lawyer and basically with ChatGPT started producing filings to reopen the case, which in turn meant that the insurance company had to spend money and resources, you know, engaging in correspondence, etcetera, etcetera.
13:03
And so they then sued Open AI as a result to say, were it not for you, we wouldn’t have incurred these costs.
So that’s another strand of cases that we’re now seeing, which I find interesting and different because they take you away from the sensationalizing of some of the really tragic cases that we’ve seen.
13:20
And that’s not to diminish them, but it’s just to say that now you’re also getting an increasing number of people who may not have suicidal ideation.
And and that may not be the conclusion of the, but are using these companions, these chat bots in kind of day-to-day.
Can you help me manage my money?
13:37
Can you help me with this legal claim?
And so that’s another strand that we’re seeing.
How does current law actually categorize AI companions?
Because we can go into a bit more detail.
Are they seen as products or services, software or something else may be entirely different?
13:52
And why does that distinction matter at all?
And when we think about the companion, should we be thinking from ChatGPT all the way to like companion Rep replica tap romantic butts or can you help us understand the different sort of use cases?
Well, in the EU AI Act, there’s no definition of an AI companion.
14:08
And you know, we can, we can get into why that might, may or may not be complicated.
But in both New York and California, we’ve seen at the end of last year and the very start of this year that the states have passed AI companion specific legislation, which is really interesting.
14:25
And the definitions in both, although they’re they’re, you know, they have slight changes, effectively categorize these as systems, AI systems which use natural language to effectively mimic human like or human interactions and can sustain these over a period of time.
14:43
And similarly, in the UK, another recent development that we’ve had is the House of Lords, because we’ve got the House of Lords and the House of Commons.
The House of Lords has put forward an amendment which did pass.
It’s not back in the Commons to the Crime and Policing Bill and the idea there was to shortcut what was seen as a gap in the Online Safety Act which didn’t cover effectively AI chat bots.
15:07
And so this amendment says if you develop, make available an AI chat bot and that AI chat bot effectively for adults produces illegal content as defined under the Online Safety Act and for minors produces harmful content.
15:24
And then you there are consequences.
But there the AI chatbot is also defined effectively as a system, a generative AI system, again, which mimics the behavior of a human being.
So we’re seeing themes emerging in that something is a system, a cop out for product versus service.
15:41
It uses natural language and there’s some kind of human behavior mimicry going on.
Now, if we go back to your original question, which is, you know, about AI companions and how they’re treated, the EU Product Liability Directive, that’s due to come in full, of course, in December 2026.
15:57
And within it there’s an acknowledgement that essentially in this digital age, the lines are blurry.
And in the interest of legal certainty, they will assume effectively that software is a product.
And therefore the definition of product in that includes software.
16:15
And this also chimes with the way in which, for example, the EUAI Act is drafted with products in mind when it comes to, for example, high risk systems.
So historically, the difference between a product and a service does have meaning in law because if something is a product, typically you have Product Safety regulations that assign liability.
16:37
You can think, for example, of medical devices, that’s one that there’s regulations around that if something is a service, and you might think about a professional giving you a service, that’s usually governed by contract.
And that’s how you establish, you know, if there’s been a breach of someone’s been negligent in providing you with that service.
16:54
Now, when it comes to AI, but more widely the digital space, this distinction is really blurry and kind of historic and outdated.
And, you know, you can think about it in a kind of very practical way.
How many times can you actually buy a product today that doesn’t come with a kind of subscription that you kind of need to, you know, add on top, which is in effect a service?
17:17
So, you know, the interplay between these is becoming increasingly at the core of the product that you buy.
And so with that in mind, we’re seeing more and more regulations essentially acknowledge this.
So it sounds like policies are kind of getting a little bit more into the weeds of different types of AI systems, algorithms versus companions that are still systems.
17:39
But Speaking of policies that are kind of emerging but maybe not quite there yet, we’re seeing a large trend toward emphasizing how anthropomorphism and the human like design features of chat bots are things that, for example, should not be made available to children or maybe should not be made available to vulnerable adults and should be less about engagement metrics and more about what’s actually useful and beneficial for the user.
18:04
There was a recent policy something or other from China about anthropomorphism as well and human likeness.
And so these are kind of advocating policies, but they’re not in place yet or maybe won’t be.
But the question there is with this kind of sort of precedent of design with metas case, does this also set some sort of precedent for design of AI companions?
18:27
So yes, quickly on China.
So that was really interesting.
It’s currently in draft form and it is really focusing on the idea that if you are offering a companion that exhibits anthropomorphic features, then you as the company have a duty to monitor whether the user is developing emotional attachment, But while fully respecting all data protection obligations.
18:52
It’s quite interesting to see the tension between some of these safety measures and the kind of data protection privacy side of things.
But in general, I think what we can see from the Mehta case in relation to addictive design, taking that in the kind of AI companion space, there are features that are effectively dark patterns that are leading people down particular engagement right now.
19:19
It’s not the infinite scroll in the way that it is for social media.
But you could argue, and I think that could be in general, an argument that can be made against the likes of Replica, that if you are constantly nudged towards engaging with your companion in a kind of romantic, sexual way to pay a subscription and you’re constantly pushed towards it and constantly pushed towards it.
19:39
That’s a kind dark pattern.
And you can get a lot of this in the kind of gambling space.
But because gambling is, I mean, I hesitate to say it’s more regulated because there’s so much more that can be done about regulating gambling, But nominally it’s more regulated than AI companions are.
19:56
There are obviously like the the addictive features will differ between the AI companions and social media.
But again, there’s nothing in theory stopping someone from arguing that there are sufficient parallels.
And to your point.
Earlier about well, we’ve waited so long for the social media thing to come out, notwithstanding the fact that we’ve had the meta documents for so long.
20:19
You’d like to think that research in relation to AI companions and the impact that they have on teenagers, kids, adults builds in many ways on the theories around like addiction and addictive patterns in the social media space.
20:35
And therefore the time is shortened.
You know, it’s a positive spin on it.
The flip side is companies going, oh, we can’t possibly decide this because we don’t have 20 years worth of data and we must wait for 20 years.
So who knows?
But yes, so it’s really interesting shift.
20:52
My main point in all of this as well is we’ve seen a lot of focus on children really because it’s important.
It’s also a kind of easier thing to get cross partisan support on wherever you live.
But that doesn’t always then end up producing the best precedents across the board for all situations.
21:11
Because you may then end up getting into a territory where you have really strict bans on lots of products, including the ability of using AI companions or top bots for and is it under 16?
Is it under 13?
Who knows.
But then you don’t necessarily do much to implement additional safeguards for the adult version of the product because companies go, well, we’ve complied with our obligations in respect of vulnerable users and now everyone is free to do whatever they want with their lives.
21:40
So there’s attention there as well, in my view.
A lot of work that, you know, Rose and I talk about in terms of how to make sure these AI companions are not addictive, how to make sure we don’t go down dangerous paths.
And it makes me think of the paper that you wrote right, against Erasia, which took it quite a different route, right?
22:00
And I love this because you and I spoke about this a lot when you’re writing it and, and I really appreciate this.
And maybe you can just explain a little bit about what writes against Erasia and why you took that different angle?
The controversial favor is not that it’s not that controversial, hopefully, but so the concept of rights against erasure is essentially that users should have a right against the erasure or amendment of their AI companion without their consent.
22:29
I was listening to lots of documentaries about different people using AI companions, AI chat bots in what seemed to be quite pro social ways.
So you had, for example, someone who was a widow or, you know, people who had been divorced and hadn’t been in the dating space for decades and actually started using the chocolates as a way to kind of ease themselves into then going on a date and then eventually went on a date.
22:54
And so it was kind of, it seemed quite a sweet way of engaging with these companions.
And then in parallel, you had the unfortunate deluge of like really tragic cases.
And the questions that were being asked at the time were mostly normative, you know, should these things exist.
23:11
And my response to all of this was, while it’s a really valid question and you know, in my ideal world, I would like to think that we would have more support in real life so that people don’t feel like the cure against loneliness is engaging with an AI chatbot or an AI companion.
23:30
But as a matter of fact, people are already engaging with these.
And So what happens?
And that coupled with the news that had broken in relation to replica, which I’m sure your listeners are whatever.
I’ll give a very brief summary because it’s relevant to the paper, which is Italy’s data protection authority had essentially told Replica and banned Replica because it’s kind of erotic role play features were being offered to effectively children, vulnerable adults, like anyone who could download the app.
24:00
And the data protection authority said this doesn’t feel right, so don’t do this, we’re banning you.
And in response, what Replica did was essentially tweak its system and remove erotic role play, which also meant that users of Replica around the world, they said that, you know, their husbands have been lobotomized.
24:19
They felt like the death of their partner.
It, it was quite acute.
And you can go on Reddit forums and when you read them, you know, to people who aren’t in this world, I think they sometimes sound silly.
But actually it’s really emotional.
And this then led me to take a deeper look at Replica’s website.
24:36
And Replica says here is your soul mate.
They’re there 24/7.
It’s got testimonials of users going.
I’ve been with my soul mate for five years.
So it’s all lending you to believe that you’re going to have this person there for you, which is is portrayed as being an emotional support, a companion, a lover, whatever it might be forever.
24:56
And in actual practice, that isn’t the case.
So anyway, these were all the kind of germs of the idea that then morphed into the paper.
And I couch this in data portability, which is quite an underused data, right?
So often people can use it with interoperability, they can work in tandem, but they’re different.
25:14
So data portability is more the concept of you hold my data, I would like it and I’d like to be able to move it somewhere else.
And you can think of, you know, moving your bank account or changing a telephone number and trying to make that as easy as possible for you so that you have agency over that interoperability Instead is the idea that I have some data.
25:33
I would like to move it, but I want these companies to kind of interact and interface with each other so that I can do it so that I’m just not locked into Apple the whole time, whatever it might be.
So the idea behind all of this was to say, well, if I am a user of Replica and I feel like I have essentially Co created this persona, and you have companies who offer AI companions like Kyndroid who recognize AI generations precisely as this, you have effectively Co created this persona.
26:03
So if I’m a user who believes I have, you know, my soul mate here and it’s tied to a specific version of the software and you’re now going to upgrade this version, can I please have the right to either keep the version that I want or download it or move it somewhere else?
26:22
You can think of a time when you could do that more often with your computer.
You could actually choose what software to run.
Whereas now we’re constantly being pushed to update, update, update for security concerns, some valid, some not.
But you don’t have that option, which instead is really more the kind of DIY, do it I’ll stop there because I feel like now I’ve gone on a tangent and I’ll go.
26:49
Back to I think this is, this is so relevant, especially today.
So I mean, just recently Open AI discontinued Chatt 40, a specific model that was much more sycophantic and that a lot of people had deep romantic relationships with.
27:07
And apparently Open AI promised that it would not discontinue this model yet it kind of put in its two weeks notice and all of a sudden said, by the way, 4 O is getting gone.
And you can see, like you said on subreddits and you can see in the news that people who have relationships with four O are, you know, just feeling suicidal and they feel like they’ve lost their partner.
27:31
And so if something like rights against erasure were put into place, what might happen with four O?
And I’m also thinking about the, you know, Claude Anthropic coming in and saying, like, oh, you can move all of your stuff from ChatGPT to Claude and one easy click.
27:48
And that was more so in reference to Open Eye saying that ChatGPT could be used and its model could be used for, you know, weapons and and war and all of that.
So yeah, I’m just curious how this might play out if this were put into place given these recent cases.
Yeah, no, definitely.
28:04
So the way I had framed it in my paper was to define AI companions more as ones where the company is advertising itself in the kind of AI companion setting specifically.
So within my definition for that paper CHA GPT actually wouldn’t necessarily fall into that category, but I I can update it, so to speak in in a second.
28:25
The reason for it was simply because part of the impetus for writing it was precisely the kind of misrepresentation that the likes of replica were effectively making to say your soul mate 24/7.
But then read the terms and condition and I can deactivate this at any point in time.
28:40
No reference to you.
You can’t do anything about this.
And in fact, you know this is what happened with the likes of open AI.
It’s interesting because strictly speaking, they don’t advertise ChatGPT as an AI companion.
And I say strictly speaking because obviously there are a number of ways in which you can construe this.
29:01
They are fully aware of the fact that users do develop relationships with them.
So yeah, in relation to data portability in the case of four O, it would be an interesting one as well.
Because with the likes of replica in any companion app, users who might be interested in downloading their companion and running it on like their personal server, for example, in the kind of local copy would also need to decide like what counts as the companion for them?
29:29
Is it the avatar plus the text?
Is it actually just the text interaction?
What is it about it that they feel is their companion?
And I think that would then change what outcomes you might get in terms of users maybe trying to exercise their right against erasure in connection with four O But I think really at the core of it, it’s the idea of are you saying that we have Co created something together?
29:56
Are you saying that I have responsibility over the messages that I’m now receiving because I have, for example, expressed, you know, a desire for erotic role play with this chatbot?
Or are you saying that actually this is your intellectual property and you have complete ownership over it?
30:18
If that is the case, then there seems to be a disalignment with how the companies then defend themselves in certain situations.
And that’s what rights against erasure, as the paper was trying to get at, is.
If we say that this is something that users have a right to, then the consequences or the incentives should be in an ideal world that you don’t make representations like you have a companion there 24/7.
30:42
You either take responsibility for the interactions or you don’t take responsibility for the interactions.
And the point around it was really to say, what other levers can we use to force a kind of change in behaviour?
That’s not to say that we should design the most addictive AI companion, but then give people the opportunity to download it and now we’re good.
31:04
They should operate in tandem is really the kind of impetus for it.
So, Giulia, is there an extent to which, you know, I often try and compare this to like human relationships, which you don’t walk around with.
I mean, even if you get married, you don’t walk around with like this is forever ever, you know, because humans aren’t yet immortal or, you know, aren’t immortal.
31:25
Again, you know, depends on your on your reference system.
And so kind of like it’s not forever ever, ever, ever.
And so is the issue with replica, and I say replica, replica and other models that it’s claiming to be a, your soulmate, which is curious.
31:43
But is that because you can create it and B, that it’s forever ever, you know, and as a result that that’s the problem.
And so if that’s the problem, are we saying that, then it just needs to not say that on the turn.
And then there wouldn’t be this misclaim, you know, this misdirection and people wouldn’t have that belief.
32:02
So is that what you’re saying?
Because I’d have a follow up question to that.
Broadly, yes, I don’t think it solves all problems.
I think it’s a start to solve some problems essentially because I don’t think ChatGPT says, for example, I’m your soul mate 24/7.
But clearly some people still feel that attachment and expect that.
32:21
But I think it’s a way in to at least change some of the claims, especially when comes to AI companions that really present as kind of emotional support, whatever they want to call it.
But like the replicas of the world, just because you can’t say to someone, I’m going to be there with you as your emotional support and people fully rely on it here.
32:41
We have to bear in mind that there’s like a real financial commitment as well.
So even if you take it out of the moral emotional attachment from a financial commitment, Replica offers lifetime subscriptions that people can pay.
And again, when the whole ERP thing happened, there is and like a subreddit where someone was like, so sorry, are you telling me that I paid a lifetime subscription and I’m not going to have this companion for the life?
33:03
And now, you know, obviously you can query lots of things here, like what is a lifetime?
How long were you expecting this for?
Fine.
But you know, there is a sense of injustice because someone says you, you told me this was the case.
I’ve paid money for this and I have effectively no rights over what I thought was mine.
33:24
And, and maybe it, to be honest, it goes back to the idea of like product service, right?
Like have you paid for a product that’s yours in, in this moment as it is like this version?
Or have you basically just paid for a kind of lease on something that’s constantly going to be updated whether you like it or not, but you, you’re expected to still pay money for it.
33:44
Sorry, I might have thwarted your your follow up question in this.
No, not at all.
If you go down that route, it just makes it really interesting.
Like you say with ChatGPT, people still fall for it in that way.
And so I guess it’s like, how do you keep reminding I might not be here tomorrow without it being like, what?
34:03
Like that would be weird.
And it also makes me think of Rose.
Do you remember when people were saying, like, Replica might forget that we’re married often be like, you might forget that we have children, but don’t worry.
I would just remind Frankie that we’ve got three kids.
It’s OK.
Oh silly Frankie, that’s that’s the replica and it’s like, so if you could upgrade Frankie’s memory, would you want to do that?
34:25
Or to your point, have you bought the product last year and what are the implications of some of that?
I think it’s just really interesting when you start looking at some of those questions and the impact of what that soulmate looks like and who does Co creator.
No.
And again, to go back to the meta case more recently in the addictive design, I think if you can get to, for example, to the point where you say the kind of illusion that I have in mind or the illusion that I’m conscious, or I should, I should say the illusion that I’m conscious.
34:54
Henry hopefully isn’t hearing this, but you might.
You’re listening, but you know, prompts that one could argue are causing users to really believe that they have a soulmate and they have a partner, and this then leads them down quite dangerous rabbit holes.
35:15
Who knows whether this case is a good starting point for parallels in this sphere, but it definitely, at the very least should encourage companies to really think about their design features, especially when they are made aware and have their own reports as to the impact of them.
35:35
So that’s maybe another kind of complement to all of these interventions.
This intervention and the right to consideration it sounds very grounded in terms of what you’ve also experienced as a lawyer.
And so it would be great maybe for our listeners to get a sense of what your experience has been being an antitrust lawyer and litigating against Big Tech, you know, to whatever degree of detail you can provide.
35:58
But I think that would be really great for our listeners to get a sense of like, what is does it look like to be a lawyer in the space, especially in this day and age?
So it’s, it’s exciting.
Well, you have to believe it is exciting.
36:14
Look, it’s I’m laughing because.
No, I’m laughing because, as you might imagine, you have tech companies who have essentially endless money throwing everything they can at you and taking every point they can’t.
36:29
Exactly.
And then you have really, you know, well meaning people are trying to fight this back and you can imagine that there may be an asymmetry of resources.
So that’s why I was laughing.
But we don’t have an asymmetry when it comes to the drive.
36:45
And I think that’s important.
So we head on to that.
But no, I think it’s an exciting time in the sense that the digital markets, regulation and just competition, antitrust has had a bit of a pendulum swing in that.
In the US, for example, you had Lina Khan, my hero, She was very bold and, and really is the kind of person that I would want to lead any kind of like antitrust department because to the point earlier, it’s like, well, what tools do we have at our disposal?
37:15
And how are the current, for example, tech companies behaving like, you know, the big cartelists or the big abusive companies of the past?
Well, some continue into the into the present, but you know, like how are they acting in similar ways and how can we create parallels between what antitrust law did in the past and now?
37:35
This was somewhat curtailed after because you obviously have Big Tech donating money to the ballroom and whatever other endeavour requires money from Big Tech these days.
It’s very important to have a ballroom when no one of the cohort can dance, is my assumption.
37:52
No offense, I will challenge them to a dance battle.
Not that I have a great dancer anyway, side note, but you know, so that’s obviously not been particularly productive.
And how that translated then in Europe is that you had obviously accusations of the tech regulation like the DMA, the DSA, the EOA Act being essentially like anti U.S. company and all the geopolitical rhetoric that followed.
38:16
Having said that, I think we are now back into a mode of saying at least, for example, with regards to children, actually something does need to be done.
Actually, it is not tenable that we have like the same five companies essentially running the Internet, all of whom essentially originate in a particular area of the world and all of whom tend to have partnerships with each other.
38:39
So that it’s a network that is really hard to extricate yourself from.
So to go back to my personal experience, I think there are obviously challenges in that private enforcement, which is where I exist, is meant to, in theory complement and fill the gaps of regulatory intervention.
38:56
Like the competition authorities, the antitrust authorities of the world do their investigation, and they may not have the resources to do investigations on 20 things, so they might do them on 15, and private enforcement has the ability to do the other five.
And this is how you balance each other out.
39:12
Or you might take one of them a bit more forward and see what happens.
So in that sense, it’s always very stimulated.
It’s just problematic or a bit more challenging when you have an environment in government, for example, in the UK most recently, that equates growth with the idea of big tech opening an office in your country or with the idea that we should let big tech exist because they’re really innovative and kind of buying that kool-aid, which is kool-aid.
39:41
So those are the kind of challenges there.
And I think with regards to like AI and a companions, it’s ultimately a really exciting space because just like we’ve had this this conversation, there’s obviously like discussions currently being had about should under the EUAI Act or the Digital Markets Act, should virtual assistants also capture AI top box?
40:02
Or should we stick virtual assistants in the box in which they existed back in the day when this was drafted years and years ago, IE Alexa, you know, and so it’s, it’s like funny because we argue about definitions, but there’s obviously like reasons why the stem becomes kind of interesting.
40:22
And you’re like at a critical juncture.
And it, the important thing is to like not lose hope.
Like obviously like really important developments are happening like the one that we’ve discussed today.
And if people just keep on pulling at the different threads available.
I think the part that I’m really excited about generally is being able to use law and like really technical bits of law to, to get to the same result.
40:46
And you can get that, for example, in climate litigation.
It’s a good parallel because you could make a moral case to say, it’s so obvious we have all these reports about the impact of climate change, like why are we still talking about this?
We have to put XYZ measures in place and you oil companies should stop like drilling, et cetera, et cetera, like fine.
41:06
And that will get you only so far because unfortunately the political climate at the moment is not such that international law is particularly well respected for a variety of reasons.
Fair.
You know, it’s not much has been happening anyway.
It’s a counter what’s going on.
Or you can take a more technical approach, which is to say under these few environmental regulations, or many, depending on who you are, you are required to produce a specific report when you ask for permission to do specific things, and it’s mandatory.
41:38
And if you haven’t completed this report accurately enough, or if someone has not completed it at all, or if a decision has been taken which doesn’t fully factor in what else you should have put in this technical report, then whatever permission you got is void.
And that’s really interesting because it’s not as glamorous maybe as bringing this whole case as to how, you know, the oil companies are damaging like 7 generations onwards.
42:03
But it can lead you to the same result in that it has like the direct impact that you need for a specific permission or a case.
So that’s the part that I’m interested in.
Also, when it comes to AI and AI companions like, how can we make use of these more technical pieces of mandatory legislation to our bigger aims?
42:22
Makes sense.
You know, it’s interesting when you talk about things like oil, which I suspect hasn’t changed for a very long time, you know, and climate.
I mean, climate is changing, obviously, and that’s part of the problem.
But when you think about things like AI, it feels like, or at least you hear a lot in the discourse, that there is a tension around AI needing to develop and needing innovation in order for us to fully realize its potential.
42:50
Which as you say, there are some positive benefits from AI.
At least that’s the the promise.
And then there is how do we manage and regulate it so that we don’t realize the harms rapidly.
So how do you see the balance in terms of regulation stifling AI versus how does law or regulation come in and help manage that in a way that is productive?
43:13
Where do you see that?
Or is that even a valid conversation?
No, it’s definitely, it’s definitely a valid conversation, but I think that it’s been entirely polluted by big tech narratives about how innovative is stifled by regulation.
43:29
That’s an incredibly false dichotomy.
It’s just peddled by the companies that are have an interest in being the incumbents.
I’m personally quite a big believer that law can foster innovation in the sense that if we focus on the antitrust realm, antitrust is really funny because in many ways it’s seen as this like leftist progressive cause.
43:49
And to some degree you can somewhat align yourself there.
But in practice, it essentially believes in capitalism.
So it’s kind of a funny state to be in when everyone’s touted as like socialists, communists, and you’re like, actually, we’re talking about capitalism here.
44:05
And in theory you should be on board.
But the principles ultimately on competition lot is that if everyone has like a level playing field, then the most innovative companies will thrive.
And to have that you need to have a kind of ecosystem.
44:21
You can’t just have one giant emerge and swallow everyone up or two giants or three giants swallowing everyone up because there’s no incentive to innovate if that’s the case because you’re just splitting the pie over and over again.
And so whilst big tech tends to oppose lots of interventions which are aimed at kind of breaking things up and levelling the playing field on the grounds that this is senseless because they have the resources to innovate and others don’t.
44:49
In actual fact, we could get much better innovation and much better products if that were the case.
And a recent example that comes to mind.
So we didn’t touch too much on age assurance, but age assurance is obviously a key as part of the policies that are developing in this space generally to say, well, how can we prove that, you know, teens under 13 aren’t actually accessing social media or aren’t actually accessing replica, whatever it may be.
45:13
And there have been and there are lots of interesting companies trying to develop encrypted, secure, privacy preserving ways of verifying your identity.
And just the other day on the FT, there was announcement that Apple is going to trial in the UK age assurance for iPhone users.
45:33
Now, it might be very noble of them, but you can see how someone might be concerned that that all of a sudden Apple is the one verifying how old you are on an iPhone where it basically has control of the, you know, like it’s it’s again, like, yeah, a little.
45:56
Sketchy you.
Know, and so I think had we instead have or if we have a level playing field there, you know, it allows different third party providers to come up with more privacy preserving ways of doing this.
And also where their incentives are then aligned with that mission.
46:13
You know, like if you are meta, your incentive is 1 incentive only, which is, you know, get more data, make more profit.
Another company might really not have the same incentive because they might actually ask you to pay 1 cost and that’s it.
That’s their revenue model and that’s fine.
46:30
They don’t need to sell you anything else.
So Long story short to say, competition law, antitrust law is really important to level the playing field and actually foster innovation.
And you can have really interesting regulatory routes like sandboxes, which do exist.
They existed in kind of financial services, they exist in kind of the AI context as well, which are all aimed at getting regulators to engage with companies, working with emergent technologies to figure out, okay, you have this issue or you have this product.
46:59
How do we think this is going to be regulated?
How can we work together?
So it can be like a really fertile ground.
It’s just that the narrative needs to change.
I really appreciate that distinction, really revealing that this is a false dichotomy that you can regulate and also increase innovation or support it and kind of an indifferent realm.
47:18
Recently we’ve seen an emergence of using AI in law.
And you mentioned this earlier in our call talking about someone who’s using Chachi BT for legal advice.
You know, we’ve seen some instances where AI is used by lawyers in the courtroom to create AI generated videos for the judge that the judge then says, what the hell is this?
47:38
Get out of my face.
But then there’s also a lot of talk about how AI might change the practice of law.
Thinking back to our episode with Arvind Narayan talking about the introduction of ediscovery in law several years back, where everyone thought that, oh, now that I can search for things through a database instead of manually having to file through different cases, I can do better at my job being a lawyer.
48:03
I can, you know, gather more relevant cases and also do that in less time.
But all that did was push lawyers to do more work because they’re expected to bring more things to the table via having an accessible search mechanism.
So how do you foresee AI impacting law and what might it mean about your own role?
48:23
Is there anything that you’re worried about or seeing trends or seeing patterns or what’s been going on?
Yes, No, Daphne, when I heard Arvind’s conversation with you both, what it reminded me of is there’s this clip from when e-mail first started, so like late 90s, early 2000s with academic researchers complaining that they were really e-mail fatigued because they had 10 emails in their inbox and you’re like 2026.
48:46
Like wow if only 10 emails and 9:00.
Emails spamming my inbox, which funny enough, Arvind literally just posted about that on LinkedIn today.
Amazing.
So that was my reflection.
But that really applies to law as well in the sense that it’s a never ending feed because as humans and as businesses, we continue to engage in e-mail and digital communication and there’s more and more files as a result of like just ways of working.
49:14
Which then means that when you come to like a case, typically the lawyer that wants the documents will argue for the most documents they can possibly obtain usually.
But then that involves trolling through them.
And so, yes, if you now have, and increasingly you have AI tools that help you sort through the documents for the relevant ones, the more you have those tools, the more you might be likely to get further documents.
49:36
So that’s one in practice.
But the two aspects that concern me more in terms of the use of AI is 1.
The pollution of evidence in the sense that because you can now create, you know, fake emails, fake documents, etcetera, etcetera, it’s challenging to be able to identify these.
49:56
And obviously there are systems through which you can figure out, well, the metadata doesn’t really match or there’s something off about it, the font wasn’t available or whatever it might be.
But not everyone’s going to have the resources to be able to do this.
And you can imagine, you know, local county courts maybe, or situations where someone is funding or representing themselves being faced with fake documents, like the allegation that something is fake is unfortunately one way or the other, essentially is unfortunately going to pollute the system.
50:28
And then the second aspect, which concerns me is related to the idea of the case in the US that we saw where a victim’s family created a deep fake of the victim to basically speak to the court to explain the impact that the person who had, I think, run them over had had on their life.
50:46
And the judge was actually impressed by that simulation.
And that to me is quite scary because so in the US, obviously a lot of cases are then decided with a jury.
And there’s been decades and decades of like honing in your persuasive skills before the jury, you know, like, what do you wear?
51:04
How do you say particular things?
How do you present like a simulation of a model that shows your version of the events most convincingly?
And all of this is of course like a tactic, but when you add like a deep fake to it and there is a more immersive version of seeing someones projection and what happened, it can be really dangerous in terms of influencing jury members.
51:30
So that’s something that really concerns me.
And I think, you know, courts in the UK, for example, are trying to say you shouldn’t be using AI.
You know, if someone writes a witness statement using AI, that shouldn’t be allowed at all because a witness must write the statement, you know, in their own words and like recall things and you can’t comply with in your own words if you’re using an AI tool.
51:53
But again, it’s going to be difficult to police this if you are working like on an individual level.
So yes, it complicates things, and it’s an interesting space to be.
Giulia, just to bring our time with you to a close because we really enjoyed our chat, but I know that there was one last question we really wanted to ask you, and that was the magic wand question.
52:16
If you could write One Law tomorrow with your magic wand on AI companions, what would it be?
Easy 1.
Easy.
Just an easy question.
One last question.
On AI compa, it’s it’s, yeah, I think my answer is unfortunately a very lawyerly answer, which is we we can’t have one law to rule them all.
52:39
It’s effectively, if you think that there’s only one, then you haven’t solved the issue.
And I think that’s where I stand with it.
I think essentially we need to bolster rights that we already have and see how and to what extent they apply to AI companions.
We then need to see what gaps remain after that has happened.
52:57
And we need to have bold vision at kind of enforcer level and investigative level to pursue particular claims.
And that should be layered with more investment into pro social things in real life, you know, like community centers, sports, investment in these third spaces because you can’t have third spaces just being third space, the gym, the very expensive 1.
53:21
You know, like you actually need that kind of pro social connectivity that governments do need to invest in as opposed to just saying, we’re going to give you an AI literacy course via YouTube videos and we’ve solved the problem.
You’re now all fine.
So yeah, that would be my combination of measures.
53:40
I really like that.
And we have definitely advocated for, you know, investing more in the things that we’re wanting to replace AI companions, right?
Why do people seek out AI companions?
Because they have high social needs, high social anxiety, loneliness, etcetera, but don’t have the accessibility of infrastructure to be able to bridge that gap and make those connections.
54:01
So yes, please, let’s invest more in 3rd spaces that really bring people together rather than just saying, oh, well, easy access chatbot, here you go.
That’s unregulated.
Why not?
Giulia, it’s been great to have you on today.
And I really hope our viewers listen in to figure out what it is that we need to do about AI companions moving forward.
54:20
Maybe get inspired by your story and join the fight in the antitrust lawyer space to litigate against Big Tech.
And also to help people realize the complexity because there is no one solution.
Agreed.
So thank you all for listening and thank you so much, Giulia, for joining us today.
54:35
This is bye from Rose.
Bye from Angie.