WEBVTT

cb4975bd-b5c4-432d-a6bb-e2ce23c9ac4e-0
00:00:04.040 --> 00:00:06.157
Well,
now I'd like to introduce our last guest

cb4975bd-b5c4-432d-a6bb-e2ce23c9ac4e-1
00:00:06.157 --> 00:00:07.600
of the day, Daniel Huntenlocker.

18f15136-b9e3-4a0f-b91c-83c5d342d637-0
00:00:07.600 --> 00:00:11.800
Dan is the inaugural Dean of the MIT
Schwartzman College of Computing.

ddd0bf4b-fecb-4aab-8c7c-6f68bf0544ac-0
00:00:12.320 --> 00:00:16.338
He's one of the more influential thinkers
out there when it comes to AI and is,

ddd0bf4b-fecb-4aab-8c7c-6f68bf0544ac-1
00:00:16.338 --> 00:00:19.954
among very many other things,
the author of The Age of AI and our Human

ddd0bf4b-fecb-4aab-8c7c-6f68bf0544ac-2
00:00:19.954 --> 00:00:22.466
Future,
which is co-authored with Henry Kissinger

ddd0bf4b-fecb-4aab-8c7c-6f68bf0544ac-3
00:00:22.466 --> 00:00:23.320
and Eric Schmidt.

0cfb1ace-4e8e-4435-a154-a801f3ee9e37-0
00:00:23.680 --> 00:00:25.040
Dan, welcome to M Tech AI.

b8822c54-c3ae-4f49-b1fb-8bcd8efc21f2-0
00:00:25.040 --> 00:00:25.560
Please join me.

e6e480ec-91c7-4354-a7fb-2cddf9b619dd-0
00:00:30.960 --> 00:00:31.360
Have a seat.

3d311366-9272-4c79-b12b-389af83f1308-0
00:00:31.400 --> 00:00:31.680
Have a seat.

c4e79864-62de-4885-905c-0967946dd75b-0
00:00:33.280 --> 00:00:33.960
Thanks for being here.

017ad8b9-9550-4dd4-b92a-0f73d5c215eb-0
00:00:34.160 --> 00:00:35.040
Hey, great to be here.

b7142bee-5e17-4f37-81d2-e5da107e28b6-0
00:00:35.120 --> 00:00:35.480
Thanks.

71cd62a9-b973-41c8-838b-741f08ec53cc-0
00:00:37.080 --> 00:00:40.800
Your book came out five years ago,
five years ago in November.

f0f6196e-1f45-4b0b-9b24-7b17c87ec7fe-0
00:00:40.800 --> 00:00:42.440
But yes, nothing's happened since then.

1c9c2360-1ee8-4df2-bdea-5e876d0506ce-0
00:00:42.440 --> 00:00:43.440
No, nothing at all.

f7f6651b-35d8-4291-8646-32774f7cae18-0
00:00:44.040 --> 00:00:47.426
I'm curious, like if so, if, if you,
as you go back and you think about the

f7f6651b-35d8-4291-8646-32774f7cae18-1
00:00:47.426 --> 00:00:50.189
process of writing that book and the
things that you learned,

f7f6651b-35d8-4291-8646-32774f7cae18-2
00:00:50.189 --> 00:00:53.442
if you were to write it today,
if you were to go back or maybe to update

f7f6651b-35d8-4291-8646-32774f7cae18-3
00:00:53.442 --> 00:00:55.760
it today, what would you,
what would you add to it?

d90333f3-2c5d-4c8e-9e5e-19d1610fe72e-0
00:00:55.800 --> 00:00:57.280
What would you,
what would you change about it?

f61c1cae-581a-4712-876a-06489b21151a-0
00:00:58.960 --> 00:01:00.920
Actually, I'd change very little.

1574e311-3aee-41a9-a621-a15105dc4018-0
00:01:02.840 --> 00:01:08.430
I I think that the most important thing
to add would be all the examples we have

1574e311-3aee-41a9-a621-a15105dc4018-1
00:01:08.430 --> 00:01:12.640
today of the kinds of issues that we were
identifying there.

37ee952f-3278-4d2a-a860-82a99de955cd-0
00:01:13.440 --> 00:01:19.326
Of the fact that AI is really this
different form of intelligence from human

37ee952f-3278-4d2a-a860-82a99de955cd-1
00:01:19.326 --> 00:01:20.320
intelligence.

be51fbc4-42bc-4b5f-9bce-992f67ae2779-0
00:01:21.000 --> 00:01:25.195
But because the only form of intelligence
we're used to dealing with,

be51fbc4-42bc-4b5f-9bce-992f67ae2779-1
00:01:25.195 --> 00:01:29.330
at least one that speaks to us and writes
in our language, etcetera,

be51fbc4-42bc-4b5f-9bce-992f67ae2779-2
00:01:29.330 --> 00:01:34.064
is human intelligence going to be very
challenging for us to figure out how to

be51fbc4-42bc-4b5f-9bce-992f67ae2779-3
00:01:34.064 --> 00:01:37.960
really work most effectively with this
new form of intelligence.

d35b7c2a-c468-4d99-af1b-51262e8ec591-0
00:01:37.960 --> 00:01:41.320
And I think we're seeing that all over
the place today.

4097bee2-4b6c-4e55-8663-054e18579133-0
00:01:42.040 --> 00:01:43.680
There are many, many more examples.

018380d7-4b74-4898-ae5c-81a9b0570447-0
00:01:43.680 --> 00:01:47.619
You know,
at the time that we wrote the book,

018380d7-4b74-4898-ae5c-81a9b0570447-1
00:01:47.619 --> 00:01:49.760
there was no ChatGPT yet.

ca5bdef6-e3f4-41b8-9884-9bafd5ec700b-0
00:01:49.760 --> 00:01:53.027
It was GPT 3 and then we did in the
paperback,

ca5bdef6-e3f4-41b8-9884-9bafd5ec700b-1
00:01:53.027 --> 00:01:58.380
which came out about a year later,
we had a forward that talked a little bit

ca5bdef6-e3f4-41b8-9884-9bafd5ec700b-2
00:01:58.380 --> 00:02:02.203
about ChatGPT,
but that's just sort of for setting the

ca5bdef6-e3f4-41b8-9884-9bafd5ec700b-3
00:02:02.203 --> 00:02:02.760
context.

de8d13d9-8bef-47d0-b970-70d9991186ac-0
00:02:03.520 --> 00:02:09.621
So a lot has happened now that, you know,
it's hard not to view AI as having human

de8d13d9-8bef-47d0-b970-70d9991186ac-1
00:02:09.621 --> 00:02:13.958
like intelligence,
but it's very different in ways that we

de8d13d9-8bef-47d0-b970-70d9991186ac-2
00:02:13.958 --> 00:02:19.692
don't understand at the moment and that
are very important to making the most

de8d13d9-8bef-47d0-b970-70d9991186ac-3
00:02:19.692 --> 00:02:25.720
effective use of it in terms of benefits
and also in terms of avoiding downfalls.

ea964088-204e-44ef-b360-6d65010138f2-0
00:02:26.920 --> 00:02:30.849
I'm reading a book about an octopus right
now, which is very interesting,

ea964088-204e-44ef-b360-6d65010138f2-1
00:02:30.849 --> 00:02:33.080
but also a different form of intelligence.

c2f56383-70d1-4c70-9a7f-1841eb58f15e-0
00:02:33.360 --> 00:02:36.536
But I'm curious what you mean by that
when you say it's a different form of

c2f56383-70d1-4c70-9a7f-1841eb58f15e-1
00:02:36.536 --> 00:02:37.080
intelligence.

8d1b7c7c-61d6-4cd7-a656-9595feec91e2-0
00:02:37.080 --> 00:02:38.440
Like how so?

2ab8f3d6-362b-4717-ae28-dc1d9aaed96c-0
00:02:39.640 --> 00:02:44.419
So when you're used to dealing with a
human, you expect, you know,

2ab8f3d6-362b-4717-ae28-dc1d9aaed96c-1
00:02:44.419 --> 00:02:48.699
somebody who has hopefully,
although some of us are amoral,

2ab8f3d6-362b-4717-ae28-dc1d9aaed96c-2
00:02:48.699 --> 00:02:53.050
but somebody who has, you know,
some deep sense of morality,

2ab8f3d6-362b-4717-ae28-dc1d9aaed96c-3
00:02:53.050 --> 00:02:56.759
has judgement that's come through human
experience.

7cdde9a3-0087-4129-a0bd-49c883fefc5d-0
00:02:56.760 --> 00:03:01.991
Various sorts of things that when you
think about the ways that machine

7cdde9a3-0087-4129-a0bd-49c883fefc5d-1
00:03:01.991 --> 00:03:05.696
learning systems train,
and I think it's inherent,

7cdde9a3-0087-4129-a0bd-49c883fefc5d-2
00:03:05.696 --> 00:03:07.440
it's not today's models.

cb1fe3ba-444b-49e8-b7cc-f22fdcf47350-0
00:03:07.440 --> 00:03:09.760
These are things that will be with us
forever.

06899d15-a9b3-439d-90fb-e685932f5764-0
00:03:10.800 --> 00:03:15.640
You just end up with different forms of
ability to reason.

7e8af385-b84e-4935-9064-2316034c8ae8-0
00:03:18.080 --> 00:03:23.794
These things can do a very good job of a
peering to follow moral judgement,

7e8af385-b84e-4935-9064-2316034c8ae8-1
00:03:23.794 --> 00:03:29.734
but the difference between appearance and
reality becomes extremely important,

7e8af385-b84e-4935-9064-2316034c8ae8-2
00:03:29.734 --> 00:03:33.720
for example,
in any high exigency kind of situation.

ed00bdee-7729-4374-bf3d-64782d2aad5d-0
00:03:34.160 --> 00:03:37.757
But it's often important in the
day-to-day, you know,

ed00bdee-7729-4374-bf3d-64782d2aad5d-1
00:03:37.757 --> 00:03:42.154
and I think we see it in everything from
the ways that, you know,

ed00bdee-7729-4374-bf3d-64782d2aad5d-2
00:03:42.154 --> 00:03:46.550
the medical profession,
which is fairly conservative about making

ed00bdee-7729-4374-bf3d-64782d2aad5d-3
00:03:46.550 --> 00:03:51.480
change, is being fairly slowly,
being fairly slow to adopt AI in clinical

ed00bdee-7729-4374-bf3d-64782d2aad5d-4
00:03:51.480 --> 00:03:52.079
settings.

fa955f8e-e7c0-4630-94fc-41c8e7f1d973-0
00:03:52.080 --> 00:03:55.413
Although is really starting to now to,
you know,

fa955f8e-e7c0-4630-94fc-41c8e7f1d973-1
00:03:55.413 --> 00:04:00.446
areas like software engineering where
it's completely transformed the way

fa955f8e-e7c0-4630-94fc-41c8e7f1d973-2
00:04:00.446 --> 00:04:03.440
people code,
the way programming gets done.

18f0e72a-5d04-444a-b52e-03281796b64f-0
00:04:04.120 --> 00:04:08.923
But it's still certainly apparent to me,
and I think to many others,

18f0e72a-5d04-444a-b52e-03281796b64f-1
00:04:08.923 --> 00:04:14.213
that this is not replacing the importance
of human judgement and developing

18f0e72a-5d04-444a-b52e-03281796b64f-2
00:04:14.213 --> 00:04:14.840
software.

5ad764c1-d836-4e2b-8ee1-e4df80cfca6f-0
00:04:15.320 --> 00:04:17.440
It's fundamentally changing that role.

ebfa9007-a5e8-45d2-beef-6d054ead7059-0
00:04:17.440 --> 00:04:21.201
But if anything,
I would say that human reason and human

ebfa9007-a5e8-45d2-beef-6d054ead7059-1
00:04:21.201 --> 00:04:26.150
intelligence is more important today in
the age of AI than it's ever been,

ebfa9007-a5e8-45d2-beef-6d054ead7059-2
00:04:26.150 --> 00:04:29.119
rather than being sort of displaced by
that.

9c0bfd1c-14cb-45c8-89b4-73dce1e89195-0
00:04:29.120 --> 00:04:32.760
And yet the narrative,
the norm is all about displacement.

2f8d0b91-5adb-45bd-8dbb-8d661f1c5b4b-0
00:04:33.040 --> 00:04:35.000
And why, why, why, why is that?

7c145243-2bc4-4ea7-8691-852581477ff1-0
00:04:35.000 --> 00:04:36.814
Like,
what is it so important about human

7c145243-2bc4-4ea7-8691-852581477ff1-1
00:04:36.814 --> 00:04:39.320
intelligence that we haven't been able to
do with AI yet?

938b4972-59ab-482d-a8b6-ead2f1e7cff0-0
00:04:41.280 --> 00:04:43.168
Well,
I think some of this becomes a

938b4972-59ab-482d-a8b6-ead2f1e7cff0-1
00:04:43.168 --> 00:04:44.240
philosophical debate.

d6ff5383-516d-4085-879b-df7b293776fb-0
00:04:45.200 --> 00:04:49.510
And so, you know, gets hard for,
I try to tie it to sort of real

d6ff5383-516d-4085-879b-df7b293776fb-1
00:04:49.510 --> 00:04:50.240
experience.

5610bbe4-7818-4257-b8b6-7a49d9a0c9d4-0
00:04:50.240 --> 00:04:54.160
But then, and in fact,
the the book with Kissinger and Schmidt

5610bbe4-7818-4257-b8b6-7a49d9a0c9d4-1
00:04:54.160 --> 00:04:58.640
had a lot of philosophical elements to it
at the time that we wrote it.

01648c3f-7c58-4b5d-a244-a573b18ea585-0
00:05:00.000 --> 00:05:04.452
But I think many of us believe that
living in a simulation of reality and

01648c3f-7c58-4b5d-a244-a573b18ea585-1
00:05:04.452 --> 00:05:06.920
living in reality are not the same thing.

16d81b20-792a-48c5-91fe-6ef424a38624-0
00:05:08.520 --> 00:05:12.680
And so it's really down to that same kind
of question.

e32024f2-9195-4e6a-bc5d-af6b427fd541-0
00:05:13.080 --> 00:05:18.173
What is it about, you know,
human decision making, human reason,

e32024f2-9195-4e6a-bc5d-af6b427fd541-1
00:05:18.173 --> 00:05:22.169
you know,
knowledge of the sort of consequences of

e32024f2-9195-4e6a-bc5d-af6b427fd541-2
00:05:22.169 --> 00:05:27.262
your decisions for yourself,
for your loved ones, for, you know,

e32024f2-9195-4e6a-bc5d-af6b427fd541-3
00:05:27.262 --> 00:05:31.650
for your coworkers,
all other humans with whom you have

e32024f2-9195-4e6a-bc5d-af6b427fd541-4
00:05:31.650 --> 00:05:34.080
various kinds of relationships.

fa52c9ce-b234-4bf1-a038-5094a7995afa-0
00:05:35.040 --> 00:05:37.730
You know, how do you,
how does your decision making,

fa52c9ce-b234-4bf1-a038-5094a7995afa-1
00:05:37.730 --> 00:05:41.080
how does your thinking affect that whole
network of other people?

787a64d8-41a7-44c2-8651-af614875b084-0
00:05:41.720 --> 00:05:45.086
For most of us,
that deeply effects how we operate and

787a64d8-41a7-44c2-8651-af614875b084-1
00:05:45.086 --> 00:05:45.760
what we do.

211a0f28-c23c-44ee-b120-79d78860d967-0
00:05:46.960 --> 00:05:51.680
And those things are absent from non
human actors.

6f5b2950-0653-4883-bc99-8e4529282542-0
00:05:52.240 --> 00:05:56.040
That doesn't mean that non human actors
aren't extremely powerful.

8e0eed57-2555-4419-851a-ca8e8f52b3bb-0
00:05:56.040 --> 00:06:00.223
It's just that trying to shove us in the
box of a non human actor or shove a non

8e0eed57-2555-4419-851a-ca8e8f52b3bb-1
00:06:00.223 --> 00:06:04.252
human actor in the box of a human actor,
which is what you're doing if you're

8e0eed57-2555-4419-851a-ca8e8f52b3bb-2
00:06:04.252 --> 00:06:07.920
trying to say, well, these these,
you know, these are the same things.

c1b6fbe9-0c50-486b-9bed-bd4f8ff6f0cd-0
00:06:07.920 --> 00:06:13.034
They're completely interchangeable,
just sort of fundamentally flawed as an

c1b6fbe9-0c50-486b-9bed-bd4f8ff6f0cd-1
00:06:13.034 --> 00:06:13.640
approach.

2b6d3c61-7b31-4a57-99f8-0c81e0984e29-0
00:06:13.880 --> 00:06:15.960
That seems to me like that doesn't stop
us from doing it.

92ad1c29-cdcd-409a-88ac-04eeabb82412-0
00:06:15.960 --> 00:06:16.360
It doesn't.

b9698418-8e55-47d8-9dc5-e2fea2ea4859-0
00:06:16.360 --> 00:06:17.000
No, it doesn't.

f0514d93-67ac-4014-856e-bc74c5b7484c-0
00:06:17.000 --> 00:06:20.127
I think certainly that's a mistake you
see people making is trying to shove it

f0514d93-67ac-4014-856e-bc74c5b7484c-1
00:06:20.127 --> 00:06:20.880
in the slot, right?

56865aaf-0b13-4c51-b007-3d62e4018f79-0
00:06:21.360 --> 00:06:26.469
But that's that also seems to me that
maybe what you're saying here is that

56865aaf-0b13-4c51-b007-3d62e4018f79-1
00:06:26.469 --> 00:06:31.848
some of the concerns that people have
about what about the effects of AI in the

56865aaf-0b13-4c51-b007-3d62e4018f79-2
00:06:31.848 --> 00:06:35.815
workplace, for example,
or just on society, are those are,

56865aaf-0b13-4c51-b007-3d62e4018f79-3
00:06:35.815 --> 00:06:37.160
are those overblown?

4fb3a28c-78d1-4d19-900b-1e6cca4dca4e-0
00:06:37.160 --> 00:06:40.000
Are those are those are those too much?

9f27da14-25cd-4e49-ac1a-1edeb8ae34e5-0
00:06:40.000 --> 00:06:42.927
Because it is,
it does turn out to be difficult to shove

9f27da14-25cd-4e49-ac1a-1edeb8ae34e5-1
00:06:42.927 --> 00:06:44.160
an AI into a human slot.

dc2df656-2ef0-451f-a77d-9796b40f5ae5-0
00:06:44.160 --> 00:06:47.040
It's, it's, it's a really good question.

c515ff02-2c79-4dc3-8358-d61a0f5d53f7-0
00:06:47.040 --> 00:06:51.255
So I mean, the, the,
the way I think about it is that AI is

c515ff02-2c79-4dc3-8358-d61a0f5d53f7-1
00:06:51.255 --> 00:06:52.520
such a big change.

552286fa-9ef6-4a17-bd7e-a999d22d24d7-0
00:06:52.520 --> 00:06:55.640
You know, I mean,
human reason was the only form of reason.

7c883170-17c6-44ab-91b9-ba8e5209be82-0
00:06:55.640 --> 00:06:57.724
I mean,
sometimes we would think our pets were

7c883170-17c6-44ab-91b9-ba8e5209be82-1
00:06:57.724 --> 00:06:59.719
reasoning, but come on, you know,
like they,

7c883170-17c6-44ab-91b9-ba8e5209be82-2
00:06:59.719 --> 00:07:01.360
they certainly weren't talking to us.

c821ea44-1112-4c0a-8a12-86f7a12cd2e9-0
00:07:01.360 --> 00:07:02.960
And let's talk about hallucination.

797e4c10-220c-4ef0-afcf-b3620ee640ae-0
00:07:02.960 --> 00:07:08.458
We were really hallucinating the but,
but with with AI it, it,

797e4c10-220c-4ef0-afcf-b3620ee640ae-1
00:07:08.458 --> 00:07:10.640
it feels very human like.

9a490498-4475-4427-a0fb-7a70b8488f8a-0
00:07:11.240 --> 00:07:13.680
And that is a really fundamental change.

b64ea74e-d931-4a8f-b4f8-f26b71edcdf2-0
00:07:13.680 --> 00:07:18.002
We're not used to interacting and spoken
and written language with anything except

b64ea74e-d931-4a8f-b4f8-f26b71edcdf2-1
00:07:18.002 --> 00:07:18.680
other people.

470e8236-e5ad-4064-9979-1873f7ccbbce-0
00:07:20.040 --> 00:07:24.452
And, and so it, it,
we're right to have that feel

470e8236-e5ad-4064-9979-1873f7ccbbce-1
00:07:24.452 --> 00:07:25.600
disorienting.

806c00fb-a760-4062-843c-491012213ccb-0
00:07:25.600 --> 00:07:28.480
There's no other way to feel about it
than that it's disorienting.

3179f8c7-9b03-4cde-b72b-f1207ea68dc5-0
00:07:29.200 --> 00:07:33.850
But I think that the, the,
what we're attributing to it in that

3179f8c7-9b03-4cde-b72b-f1207ea68dc5-1
00:07:33.850 --> 00:07:37.484
disorientation is actually in,
in many instances,

3179f8c7-9b03-4cde-b72b-f1207ea68dc5-2
00:07:37.484 --> 00:07:43.007
causing us to worry about some things
that maybe we should be a little less

3179f8c7-9b03-4cde-b72b-f1207ea68dc5-3
00:07:43.007 --> 00:07:48.021
concerned about and maybe not as
concerned about some things that we

3179f8c7-9b03-4cde-b72b-f1207ea68dc5-4
00:07:48.021 --> 00:07:52.600
should be more concerned about because in
this different form.

42d8d067-7b0a-4d19-a863-fb189538ec4f-0
00:07:52.640 --> 00:07:54.320
And, you know,
we're seeing it all over the place.

01274f8a-de7e-431d-b949-36fa8ce99538-0
00:07:54.320 --> 00:07:57.949
So like, you know, when you look at,
you know, great software engineering,

01274f8a-de7e-431d-b949-36fa8ce99538-1
00:07:57.949 --> 00:08:01.336
software engineers and software
engineering teams that are making the

01274f8a-de7e-431d-b949-36fa8ce99538-2
00:08:01.336 --> 00:08:03.320
most effective use of these coding tools.

ef9b8a1d-ddca-4b40-8349-fab0dc582d87-0
00:08:03.680 --> 00:08:08.847
It was just in the previous discussion
you were having, you know,

ef9b8a1d-ddca-4b40-8349-fab0dc582d87-1
00:08:08.847 --> 00:08:15.424
what you really want to do is pair one or
a small number of your very best software

ef9b8a1d-ddca-4b40-8349-fab0dc582d87-2
00:08:15.424 --> 00:08:22.080
engineers with these coding tools and you
know, maybe an army of these coding tools.

82beb0af-bb7b-4d5f-8d89-c528971832ea-0
00:08:22.400 --> 00:08:26.237
But it's again,
it's about this figuring out how to bring

82beb0af-bb7b-4d5f-8d89-c528971832ea-1
00:08:26.237 --> 00:08:30.670
these two forms of intelligence,
human and machine together that's

82beb0af-bb7b-4d5f-8d89-c528971832ea-2
00:08:30.670 --> 00:08:34.640
delivering the best outcomes in big
coding projects by far.

d99961c5-01f5-4e19-827c-fc132e863f6d-0
00:08:34.640 --> 00:08:39.070
I mean, anybody I know who's, you know,
a senior, you know, VP level or,

d99961c5-01f5-4e19-827c-fc132e863f6d-1
00:08:39.070 --> 00:08:44.106
or engineering manager or even individual
coder on one of those kinds of projects,

d99961c5-01f5-4e19-827c-fc132e863f6d-2
00:08:44.106 --> 00:08:49.083
It's not like a completely green software
engineering throwing these things loose

d99961c5-01f5-4e19-827c-fc132e863f6d-3
00:08:49.083 --> 00:08:52.360
is going to develop great production
quality systems.

73853f75-c433-4f98-a166-37e2cb9aa1fe-0
00:08:52.960 --> 00:08:57.550
They can build stuff and sometimes that
stuff is very interesting proof of

73853f75-c433-4f98-a166-37e2cb9aa1fe-1
00:08:57.550 --> 00:08:58.040
concept.

046c0bb8-56be-4d09-9637-f85861cf518e-0
00:08:59.040 --> 00:09:02.321
But you know,
anybody can write a novel today too with,

046c0bb8-56be-4d09-9637-f85861cf518e-1
00:09:02.321 --> 00:09:04.080
with, you know, generative AI.

a94a09a1-5082-423e-96ed-aa09c2db6234-0
00:09:04.080 --> 00:09:07.729
And you know,
we're used to calling that AI slop today,

a94a09a1-5082-423e-96ed-aa09c2db6234-1
00:09:07.729 --> 00:09:08.120
right?

698eb6ca-8df9-408a-993a-8e1efecdb0c3-0
00:09:08.120 --> 00:09:12.656
So it, you know, I think that to me,
part of the difference between slop and,

698eb6ca-8df9-408a-993a-8e1efecdb0c3-1
00:09:12.656 --> 00:09:15.563
you know,
really fundamental advances is the ways

698eb6ca-8df9-408a-993a-8e1efecdb0c3-2
00:09:15.563 --> 00:09:19.285
in which we're using this,
the fact that we're recognizing that

698eb6ca-8df9-408a-993a-8e1efecdb0c3-3
00:09:19.285 --> 00:09:23.415
they're different strengths and that
we're learning how to bring those

698eb6ca-8df9-408a-993a-8e1efecdb0c3-4
00:09:23.415 --> 00:09:24.519
strengths together.

d70433d6-7970-4b21-a4cd-a29724512ed9-0
00:09:24.520 --> 00:09:28.693
And those are things both about
individuals and how we're interacting

d70433d6-7970-4b21-a4cd-a29724512ed9-1
00:09:28.693 --> 00:09:32.091
with these systems,
but also how these systems are being

d70433d6-7970-4b21-a4cd-a29724512ed9-2
00:09:32.091 --> 00:09:35.906
designed and developed,
which right now it's not so well suited

d70433d6-7970-4b21-a4cd-a29724512ed9-3
00:09:35.906 --> 00:09:39.662
to sort of, you know,
bringing together two different forms of

d70433d6-7970-4b21-a4cd-a29724512ed9-4
00:09:39.662 --> 00:09:44.311
intelligence and also how, you know,
organizations and employers are bringing

d70433d6-7970-4b21-a4cd-a29724512ed9-5
00:09:44.311 --> 00:09:46.160
in these things to bear, right?

42ec6e1a-26ca-4c75-8186-50d177b36380-0
00:09:46.160 --> 00:09:51.541
So it's, it's really, you know, a, a,
a joint sort of mindset, responsibility,

42ec6e1a-26ca-4c75-8186-50d177b36380-1
00:09:51.541 --> 00:09:54.947
whatever you want to call it,
of people using AI,

42ec6e1a-26ca-4c75-8186-50d177b36380-2
00:09:54.947 --> 00:10:00.601
organizations using AI and the, you know,
those of us who are developing and, and,

42ec6e1a-26ca-4c75-8186-50d177b36380-3
00:10:00.601 --> 00:10:01.760
and deploying it.

1d72479e-0652-4f9c-838b-6899d423bdda-0
00:10:02.520 --> 00:10:05.359
I don't know how much of our previous
converse of the previous conversation

1d72479e-0652-4f9c-838b-6899d423bdda-1
00:10:05.359 --> 00:10:06.480
you're able to hear backstage.

c5378cf1-ab14-4fff-87c6-e4fd5b9c8d62-0
00:10:06.480 --> 00:10:09.483
But one of the things that I was talking
about is a friend of mine who's ACTO at a

c5378cf1-ab14-4fff-87c6-e4fd5b9c8d62-1
00:10:09.483 --> 00:10:12.196
company who had said to me something
along the lines of I'm glad I'm not a

c5378cf1-ab14-4fff-87c6-e4fd5b9c8d62-2
00:10:12.196 --> 00:10:14.657
young software engineer today,
Glad I'm not an entry level software

c5378cf1-ab14-4fff-87c6-e4fd5b9c8d62-3
00:10:14.657 --> 00:10:15.199
engineer today.

d3eb06bd-3fae-43e9-881b-b1b7a3066265-0
00:10:15.320 --> 00:10:17.778
There are going to be short term
challenges that people face,

d3eb06bd-3fae-43e9-881b-b1b7a3066265-1
00:10:17.778 --> 00:10:19.880
especially as they're trying to enter the
workplace.

5589a02a-9960-4aa8-9e24-b1af1713e089-0
00:10:19.880 --> 00:10:25.762
What are your thoughts on on how you know
someone who is graduating just just out

5589a02a-9960-4aa8-9e24-b1af1713e089-1
00:10:25.762 --> 00:10:26.480
of school?

21d32262-60d1-44e4-8f80-a0b482c3bbbb-0
00:10:26.480 --> 00:10:28.440
How should they think about working with
AI?

f5e93cca-c6fb-4833-954a-5ddabaf5381b-0
00:10:28.680 --> 00:10:31.645
Yeah,
this is a questions how to tease this

f5e93cca-c6fb-4833-954a-5ddabaf5381b-1
00:10:31.645 --> 00:10:36.633
question apart in a good way in a short
time because it's a very involved

f5e93cca-c6fb-4833-954a-5ddabaf5381b-2
00:10:36.633 --> 00:10:37.240
question.

f1e7976b-72d9-484d-866b-afa78c5dfb1d-0
00:10:40.480 --> 00:10:47.007
You know, I think MIT in schools like us,
our students often used to complain in

f1e7976b-72d9-484d-866b-afa78c5dfb1d-1
00:10:47.007 --> 00:10:51.440
computer science program and computer
science classes.

eb3546a8-70ae-4a60-950c-35988cc6a449-0
00:10:52.680 --> 00:10:55.483
You know,
you're not teaching us enough about the

eb3546a8-70ae-4a60-950c-35988cc6a449-1
00:10:55.483 --> 00:11:00.080
practical things so we can run out and be
a, you know, productive coder tomorrow.

c48ec6f3-4eb2-4c17-9e85-1647d5a0a358-0
00:11:00.520 --> 00:11:05.380
And we said correct,
because we're trying to teach you how to

c48ec6f3-4eb2-4c17-9e85-1647d5a0a358-1
00:11:05.380 --> 00:11:11.338
think computationally for an entire
career where the ways that these things

c48ec6f3-4eb2-4c17-9e85-1647d5a0a358-2
00:11:11.338 --> 00:11:17.531
develop are going to change multiple
times and going to accelerate during your

c48ec6f3-4eb2-4c17-9e85-1647d5a0a358-3
00:11:17.531 --> 00:11:18.079
career.

fd10361e-c998-4f8c-b104-c6e1cafce04e-0
00:11:18.800 --> 00:11:24.562
This is probably the biggest proof point
in my career of that kind of a change

fd10361e-c998-4f8c-b104-c6e1cafce04e-1
00:11:24.562 --> 00:11:29.595
where you go from, you know,
coding tools that help you develop more

fd10361e-c998-4f8c-b104-c6e1cafce04e-2
00:11:29.595 --> 00:11:35.211
code to coding tools help you develop
less code but have more code produced,

fd10361e-c998-4f8c-b104-c6e1cafce04e-3
00:11:35.211 --> 00:11:38.640
which is a very different take on the
problem.

9e8248b1-2a79-4763-bb50-2732f7ab6ed8-0
00:11:39.200 --> 00:11:45.274
So I think the the the education and the
programs that have long thought about how

9e8248b1-2a79-4763-bb50-2732f7ab6ed8-1
00:11:45.274 --> 00:11:51.202
to educate people to think deeply about
the objectives of building a large scale

9e8248b1-2a79-4763-bb50-2732f7ab6ed8-2
00:11:51.202 --> 00:11:56.910
software systems and how to get those
done most effectively what we sometimes

9e8248b1-2a79-4763-bb50-2732f7ab6ed8-3
00:11:56.910 --> 00:11:58.960
call computational thinking.

e8c67a6a-72af-4e75-8f83-151016c13ea8-0
00:12:00.840 --> 00:12:04.760
I don't think that I think that those
jobs are going to get better, much better.

3378ca2b-f54b-443c-9189-077f949f73c9-0
00:12:05.000 --> 00:12:06.360
They are getting better today.

115fcacb-88a3-480b-9cf8-9402dc344ebf-0
00:12:06.360 --> 00:12:07.120
They're getting better.

b3eaa343-3690-4bec-b5f6-c45b2e44cd08-0
00:12:07.480 --> 00:12:11.555
Talk to some of those developers,
whether they're relatively junior or

b3eaa343-3690-4bec-b5f6-c45b2e44cd08-1
00:12:11.555 --> 00:12:15.458
super senior, who you know,
are now working in an environment where

b3eaa343-3690-4bec-b5f6-c45b2e44cd08-2
00:12:15.458 --> 00:12:19.361
they have the support from their
management and where the tools are

b3eaa343-3690-4bec-b5f6-c45b2e44cd08-3
00:12:19.361 --> 00:12:21.600
getting better, you know, very rapidly.

6df7a120-3087-4b8d-b2be-b80ff3a7381c-0
00:12:21.840 --> 00:12:22.760
Nobody's telling them.

b1f3b2ba-6e2b-4a2d-b421-de2467029d62-0
00:12:22.760 --> 00:12:25.640
Just set this thing loose and don't care
what it's doing right there.

e2d04e58-ec44-4afe-b0bc-df0a09677b5d-0
00:12:25.920 --> 00:12:30.120
They're stupid ways to use these tools,
those people.

e7d4eeb4-442f-4738-991c-9902c9a954be-0
00:12:30.120 --> 00:12:31.400
It's exhilarating.

743053a0-7730-485e-85b2-dee5422d499f-0
00:12:32.080 --> 00:12:35.675
You can in a, you know,
a team of 6 good software engineers in a

743053a0-7730-485e-85b2-dee5422d499f-1
00:12:35.675 --> 00:12:38.497
few weeks,
do something now that might have taken,

743053a0-7730-485e-85b2-dee5422d499f-2
00:12:38.497 --> 00:12:42.148
you know, a team of 50,
which involves a lot more of those boring

743053a0-7730-485e-85b2-dee5422d499f-3
00:12:42.148 --> 00:12:45.080
coordination meetings that software
developers hate.

7d835ab6-97c0-48cc-bbc2-2f66a7e43d50-0
00:12:46.120 --> 00:12:50.000
Might have taken a team of 56 months a
year.

57ac135f-c123-48a3-84ff-3ee06ca125ba-0
00:12:51.640 --> 00:12:55.821
But you're not going to get that by not
understanding the different roles of

57ac135f-c123-48a3-84ff-3ee06ca125ba-1
00:12:55.821 --> 00:13:00.002
somebody who really understands how to
think at the big systems architecture

57ac135f-c123-48a3-84ff-3ee06ca125ba-2
00:13:00.002 --> 00:13:02.771
kind of level of a good season software
developer,

57ac135f-c123-48a3-84ff-3ee06ca125ba-3
00:13:02.771 --> 00:13:06.790
someone who understands broader
computational thinking and what I'll call

57ac135f-c123-48a3-84ff-3ee06ca125ba-4
00:13:06.790 --> 00:13:09.885
a code jockey,
somebody who's good at producing a lot of

57ac135f-c123-48a3-84ff-3ee06ca125ba-5
00:13:09.885 --> 00:13:11.080
high performance code.

bd32315f-771f-4b2f-867a-24bf9d91886e-0
00:13:11.600 --> 00:13:12.800
Those are different things.

584f687e-c480-4c31-9d73-6993796ec227-0
00:13:12.800 --> 00:13:19.280
The latter 1 is much more subject to what
some of these tools today are able to do.

fcf61e67-0078-4726-bfe9-89cc2d69af80-0
00:13:19.880 --> 00:13:26.149
So I think, you know,
there are a number of people who trained

fcf61e67-0078-4726-bfe9-89cc2d69af80-1
00:13:26.149 --> 00:13:28.040
more to write code.

be89926e-69ff-4ef0-af19-640d75a7b30b-0
00:13:29.280 --> 00:13:33.160
That's,
that's probably a less good career path.

35a222f7-4116-4dcf-aa4d-352d6a3bb82a-0
00:13:33.960 --> 00:13:36.960
There are people who really trained and
have the experience.

16a3ee8d-1557-4028-aa25-7513581fa211-0
00:13:36.960 --> 00:13:39.600
And this doesn't just mean, you know,
great university education.

62bc327d-23ee-465f-853a-3f5317019a05-0
00:13:39.600 --> 00:13:43.522
I know people who don't even have college
degrees who are unbelievable senior

62bc327d-23ee-465f-853a-3f5317019a05-1
00:13:43.522 --> 00:13:44.880
software architects, right?

85381992-f059-464d-927d-abb12ba2e0a4-0
00:13:44.880 --> 00:13:47.085
This,
this comes from some mixture of

85381992-f059-464d-927d-abb12ba2e0a4-1
00:13:47.085 --> 00:13:49.000
experience and the right mindset.

d3b852a5-9df1-401e-a8d4-9a8b291eb38a-0
00:13:49.320 --> 00:13:52.266
If you don't have the education,
and even if you do have the education,

d3b852a5-9df1-401e-a8d4-9a8b291eb38a-1
00:13:52.266 --> 00:13:54.600
you didn't have the right mindset,
it's still a problem.

dad8d762-4feb-4086-b772-92ff21ba8374-0
00:13:55.760 --> 00:13:57.440
This is the golden age of that.

d65d4d10-0173-4d88-b795-bcbe818ac3c3-0
00:13:58.480 --> 00:14:00.880
So there's a big bifurcation happening
there.

80bf3a93-3dd7-4530-bc69-6067f3e40c84-0
00:14:00.880 --> 00:14:03.654
And I, I think frankly,
it's going to happen everywhere that

80bf3a93-3dd7-4530-bc69-6067f3e40c84-1
00:14:03.654 --> 00:14:05.520
those I, I really appreciate that answer.

430920bb-61c5-4ba6-ae0c-dbc34aeaf708-0
00:14:05.520 --> 00:14:06.160
That was very thoughtful.

8c17b5e4-f4b5-4edb-bb47-622d4216b744-0
00:14:06.160 --> 00:14:07.760
I'm starting to get some questions.

bc77587e-1936-4ad4-896f-f385c374f5d3-0
00:14:07.760 --> 00:14:09.040
The chat that was a long answer.

7d59e6ad-0bc3-444c-b985-4f20b160cc5c-0
00:14:09.040 --> 00:14:09.720
It was a long answer.

281936d7-f389-42bd-9e3b-408f0f3c5303-0
00:14:09.720 --> 00:14:10.520
It was very interesting.

1daf5c5f-e68f-4be5-be34-862744b3e5e6-0
00:14:10.520 --> 00:14:10.880
Thank you.

b628909b-4f79-4632-aabb-f3f12fc892fa-0
00:14:12.240 --> 00:14:14.562
Starting with some questions to remind,
if you do have questions,

b628909b-4f79-4632-aabb-f3f12fc892fa-1
00:14:14.562 --> 00:14:17.237
you can come to the microphones in the
room or you can ask them on the chat

b628909b-4f79-4632-aabb-f3f12fc892fa-2
00:14:17.237 --> 00:14:18.680
online and we'll, we'll get to them here.

7582440e-fcd0-4ca5-bd03-4775c2f2aa36-0
00:14:19.960 --> 00:14:23.325
You, you mentioned that, you know,
maybe we're thinking about the wrong

7582440e-fcd0-4ca5-bd03-4775c2f2aa36-1
00:14:23.325 --> 00:14:23.840
challenges.

dd7e04fe-d270-4b6c-aeeb-0b09aff61e5a-0
00:14:25.160 --> 00:14:28.089
What, what are the,
what are the challenges we should be

dd7e04fe-d270-4b6c-aeeb-0b09aff61e5a-1
00:14:28.089 --> 00:14:31.379
thinking about in terms of AI adoption as
it becomes, you know,

dd7e04fe-d270-4b6c-aeeb-0b09aff61e5a-2
00:14:31.379 --> 00:14:35.594
as it becomes more fully integrated into
society in the ways that we're seeing it

dd7e04fe-d270-4b6c-aeeb-0b09aff61e5a-3
00:14:35.594 --> 00:14:36.520
happening rapidly?

fbd66219-2293-4fc1-983d-07f3c2d4570d-0
00:14:37.240 --> 00:14:37.440
Yeah.

1b34a9a1-64bd-4508-84ae-5173054f0eb0-0
00:14:37.440 --> 00:14:42.105
So there's one that's sort of at the
center of not just me,

1b34a9a1-64bd-4508-84ae-5173054f0eb0-1
00:14:42.105 --> 00:14:47.858
but a number of us at MIT have been
talking and thinking about, you know,

1b34a9a1-64bd-4508-84ae-5173054f0eb0-2
00:14:47.858 --> 00:14:51.280
at least since the the the last five
years.

6eff266c-bf21-4361-bedc-388e9a91f544-0
00:14:52.760 --> 00:14:58.812
And certainly the history of this at MIT
goes all the way back to JC or Licklider

6eff266c-bf21-4361-bedc-388e9a91f544-1
00:14:58.812 --> 00:15:03.757
or like, as he was called,
a faculty member here in the, you know,

6eff266c-bf21-4361-bedc-388e9a91f544-2
00:15:03.757 --> 00:15:04.200
1960s.

6bef068d-dcf7-4e76-a930-53dd1ea18d3b-0
00:15:06.160 --> 00:15:12.162
And it's really about how do we focus on
collaboration of human and intelligent

6bef068d-dcf7-4e76-a930-53dd1ea18d3b-1
00:15:12.162 --> 00:15:17.639
machines rather than just focus on
intelligent machines and what they're

6bef068d-dcf7-4e76-a930-53dd1ea18d3b-2
00:15:17.639 --> 00:15:20.040
able to accomplish on their own.

142f7253-cbdb-4754-b6ac-060868951c12-0
00:15:20.040 --> 00:15:24.623
And when you think about the whole early
development of AI, you know,

142f7253-cbdb-4754-b6ac-060868951c12-1
00:15:24.623 --> 00:15:28.027
the Turing test or Imitation game as it
was called,

142f7253-cbdb-4754-b6ac-060868951c12-2
00:15:28.027 --> 00:15:31.759
that was all about replicating human
performance, right?

0f26a856-2a08-44c7-bd3d-b1225e7ba7d0-0
00:15:31.760 --> 00:15:34.492
In fact, it was about,
could you tell if this was a human or an

0f26a856-2a08-44c7-bd3d-b1225e7ba7d0-1
00:15:34.492 --> 00:15:34.920
AI or not?

f04a76b7-adc0-4573-949d-869686f0e01c-0
00:15:34.920 --> 00:15:36.440
Well, that's long gone.

c40b114c-87c4-4c26-bb96-7263df11358a-0
00:15:36.840 --> 00:15:40.036
Usually if it seems a little less clueful
in like, you know,

c40b114c-87c4-4c26-bb96-7263df11358a-1
00:15:40.036 --> 00:15:44.490
some kind of test you would give somebody,
it's probably human because humans aren't

c40b114c-87c4-4c26-bb96-7263df11358a-2
00:15:44.490 --> 00:15:45.800
good at pop tests, right?

d0b9f66f-e244-410d-b781-4cca8b99fa5c-0
00:15:45.800 --> 00:15:49.137
So,
but that really oriented us toward not

d0b9f66f-e244-410d-b781-4cca8b99fa5c-1
00:15:49.137 --> 00:15:54.958
thinking in the whole design and
development and deployment process toward

d0b9f66f-e244-410d-b781-4cca8b99fa5c-2
00:15:54.958 --> 00:15:57.520
this human machine collaboration.

97f55633-da46-47ae-b35a-96806eaa14a9-0
00:15:58.400 --> 00:16:00.120
And it's a harder problem.

8428a621-6bda-4636-865b-71709e161d07-0
00:16:00.720 --> 00:16:03.916
So if you even just think about the
challenge of, OK,

8428a621-6bda-4636-865b-71709e161d07-1
00:16:03.916 --> 00:16:07.172
I have some benchmark that I'm
benchmarking, you know,

8428a621-6bda-4636-865b-71709e161d07-2
00:16:07.172 --> 00:16:11.612
a machine learning model against,
it's all kinds of challenges and sort of

8428a621-6bda-4636-865b-71709e161d07-3
00:16:11.612 --> 00:16:15.993
deciding is the benchmark really
reflective of like reality of using this

8428a621-6bda-4636-865b-71709e161d07-4
00:16:15.993 --> 00:16:17.000
this large model?

bb81b319-5ab3-4b7b-beef-7a9dab5eb3a0-0
00:16:18.480 --> 00:16:23.820
But now if you think about how do I
benchmark how much better this AI is

bb81b319-5ab3-4b7b-beef-7a9dab5eb3a0-1
00:16:23.820 --> 00:16:29.600
together with a human than a human is
alone or an AI is alone on this problem.

cbd8aa5b-be99-4290-a3c3-843cdba23ac3-0
00:16:30.200 --> 00:16:34.314
Just even trying to frame that as a
benchmarking challenge,

cbd8aa5b-be99-4290-a3c3-843cdba23ac3-1
00:16:34.314 --> 00:16:39.731
much less than how do you sort of adapt
it to a particular work scenario is is

cbd8aa5b-be99-4290-a3c3-843cdba23ac3-2
00:16:39.731 --> 00:16:41.240
much more challenging.

1bcaf614-a36c-4889-896f-0557877b005e-0
00:16:41.240 --> 00:16:42.680
This is a harder problem.

f1946c65-12de-4857-8a83-77fee990299b-0
00:16:42.680 --> 00:16:47.364
It's not like we're sort of not working
on something that would be low hanging

f1946c65-12de-4857-8a83-77fee990299b-1
00:16:47.364 --> 00:16:47.720
fruit.

a2c8a71f-642c-4539-87ba-e42132260b55-0
00:16:48.600 --> 00:16:52.583
But I think that makes it all the more
important that in addition to developing

a2c8a71f-642c-4539-87ba-e42132260b55-1
00:16:52.583 --> 00:16:54.974
systems,
the way we're developing them today is

a2c8a71f-642c-4539-87ba-e42132260b55-2
00:16:54.974 --> 00:16:58.460
stand alone forms of intelligence that
there's much more focus on the

a2c8a71f-642c-4539-87ba-e42132260b55-3
00:16:58.460 --> 00:17:00.999
integration of,
of human and machine intelligence.

13001108-c1bd-425c-81a3-d2728cd1b2e5-0
00:17:01.720 --> 00:17:06.949
So as you mentioned, we, we, we,
we build AI by design to be human like

13001108-c1bd-425c-81a3-d2728cd1b2e5-1
00:17:06.949 --> 00:17:10.217
it's,
it's trained on human data to do human

13001108-c1bd-425c-81a3-d2728cd1b2e5-2
00:17:10.217 --> 00:17:12.760
tests or to succeed at human tests.

7092e68e-466f-44ea-9297-86f7acedddab-0
00:17:14.880 --> 00:17:16.880
Are there alternatives to building an
intelligence?

5681996b-386a-4217-8f8d-220dffd03059-0
00:17:16.880 --> 00:17:18.040
What what alternatives are there?

2be76138-b305-406e-b5cc-96294517564d-0
00:17:19.160 --> 00:17:20.680
Well, so a couple of things.

d6e9498b-f694-4adc-9569-4022a879b9f3-0
00:17:20.680 --> 00:17:24.080
So one is you could,
I guess the first thing I would say is

d6e9498b-f694-4adc-9569-4022a879b9f3-1
00:17:24.080 --> 00:17:28.841
actually one of the big advances in large
scale machine learning in the last couple

d6e9498b-f694-4adc-9569-4022a879b9f3-2
00:17:28.841 --> 00:17:33.376
of years is less and less training on
what humans do and more and more training

d6e9498b-f694-4adc-9569-4022a879b9f3-3
00:17:33.376 --> 00:17:34.680
on other things, right.

d972dbb9-62d1-40c1-ae04-e434d1b3f718-0
00:17:34.680 --> 00:17:39.138
So when you start, you know,
so like self play in any sort of

d972dbb9-62d1-40c1-ae04-e434d1b3f718-1
00:17:39.138 --> 00:17:44.243
situation that you can set up multiple
AIS playing against each other,

d972dbb9-62d1-40c1-ae04-e434d1b3f718-2
00:17:44.243 --> 00:17:49.205
a lot of training now is using that and
not using sort of, you know,

d972dbb9-62d1-40c1-ae04-e434d1b3f718-3
00:17:49.205 --> 00:17:52.368
human reinforcement learning type
feedback,

d972dbb9-62d1-40c1-ae04-e434d1b3f718-4
00:17:52.368 --> 00:17:56.899
but actually just these systems judging
each other or, or, or,

d972dbb9-62d1-40c1-ae04-e434d1b3f718-5
00:17:56.899 --> 00:17:59.200
or interacting with one another.

1dee0651-10d2-4705-aad0-98007455d4fc-0
00:18:00.080 --> 00:18:04.664
But also when we do think about any sort
of use of AI in the physical world,

1dee0651-10d2-4705-aad0-98007455d4fc-1
00:18:04.664 --> 00:18:07.283
more and more, it's it's sort of,
you know,

1dee0651-10d2-4705-aad0-98007455d4fc-2
00:18:07.283 --> 00:18:12.106
raw sensing data completely uninterpreted
by humans, you know, images, you know,

1dee0651-10d2-4705-aad0-98007455d4fc-3
00:18:12.106 --> 00:18:15.261
audio, any,
and I don't mean of people talking like,

1dee0651-10d2-4705-aad0-98007455d4fc-4
00:18:15.261 --> 00:18:18.238
you know,
just large scale sensing data listening

1dee0651-10d2-4705-aad0-98007455d4fc-5
00:18:18.238 --> 00:18:22.585
to the machines running in a plant and
you start hearing noises that are

1dee0651-10d2-4705-aad0-98007455d4fc-6
00:18:22.585 --> 00:18:25.800
indicative that something's going to go
wrong, right?

46220521-3aa6-46cf-80c4-9e7d33ddcbf3-0
00:18:25.800 --> 00:18:29.501
I mean, it's just,
it's a lot of this kind of data going in

46220521-3aa6-46cf-80c4-9e7d33ddcbf3-1
00:18:29.501 --> 00:18:34.560
directly in a way that that that's not
encumbered by the human perceptual system.

5ab1d85d-104d-42e0-a4fa-b1a0ff61aa19-0
00:18:34.680 --> 00:18:39.006
It's encumbered by whatever system is
recording is again a difference between

5ab1d85d-104d-42e0-a4fa-b1a0ff61aa19-1
00:18:39.006 --> 00:18:40.560
the machines and the humans.

5fa208e7-6d13-4044-9daf-8bcb7bd6772b-0
00:18:41.000 --> 00:18:45.876
The human visual auditory systems are
extremely powerful in ways that, you know,

5fa208e7-6d13-4044-9daf-8bcb7bd6772b-1
00:18:45.876 --> 00:18:48.586
cameras and microphones and so forth
aren't,

5fa208e7-6d13-4044-9daf-8bcb7bd6772b-2
00:18:48.586 --> 00:18:52.560
but they are also limited in ways that
those other things aren't.

e7ff1ccb-9863-4297-bb4e-370815061cd9-0
00:18:52.600 --> 00:18:54.880
So it's maybe a more concrete version of
this.

0fd573df-a2af-4773-9358-5dba0368b596-0
00:18:55.200 --> 00:18:58.936
These are different systems,
different machine learning algorithms,

0fd573df-a2af-4773-9358-5dba0368b596-1
00:18:58.936 --> 00:19:00.640
different sensing capabilities.

28f790a7-9eba-4fa3-bdf3-fc23840fb33f-0
00:19:01.240 --> 00:19:04.334
And so it shouldn't be hard to understand
that at least in principle,

28f790a7-9eba-4fa3-bdf3-fc23840fb33f-1
00:19:04.334 --> 00:19:06.412
if we could harness the two of those
together,

28f790a7-9eba-4fa3-bdf3-fc23840fb33f-2
00:19:06.412 --> 00:19:07.959
we're going to get better outcomes.

b375a9f7-38e7-4db7-b01f-f05effbceaf9-0
00:19:08.440 --> 00:19:10.767
Like if you and I think exactly the same
way,

b375a9f7-38e7-4db7-b01f-f05effbceaf9-1
00:19:10.767 --> 00:19:14.360
we're more likely to come up with a less
good answer than if we don't.

4d8353aa-0084-47e5-a95c-71bd6a17e6d3-0
00:19:14.400 --> 00:19:14.720
Right.

a27fcfca-5260-422b-8e48-6216dc13d8c1-0
00:19:14.720 --> 00:19:18.000
So it just it's it this is not just about
humans and machines.

2c69e0db-87fb-433c-87f4-1bed88782c30-0
00:19:19.320 --> 00:19:23.426
And so that was sort of the the the first
piece that this is all all already

2c69e0db-87fb-433c-87f4-1bed88782c30-1
00:19:23.426 --> 00:19:23.960
happening.

042ea23f-27d4-480a-829d-5ec8febf4d2e-0
00:19:23.960 --> 00:19:27.840
And now I need your question again,
because what are the alternative?

a4ad040f-2199-473d-8471-742600d26999-0
00:19:28.120 --> 00:19:30.640
Like what are the alternative approaches
to, to developing an intelligence?

1bcec4f3-4cb2-4947-8d38-111355520ede-0
00:19:30.640 --> 00:19:30.840
Oh, yeah.

a36d3919-e9fa-4577-95a6-59d60f44b78a-0
00:19:30.880 --> 00:19:31.040
Yeah.

eb7228e1-1cac-4658-99cc-1304f3f8e3fe-0
00:19:31.040 --> 00:19:31.400
So that.

d2427c99-add7-4d04-8796-ede4a2cc6895-0
00:19:31.400 --> 00:19:31.600
Yeah.

f2d3c373-4586-4f8e-a6bf-38f12e0c3f63-0
00:19:31.600 --> 00:19:32.440
So other approaches.

f02aef99-4f66-4d8a-88cc-7e60c3e1f42c-0
00:19:32.440 --> 00:19:37.600
So then I think the other thing is that
we really need to work more and more.

1d6ff659-97b6-4fcb-b337-806b968a9058-0
00:19:37.600 --> 00:19:41.935
And I think this will start to happen in
the software development arena because of

1d6ff659-97b6-4fcb-b337-806b968a9058-1
00:19:41.935 --> 00:19:46.271
this partnering of experienced engineers,
bringing different kinds of knowledge to

1d6ff659-97b6-4fcb-b337-806b968a9058-2
00:19:46.271 --> 00:19:49.040
the table than these machines is starting
to become.

a6bb555c-d618-4d1c-aaa9-64022090c102-0
00:19:49.040 --> 00:19:52.213
So valuable companies,
at least that are at the forefront of

a6bb555c-d618-4d1c-aaa9-64022090c102-1
00:19:52.213 --> 00:19:54.815
doing this,
are going to start to learn how to do

a6bb555c-d618-4d1c-aaa9-64022090c102-2
00:19:54.815 --> 00:19:55.960
that more effectively.

f0b7d489-7a03-4f41-a654-0c62b7539ff0-0
00:19:56.480 --> 00:19:59.505
But for instance,
most of what AI does is relatively

f0b7d489-7a03-4f41-a654-0c62b7539ff0-1
00:19:59.505 --> 00:20:01.560
uninterpretable to people right now.

0543b0ef-c5c4-4b60-8a6b-4cf988aacc0c-0
00:20:01.760 --> 00:20:03.160
It's a bit of a black box.

1ef9b92c-5b0b-45f9-a32d-efe28e83849e-0
00:20:03.840 --> 00:20:05.933
Well,
if you really want human machine

1ef9b92c-5b0b-45f9-a32d-efe28e83849e-1
00:20:05.933 --> 00:20:08.080
interaction to work in a reasonable way.

c52683be-cd8f-49ed-a526-75a5755f6238-0
00:20:09.040 --> 00:20:12.800
These things have to somehow be legible
to human beings what it's doing.

555a6245-ddb6-4281-bbf7-4f5e1c01287d-0
00:20:13.520 --> 00:20:19.720
This is not necessarily the same as,
you know, explainable or justifying.

dafff05e-8e14-4956-a149-84185d7b1795-0
00:20:19.720 --> 00:20:21.964
They're just,
there has to be some things that I get

dafff05e-8e14-4956-a149-84185d7b1795-1
00:20:21.964 --> 00:20:24.040
some understanding of what that thing was
doing.

a37ba378-b443-46d6-8bce-4bbbbfdb3981-0
00:20:25.120 --> 00:20:29.463
And in fact, you know,
one of the complaints that I do still

a37ba378-b443-46d6-8bce-4bbbbfdb3981-1
00:20:29.463 --> 00:20:32.809
hear from,
from from senior software folks is,

a37ba378-b443-46d6-8bce-4bbbbfdb3981-2
00:20:32.809 --> 00:20:36.370
you know,
if there's something wrong with some AI

a37ba378-b443-46d6-8bce-4bbbbfdb3981-3
00:20:36.370 --> 00:20:42.280
generated code, some of some subtle bug,
just throw it out and have a human do it.

323a527a-e115-4f5c-9bf5-c65081c7ddd6-0
00:20:44.120 --> 00:20:48.955
Because it just, you know,
part of debugging subtle bugs is you try

323a527a-e115-4f5c-9bf5-c65081c7ddd6-1
00:20:48.955 --> 00:20:52.440
to get yourself in the mindset of the
developer.

d8038f44-bc1e-4e09-a80a-448fbccc5ff5-0
00:20:52.440 --> 00:20:57.105
Why did they make that stupid mistake in
this absence that there is no sort of

d8038f44-bc1e-4e09-a80a-448fbccc5ff5-1
00:20:57.105 --> 00:21:00.058
mindset,
there is no sort of legible kind of more

d8038f44-bc1e-4e09-a80a-448fbccc5ff5-2
00:21:00.058 --> 00:21:03.720
abstract understanding of what these
systems are doing today?

c04ffb5b-33d3-441f-a53b-f90d1b6a4be7-0
00:21:04.800 --> 00:21:08.064
And I think those kinds of advances will
be very powerful there and will be

c04ffb5b-33d3-441f-a53b-f90d1b6a4be7-1
00:21:08.064 --> 00:21:08.880
powerful elsewhere.

d224ecbd-70ad-4563-a746-c55bf3d98a97-0
00:21:08.880 --> 00:21:11.480
But it's a different framing of the
problem.

99fe44da-5b5f-4025-a4e2-da833b68625f-0
00:21:12.040 --> 00:21:14.363
Like,
I think when people have talked about

99fe44da-5b5f-4025-a4e2-da833b68625f-1
00:21:14.363 --> 00:21:17.531
interpretable AI,
they've more right now been worried about

99fe44da-5b5f-4025-a4e2-da833b68625f-2
00:21:17.531 --> 00:21:20.382
sort of, you know,
sort of moral and ethic, you know,

99fe44da-5b5f-4025-a4e2-da833b68625f-3
00:21:20.382 --> 00:21:23.920
like making more ethical AI, you know,
sort of is it being biased?

7ffe4e2a-4e31-4644-85fa-4258a8bcbb29-0
00:21:23.920 --> 00:21:28.524
Is it Those are also important things,
but it's a very different domain and not

7ffe4e2a-4e31-4644-85fa-4258a8bcbb29-1
00:21:28.524 --> 00:21:32.897
one that's oriented toward effective
collaboration where you could start to

7ffe4e2a-4e31-4644-85fa-4258a8bcbb29-2
00:21:32.897 --> 00:21:35.200
measure the collaboration effectiveness.

ab68695c-0d9b-4d83-be66-0121b53c399b-0
00:21:35.200 --> 00:21:39.317
So I think that those are some of the
kinds of things that we need to start to

ab68695c-0d9b-4d83-be66-0121b53c399b-1
00:21:39.317 --> 00:21:42.080
add to the stable of the ways that we're
developing.

0c6ecf93-da3b-44ad-b5fb-33cfa46a8373-0
00:21:42.080 --> 00:21:44.040
It's not like we should stop doing what
we're doing.

a8cf59e4-136c-4e4f-a4d3-6442a65b9eb7-0
00:21:44.040 --> 00:21:45.760
It's it's paying off a lot.

24f0347e-1cc1-4f1b-a89b-1a480a5f8905-0
00:21:45.760 --> 00:21:50.627
But it's that these other things are
harder and therefore probably going to

24f0347e-1cc1-4f1b-a89b-1a480a5f8905-1
00:21:50.627 --> 00:21:53.573
take longer and are going to offer,
you know,

24f0347e-1cc1-4f1b-a89b-1a480a5f8905-2
00:21:53.573 --> 00:21:56.840
even more impressive set of things that
we can do.

5330707c-c0e1-46a5-9497-2778ad8238c5-0
00:21:57.440 --> 00:21:57.920
Appreciate that.

52916e72-1658-4328-aa5f-281d56d7d840-0
00:21:57.920 --> 00:21:58.040
Yeah.

23db3603-aad4-4850-bec7-cd871b3a9967-0
00:21:58.560 --> 00:22:03.920
Hi, My name is Ashmita Gupta,
and I'm a computer engineer by education.

dc8ab1aa-f699-42ce-b742-779a8880b01e-0
00:22:03.920 --> 00:22:09.924
And we talked a lot about for, you know,
graduates coming out of school or going

dc8ab1aa-f699-42ce-b742-779a8880b01e-1
00:22:09.924 --> 00:22:13.852
into school,
what should how should they be thinking

dc8ab1aa-f699-42ce-b742-779a8880b01e-2
00:22:13.852 --> 00:22:14.520
about it?

2ecada03-899f-4aee-af8c-4062a5d2ae69-0
00:22:14.520 --> 00:22:18.047
My question is,
how should educational institutions be

2ecada03-899f-4aee-af8c-4062a5d2ae69-1
00:22:18.047 --> 00:22:23.243
thinking about adopting their curriculum
in the light of how AI is now changing,

2ecada03-899f-4aee-af8c-4062a5d2ae69-2
00:22:23.243 --> 00:22:25.360
you know, once the kids graduate?

29b4432f-24bd-43fb-9126-78a77fb549c6-0
00:22:25.480 --> 00:22:26.680
So what's your thought on that?

8191009b-17e3-4a30-8a0e-e7ff0102e9e4-0
00:22:27.040 --> 00:22:27.280
Yeah.

382e91ff-ea83-4597-b520-5449262b5076-0
00:22:30.160 --> 00:22:31.840
Again, many layers to this.

1f04cc25-95a1-4c4a-8470-6f93ee00b508-0
00:22:31.840 --> 00:22:38.511
So I'm actually going to start with
academic content about AI and how it

1f04cc25-95a1-4c4a-8470-6f93ee00b508-1
00:22:38.511 --> 00:22:43.721
works and its systems,
which is very different from say,

1f04cc25-95a1-4c4a-8470-6f93ee00b508-2
00:22:43.721 --> 00:22:46.920
how AI might be used pedagogically.

50a8bd4d-8d2d-498c-a048-cc50971e8ac1-0
00:22:47.080 --> 00:22:51.438
So I'll take those as two separate
questions that they sort of meet in some

50a8bd4d-8d2d-498c-a048-cc50971e8ac1-1
00:22:51.438 --> 00:22:51.840
places.

c1521829-68bf-437e-a6ad-1799b43963bb-0
00:22:52.360 --> 00:22:57.639
So I think when it's understanding about
machine learning and other AI approaches,

c1521829-68bf-437e-a6ad-1799b43963bb-1
00:22:57.639 --> 00:23:02.727
it's extremely important that the that
gets more and more integrated with every

c1521829-68bf-437e-a6ad-1799b43963bb-2
00:23:02.727 --> 00:23:04.000
academic discipline.

b8d8e5a3-9249-4166-ba43-7c2980c458e6-0
00:23:05.400 --> 00:23:10.023
And that's something that partly by luck
and partly by design,

b8d8e5a3-9249-4166-ba43-7c2980c458e6-1
00:23:10.023 --> 00:23:16.188
with the Schwartzman College of Computing
at MIT having started in 2019, which was,

b8d8e5a3-9249-4166-ba43-7c2980c458e6-2
00:23:16.188 --> 00:23:20.005
you know,
ahead of this gigantic tsunami that we're

b8d8e5a3-9249-4166-ba43-7c2980c458e6-3
00:23:20.005 --> 00:23:21.840
living through right now.

0ac1bb4f-4aff-4f17-abfc-c2b3b74259ac-0
00:23:22.600 --> 00:23:28.270
We've put in place a whole educational
set of educational offerings that we call

0ac1bb4f-4aff-4f17-abfc-c2b3b74259ac-1
00:23:28.270 --> 00:23:33.660
the common ground for computing education,
which integrates the forefront of

0ac1bb4f-4aff-4f17-abfc-c2b3b74259ac-2
00:23:33.660 --> 00:23:37.160
computing,
both CS and AI with disciplines across

0ac1bb4f-4aff-4f17-abfc-c2b3b74259ac-3
00:23:37.160 --> 00:23:37.440
MIT.

d325c1d3-7f03-4e38-91b1-b6b6f5ea59fb-0
00:23:37.440 --> 00:23:41.400
And these are classes that are Co taught
with 20 different departments across MIT.

bdbe8171-fcb5-4cd3-85ed-fb35aab10d04-0
00:23:41.920 --> 00:23:47.604
So you get faculty and engineering
departments and you know, in Sloan, in,

bdbe8171-fcb5-4cd3-85ed-fb35aab10d04-1
00:23:47.604 --> 00:23:51.242
you know,
in humanities and social sciences all

bdbe8171-fcb5-4cd3-85ed-fb35aab10d04-2
00:23:51.242 --> 00:23:57.002
across MIT working on how to engage and
sort of like thinking about machine

bdbe8171-fcb5-4cd3-85ed-fb35aab10d04-3
00:23:57.002 --> 00:23:59.199
learning in their discipline.

98a6b6e1-a1a5-4ff4-a882-a388d4e01beb-0
00:23:59.960 --> 00:24:04.964
So I think that's extremely important
when I think it comes to the,

98a6b6e1-a1a5-4ff4-a882-a388d4e01beb-1
00:24:04.964 --> 00:24:10.558
I think when it comes to the pedagogical
side, it's extremely important to,

98a6b6e1-a1a5-4ff4-a882-a388d4e01beb-2
00:24:10.558 --> 00:24:16.152
to figure out how to use AI effectively
in your teaching in any department,

98a6b6e1-a1a5-4ff4-a882-a388d4e01beb-3
00:24:16.152 --> 00:24:18.360
any field, any faculty member.

908e712b-012b-4a52-ae68-8594a9921c23-0
00:24:19.760 --> 00:24:21.760
Like, I think back to like, I'm old.

c14b377d-cf14-4b3c-82db-82409d93f219-0
00:24:21.800 --> 00:24:25.232
So the calculator, you know,
replacing the slide rule is something

c14b377d-cf14-4b3c-82db-82409d93f219-1
00:24:25.232 --> 00:24:26.360
that I still remember.

bf6f71aa-ad53-4f49-a8b1-891a0404df68-0
00:24:27.480 --> 00:24:31.343
And I remember all these faculty,
I was a pretty young student at the time

bf6f71aa-ad53-4f49-a8b1-891a0404df68-1
00:24:31.343 --> 00:24:34.228
who, you know,
banned programmable calculators in their

bf6f71aa-ad53-4f49-a8b1-891a0404df68-2
00:24:34.228 --> 00:24:37.320
classes that was really going with where
things were going.

5b4519c8-438c-40ec-b76d-cfbe6edd5961-0
00:24:38.520 --> 00:24:42.092
The, and you know,
the banning of AI is it just like it's

5b4519c8-438c-40ec-b76d-cfbe6edd5961-1
00:24:42.092 --> 00:24:46.280
another we have to figure out this is
ubiquitous, it's in people's.

8f32f961-8335-4f37-8185-bb8dcbfca4cc-0
00:24:46.360 --> 00:24:51.245
You can't take something that people use
all the time in every aspect of their

8f32f961-8335-4f37-8185-bb8dcbfca4cc-1
00:24:51.245 --> 00:24:56.378
daily lives a lot more than engineers use
slide rules back in the day and say, no,

8f32f961-8335-4f37-8185-bb8dcbfca4cc-2
00:24:56.378 --> 00:24:59.594
you have to,
you may have to say pause a little bit

8f32f961-8335-4f37-8185-bb8dcbfca4cc-3
00:24:59.594 --> 00:25:04.666
while we figure out how we can guide you
because this is something that can wreak

8f32f961-8335-4f37-8185-bb8dcbfca4cc-4
00:25:04.666 --> 00:25:09.551
havoc with your ability to learn or it
can actually position you to learn more

8f32f961-8335-4f37-8185-bb8dcbfca4cc-5
00:25:09.551 --> 00:25:11.840
effectively than you currently learn.

42f6ae8c-bcdc-440d-8021-1cd674156391-0
00:25:12.480 --> 00:25:16.450
But so we don't want to just unleash it
without thinking about how to do that and

42f6ae8c-bcdc-440d-8021-1cd674156391-1
00:25:16.450 --> 00:25:19.597
how to guide students and how to be sure
that they're, you know,

42f6ae8c-bcdc-440d-8021-1cd674156391-2
00:25:19.597 --> 00:25:20.760
following that guidance.

c64d2086-0de4-444e-97f5-f6a28c3a22b9-0
00:25:21.200 --> 00:25:24.360
But those are those are the things that I
think are really, really important there.

98d623e2-25a4-4b8b-af7c-98afa42ac2f0-0
00:25:24.360 --> 00:25:26.360
Thank you.

e094d655-b8c4-4fa0-b62b-79793929dbe4-0
00:25:26.400 --> 00:25:29.977
I'm going to do one more question if we
can and then think we'll be out of time,

e094d655-b8c4-4fa0-b62b-79793929dbe4-1
00:25:29.977 --> 00:25:30.640
Sir, thank you.

78eda92e-4c04-47b9-b6ca-bded69302ea9-0
00:25:31.720 --> 00:25:35.111
Right now,
my understanding about like AI is about

78eda92e-4c04-47b9-b6ca-bded69302ea9-1
00:25:35.111 --> 00:25:37.240
data algorithm, computing power.

7996c2dd-20c2-4156-b143-86dae94748d2-0
00:25:37.680 --> 00:25:41.520
But as long as you talk more,
I just feel a little pause.

3e6fab5b-4593-4d59-aecf-072ed55d9419-0
00:25:41.840 --> 00:25:46.978
Do you think after five years or 10 years
when AI keep developing,

3e6fab5b-4593-4d59-aecf-072ed55d9419-1
00:25:46.978 --> 00:25:52.501
if lease pattern are going to be changed
or switching to a new pattern,

3e6fab5b-4593-4d59-aecf-072ed55d9419-2
00:25:52.501 --> 00:25:57.640
maybe having like a human brain connected
to like algorithms more?

0c6a9625-2a1e-4863-a313-60e944ea0979-0
00:25:57.640 --> 00:26:03.960
Or you think lease will still layer data
algorithm and computing power?

aa03026a-3e16-4945-aaa7-cc121586141f-0
00:26:03.960 --> 00:26:04.320
Thank you.

f4fbad0c-bb3b-4399-9553-8f00d5b85209-0
00:26:05.080 --> 00:26:07.553
I mean,
I think data algorithms and computing

f4fbad0c-bb3b-4399-9553-8f00d5b85209-1
00:26:07.553 --> 00:26:09.920
power are going to be true for a long
time.

6f82d0b5-060a-454a-8c56-8054ee57689a-0
00:26:10.640 --> 00:26:13.520
You know,
I think algorithms and algorithmic

6f82d0b5-060a-454a-8c56-8054ee57689a-1
00:26:13.520 --> 00:26:17.040
advances can change the computing power
picture a lot.

c5c5b34e-3b63-4cae-a9fa-22c4aab32843-0
00:26:18.040 --> 00:26:21.659
And sometimes people are not paying that
much attention to that right now because

c5c5b34e-3b63-4cae-a9fa-22c4aab32843-1
00:26:21.659 --> 00:26:25.234
the algorithms have not been advancing as
quickly as they might at some point in

c5c5b34e-3b63-4cae-a9fa-22c4aab32843-2
00:26:25.234 --> 00:26:25.719
the future.

5dfb8eb4-3afd-4fe4-a7bb-802643082aeb-0
00:26:27.840 --> 00:26:32.244
And, you know, the, the form, the,
the mode by which humans and AI interact,

5dfb8eb4-3afd-4fe4-a7bb-802643082aeb-1
00:26:32.244 --> 00:26:34.876
you know,
whether that's going to become more

5dfb8eb4-3afd-4fe4-a7bb-802643082aeb-2
00:26:34.876 --> 00:26:38.480
direct than than written and spoken
language, I, I don't know.

d33b1a3f-d1e1-4bc9-8369-9b99aa0cdf78-0
00:26:38.480 --> 00:26:41.566
It's kind of it's,
it's certainly happening at some levels

d33b1a3f-d1e1-4bc9-8369-9b99aa0cdf78-1
00:26:41.566 --> 00:26:44.129
these days,
but I I think that those things will

d33b1a3f-d1e1-4bc9-8369-9b99aa0cdf78-2
00:26:44.129 --> 00:26:45.279
remain very important.

2d22e168-8947-424c-934d-cb09c1624918-0
00:26:45.600 --> 00:26:48.720
So we still can use in these three
element to like a thing.

b46cbccb-f154-4b28-baba-94a67da3f788-0
00:26:48.720 --> 00:26:48.960
Yeah, yeah.

3d58221e-04a6-4908-9767-3d57d65f4797-0
00:26:49.320 --> 00:26:50.840
Thank you, Dan.

48025081-f0c2-4c34-96e5-c5d523bf86ef-0
00:26:50.960 --> 00:26:51.640
Thank you so much.

f4e0d351-761e-4748-ba24-06b79c28590b-0
00:26:51.640 --> 00:26:52.600
No, thank you for having us.

7f20c12d-3f92-4f24-8438-f5d1ea022c22-0
00:26:52.640 --> 00:26:53.200
Great to see you.

55e9032c-a55d-4736-a266-a50b546f411c-0
00:26:53.360 --> 00:26:53.840
Great to see you.

715c8e50-4881-4a63-b573-3e39b607ffb8-0
00:26:54.080 --> 00:26:54.640
Thank you all.