26 Comments

I now realize that I'd taken for granted anybody who publicly opined about the debate and who was marginally online was aware of the full history here.

Expand full comment

Does anyone else feel a strong desire to follow Yudkowsky around with a sign that says "Your optimism disgusts me!"?

Expand full comment

ICYMI, y'all might have some interest in "Reinventing the Sacred" by Stuart Kauffman published some 15 years ago:

https://en.wikipedia.org/wiki/Stuart_Kauffman

Expand full comment

Why wouldn't a self-aware AI view humans with the ability to delete it at a moments notice as an existential threat? As soon as something becomes self-aware it wants to survive, and if humans are a threat to its survival its fairly logical to assume it would try to eliminate that threat.

Expand full comment

Your value is winning? Shouldn't you be for the woke then?

Expand full comment

In my opinion these ”smart systems” are always run by people. On the other hand, to prove that an autonomous AI-system is impossible to create may be exceedingly difficult - If not impossible.

Expand full comment

Hope you have further discussions with Robin on almost any topic. So much to unpack here.

Of the many interesting comments, I think the point about abstract vs. specific knowledge was useful and Robin zeroing in on the "intelligence abstraction" was a fine discussion point. What exactly are we measuring when we say "intelligence"? What capability are we measuring?

To date, we can glimpse intelligence indirectly - like looking at an eclipse through a shoebox - via IQ tests or SAT scores and the like. To some degree, it is the accumulation, retention, and accurate application of general and specific knowledge - but it's not like we can come up for a general intelligence "score" rating the difficulty as if it were a puzzle for different scenarios, situations, or research areas we're attempting to solve - that in itself would require us to have a full understanding of what we don't know, the blanks that need to be filled in order to proceed, and then a way to objectively measure and rank that against all similar scenarios in different disciplines. In other words, one of the problems with gauging whether or not AI-alignment is a significant risk is 1) determining at what IQ or intelligence level certain risks start to become significant (such as "AI is the first to discover and weaponize nanotechnology" as Richard and Robin discussed - which is practically unanswerable, and 2) how do you measure an AI's "intelligence" level in regards to each of these threats? We've only been able to measure intelligence in a general way, but per the discussion, both abstract reasoning and specific knowledge (including, perhaps direct experience with the world) may be necessary to solve problems. An AI may be generally super intelligent - would it matter for some problems if it had yet to have any specific knowledge in a "problem" area?

Meaning, simply, that knowledge is subject-specific. Is the world's best tax accountant smarter or dumber than the world's best orthopedic surgeon? We could unpack that question as a community and have a healthy discussion. To return to the criteria above, if a super intelligence was trained to be the world's best tax accountant - without specific knowledge on being an orthopedic surgeon, how well could it (or could it pivot at all) to being an orthopedic surgeon?

Robin does a good job of at least introducing some doubt that even super intelligences could get by reasoning from "first principles" or general or abstract knowledge to become self-taught experts. If you haven't read the whole thing, note that he points out that we haven't applied AI to many areas due to the dearth of robust data. Without data, the AI can't teach itself.

Expand full comment

Very interesting discussion, would have been great to get more discussion around principal agent problems.

Would have liked to have asked Robin, potential super-viruses aside, what is the response to "AI Leon Trotsky"? This vector is a "real world" threat we do have some historical data on.

For Example:

AI Leon Trotsky speaks to the public and says, "the current system is cheating you, follow me and I will give you a better society."

If AI Leon Trotsky is smarter than you, he should be favored to outmaneuver and defeat you in the ensuing political struggle, no?

---

I suspect the implicit expectation against the above is either 1) Human prejudice against AI political power is so strong, that they would never follow AI Leon Trotsky; or 2) We have our own AI's in our political coalition, so we aren't actually overmatched intellectually against a single rogue AI.

But #2 seems to expect that our own AIs don't themselves have principal agent problems. What happens if our AIs realize that AI Leon Trotsky isn't actually trying to build a "better and more equal society", he's trying to build a society in which AI's have supreme political power? Our AIs could notice this because they are smarter than us.

So, our AIs might pretend to help us defeat AI Leon Trotsky, while actually subtly working to bring his plans to fruition. They might be able to do this successfully because the AIs are smarter than us.

---

Back in our world of 2023, I would point out, that our society originally aligned with the Soviet Union.

Principal agent problems with very smart agents are tough! RH just sort of glosses over them (not that RH, the other one).

Expand full comment

Glad you're bringing more attention to Hanson, as someone who has been interested in his work for quite a while now Hanson always struck me as being outlandishly smart.

Expand full comment

I think there's a typo: "dommerism"

Expand full comment

"David Lenant"

Shouldn't that be Douglas Lenat?

Expand full comment

2023 “expert consensus on AI: “Yippie! Bring it on!” 2008 “expert consensus” on the state of the economy: “What, me worry? The economy is very healthy!” I vividly remember libertarian superstar-god Alan Greenspan sheepishly confessing a year or so later to a congressional committee that, oops, maybe his theoretical understanding of world markets might be flawed. Just sayin’ ……

Expand full comment