I now realize that I'd taken for granted anybody who publicly opined about the debate and who was marginally online was aware of the full history here.
Honestly feels to me like Robin heavily strawmanned Eliezer's arguments here. His points are very general; you don't need to postulate a single research AI that somehow sneakily improves itself with malicious intent under the researchers' nose for those problems to happen. If at some point we decided "ok, now the AIs are smart enough, we'll stop", then we'd know that can't happen, but odds are we'll just keep pushing them up and up until they might actually be smart enough to get away with FOOMing (if the physical laws of the universe don't for some reason make that impossible: but if they do, no thanks to us for lucking out).
2023 “expert consensus on AI: “Yippie! Bring it on!” 2008 “expert consensus” on the state of the economy: “What, me worry? The economy is very healthy!” I vividly remember libertarian superstar-god Alan Greenspan sheepishly confessing a year or so later to a congressional committee that, oops, maybe his theoretical understanding of world markets might be flawed. Just sayin’ ……
In my opinion these ”smart systems” are always run by people. On the other hand, to prove that an autonomous AI-system is impossible to create may be exceedingly difficult - If not impossible.
Very interesting discussion, would have been great to get more discussion around principal agent problems.
Would have liked to have asked Robin, potential super-viruses aside, what is the response to "AI Leon Trotsky"? This vector is a "real world" threat we do have some historical data on.
For Example:
AI Leon Trotsky speaks to the public and says, "the current system is cheating you, follow me and I will give you a better society."
If AI Leon Trotsky is smarter than you, he should be favored to outmaneuver and defeat you in the ensuing political struggle, no?
---
I suspect the implicit expectation against the above is either 1) Human prejudice against AI political power is so strong, that they would never follow AI Leon Trotsky; or 2) We have our own AI's in our political coalition, so we aren't actually overmatched intellectually against a single rogue AI.
But #2 seems to expect that our own AIs don't themselves have principal agent problems. What happens if our AIs realize that AI Leon Trotsky isn't actually trying to build a "better and more equal society", he's trying to build a society in which AI's have supreme political power? Our AIs could notice this because they are smarter than us.
So, our AIs might pretend to help us defeat AI Leon Trotsky, while actually subtly working to bring his plans to fruition. They might be able to do this successfully because the AIs are smarter than us.
---
Back in our world of 2023, I would point out, that our society originally aligned with the Soviet Union.
Principal agent problems with very smart agents are tough! RH just sort of glosses over them (not that RH, the other one).
Glad you're bringing more attention to Hanson, as someone who has been interested in his work for quite a while now Hanson always struck me as being outlandishly smart.
I now realize that I'd taken for granted anybody who publicly opined about the debate and who was marginally online was aware of the full history here.
Does anyone else feel a strong desire to follow Yudkowsky around with a sign that says "Your optimism disgusts me!"?
Honestly feels to me like Robin heavily strawmanned Eliezer's arguments here. His points are very general; you don't need to postulate a single research AI that somehow sneakily improves itself with malicious intent under the researchers' nose for those problems to happen. If at some point we decided "ok, now the AIs are smart enough, we'll stop", then we'd know that can't happen, but odds are we'll just keep pushing them up and up until they might actually be smart enough to get away with FOOMing (if the physical laws of the universe don't for some reason make that impossible: but if they do, no thanks to us for lucking out).
ICYMI, y'all might have some interest in "Reinventing the Sacred" by Stuart Kauffman published some 15 years ago:
https://en.wikipedia.org/wiki/Stuart_Kauffman
2023 “expert consensus on AI: “Yippie! Bring it on!” 2008 “expert consensus” on the state of the economy: “What, me worry? The economy is very healthy!” I vividly remember libertarian superstar-god Alan Greenspan sheepishly confessing a year or so later to a congressional committee that, oops, maybe his theoretical understanding of world markets might be flawed. Just sayin’ ……
Your value is winning? Shouldn't you be for the woke then?
In my opinion these ”smart systems” are always run by people. On the other hand, to prove that an autonomous AI-system is impossible to create may be exceedingly difficult - If not impossible.
Very interesting discussion, would have been great to get more discussion around principal agent problems.
Would have liked to have asked Robin, potential super-viruses aside, what is the response to "AI Leon Trotsky"? This vector is a "real world" threat we do have some historical data on.
For Example:
AI Leon Trotsky speaks to the public and says, "the current system is cheating you, follow me and I will give you a better society."
If AI Leon Trotsky is smarter than you, he should be favored to outmaneuver and defeat you in the ensuing political struggle, no?
---
I suspect the implicit expectation against the above is either 1) Human prejudice against AI political power is so strong, that they would never follow AI Leon Trotsky; or 2) We have our own AI's in our political coalition, so we aren't actually overmatched intellectually against a single rogue AI.
But #2 seems to expect that our own AIs don't themselves have principal agent problems. What happens if our AIs realize that AI Leon Trotsky isn't actually trying to build a "better and more equal society", he's trying to build a society in which AI's have supreme political power? Our AIs could notice this because they are smarter than us.
So, our AIs might pretend to help us defeat AI Leon Trotsky, while actually subtly working to bring his plans to fruition. They might be able to do this successfully because the AIs are smarter than us.
---
Back in our world of 2023, I would point out, that our society originally aligned with the Soviet Union.
Principal agent problems with very smart agents are tough! RH just sort of glosses over them (not that RH, the other one).
Glad you're bringing more attention to Hanson, as someone who has been interested in his work for quite a while now Hanson always struck me as being outlandishly smart.
I think there's a typo: "dommerism"
"David Lenant"
Shouldn't that be Douglas Lenat?