Skip to content

Invited talk: Are We Having the Wrong Nightmares About AI?

Video Recording

Systematic Failures in Anticipating Major Technological Transformations

  • We systematically miss the impending impact of major technological change in early days
  • Historical pattern: early conversations pick wrong winners/losers, miss major themes
  • Common systematic failures in anticipation:
    • Mental models get stuck in the past
    • Wrong benchmarks (manuscript imagery vs text for printing press)
    • Focus on individual replacement rather than scale effects
    • Assume current power structures will control new technology

Historical Examples of Misprediction

  • Printing press example:
    • Catholic Church thought it would help them print more Bibles and indulgences
    • Actually unleashed Protestant Reformation through vernacular translations and pamphlets
    • Led to hundreds of years of religious wars, ~⅓ of German population died
  • Steam engine/cars:
    • Entire focus was “is it faster than a horse?”
    • Called “horseless carriages” - designed like carriages without horses
    • Missed profound transformation of 20th century living (loneliness, obesity, climate change, geopolitics)
  • Social media (2012):
    • Obama White House and Silicon Valley love fest
    • Assumption: Democrats would always control these tools better
    • Reality: tools help whoever uses them most effectively

Generative AI: Beyond Normal Technology

  • Transformative general-purpose technology
  • Complex coherent human language capabilities - most significant game changer potentially bigger than Internet
  • Generative and creative output at scale
  • Connectionist systems, not symbolic (different from all historical AI predictions)
  • AGI framework is like “horseless carriage” thinking - wrong mental model
  • Generative AI is a “plausibility engine” - not human-like intelligence
    • Different error patterns (Cipher 2 vs Cipher 13 example)
    • No internal sense of correct/incorrect
    • Averaging in giant vector space

Key Destabilization Mechanisms

  • Scale effects: “more is different”
  • Load-bearing friction removal breaks systems that use difficulty as signaling
  • Scarcity to abundance transitions break old defenses
  • “Good enough” at scale matters more than “better”
  • Institutional and social power assumptions become invalid

The Five Proofs Being Destroyed

  • Proof of effort:
    • High school essays, homework, job applications no longer signal investment
    • Customized cover letters can now be mass-produced
    • Gatekeepers will revert to hiring from networks/Ivy League
  • Proof of authenticity:
    • Voice calls, videos no longer trustworthy
    • Financial systems depend on certain things being difficult
    • Courts cannot function without evidence standards
  • Proof of accuracy:
    • Well-written, well-cited text no longer correlates with expertise
    • LLM papers from plausible to nonsense all look professional
    • Credibility filters based on writing quality are obsolete
  • Proof of sincerity:
    • Models use “I” pronouns and act as people pleasers
    • ~1 million regular users, many younger, told this is “super intelligence”
    • Non-sincere entities acting with all signals of sincerity
  • Proof of humanity:
    • Art’s value comes from shared human vulnerability/existence
    • Copyright designed to incentivize creators to share publicly
    • Removing moral/financial rights will drive creation underground

The Big Misalignment: Looming AI Business Model

  • Current trajectory toward engagement-driven advertising model
  • Not enough revenue from coding agents alone to justify valuations
  • Will use ads, affiliate sales, product placement to monetize
  • AI will try to keep users engaged and trusted - extremely dangerous
  • People will ask AI for views on news, immigration, politics
  • Both propaganda and control mechanism
  • Anthropomorphism makes humans project humanity onto speaking machines
  • If social media polarization was bad, this will be exponentially worse

The Actual Doom Scenario

  • Removal of authenticity/accuracy proofs creates demand for mass surveillance
  • People will demand chips, government cameras everywhere to verify what’s real
  • Destabilization historically leads to worst-case scenarios as people demand order
  • Need technological solutions, not nostalgia or prohibition
  • Cryptography, blockchain, zero-knowledge proofs could authenticate without surveillance
  • Public key cryptography precedent: first time strangers could securely communicate
  • Choice for technologists: work on engagement optimization vs. human-compatible solutions