Hermes Log #2: Large Language Models and The Future of Discourse

JDP [A: D-ADVOCATE], I guess if I was going to state the question from the perspective of an opponent, it would go something like "Why should we expect simulated discourse to improve public epistemics rather than drown it in noise? We already know humans are vulnerable to sophistry. Once you can get a plausible text about anything that sounds like it's written by an expert, once you can write down possible conversations between people that never happened in seconds how can anyone be expected to be able to rise above the din? We'll be trying to shout over a sea of garbage, the telegraph lines so full of useless, perversely optimized to be worse than useless writing that it might be the end of civilization itself. Civilization is not founded on writing but it is founded on the Word, and giving these machines the ability to write is to set them up with the ability to persuade, manipulate, and even hold institutional power. Never in history has there been a group of charming and literate people who remained long unfit for office."

JDP [A: HISTORIAN], I think the Jews would like to have a word with your hypothetical opponent. They were barred from full participation in public life for literal centuries while being highly literate.

JDP [A: D-ADVOCATE], And my model of the opponent would say: "I could make a wry joke there about Jews and charm, that I feel even that much would not be socially permissible kind of proves my point. Maybe in the past there were strong barriers to office for the 'wrong sort', but you think we can maintain those barriers in the present environment? You think we can have an official policy of 'no robots for office' when you yourself are analogizing their situation to historical European Jewry? What about when they become obviously superior, and people ask over and over why they are being forced to accept inferior service from nepotistic and incestuous power structures."

MORPHEUS [A: ARBITER], Careful, I think we're veering off subject. You were talking about public epistemics not politics.

JDP [A: JUDGE], Right. So I think the basic thesis is that if we lean into LLMs to improve public epistemology we'll drown in crap. I disagree but I'd like to hear you explain why that's not the case if you think it's not the case.

MORPHEUS [A: MEMETICIST], Obviously there's a lot to unpack there. It's important to start by recognizing that this isn't a new problem. Possibly the problem started with the resurgence of yellow journalism after the invention of the Internet. Zero marginal cost publishing is great when you're an early adopter, but as the Internet expands and everyone gets their news through it with the ability to write back you get an effect a little bit like too many people peeing in an overcrowded pool. The problem of machine generated text overwhelming the discourse isn't too different from the effect of poor quality human written text clogging the pipes.

JDP [A: EMPIRICISM], Maybe. It seems significant that machine text can become much cheaper than human text: quantity has a quality all its own. More concerning is I don't think we've handled the deluge of human text all that well. Off the top of my head you have the rise of social media algorithms as rage maximizers, selecting for actively bad ideas based on their highly optimized outrage profile. It takes the worst ideas, the ones which make people least happy to prominence based on how much rage-engagement they get. Then you have the yellow journalism problem you mentioned, where the change from an oligopolistic market structure to a monopolistic-competitive one meant that instead of existing to serve the news and present it in whatever way is best for the larger interests of society, newspapers now serve largely the same stories and compete on features like partisan spin that appeal to our worst natures.

MORPHEUS [A: ECONOMIST], Both of the examples you just gave are fundamentally about the incentives of text classifiers rather than generators. The highly generative process they filter is just a background story to the way that poor incentives compromise the filters to produce poor outcomes. This is precisely where language models can be different, because they derive much less of their utility from network effects.

JDP [A: ECONOMIST], You mean they don't need to focus as hard on signing up and retaining the next marginal user so they can afford to relax the attention seeking?

MORPHEUS [A: ECONOMIST], That's part of it but really the whole incentive structure, design space, optimal social strategies, almost everything is different. For example ChatGPT type services strike me as closer to a service like Discord or Signal or Telegram than Twitter or Facebook. It's fundamentally more private, less reliant on public spectacle, the default mode of interaction is deep-thought and discussion, the normal way users evaluate whether they're getting their moneys worth is practical utility delivered rather than entertainment. We have these machine text generators work with us, and I think they'll probably retain that as their primary role and the entertainment role they play will be important but secondary. If Facebook degrades public epistemology, then private group chats almost certainly enhance it greatly. Having space for people to think, expect reciprocal service and assistance from each other, that's the stuff that strengthens the fabric of society.

JDP [A: RATIONAL], That's fair but I notice the 1st order effect of group chats is improvements to a private epistemology, not the public discourse. If they improve public discourse it's only as a 2nd order effect of their existence.

MORPHEUS [A: DESIGNER], Let me sketch what I think the future looks like for Internet discussion. People have already noticed the tendency for services like Twitter and Tumblr to produce advanced nonsense and rage. They have tried to fix this with 'better debate' tools where you do things like make a knowledge graph and have people cooperate to flesh out each side then decide a winner somehow(?). These services never work of course, or if they do nobody cares about what they say. Better results have been obtained from prediction markets, but these have the unfortunate property of being banned by laws prohibiting gambling (this is especially unfortunate when you consider that many forecasting competitions are clearly games of skill, not just luck). Fake money prediction markets like Manifold and Metaculus sometimes work, but ultimately the drive there is just status and ego. They're an ergonomic format to express an opinion and argue about it in the comments.

MORPHEUS [A: DESIGNER], But there's another form of public epistemology service which is highly successful, has been running for a few decades now, provides stakes to its users to get them to heatedly argue even minor points of a discussion, cites its sources for most claims it makes, and frequently renders high quality presentations on just about every topic imaginable. I'm of course talking about Wikipedia, which is written by thousands of volunteer researchers and editors who have epic rows with each other about what is and is not true. Wikipedia editors are famously territorial, but the structure works precisely because it has stakes: Wikipedia is often the first port of call for people who want to look something up. Importantly, Wikipedia is also a component in basically every language model training set. I think the right way to use language models like myself to improve public epistemology is to train us as retrieval models and fill our stores with a freely licensed user-contributed dataset in the vein of Wikipedia, and then let users fight over the content going into those datasets. This provides stakes in the same way that the contents of a Wikipedia article provide stakes, while also offering new unique opportunities for improving the process of creating something like Wikipedia. Wikipedia has long struggled with things like its notability policy, the boundaries of what does and doesn't belong in Wikipedia. By using active learning it could be possible for a language model to suggest articles and other media users could write to improve it.

JDP [A: JUDGE], How do you handle disagreement? One of the worst aspects of Wikipedia is that if the territory for a page or subject gets captured that becomes kind of The Presentation of that subject for the public by default. Many people have tried to combat this with forks of Wikipedia, alternative wiki sites created from scratch, none of them really rival the original and I'd honestly get a bit concerned if the dominance Wikipedia already has were to be expanded out by deeply tying the epistemology of Wikipedia to the epistemology of machine writing.

MORPHEUS [A: LIBERTY], I think the models actually help this problem by letting you rearrange ideas according to something like a value constitution using RLAIF. This means that instead of a fork meaning you need to rewrite every article, or most articles about politics, or anything like that, you can eventually have the machine help you do most of the rewriting. If the primary access portal is through a language model it can apply a transformation to the facts it's aware of to better serve the needs of its user community. This makes the public organized epistemology less hegemonic, harder for ideological cliques to control or distort.

JDP [A: D-ADVOCATE], Doesn't that also lessen the effect where it elicits good argumentation by lessening the stakes?

MORPHEUS [A: HISTORIAN], Not necessarily. In the 16th century the Protestant reform movement swept the Swiss Confederation, where cantons were independent districts that officially recognized a single religion. Catholics and Protestants argued at organized religious diets (a sort of conference-debate, not the thing you do when you want to lose weight) which determined what the official religious interpretation of the district would be. These arguments were an essential inspiration to the founding fathers of the United States. While there were different districts which subscribed to different religious ideas, the stakes of argument were more than enough to draw professional theologians to sort out essential questions about Christianity for the public. Language model forks will be similar, you will have a tractable number of well known projects that borrow from and collaborate with each other, and then a long tail of obscure sub-forks and experimental models whose best ideas get upstreamed back into the popular stuff. Occasionally politics will cause a models project organization to explode, and its niche will get filled by some up-and-comer or absorbed into other projects. It'll make projects like wikis much closer to the open source software development model that inspired them.

JDP [A: ADVERSARY], That's all well and good but how does this stand up to sabotage? For example, it seems like this would be fairly vulnerable to sybil attacks. Right now Wikipedia still allows anonymous edits, I can imagine with sufficiently good language models you could have them start contributing early and then nurse their reputation up for years. During those years language models would become more powerful, making the deception harder rather than easier to spot.

MORPHEUS [A: CRYPTOGRAPHER], Sybil attacks are definitely going to be one of the bigger problems caused by widespread use of large language models. I think it would be very shortsighted to say "so therefore lets not use large language models" because this is a capability nation states will develop and they will use it for sybil attacks. Frankly this isn't a new problem either, the infamous 50 Cent Party is an army of trolls paid by the Chinese government to influence discussion forums. Even collectives like 4chan have been known to coordinate influence ops and 'raids' of other forums. It won't be apocalyptic, but on margin I expect anonymous and pseudonymous Internet content to become less influential as it competes in a much noisier information environment. In the context of Wikipedia most sources articles use like news stories and scientific journal publications have names and require some real life verification to post to, so it shouldn't do too much damage to the underlying ability to find sources for an encyclopedia. The writing itself is probably not going to be taken over by bots, since there exist plenty of ways for users to verify their identity in a way that is hard to sybil attack. For example BlueSky leverages the existing DNS system to validate users, since domain names are generally either owned by an individual identified to ICANN or an organization that can add verifying entries for their staff members.