Cultural col­lapse

There has been a bit of talk about model col­lapse, spurred by this Nature paper that was pub­lished a few weeks ago. The pa­per demon­strates some the­o­retic and em­pir­i­cal re­sults about how train­ing new mod­els from data gen­er­ated by pre­vi­ous it­er­a­tions of the model can re­sult in de­gen­er­ate mod­els.

There’s been a fair amount of crit­i­cism of this pa­per, mainly cen­ter­ing on the fact that the con­di­tions tested in the pa­per don’t re­ally re­flect the way syn­thetic data is used to train mod­els in the real world.

I’m not par­tic­u­larly wor­ried about the po­ten­tial of model col­lapse. There’s lots of ac­tive re­search on how we can make use of syn­thetic data, and post-fil­ter­ing seems like a tractable and valid way of avoid­ing some de­gen­er­ate cases. What I’ve been think­ing about more is a so­ciotech­ni­cal prob­lem: the in­flux of AI-gen­er­ated con­tent may not be harm­ful for train­ing fu­ture mod­els, but I worry that the reifi­ca­tion of con­tent gen­er­ated by mod­els that only have the shal­low­est con­cep­tion of mean­ing can re­sult in an ecosys­tem in which cul­tural ar­ti­facts are slowly leached of any depth. Model col­lapse does­n’t bother me as much as the po­ten­tial of cul­tural col­lapseMaybe this term is a lit­tle hy­per­bolic, but coun­ter­point: maybe not? does.

Layers of mean­ing

Culture is a fuzzy ob­ject. I ap­proach cul­ture as so­cially ne­go­ti­ated mean­ings of signs. Since I work on lan­guage (broadly de­fined), let’s first con­sider how this de­f­i­n­i­tion plays out in lan­guage as a cul­tural ar­ti­fact. At the most ba­sic level, we con­sider words (signs) to have ref­er­en­tial mean­ings. The word “dog” evokes some con­cep­tion in your mind of the four-legged mam­mal that wags its tail and barks.

But there are other kinds of mean­ing as well: there is the so­cial mean­ing about re­gional iden­tity em­bed­ded when some­one, for in­stance, says “pop” in­stead of “soda”.https://​www.popvs­soda.com There is mean­ing in in­ter­ac­tion: it is mean­ing­ful to re­spond to a ques­tion with si­lence, even though no words were ever ut­tered. And speech is not only ref­er­en­tial, but per­for­ma­tive. Uttering “I do” can marry a cou­ple; say­ing “I bet you it will rain to­mor­row” ini­ti­ates the wa­ger (if the in­ter­locu­tor ac­cepts).These ex­am­ples are from J. Searle, How to do things with Words. There is even mean­ing em­bed­ded in the way we dressPenelope Eckert, Variation and the Indexical Field and the make-up we ap­ply.Norma Mendoza-Denton, Muy Macha Within all of these semi­otic fields, there is struc­ture and mean­ing.

If there are mul­ti­ple as­pects of mean­ing, what kind of mean­ing do lan­guage mod­els learn? They are built on the idea of dis­tri­b­u­tional se­man­tics: words take on mean­ing by how they re­late to other words. Food is the set of things that you “eat”. Eating is what you do to “foods”. This re­la­tional sys­tem of mean­ing is in­ter­nally use­ful for the model, and gen­er­ally maps on rel­a­tively well to how we in­ter­pret lan­guage, but the mean­ing mak­ing for us hap­pens when we read the to­kens that the model out­puts. It’s at that point that we de­cide that “dog” means the same thing to the LLM as it does to you and I.

This is pretty ef­fec­tive in prac­tice, and I find dis­tri­b­u­tional se­man­tics to be a pretty com­pelling ap­proach to un­der­stand­ing mean­ing. But it’s worth con­sid­er­ing also what as­pects of mean­ing aren’t learned in this scheme. In­tro­duc­tions of dis­tri­b­u­tional se­man­tics of­ten quote com­pu­ta­tional lin­guist John FirthFirth, A Synoposis of Linguistic Theory as say­ing “You shall know a word by the com­pany it keeps.” But in that sec­tion, he ex­plains that col­lo­ca­tionalA word’s ap­pear­ance in re­la­tion to other words. meaning is but one as­pect of mean­ing, and that col­lo­ca­tion should not be mis­taken for con­text, which is so­cio­cul­tural. Meaning takes place across words, but also all kinds of other signs. Modern sys­tems are learn­ing as­pects of mean­ing that are largely re­moved from so­cial­cul­tural con­text!

What’s con­cern­ing is that, for the most part, com­pa­nies de­vel­op­ing AI prod­ucts are okay with this. At min­i­mum, there is a lack of crit­i­cal en­gage­ment with how lan­guage can con­struct power dy­nam­ics and so­cial struc­ture: to build sys­tems that serve a par­tic­u­lar lan­guage va­ri­ety means ne­glect­ing the di­ver­sity of other ones. And this is not just hy­po­thet­i­cal. Studies have shown, for ex­am­ple, that speech recog­ni­tion tech­nol­ogy works much worse for Black English speak­ers than white ones.Koenecke et al., Racial dis­par­i­ties in au­to­mated speech recog­ni­tion

What is per­haps more in­sid­i­ous, I think, is when peo­ple build these sys­tems to en­gage with cul­ture. Like this startup whose ad was float­ing around the in­ter­net a lit­tle bit ago:

What both­ers me is­n’t even the prod­uct nec­es­sar­ily; there are in­stances where sim­pli­fy­ing text might be use­ful! But I ob­ject to the prob­lema­ti­za­tion of “dif­fi­cult lan­guage”, and es­pe­cially to the im­pli­ca­tion that your read­ing po­ten­tial is mea­sured by the num­ber of book sum­maries you can in­gest. What aspects of mean­ing are we los­ing when we fo­cus solely on the plot? Language is mean­ing­ful, and we need to be in­ten­tional about which as­pects of mean­ing we keep and which we dis­card.

This temp­ta­tion to boil every­thing down to the essence re­flects a to­tal mis­un­der­stand­ing of the point of the ex­er­cise in cre­at­ing and con­sum­ing cul­ture. In try­ing to dis­till the plot, we may have lost it al­to­gether.

The na­ture of the en­ter­prise

So what are we try­ing to do here? Recently, there was an ad for Google Gemini that caught some view­ers off guard. A fa­ther asks Gemini to write a fan let­ter on be­half of his daugh­ter.

To many peo­ple, my­self in­cluded, this seems to miss the point of writ­ing a fan let­ter en­tirely. But why? Pegah Moradi, a PhD stu­dent at the Cornell I-School, wrote an ex­cel­lent piece about the au­topen, the ma­chine that can repli­cate sig­na­tures.Moradi and Levy, “A Fountain Pen Come to Life”: The Anxieties of the Autopen In the es­say, she cov­ers three cases in which use of the au­topen elides so­cial val­ues that we at­tach to the act of cre­at­ing a sig­na­ture: au­then­tic­ity, ac­count­abil­ity, and care.

That is to say, the shape of the sig­na­ture does­n’t con­tain the mean­ing, but rather what the sig­na­ture sig­ni­fies. And this sig­ni­fied mean­ing comes be­cause we have his­tor­i­cally agreed on the cul­tural sig­nif­i­cance of a signed name. To mechanically cre­ate or recre­ate the sig­na­ture out­side of that con­text re­duces its mean­ing. To gen­er­ate a let­ter re­moves the la­bor of care that gives the let­ter mean­ing in the first place. The “grammar” of the fan-mail rit­ual re­quires the work that goes into con­struct­ing the let­ter; to re­move that work re­sults in a se­man­ti­cally vac­u­ous in­ter­ac­tion.

Google’s as­ser­tion that gen­er­at­ing this let­ter is equiv­a­lent (or prefer­able, even) to the act of writ­ing one is an era­sure of the rich­ness of cul­tural mean­ing that gives the rit­ual of fan-mail its sig­nif­i­cance. It rei­fies the idea that cul­ture is re­duced to the to­kens which com­prise the ar­ti­fact; cul­ture be­comes a de­con­tex­tu­al­ized fac­sim­ile of it­self. This is what I mean by cul­tural col­lapse.

In my opin­ion, the point of writ­ing, of art, of cre­at­ing and con­sum­ing cul­ture, is to cre­ate mean­ing, not to trans­mit it. I write these words not with the hope that you will think my thoughts, but that they will play a role in in­spir­ing your own.

I don’t mean to sug­gest that cul­ture and com­pu­ta­tion are mu­tu­ally ex­clu­sive. My own re­search is all about com­pu­ta­tion­ally study­ing these other as­pects of mean­ing – for ex­am­ple, what are the dif­fer­ent mean­ings con­tained in con­do­lence-giv­ingZhou and Jurgens, Condolence and Empathy in Online Communities or em­bed­ded in the meme tem­plates that we choose to use on Reddit?Zhou, Jurgens and Bamman, Social Meme-ing In fact, this kind of work has re­ally been en­abled by the great progress in NLP and com­puter vi­sion. But cul­ture as it ex­ists around is in­cred­i­bly rich with lay­ers and lay­ers of mean­ing, and we need to think crit­i­cally about which mean­ings we are com­put­ing on, and which we are ig­nor­ing.