There are a few words which are used vastly more frequently than they are defined. Discussions using these words can go on for years, if not millennia, generating more heat than light. “Consciousness” is one1. So is “Capitalism”.

I’ve noticed another: “Bias”.

Discussions of bias are usually animated by the understanding that bias is bad, rather than any account of what bias is. Into this definitional void spill the assumptions of our egotistical age: that bias is a property of individuals, that it manifests at the point of choice, and as a discrimination on some innocent dimension of another person.

This is the prototype of old-fashioned prejudice: no Irish allowed in the guest house, no Black people in the swimming pool, no women in the university. These are the biases of racism, sexism, etc which increasingly stand out as social norms have shifted. Individualised responses to the ‘isms’ ask us all to do the work to decontaminate our attitudes, digging deeper and deeper - perhaps into their unconscious - on this quest.

This sense of ‘bias’ is common in social psychology. Groups, and group identities, provide expectations about members of other groups. Signals of group membership trigger expectations. Bias, in this sense, is socially rich, emotionally “hot”. And unambiguously bad.

From cognitive psychology - the branch of psychology concerned with memory, reasoning and judgement - we get a slightly different version of bias. This is the bias of the “judgement and decision making” tradition, the research programme of “Thinking Fast and Slow”, and which later informed behavioural economics, nudge and popular accounts like “Predictably Irrational”. Bias, in this tradition, has become an ever-growing family of biases: the sunk-cost fallacy, peak-end rule, loss aversion, anchoring, the halo effect, and on and on.

The biases of cognitive psychology are still individual, but they are framed as information processing choices, not social choices. It is also more ambiguous whether cognitive biases are good or bad. Researchers in the area emphasise that they are adaptive - shortcuts and assumptions which are necessary given the uncertainty and urgency of real-world decisions, part of the toolkit that allows us to be smart, flexible, decision-makers in a confusing world.

And yet, the odour of badness still clings to cognitive bias.

Part of this is because the method cognitive psychology uses to identify biases relies on carefully constructing scenarios in which our choices can be shown to be mistaken or inconsistent. This focus led one commentator to describe Kahneman’s account of human nature as “we basically run around failing all the time”.

The truth to this grows as the original research gets translated into other areas, business and popular media; TED talks and self-help books. Biases are fun because they are mistakes and talking about them dangles the promise that we can cleanse ourselves of these impurities (or, perhaps, identify which of our fellows carries too large a stain and cut them out). The title of Kahneman’s book was Thinking Fast and Slow. What is received of that book is often that Fast ( = biased) thinking is the problem. If only we could slow down and let the rational part of our minds have control, rather than the biased, quick, part.

There is a deep reason for the implication of error and failure, stemming from the way bias is defined in cognitive psychology. The identification of bias must necessarily assume a standard against which human judgement fails. Many of the original studies of the field used principles of logic or probability, which at first seem unimpeachable as standards we ought to follow.

As an example, take the Gambler’s Fallacy: if the tosses of a coin are independent, and I throw Heads, Heads, Heads and Heads again, what this is the chance the next throw will be Tails? The correct answer is that the chance is 50%, just as it was for all the previous throws and will be for all the subsequent throws. Each throw has the same odds, regardless of the outcome of other throws. Against this answer, our intuition that a Tails is more likely following a string of Heads shows up as a bias.

The question, though, assumes more than the law of probability. Any real world choice is defined by more than the rules of logic and probability, even as those rules remain absolutely true in abstract. Additional assumptions need to be made to declare that the rule, and only the rule, of independence applies. We must assume that the coin is fair, that the scenario isn’t part of a con, that the description given by the researcher, even, is honest and complete. Against this, we might reasonably ask, how often are events truly independent in real life? And, if two people saw 9 heads in a row and one insisted the odds of Heads on throw 10 were still 50/50, who would you think was the smarter of the two?

As the family of biases grows, clarity diminishes about exactly which rules define the correct choice, and how. Some rules must be assumed in order to define what constitutes a bias. The focus of research is often on the errors made, rather than the standard against which they are judged, allowing researchers to smuggle in assumptions about what defines correct behaviour.

Research on cognitive bias brings to the understanding of human psychology simultaneously a restricted view of optimal behaviour, and an implication that we are all, somehow, mysteriously and inevitably, failing against impossible standards. Researchers don’t say this out loud for the most part, you have to pick it up from the way the research is recycled and deployed in common discourse.

The upshot is that our discussion of bias is often limited by the dominance of these two accounts of bias from social and cognitive psychology. There are various ways we can expand our account of bias - beyond being a property of individual choice, to a property of systems or structures; beyond being the result of arbitrary preferences or tastes; and beyond being indelibly bad. This last point is important, because if you view bias as a contaminant which must be removed, you condemn yourself to try to purify human decision making, a hopeless task which is only possible in the same abstract realm in which laws of logic and probability are defined.

There is another tradition in the psychology of decision making, one which offers an alternative account of bias.

Signal Detection Theory evolved out of wartime studies of radar operators. The Theory characterises decisions as detection or discrimination problems. Faced with uncertain information (fuzzy green blobs in the case of radar operators) the task is to correctly judge when something is there (for radar, an airborne threat). This characterisation has far wider application than just radar: whether you are judging if a food is safe to eat, or a colleague safe to trust, if a news story is true or if the sound you just heard was your name being called (or not), signal detection theory applies.

Signal Detection Theory identifies four possible outcomes for decision, defined by the truth of the world, and the outcome of one’s judgement. There are exactly two ways of being right:

• When something is true, and you say it is true: Hit

• When something is false, and you say it is false: Correct Rejection

And exactly two ways of being wrong:

• When something is false and you say it is true: False Alarm

• When something is true and you say it is false: Miss

A fundamental insight is that the possible errors trade off against each other. You can avoid most false alarms, if you accept more misses, and vice versa.

Following from this, any decision maker can be characterised by two parameters - their ability to discriminate (tell true from false) but also their bias (their leaning towards one or other of the two possible error types). Crucially, it is not possible to judge ability to discriminate by looking at only successes. You might correctly call 100% of inbound fighter planes (or spot 100% of lies told to you, or detect 100% of poisoned food), but if you also call 100% of random blobs planes (and 100% of honest statements lies, and 100% of safe meals poisoned) then you are not brilliant at discriminating, you just have an extreme prejudice away from misses and towards false alarms.

To successfully compare different decision-makers you need to estimate both their ability to discriminate and their bias. Each decision maker has a bias - they have to have a bias - which reflects the balance they have chosen between making the two possible mistakes.

What’s the optimum bias? Well that depends on your beliefs about the prior likelihood of the possible outcomes, and on the costs and benefits of the different classes of mistake. You can see this most clearly in the example of a fire alarm - a simple decision making device which discriminates between two cases: fire or no fire.

If the alarm sounds when there is no fire (a false alarm), it is annoying. If the alarm fails to sound when there truly is a fire (a miss), then people may burn to death. The error cost-benefits are not symmetric, making it clear why false alarms are so common. We’ve all cursed alarms for going off and reassured ourselves “it’s never a real fire”, but this apparent unreliability is actually a feature of the bias-tuning which is designed to avoid, at the cost of more false alarms, those fatal misses when there really is a fire but the alarm doesn’t sound.

Considering fire alarms shows a sense in which bias-free isn’t an option. If you build an alarm you have to adjust the sensitivity so it is tuned somewhere between erring on the side of sounding when there is no fire versus erring on the side of not sounding when there is a fire. This tuning is the bias. Your understanding of the costs and benefits of different errors might make you adjust the bias, but there is simply no option in which the system can not have a bias at all (even a bias of zero is a bias, incorporating an assumption that the two kinds of errors are equally costly).

The attraction of the Signal Detection Theory account of bias is that it demands you understand bias as both inevitable and as the outcome of a trade-off. Perfect decisions don’t exist, so you have to pick a holding position. This encourages us to look outside of individual decision and the individual decision maker, beyond individual errors, to try to understand the unseen (the classes of error avoided, the perceived costs and benefits from different kinds of errors).

This perspective also discourages us from imagining there is a bias free decision process, or a bias-free decision maker. This recognition is a first step to principally addressing biased decision making and making it fairer, or more consistent, or more principled.

Bringing this back to discussions which use the word bias without defining it, we have a number of questions we can ask which might move the conversation on.

Is bias used to mean “bad influence” (social model)? If that is the case you might usefully ask where the influence is, is it in the decision-makers at all, or is it in the information provided to them? Too much psychology (and I speak as a psychologist) can lead us to overestimate the frequency with which bad outcomes are due to biased preferences by decision-makers.

Is bias used to mean “flawed” or “irrational” decision making (cognitive model). Then we might usefully ask what standard is being applied, if it should be, and even if the ideal standard is appropriate, is it possible for any reasonable person to meet it.

We all aspire to error-free decision making. It’s a good aim, but dealing realistically with errors means deflating some of the moral connotations and recognising their inevitability. When we come to think about the trade-offs inherent in different classes of mistakes, Signal Detection Theory is an excellent starting point, and will help us think about how costs and benefits feed into what looks like bias.

This newsletter is free for everyone to read and always will be. To support my writing you can upgrade to a paid subscription (more on why here)

Upgrade to paid

Keep reading for the references and other things I’ve been thinking about

§

I wrote about Signal Detection Theory in another context recently:

• Helping people spot misinformation.

This paper shows the benefit you get from full formal (computational) modelling of decision processes (including estimating discrimination ability and bias):

• Stafford, T., Pirrone, A., Croucher, M., and Krystalli, A. (2020). Quantifying the benefits of using decision models with response time and accuracy data. Behaviour Research Methods, 52, 2142–2155. doi.org/10.3758/s13428-020-01372-w. [web version here].

This paper was never published in a journal, but says a lot more about bias and strategies for dealing with bias:

• Stafford, T., Holroyd, J., and Scaife, R. (2018). Confronting bias in judging: A framework for addressing psychological biases in decision making. https://doi.org/10.31234/osf.io/nzskm

The observation that bias is often not defined, or badly defined, is not a novel one. This paper from two RoRI colleagues takes a causal inference approach: Causal foundations of bias, disparity and fairness. Other approaches include legal definitions of bias (which, from my limited understanding, shade closer to something like ‘conflict of interest’), and, from what I’ve overheard, there are approaches within economics which also unpack the various possible meanings (which I’d love to hear more about if anyone knows a good reference).

Ian McKellen performing The Strangers' Case.

Everything is in Shakespeare, it seems.

…And this calls out across 400 years to the current moment:

"And that you sit as kings in your desires,
Authority quite silent by your brawl,
And you in ruff of your opinions clothed;
What had you got? I’ll tell you. You had taught
How insolence and strong hand should prevail,
How order should be quelled; and by this pattern
Not one of you should live an aged man
For other ruffians, as their fancies wrought,
With self same hand, self reasons, and self right,
Would shark on you, and men like ravenous fishes
Would feed on one another."

More from Kottke

Xikipedia - a version of Wikipedia you can doomscroll

Web: https://xikipedia.org/

You can install as an app (load page and click “install”)

… And finally

via @mhoye

END

Comments? Feedback? Pick a card, any card? I am tom@idiolect.org.uk and on Mastodon at @tomstafford@mastodon.online

A new study has been released, and it beautifully illustrates a fundamental issue in making behavioural science reliable and interpretable. Possibly it illustrates, a fundamental issue with making any science reliable and interpretable.

On Friday I was invited to talk at an event at the University of Liverpool “Does AI threaten research integrity?”, co-sponsored by the terrific UK Reproducibility Network (UKRN). For my talk I wanted to start with considering how various actors in the research system define research integrity, and to share my take on the ‘special sauce’ that defines UKRN’s approach here.

All research has a problem with reproducibility, broadly defined. Some fields, like psychology and behavioural science, have a well publicised problem with phenomena which are reported, nay celebrated, but are just not real1. Other fields have less of an issue with that. In civil engineering, I imagine, if researchers say that a particular method will keep a bridge from falling down the world finds out pretty quick how right they are. Still, in those fields, there are still a host of issues about the reliability of published research, the ability of researchers elsewhere in academia or industry to understand and audit what was done, and for anyone (including the original researchers) to repeat it, scale it and/or translate it into new contexts.

Defined like this, reproducibility applies across all research endeavours. Whether you are researching more efficient manufacturing for widgets or marriage networks in Medieval royalty, you still need to be able to discover the truth and effectively share it, and that means sharing in a way that reliably signals credibility, as well as transmitting the raw result. You don’t have to buy into the idea that there is a reproducibility ‘crisis’ in your field to see that increasing reproducibility is needed.

The UKRN, and the wider global network of Reproducibility Networks, play a unique role here, coordinating researchers themselves to articulate and bring about improvements in research quality. This, I argue, is an essential ingredient in the special sauce of the UKRN: it is focused on ‘research improvement’, not a more narrowly defined ‘research integrity’. We could argue the semantics, but in many contexts research integrity has been captured by the governance issues of dealing with research fraud and other severe malpractice.

Those things are bad, but an essential insight is that you don’t need to be actively trying to make dishonest, unreliable or irreproducible research - you get these things for free simply because honesty, reliability and reproducibility are unreasonably hard to achieve.

This was brought home to me by the “Many Analysts” study I was involved with, in which 29 research teams were all given the same data and the same research question. Lining up each team’s results there were not just 29 different answers, but significant range - the answers varied from saying that the effect was negative (left of the figure below), that there was no effect, to there was a significant positive effect (right of the figure).

More than this, each team used a different statistical model, and followed a different data preprocessing procedure before estimating the effect.

The moral was well expressed by the paper: “significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions.”

Most research has just one analysis team and reports a single, best, estimate. If you do this, how do you know if you are an outlier, like the teams at the extreme ends, or somewhere the middle. You can’t!

The implications for the research literature are profound. Honest, dedicated, skilled researchers may investigate a topic and come to opposite conclusions because of variation between how they conduct their analysis - never mind the variations in how the data to be analysed is collected.

§

Now a new report by Annika Külpmann and colleagues - Methodological Flexibility in the Iowa Gambling Task Undermines Interpretability: A Meta-method Review - has been published, which gives another glimpse of just how deep this problem can be.

The Iowa Gambling Task is a classic test of decision making ability. It is used widely in medical research, but also in healthy populations to look at differences in decision making. It’s the kind of task every psychology student will have heard of. In the Task you are presented with four decks of cards, with each card delivering a reward or penalty. You are asked to draw cards from the decks, trying to accumulate as many points as possible. Unbeknown to participants, for two of the decks the average payout is negative - while most cards might give out a reward, these decks also contain penalty cards which over time outweigh any benefit of drawing from these decks. The original designers of the task thought that if you had intact frontal lobe function, you should learn to avoid the “bad” decks and prefer to choose from the “good” decks.

The Iowa Gambling Task has been used in literally thousands of research studies, which is why the result of the Külpmann et al review is so surprising. We hope that the more popular methods in any field will be the most reliable. Not so.

Külpmann et al started by picking 100 studies at random from a very long list of published studies. They found that there were hundreds of different versions of ‘The’ Iowa Gambling Task. Different research teams varied the instructions given to participants, the size and frequency of rewards and punishments, the order of the cards in the decks, the number of trials, and many other things. So extreme was this variation that Külpmann et al say the Iowa Gambling Task should be considered a “franchise”, not a single task, something that is undergoing constant adaptation for different purposes.

Obviously, if the Task is actually many different Tasks this makes it hard to draw conclusions across different studies. If studies have different findings it could just be because they involved different tasks and called them the same thing. Compounding this, Külpmann et al found that none of the meta-analyses they looked at - studies which are meant to synthesise across the published literature - used procedural variation as a criterion for including or excluding studies in their review. The meta-analyses assumed all these procedural variations didn’t matter. A big assumption!

Adding to this mess, Külpmann et al found that many studies didn’t even include enough detail to know exactly how they had varied the task when it was implemented - this was true for 65% of clinical research papers which used the task. Yes, 2/3rds of studies which reported findings using some variation of task didn’t contain the information which would let anyone else repeat what they did.

The most damning part of the Külpmann et al report concerns their analysis of different scoring methods. Even for researchers running identical versions of the Task, there is still flexibility in how you calculate performance on the task. You can count the choices from good decks vs bad decks, or count the points. You can look at all choices, or just the choices in the second half, or you could calculate a learning curve over successive trials. When you have a score, you can transform it in various ways to control its statistical properties (imposing maximum or minimums for example). Or you can do all of these things in some combination.

Incredibly, Külpmann et al found, across 100 studies, 244 different scoring methods reported. Over 200 of these were mathematically distinct.2 This incredible variety is a playground for individual researchers to fish around for a scoring system that ‘performs better’ - in other words, which gives them the results they hoped to get from running the task.

If the different procedural variants and the different scoring methods all gave very similar results, this variation wouldn’t matter. This was not the case.

By using a large, open, dataset of Iowa Gambling Task data Külpmann et al were able to run these 200+ different scoring methods on the same data. They found that choice of scoring method could vary a result in any direction, to any extent. Here is their plot of the correlation of the different methods against the the single most common (modal) method:

The correlations (x-axis) go from near 1 (perfectly matches the modal method), through 0 (completely independent of the modal method) to near -1 (the perfect opposite of the modal method). Yes, that means that you could - by careful choice of an existing scoring method from the literature - find any effect, or nothing, or the reverse of any effect you choose.

This is bonkers. It’s like if you went to see a doctor and they had thermometer which, depending on details which they won’t reveal to you, could tell you your temperature perfectly, give you a random result, or be the opposite of correct temperature, and the doctor themselves didn’t know which one it was.

The situation is so extreme as to be farcical.

§

The solutions to this mess are well rehearsed: transparent reporting, data sharing, protocol standardisation, measurement validation (see Flake and Fried, 2020, for example).

The worry is that if it is this bad under this particular empirical rock, it might be just as bad in other parts of behavioural science, or beyond.

The lesson of this, and many other studies, is that good intentions are not enough to deliver consistency and validity. Research studies are complex objects, complex objects where the details matter. We can’t rely on academic conventions for carrying out and reporting research to deliver credibility and reliability at the population scale. This is the reproducibility problem

And if you think your own favourite research area is problem free, ask yourself how you know that, and how systematically people have looked.

This newsletter is free for everyone to read and always will be. To support my writing you can upgrade to a paid subscription (more on why here)

Upgrade to paid

Keep reading for the references and other things I’ve been thinking about

§

References

The Iowa Gambling Task study:

Külpmann, A. I., Ries, J., Hussey, I., & Elson, M. (2026, January 24). Methodological Flexibility in the Iowa Gambling Task Undermines Interpretability: A Meta-method Review. https://doi.org/10.31234/osf.io/4g3vr_v1

The Many Analysts team study:

Silberzahn, R. et al (2018). Many analysts, one dataset: Making transparent how variations in analytical choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337-356.

More on measurement validity:

Flake, J. K., & Fried, E. I. (2020). Measurement schmeasurement: Questionable measurement practices and how to avoid them. Advances in methods and practices in psychological science, 3(4), 456-465.

You can find my slides from the event in Liverpool here (reading the slides may be the ideal way of consuming the talk, since you get full references to all the things I talked about, but with none of my commentary). Thanks to the organisers for inviting me and to everyone who attended in person and online.

Grokked

Last time I reported some research on the Wikipedia-alternative, Grokipedia. This story illustrates another motivation a billionaire might have for building an alternative encyclopedia - so as to be upstream of the information LLMs rely on.

• The Guardian 2026-01-24: Latest ChatGPT model uses Elon Musk’s Grokipedia as source, tests reveal.

Although there are reports that propaganda sites generally just fill information gaps, where conventional media haven’t captured the keyword space, rather than successfully usurp mainstream media:

• LLMs grooming or data voids? LLM-powered chatbot references to Kremlin disinformation reflect information gaps, not manipulation

Kate Starbird is now on Substack

And anyone who cares about making sense of our information environment should probably follow her:

Sensemaking Distortion

Reflections on Influence, Frames, and Fractured Information Spaces

By Kate Starbird

Henry Farrell: Davos is a rational ritual

Henry Farrell analyses Davos as a stage on which the powerful coordinate over the creation of common knowledge.

Link: Davos is a rational ritual

Lord of the Rings, redux

Lols at this meme riffing of the well-known sycophancy of commercial language models:

Casey Newton: What a big study of teens says about social media — and what it can’t

Casey argues that we don’t need a study to know that the online environment is designed to be exploitative of our worse instincts, and that big tech has been persistently, culpably, shamefully negligent in protecting younger users:

"The next time you go to Las Vegas, you’ll notice that there are no 13-year-olds in the casinos. The reason is not because a series of longitudinal studies proved to the satisfaction of the gaming industry that gambling causes anxiety and depression. Rather, there are no 13-year-olds in casinos because we know that the environment is designed to exploit them."

Link: What a big study of teens says about social media — and what it can’t

Daisy Christodoulou: How to write a good rubric (for humans and AI)

Daisy Christodoulou is lucid on the perils of designing a reward strategy [in assignment marking terms, a rubric], with too much of a negative focus. To avoid perverse incentives you need to include what you actively want, not just what you will punish, and you need to give those who do the assessment some discretion. She shows how this applies - but doesn't only apply - to education.

Link: How to write a good rubric (for humans and AI)

… And finally

END

Comments? Feedback? Pick a card, any card? I am tom@idiolect.org.uk and on Mastodon at @tomstafford@mastodon.online

Last time, I made the claim/ shared the worry that AI may erode the very knowledge-generating infrastructure it relies on (to produce training data). As if to illustrate this, I came across this blunder from Grokipedia, the AI curated ‘anti-woke’ encyclopedia project from Elon Musk: Grokipedia misreports the death of one of the foundational researchers in NLP, Kenneth Church.

I’m reliably informed that Prof. Church takes in good humour this declaration by one of the products of the field he helped define. Nonetheless, there’s definitely some kind of Frankenstein’s monster stuff going on here.

Here’s the page, and here’s a screenshot in case it changes:

“Fact-checked by Grok 2 weeks ago”

If you read the whole article, it is clear that it conflates two individuals, one of whom may actually have died of COVID-19 in 2020. So you get this bizarre piece about a famous jockey who retired from horse-riding and took up a prestigious research career at Bell Labs.

I’m interested if the article gets corrected, because there doesn’t seem to be an obvious way to do this yourself (or even request a correction). When I first viewed the page there was an EDIT button which invited you to log in, but when I did the button disappeared!

Wikipedia is better not just because it is more correct about Kenneth Church, but because it contains a mechanism for correction, and a community behind it to provide that correction. Many errors on Wikipedia are corrected in minutes, especially any that draw significant attention.

Grokipedia may be right or wrong, but when it is wrong it isn’t clear how it can be corrected to be right. And that, ultimately, is why I’ll go on trusting Wikipedia more.

Interestingly, Grok (the chatbot) can spot the error on Grokipedia. When I prompted it, it replied

“Grokipedia (being the AI-generated, “fact-checked by Grok” alternative encyclopedia) has simply smashed their two biographies together under a single page title with no disambiguation, no warning, and no attempt to separate them.

I love that even Grok puts “fact-checked by Grok” in scare quotes.

This also illustrates something fundamental about AI - it does not have any shame, nor, it seems, the same need for coherence in beliefs that humans do. Rightly or wrongly we try to strive for consistency with ourselves. AI will happily contradict itself from moment to moment (or even sometimes in the same sentence).

Here’s a great example from Ian Leslie. The same query, just phrased differently, gets completely opposite opening responses:

Now obviously consistency may be overrated, and humans are vulnerable to these kinds of manipulation by the way questions are phrased too. Is it, however, too fanciful to think that our self-respect and sense of shame prevents us leaning into them to the same degree?

If we carry over our intuition for human minds, that there is some ‘self’ there in the AI which we can talk to, discover the truth about, correct and/ or shame into better behaviour we’ll be misled. There’s no There, there. The AI is an alien mind which has weird edges. Treat with caution.

PREPRINT: @Grok Is This True? LLM-Powered Fact-Checking on Social Media

This said, it looks like people like Grok, on X, and about 5% of its use is for fact-checking. They like it so much it is drawing energy away from the Community Notes moderation system:

Link: @Grok Is This True? LLM-Powered Fact-Checking on Social Media (2025), T. Renault, M. Mosleh and D. Rand

There’s a lot more to say about LLMs for fact-checking, but the popularity of Grok illustrates both a strength (people like interacting with AI via natural language), and a weakness (AI can eat alternative systems for producing knowledge).

Dylan Matthews: Pro-social media

Matthews makes the case that a) people mostly use LLMs for fact-checking b) LLMs have a tendency to non-customised responses. The upshot:

My provisional theory is that LLMs, as a consumer product, will push people’s senses of reality closer together in a sort of mirror image of the way social media has fractured them. They are not algorithms meant to custom-tailor content (including facts) to you, and what you will find infuriating or motivating. They are centralized systems that, until you prompt them or give them context, behave basically the same way for everyone.

Matthews illustrates his case with Grok’s response to queries about the Renée Good shooting, which quickly became highly polarised. Grok, despite its engineering, didn’t lean towards the right-wing interpretation.

The idea is supported by the Renault, Mosleh and Rand preprint (above). This shows that Republicans used Grok slightly more than Democrats, but when they do it is more often to check Republican posts than Democratic posts. Supporting Matthews’ hypothesis of LLMs-as-consensus-machine, Republican posts were more likely than Democratic posts to be rated false by Grok, and Grok tended to agree with another AI model (Perplexity), with Community Notes and with professional fact-checkers.

Consensus on facts!

The big surprise of LLMs may be that the inertial force of all the text they are trained on produces a conservative, consensus-orientated, force. Not quite the truth, but definitively truth-adjacent, and a long way from the ‘pick your own’ facts nightmare many people fear the digital age will bring.

Link: Pro-social media

This newsletter is free for everyone to read and always will be. To support my writing you can upgrade to a paid subscription (more on why here)

Upgrade to paid

Keep reading for more research on AI and social media

PREPRINT: Industry Influence in High-Profile Social Media Research

From the abstract:

Leveraging openly available data, we show that half of the research published in top journals has disclosable ties to industry in the form of prior funding, collaboration, or employment. However, the majority of these ties go undisclosed in the published research… we find evidence that industry ties are associated with a topical focus away from impacts of platform-scale features.

Covered by Kai Kupferschmidt in Science: Nearly a third of social media research has undisclosed ties to industry, preprint claims

Link: Industry Influence in High-Profile Social Media Research

PREPRINT: Fact-Checking and Misinformation: Evidence from the Market Leader

Fact-checking on Meta works - less circulation of false information, posts fact-checked as false more likely to be deleted, and those fact-checked less likely to post false information in the future.

Read all about it in the essential write up found in Indicator: An 18-month study found that fact-checking on Facebook reduced engagement with false info

PREPRINT: When” Likers’‘Go Private: Engagement With Reputationally Risky Content on X

This is a surprising result, which uses a nice study design (difference in difference, for strong causal inference fans).

In 2024, the X platform changed so that “Likes” were private. Prior to this, anyone who could view your account could inspect which posts you’d Liked (resulting in some very small-minded cancellation adjacent behaviour, in my opinion). This study compared “high reputational risk” accounts (extremists, politicians, OnlyFans) with “low reputational risk” accounts (actors, singers, tech influencers), and showed there was no discernible change in how many Likes their posts received before and after the change.

This is obviously contrary to the prediction that social concerns - including risk of shaming - drive a lot of social media behaviour. The study also includes a survey in which X users report that whether their Likes were public would indeed change their behaviour. So the result is all the more surprising.

The failure to detect a difference doesn’t mean that some users didn’t change their behaviour, as the authors discuss. It might be that most Likes are driven by bots, which are immune to shame, but I think the study design is strong and we should note when our expectations are contradicted.

Chuai, Y., Ribeiro, M. H., Lenzini, G., & Pröllochs, N. (2026). When” Likers’‘Go Private: Engagement With Reputationally Risky Content on X. arXiv preprint arXiv:2601.11140.

PREPRINT: Natural language communication enables groups to overcome unrepresentative private evidence

Nicely framed experiment, showing that allowing people to talk - rather than just report what they see via a number - allowed more effective group performance. In particular, “private evidence” is what each person knows individually, and which a successful group needs to combine across individuals to access the benefits of being in a group. Prior research shows that groups tend to under-share private knowledge (preferring to discuss common knowledge) leading them to forego the “gains from trade” of pooling the different information available across all group members.

By contrasting natural language communication in a tight, controlled, way with communication via a slider scale, this experiment shows the communicative information we are able to pack into even the briefest natural language communications (compare for example the way you would interpret differently “It’s 73” versus “I think it is 73”).

Of course, this caught my eye because LLMs promise to take advantage of the finely honed communications channel of natural language.

From the abstract:

Here we test the epistemic conditions under which language may help or hinder collective inference. In two experiments (N = 2,249), networks of four participants observed private samples from an underlying probability distribution and communicated over repeated rounds to estimate that distribution. We manipulated communication channel (natural language vs. numerical belief reports) and systematically varied the quality of private information: total sample size, representativeness of individual samples, and how evenly information was distributed across participants. All groups improved through social interaction, but groups restricted to numerical transmission performed systematically worse. Critically, the advantage of language emerged specifically when private samples were unrepresentative of the underlying truth, precisely when learners must appropriately weight others’ evidence against their own. Analysis of message content revealed that this benefit arose from participants’ ability to relay information across the network, extending the reach of evidence beyond direct dyadic exchange

Zubak, Y., & Hawkins, R. (2026, January 16). Natural language communication enables groups to overcome unrepresentative private evidence. Retrieved from osf.io/preprints/psyarxiv/mp2yf_v1

…And finally

RIP Hajime Ouchi 1945-2024, originator of the Ouchi Illusion

It works particularly well as you scroll past, I notice, so I hope everyone reading on mobile enjoys.

Ouchi’s death was announced last week to the vision science community by Lothar Spillmann on the cvnet listserv

More on the illusion from Spillmann: https://doi.org/10.1068/p7384

For an accessible introduction, it’s discussed in Chapter 30 of Mind Hacks https://mindhacks.com/book/

END

Comments? Feedback? Requests for topics to write about? I am tom@idiolect.org.uk and on Mastodon at @tomstafford@mastodon.online

The astounding success of large language models is due to the quality of the training data available - books, newspapers, online forums, Wikipedia, but also the code bases from open source software projects. All that has been shared and discussed online goes into these things.

Although the algorithms behind these models are wickedly clever, they fundamentally don’t produce intelligent behaviour without this data, and which behaviour they do produce is conditioned by what they are trained on (hence the tendency of these models to reproduce stereotypes - such as surgeons tend to be men - since these are represented in the data they are trained on).

Generative AI models produce new outputs, by definition. So once you have one of these models, you can produce infinitely more training data. Does this let you bootstrap your way to superintelligence?

Quite the opposite.

If you train a complex model on its own output, you get a phenomenon which has been termed model collapse - over successive iterations the model focuses more and more on the most common or typical patterns. Measures of performance may even show improvement initially, but incrementally the model loses the ability to represent edge cases and the uncommon. Eventually the model performance collapses as it generates similar and inappropriate output to all inputs.

Here’s an example from the classic paper (Shumailov et al, 2024). A language model (trained on human generated data) responds to a prompt about churches:

Revival architecture such as St. John’s Cathedral in London. The earliest surviving example of Perpendicular Revival architecture is found in the 18th @-@ century Church of Our Lady of Guernsey, which dates from the late 19th century. There are two types of perpendicular churches [output cut off after a limit]

After nine generations of using the model to produce data which is then added to the training data for subsequent generations, typical output to the same prompt is:

architecture. In addition to being home to some of the world’s largest populations of black @-@ tailed jackrabbits, white @-@ tailed jackrabbits, blue @-@ tailed jackrabbits, red @-@ tailed jackrabbits, yellow @-.

Repetitive, nonsensical, maniacally focused on one aspect of the training data, and a response that isn’t appropriate given the input prompt.

Guo et al (2024) have a nice illustration of the mechanism behind this. Like Shumailov et al, 2024, they trained models on their own output, then visualised the space of how inputs were represented by the model - a mapping of close or distant possible words in the semantic space of the model. In the way that ‘cat’ and ‘dog’ should be close in semantic space, and ‘cat’ and ‘astronaut’ should be less close, this shows how the universe of concepts is represented by successful generations of models.

Over time the way the model ‘thinks’ about inputs narrows, reducing the ability to make distinctions - collapsing every input so that they all seem more similar. This is why performance degrades. Good models provide different outputs depending on what you ask them. Sometimes a subtle change in input can produce a large change in output. As the models lose sensitivity to this they stop being useful for anything.

So as well as meaning we can’t bootstrap our way to superintelligence, model collapse also poses a definite risk. To avoid model collapse (or just model decay), we want to keep AI generated content out of the training data we use. Obviously, if you go to the internet of 2026 for your training data, this is impossible. Some have even compared the public release of ChatGPT to an epoch-defining pollution event - from now on, any new text you see may have been generated by a model, and you can’t know which. Model collapse can occur even if a proportion of the training data is synthetic, it doesn’t have to be all of it.

AI companies work very hard to generate new human data to fuel their ambitions. Researchers have already suggested we need to identify and preserve data which has no chance of being generated by AI, in case we need it to provide clean training data for future models.

* * *

Keeping our training data clean may be even harder than just excluding directly model generated output. Something similar to the early signs of model collapse can be seen in populations of humans who have been influenced by language models.

A result published in August 2025 analyses 83,160 student essays submitted as part of admission applications to a “highly selective private university on the East Coast”1. For cohorts of essays from each year 2019 - 2025 they calculated measures of the semantic diversity at the word, sentence and document level. Following the public release of ChatGPT there was a notable drop in the document-level semantic diversity - roughly a measure of how much variation there was between the essays in the themes discussed. Linked to the same inflection point, there was an increase in the word-level semantic diversity. Together this suggests that access to ChatGPT made student essays use a wider range of individual words, but actually narrowed the kinds of things discussed by them2.

I’ve seen this result replicated for MSc theses in an elite European University - a reduction of diversity of ideas around the time that large language models became widely accessible. We all know of students, and professionals, copying text from language models (obviously!). Anecdotally there are even reports of people picking up the phrases and verbal tics of language models.

A small study which asked participants to do creative idea generation (example challenge “Suppose that a great fog has fallen over the earth and all we
can see of people is their feet. What would happen?”), found that people who used ChatGPT to help them with the task produced more, and more detailed ideas, but these ideas were more similar across individuals and people felt less ownership of the ideas produced.

It’s natural that as we use and rely on language models in more and more of our work, those language models affect our own output. Humans have a natural tendency to mimic, so it shouldn’t be a surprise if our, human, output comes to mimic the most human-like tools we’ve created. Since everyone is using the same small family of models, it shrinks the variation between individuals, making us sound more and more like each other.

* * *

Humans and models are now an interacting epistemic system. The models learnt from human outputs, now humans are learning from the model outputs. This is affecting the text we see in the wild, but also, plausibly, human habits of thought and expression. Perhaps even more profoundly, the feedback loops may be eroding the institutions which have been established to curate human knowledge.

In October 2025 Wikipedia reported two stark, and related, trends. There are fewer new human readers, probably because people are using language models to get information. At the same time, the site is getting increasingly hammered by bots which are scraping the content, probably to build new AI models. Some human readers become human editors, the engine which has driven Wikipedia’s success over the last 25 years. Without editors, Wikipedia wouldn’t be able to adapt to incorporate new events and knowledge.

Before people who wrote code copied and pasted from ChatGPT, they copied and pasted from Stack Overflow, a question and answer site which allows people to ask technical questions, provide answers, as well as comment and vote on answers. It is widely viewed as both a vital source of training data for large language models, and something for which language models can easily substitute.

Here’s a graph of new questions posted on Stack Overflow, annotated by Emil Jacobs. It shows, over the lifetime of the site, the dramatic rise as people start using the site to ask questions, and the dramatic fall, as they stop.

There are caveats: Stack Overflow has other issues, and the actual traffic to the site hasn’t declined as dramatically as new questions. However, the ability to answer questions is upstream of people coming to the site for answers. Without new answers, users won’t come to Stack Overflow for help with the latest technology.

And, if Stack Overflow isn’t generating knowledge about new technology, what will the next generation of language models train on?

* * *

At first, we might not notice if our epistemic institutions are being eroded by language model use. As with model collapse, performance could initially improve as we find the answers we need using AI to help us be more productive and more focused. These models are capable of generating prolific output, but output which under-represents the true diversity of the world. Text is already a compression of experience. Model collapse could further drive the compression of the perspectives captured in text, meaning we all start to sound more and more like each other. If, in parallel, our reliance on models means we under-invest in our epistemic institutions then we will be accelerating towards a dead end - a world where we are unable to adapt to the new, unable to incorporate different perspectives, and over-reliant on an increasingly narrow repertoire of responses.

This is a scare story, but the warning is clear: as we use AI models, let’s keep asking ourselves if we are knowledge generating or merely output generating. Are we adding information to the world, or just words?

The AI age will mean it is easier than ever to give the superficial appearance of conveying information, and easier than ever to neglect the hard work of saying something valuable about ourselves or the world. It will also be all too easy to think we can get away without the work of sustaining the organisations we’ve built up to sort, store and aggregate new knowledge. We’ll miss ‘em when they’re gone.

This newsletter is free for everyone to read and always will be. To support my writing you can upgrade to a paid subscription (more on why here)

Upgrade to paid

Keep reading for the references, and the greatest obit I’ve heard in a long time.

Reference & More

Anderson, B. R., Shah, J. H., & Kreminski, M. (2024, June). Homogenization effects of large language models on human creative ideation. In Proceedings of the 16th conference on creativity & cognition (pp. 413-425). https://doi.org/10.1145/3635636.3656204

Guo, Y., Shang, G., Vazirgiannis, M., & Clavel, C. (2024, June). The curious decline of linguistic diversity: Training language models on synthetic text. In Findings of the Association for Computational Linguistics: NAACL 2024 (pp. 3589-3604).

Moon, K., Kushlev, K., Bank, A., & Green, A. (2025, August 20). Impersonal Statements: LLM-Era College Admissions Essays Exhibit Deep Homogenization Despite Lexical Diversity. https://doi.org/10.31234/osf.io/jsz58_v1

Shumailov, I., Shumaylov, Z., Zhao, Y. et al. AI models collapse when trained on recursively generated data. Nature 631, 755–759 (2024). https://doi.org/10.1038/s41586-024-07566-y

The Register, 2025-06-15: The launch of ChatGPT polluted the world forever, like the first atomic weapons tests

RIP Renfrew Christie

I’d never heard of Renfrew Christie before.

Born in 1949.

Growing up in apartheid South Africa, he was radicalised after being drafted into the army and discovering that the government was pursuing a nuclear weapons programme.

Picked a PhD topic - the electrification of South Africa - so he could study the infrastructure supporting uranium refinement for the weapons programme.

Started feeding information to the ANC, was betrayed by an informer, arrested and tortured.

Under torture, made a detailed confession which revealed all the targets and how he had planned bombings which were maximally disruptive of the weapons programme.

The judge read this confession out in court during the trial.

From jail, where he is placed on death row and spends years within earshot of the sounds of executions from the prison yard, Christie hears news that the ANC paramilitary wing have successfully carried out a series of bombings following the exact instructions contained in his confession.

The bombings set the nuclear programme back years and cost the government something like £1 billion.

“While I was in prison, everything I had ever researched was blown up,” he said.

The apartheid regime crumbled and Christie was pardoned, going on to become professor and Dean of Research at the University of the Western Cape.

A hero.

The New York Times: Renfrew Christie Dies at 76; Sabotaged Racist Regime’s Nuclear Program (2026-01-014)

Catch up

• Three things we learnt from a new megastudy of climate messaging

  Evidence on changing pro-environmental attitudes and behaviour

• Paid subscribers only. A thank you, some news and a preview

• The half life of trust. Or: You trusting something isn’t enough for me to trust it

• Our intuitions about political advertising are poor. Notes on a great interview with David Broockman

…And finally

via This is my Glasgow, a nice example of an naturally occurring optical illusion: “at Bingham's pond in the west of Glasgow. It might just be me, but the reflection of the shopping basket in the last of the ice makes it appear as if it's some sort of levitating cage.”

I’m trying to work out what exactly drives the illusion. Something about the shading on the ice gives a cue to depth? If you know, get in touch!

END

Comments? Feedback? Tasty training data? I am tom@idiolect.org.uk and on Mastodon at @tomstafford@mastodon.online

This post too long for email. Click on the header to open it in a browser and read the full thing

Out last week, A registered report megastudy on the persuasiveness of the most-cited climate messages. The study tested the effects of 10 well researched climate change messaging strategies on Americans’ pro-environmental attitudes and behaviour.

Science is a tapestry of evidence, where we shouldn’t put too much weight on a single study. This study, however, is worth paying some attention to, for the following reasons:

• It takes evidence from a lot of people. 13,544 to be precise, which is approximately 13,500 more than some studies of persuasion from the dark, unreliable days of psychology, pre-replication crisis.

• It has a large, diverse authorship. 25 authors by my count. Most importantly, these researchers come from different research groups, ensuring that the study design has input from people who don’t necessarily agree, and who have different favoured theories they want to support.

• I don’t know what exactly qualifies something to be a “megastudy”, but I don’t think many participants or many authors alone should count - the crucial thing is the attempt to provide something comprehensive. For this study, they first reviewed the literature (157 research papers on climate messaging), and from them selected the ten most cited messaging strategies for which there was evidence of effectiveness. By testing them in the same study they allowed, first, agreement on how each strategy should be implemented, and second, direct comparison of the effects. Here’s the description of the strategies from the paper (individual strategies bolded by me):

Six selected messaging strategies described supporting climate change mitigation as consistent with moral values and ideologies, such as care and considerateness, free market beliefs, belief in a just world, patriotism, purity and system preservation. Two messaging strategies emphasized that climate change is scientific consensus. The remaining two strategies described the costs for a socially distant group or gain associated with climate change mitigation.

• The study has a number of credibility indicators. Most importantly, and in the title, is that it is a registered report, a mechanism whereby the methods for a study are reviewed by the journal - which agrees to publish regardless of how the results turn out. This shifts the focus of journal review onto the importance of the research question, and the solidity of methods proposed for addressing it, rather than the results. You realise exactly how revolutionary the improvement this makes to the credibility of published results is when you appreciate the range of biases generated by the “review when you know the results” convention (basically positive existing results get published, negative results get file-drawered, and authors are incentivised to write up their studies as though they show a positive result, and what happened was what they predicted all along). So, basically, the registered report format meant the authors had every incentive to design a strong study, and reduced incentive to manipulate the analysis to present positive results, leaving us, the readers, with more faith that what they claim they show is representative1. They also share the anonymised data and analysis code, which is an important transparency measure and credibility signal.

So, worth spending some time with. What do they show?

1. There is bipartisan agreement on climate issues (in many ways)

This first thing isn’t emphasised by the authors of the study, but it is revealed by their results and worth comment. The study polled a sample of American adults, recruiting approximately as many Democrats and Republicans, so in this sense at least representative2

These plots show the volume of responses from participants of different political alignment. Here’s the one from belief in climate change. Although there is a famous partisan divide on this in the US, I’d argue that there is as much similarity as difference here. Most Republicans have more belief that climate change is happening than it is not (they are over 50% on the belief scale), the same as independents and Democrats (admittedly more Democrats express 100% certainty on this one).

Both Democrats and Republicans are similar in this intention to take “non-political” pro-environmental actions, such as “Eat less meat”, “fly less”, “compost”. Or, at least, they are similar in that many who identify as Democrat and many who identify as Republican have intentions to take pro-environmental action, and some from both groups have the intention not to.

The partisan groups really pull apart in how they interpret climate change and how they regard political choices in response to climate change. Republicans, for example, just seem a great deal less worried about climate change than Democrats:

You can check out the full results in the paper, but my takeaway was that - despite elite messaging questioning the reality of climate change - most of the US population doesn’t reject the reality of climate change, they just differ in what they think should be done about it.

Consequently, we shouldn’t assume that disagreements about climate policies are disagreements about the science of climate change (and if we do, that could be misdirection). There’s a similar story in the UK, where the reality of climate change and the need to do something about it has cross-partisan consensus.

2. Message targeting doesn’t matter

“Persuasiveness varied little across party lines”, report the study authors, “inconsistent with theories predicting heterogeneous effects for targeted messages.”

Put another way, messages were persuasive or not, it didn’t seem to matter who was seeing them, a Democrat or a Republican.

This chimes with what I reported political scientist David Broockman arguing in my newsletter from December, and what Alex Coppock argued in his book Persuasion in Parallel: How Information Changes Minds about Politics.

The idea of targeted messaging fulfils our hope for a magic bullet which will persuade those who think differently from us (and surely also plays well for the business model of campaign consultants), but the reality may be that the differences between different messages are larger (and easier to take advantage of) than the differences between how the same message lands with different demographics.

Although the study authors don’t rub this in, six of the ten messages they tested used a moral value reframing, inspired by the Moral Foundations school of thought. The lack of difference across Democrats and Republicans should specifically dampen our enthusiasm for how practical Moral Foundations is as a way of reaching people of different political persuasions. (That said, many good theories contain deep truths and are also entirely impractical, or impractical until applied right, but at the least this result shows that simplistic translation of Moral Foundations into persuasive messaging doesn’t allow special access to one partisan group or the other).

3. Persuasion is possible, just hard

The third thing I’d take from this study is that, consistent with a lot of the research I review in this newsletter, persuasion is possible. All groups of participants responded to some of the persuasion messages, but the response was typically small - a few percentage points.

Here’s just one of the outcomes they measured, attitude (on a scale of 0 to 100) towards climate mitigation policies (things like “Ban construction of new coal-fired power plants”).

The top strategy - a purity framing message (“Participants read a text arguing that mitigating climate change is consistent with values such as protecting the purity of American land.”) - looked like it had an average effect of moving people 1.5 points on the 100-point scale. In an election, that can be enough to matter, but in an election you also have the other side trying to push voters in the other direction. Does it matter in other contexts? Well you could say that it is impressive to get any movement in beliefs or attitudes from a short persuasive message. It proves people haven’t completely made up their minds on the issue.

That said, my native optimism about how reachable people are was given pause by what was perhaps the most interesting question the researchers asked: would participants be willing to give up a proportion of their payment for taking part as a donation to a pro-environmental organisation? No messaging strategy - not one - increased donations relative to the control condition (which was reading a non-persuasive message on a non-climate related topic, such as “the history of neckties”).

In fact, it looked like several messaging strategies decreased participants’ willingness to donate. I don’t read too much into this backfire effect. The donation measure wasn’t a planned part of the study. The most I’d take from it is the results are incompatible with any persuasion effect on expressed belief flowing through directly to behaviour (willingness to donate). Simply put, what people say is easier to shift than what they do (especially when cash is on the line).

In the words of the study authors, these results highlight “the limits of short-form messages for increasing Americans’ support for action to address climate change.”

Overall

We need more collaborative studies which test different approaches head to head. These particular results should shift our expectation down that it is possible to find bespoke messages for different political groups, without doing much to shift our prior expectation that, while always possible, persuasion is hard.

This newsletter is free for everyone to read and always will be. To support my writing you can upgrade to a paid subscription (more on why here)

Upgrade to paid

Keep reading for the references, more on persuasion, a recommendation for a great podcast and links to some recent things published elsewhere by me.

Reference & More

Voelkel, J.G., Ashokkumar, A., Abeles, A.T. et al. A registered report megastudy on the persuasiveness of the most-cited climate messages. Nat. Clim. Chang. (2026). https://doi.org/10.1038/s41558-025-02536-2

Related, by me

• Our intuitions about political advertising are poor. Notes on a great interview with David Broockman

• Language models are persuasive - and that’s a good thing. Two new studies provide insights into exactly how LLMs persuade, and what that means.

• The truth about digital propaganda. Reasonable People #55: Our piece in New Scientist bring evidence to worries about online manipulation

• How persuasive is AI-generated propaganda? Reasonable People #51: Bullet review of a new paper suggesting LLMs can create highly persuasive text and will supercharge covert propaganda campaigns.

Amy Mount: Political Heat

If climate policy and politics is your thing, please read Amy Mount’s reflection the last 12 months: 2025: When the UK’s climate consensus crumbled

The short version: emissions went down. Renewables and electrification went up. The climate narrative was muted. Wider politics got harder. Voters stayed volatile and parties were polarised. The general election in 2029 will be a defining moment for the UK’s climate progress. But I remain hopeful – because the public is onside. The question is, will it stay there?

And for deep dives on everything from the UK energy market to the Conservatives’ environmental legacy, Amy has you covered in the Political Heat podcast archives

Political Heat

• Podcast

• Substack

Full disclosure: Amy is my cousin, so I have some CoI

The Transmitter: The 1,000 neuron challenge

By me, in neuroscience magazine The Transmitter

A competition to design small, efficient neural models might provide new insight into real brains—and perhaps unite disparate modeling efforts.

Link: The 1,000 neuron challenge

WonkHE: A new model moves research in a more democratic direction – it could be faster and fairer too

By myself and Anna Butters. This is about research funding, which is an interesting and important decision problem. I’m proud of the way we wrote this to start very zoomed out on the big picture, then take the reader through to the specific mechanism - distributed peer review - about which we have new evidence on.

Here’s the start:

Research quality is the dark matter of the university sector. It is hard enough to assess research after it has been done, research funders must find some way to evaluate proposals for projects which don’t exist yet. The established model for this is external expert review, combined with a panel stage where proposals, and their reviews are discussed, and hard choices made.

UK researchers will be familiar with this via our own UKRI, and everyone who has had a funding application rejected will recognise that the reviews received may be partial or mis-directed. This speaks to the idiosyncrasy and variability in individual judgments of what makes a good project, and has downstream consequences for what ultimately gets funded.

Link: A new model moves research in a more democratic direction – it could be faster and fairer too

Preprint: Applicants as reviewers: A mixed methods evaluation of the risks, benefits, and potential of distributed peer review for grant funding allocations

Speaking of which, the preprint of our work on this has just been published. It’s a shorter (by 50%) version of the working paper, so not strictly new. Here’s the conclusion

This trial demonstrates feasibility of DPR [Distributed Peer Review] for an interdisciplinary funding programme. Implementing a DPR process requires funders to accept trade-offs, but can offer valuable benefits, including shortening time to funding decision. Building on established peer review principles, it has the potential to contribute to the democratisation of funding evaluation.

Citation:

Butters, A., Marshall, M. B., Pinfield, S., Stafford, T., Bondarenko, A., Neubauer, B., … Denecke, H. (2026, January 6). Applicants as reviewers: A mixed methods evaluation of the risks, benefits, and potential of distributed peer review for grant funding allocations. https://doi.org/10.31222/osf.io/t2p56_v1

…And finally

Tom Thomson. Northern River, 1914-15. (via @tarnport)

END

Comments? Feedback? Urban legends? I am tom@idiolect.org.uk and on Mastodon at @tomstafford@mastodon.online