An interesting paper was published over the weekend, arguing plausibly that ChatGPT is bullshit – in the technical sense defined by the philosopher Harry Frankfurt. Certainly its output exhibits ‘soft’ bullshit, meaning that it has no concern for truth or the reality of the world the text purports to describe, and there are at least some grounds for considering the much more controversial position that it displays ‘hard’ bullshit, an active intent to deceive its users, at least insofar as its creators present it and its capabilities in a misleading way, implying that ‘truthiness’ is in fact truth, as well as that a glorified autocomplete programme exhibits ‘intelligence’ in any meaningful sense.
What the paper neglects, in my view, is what could be called the rhetorical aspect of bullshitting; like most discussions of gAI, it focuses on the content of the output rather than its form. Clearly, however, part of the bullshitting power of LLM output is that it reads persuasively. Asked to comment on a historical topic (which obviously is what I’ve been focusing on over the last year – and this is why I haven’t posted anything on this blog in the last few weeks, as I’ve been trying to catch up with that project as well as finishing marking), it offers a series of declarative sentences making points that are clearly relevant to the prompt. There is no hesitation, uncertainty or equivocation, which creates, I would suggest, the impression of knowledge and authority. It echoes the confident assertions of the texts on which it was trained, which were however at least partly based on actual knowledge and understanding of the material (and one might suspect that, even if its training data included academic publications which might express themselves more cautiously on contentious issues, that gets stripped out in favour of more forthright declarations; I’m reminded of how much the Word grammar tool hates my sentences and thinks that they’re too wordy, and how it would clearly prefer a lot less caution and nuance…).
Accidentally or not, the resulting output sounds confident and authoritative. As I’ve suggested before, it sounds like a certain sort of student bluffing his way through an exam on the basis of half-remembered scraps of information and a plausible prose style – one of the archetypes of bullshitting. I think this is one reason why most students don’t in fact trust it to write their essays. As someone commented in one of my focus groups, it doesn’t write like a student – and the point is not the snark that, yes, it can place apostrophes correctly for a start, but rather that it shows no sign of the caution and humility (sometimes in excess), the sense that things are always more complicated, that undergraduates have learnt is appropriate. Only someone who really hasn’t been paying attention or done any work would think that a series of bold, bald and unsupported assertions is the way to write a decent history essay.
However – and this is what I increasingly worry about – ChatGPT output does sound like some of the sources that students might use for researching their essays, or even some of their lecturers. Its apparently authoritative tone may therefore incline them to trust its summaries of scholarly debates, paraphrases of pieces of scholarship and evaluation of their own work (all things which a more substantial number of students say they sometimes use LLMs for, rather than actually writing coursework). Judging from the work I’ve read this year, students are well aware that gAI output is too simplistic, that things are more complex and debated – but they take it as a sound starting-point for further work, as if it’s on a par with, say, an Oxford Classical Dictionary entry or an introductory book.
Why does this matter? As far as their own essays are concerned, it means they are starting not with an outline of the state of the subject from, say, 10-15 years ago, but with a compilation of the most conventional ideas from the last century or more – bearing in mind that the LLM’s statistical approach will lead it to echo the things that have been most often written, regardless of their quality – and no conception of how much, or what, has been left out. Decontextualised statements, that need to be understood in relation to different eras of scholarship and lines of enquiry and forms of evidence, are read as if they are part of a coherent account of the topic.
And that’s the most positive version. I haven’t tested it properly, but I am not instinctively confident that an LLM summary of a published article or book will actually be reliable when it comes to key arguments and points of analysis, rather than just being something that looks like a trustworthy summary but actually just repeats some key terms and phrases in a plausible manner. Likewise using ChatGPT to check one’s own work; at best it’s a grammar-checker, rather than being able to offer a useful evaluation of the quality of the argument, since it has no conception of accuracy or historical argument against which to measure it – and again I’m not sure how far it will actually be evaluating the essay, rather than producing a text that looks like an evaluation.
So, the end result is likely to be quite mediocre at best, which is why I am less worried than I was a year ago about LLMs being a threat to the integrity of assessment – so long as we focus on analytical skills, interpretation and understanding rather than either content or vacuous bullshitting, so long as we have the time to evaluate work properly, and so long as there is no external pressure to mark students leniently if they turn in such mediocrity.
The much bigger problem is student learning. The key promise of the LLM is to save time: to read articles and books so you don’t have to, to digest vast amounts of scholarship that you would never have time to read outside of a PhD. But the process of reading and understanding publications, of laboriously building up understanding of a topic, is the point of the exercise, rather than being merely the means for extracting content. Delegate that to someone or something else, and nothing will actually be learned. Likewise the process of revising an essay: the point is to learn to evaluate and improve one’s work, not for it simply to be better without the active involvement of the author.
There are plenty of things we can try to do to push students in this direction: make the reading of scholarship a key part of seminars, so they get practice in this; help narrow down their reading still further (I’m coming round to the view that it’s better that they read a couple of things properly than that they feel they need at least eight references but don’t actually engage with any of them); set assessment tasks that explicitly require such engagement, rather than it appearing as a means to an end; spend time educating them in the workings and limitations of gAI so they steer clear of it (I am struck by how successfully Wikipedia has been discredited as a source through, I assume, constant repetition of warnings by teachers, even though it’s far better than ChatGPT for what it’s good at).
Above all, however, WE need time, for offering and marking formative as well as summative assessments, for giving individual feedback and advice, for working through examples, for multi-stage assessment tasks in which we comment on drafts so they can learn to revise them. And this is time we don’t have (I try not to think about how much extra work my preferred ‘draft and revision’ approach involves, beyond the workload allowance for the module – and that’s just in smallish final-year seminar modules). But obviously farming assessment and feedback to LLMs is not the answer…
Thank you, Neville- I’m returning to the classroom for the first time since 2019 and have been thinking/reading a bit about gAI to help prepare me for the new world. This is one of the smartest and most practically helpful things I’ve read – I’m going to make all my tutors read it!
Thanks, Ika. As mentioned in the post, this is part of an ongoing project, so there will be a publicly-available report and guidance at some point when I find time to write it…