University of Cambridge > Talks.cam > Artificial Intelligence Research Group Talks (Computer Laboratory) > Vibe checks and red teaming: why ML researchers are increasingly reverting to manual evaluation

Log in

University Account

External (via Google)

Information on

Subscribing to talks details

Finding a talk details

Adding a talk details

Disseminating talks details

Help and Documentation details

Vibe checks and red teaming: why ML researchers are increasingly reverting to manual evaluation

Download to your calendar using vCal

Arduin Findeis (University of Cambridge)
Tuesday 16 January 2024, 13:00-14:00
Lecture Theatre 2, Computer Laboratory, William Gates Building.

If you have a question about this talk, please contact Mateja Jamnik .

There is a curious trend in machine learning (ML): researchers developing the most capable large language models (LLMs) increasingly evaluate them using manual methods such as red teaming. In red teaming, researchers hire workers to manually try to break the LLM in some form by interacting with it. Similarly, some users pick their preferred LLM assistant by manually trying out various models – checking each LLM ’s “vibe”. Considering that LLM researchers and users both actively seek to automate all sorts of other tasks, red teaming and vibe checks are surprisingly manual evaluation processes. This trend towards manual evaluation hints at fundamental problems that prevent more automatic evaluation methods, such as benchmarks, to be used effectively for LLMs. In this talk, I aim to give an overview of the problems preventing LLM benchmarks from being a fully satisfactory alternative to more manual approaches.

You can also join us on Zoom

This talk is part of the Artificial Intelligence Research Group Talks (Computer Laboratory) series.