The automated assessment of texts produced by learners of English

Add to your list(s) Download to your calendar using vCal

• Helen Yannakoudakis (University of Cambridge)
• Thursday 03 March 2011, 12:00-13:00
• GS15, Computer Laboratory.

If you have a question about this talk, please contact Jimme Jardine.

Helen writes:

We are working on the automated assessment of texts produced by learners of English (as a second language) in response to prompts eliciting free-text answers. We have built a system that analyses and assesses the quality of writing competence automatically. Our system uses ranking SVMs and assigns a score to a text based on textual features. The text is between 200 and 400 words.

We are interested in collecting real data to test the validity of our system, and thus would like to invite you to trick its scoring capabilities. For example, preliminary experiments on our current version have shown that discourse incoherent texts give rise to large discrepancies between the examiner’s scores and the system’s predicted values. A paper draft describing how our system works as well as sample texts, prompts and the marking criteria can be found in:

http://www.cl.cam.ac.uk/~hy260/AA.html

There is a prize of 50 pounds to the person that will compose a text that creates the highest disparity between the system’s output and the mark awarded by an examiner. Multiple submissions are welcome (in plain text), with a slight preference on getting texts that trick the system into assigning scores higher than deserved.

In this week’s reading group I will be giving more details about the competition as well as our automated assessment system.

If you would like to participate, please send me your texts by March 20. If possible, please also include two (short) sentences, one describing the topic of your text and another the reasons why you think the system will be tricked.

Many thanks, Helen

This talk is part of the Natural Language Processing Reading Group series.

This talk is included in these lists:

• Cambridge Forum of Science and Humanities
• Cambridge Language Sciences
• Cambridge talks
• Chris Davis' list
• GS15, Computer Laboratory
• Guy Emerson's list
• Natural Language Processing Reading Group

Note that ex-directory lists are not shown.