港大中文學院

School of Chinese Seminar

Is AI-generated language appropriate?

Speaker: Dr. Xi Chen (University of Central Lancashire)

Moderator: Professor Eva Ng (HKU)

Date: May 28, 2024 (Tuesday)

Time: 4:30 - 6:00 p.m.

Venue: Room 7.30, Run Run Shaw Tower, Centennial Campus, HKU

Language: English

Abstract

In this talk, I examine the performance of GPT-3.5 (ChatGPT) and GPT-4 in relation to the evaluation of in/appropriateness, employing three metrics for the evaluation—language resources deployed, human evaluation, and AI evaluation. Specifically, the first metric measures how much GPT resembles humans in the deployment of language resources in accordance with given contexts, namely, the capability of making form-context connections. The second metric adopts human emic judgment to examine whether GPT-generated language is appropriate in context. Then, the third metric requires GPT to make the judgement to test whether AI ‘knows’ its own language is appropriate or not. The tests involve more than 750 participants and over 1,500 data points from either AI-generated or human-generated output. They are analysed using a combination of computational, statistical, and qualitative approaches. Findings demonstrate that GPT performs equally well as humans in four out of five tested parameters of language resources. It even outperforms humans in terms of syntactic diversity. Based on human judgments, GPT-generated language has no significant difference from that produced by humans in terms of appropriateness. However, GPT itself does not ‘know’ very well whether the language is appropriate or not, resulting in a significant difference in the appropriateness judgments between AI and humans. Overall, GPT may have a human-like capability of language production, but is potentially fallible in connecting the language produced to its pragmatic meanings.

Biography

Dr. Xi Chen is an associate professor at the University of Central Lancashire, U.K. She received her Ph.D from SOAS, University of London in 2017. Her research incorporates AI language and complex social network analysis into pragmatic studies. She serves as the book reviews editor for the Journal of Pragmatics and also on the editorial board of Frontiers in Communication. She was a visiting scholar at Harvard University in 2022, and is currently visiting the School of Chinese at the University of Hong Kong, which is supported by the Leverhulme Research Fellowship.

All are welcome

Registration: https://hku.au1.qualtrics.com/jfe/form/SV_9X1XjAgGgqWyzPM