Study Finds AI-Generated Summaries Simpler and Easier to Comprehend

A new study published in PNAS Nexus finds that many general readers find AI generated summaries of scientific studies to be more comprehensive and easy to read. AI generated summaries can also make public perceptions of scientists more trustworthy, as the summaries can be read easily by the public and they can develop a more positive attitude towards scientific information. Many large language models like ChatGPT can be used in text summarization because they have the ability to process and generate natural language using deep learning.

The researcher of the study, David M. Markowitz explored how using AI summaries for scientific studies can improve our everyday life, as they can help people understand scientific context which would ultimately help scientists and researchers. For the study, Markowitz analyzed lay summaries and technical abstracts of 34,000 articles from Proceedings of the National Academy of Sciences (PNAS). To access the simplicity in the summaries, he used a linguistic tool called Linguistic inquiry and Word Count (LIWC) which evaluates texts on the basis of common words used, writing style and readability. Common words refer to the words we use in our everyday lives, writing style means if the text is formal or informal and readability means sentence length and how complex the vocabulary is. The results of the analysis showed that lay summaries were simpler than scientific abstracts, with more common words, short sentences and less formal writing styles. This made Markowitz explore AI generated summaries of scientific research.

The researcher wanted to find out if AI generated summaries can provide more simple texts than lay summaries authored by humans. Using ChatGPT-4, he selected 800 scientific abstracts from PNAC and asked the model to generate summaries. In the other study, 2274 participants were asked to evaluate AI and human summaries. They were asked to evaluate credibility, trustworthiness and intelligence of authors of the summaries provided to them and were also asked to tell whether they believe the summaries given to them were written by AI or a human.

• Also read: Study Reveals ChatGPT-4's Remarkable 'Theory of Mind' Abilities, Outperforming Previous Models

The results of the second study showed that AI generated summaries were perceived as simpler and easier to understand by the participants. The interesting thing that the study found was that most participants mistook AI generated summaries for human written ones and complex human written summaries for AI generated ones. For the third study, Markowitz gave 250 participants 20 pairs of AI generated and human written summaries. Five pairs each were given to participants randomly, with one pair being AI generated and other being human written one.

Then the participants were asked to read these summaries and write what they understood. They also answered multiple choice questions which were evaluated on the basis of accuracy and detail. The results showed that participants showed better comprehension after reading AI generated summaries. However, participants rated perception of intelligence lower for AI generated summaries. Credibility and trustworthiness had no major differences on both types of summaries.

Read next: From Second to First: Why India Could Overtake the USA in Developer Numbers by 2028
by Arooj Ahmed via Digital Information World