Meta-Evaluation of Sentence Simplification Metrics

Noof Alfear; Dimitar Lubomirov Kazakov; Hend Al-Khalifa

Meta-Evaluation of Sentence Simplification Metrics

Noof Alfear^*, Dimitar Lubomirov Kazakov, Hend Al-Khalifa

^*Corresponding author for this work

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Automatic Text Simplification (ATS) is one of the major Natural Language Processing (NLP) tasks, which aims to help people understand text that is above their reading abilities and comprehension. ATS models reconstruct the text into a simpler format by deletion, substitution, addition or splitting, while preserving the original meaning and maintaining correct grammar. Simplified sentences are usually evaluated by human experts based on three main factors: simplicity, adequacy and fluency or by calculating automatic evaluation metrics. In this paper, we conduct a meta-evaluation of reference-based automatic metrics for English sentence simplification using high-quality, human-annotated dataset, NEWSELA-LIKERT. We study the behavior of several evaluation metrics at sentence level across four different sentence simplification models. All the models were trained on the NEWSELA-AUTO dataset. The correlation between the metrics’ scores and human judgements was analyzed and the results used to recommend the most appropriate metrics for this task.

Original language	English
Title of host publication	Proceedings of LREC-COLING 2024
Subtitle of host publication	The 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation
Place of Publication	Turin, Italy
Number of pages	7
Publication status	Published - 25 May 2024
Event	Joint International Conference on Computational Linguistics, Language Resources and Evaluation - Torino, Italy Duration: 20 May 2024 → 25 May 2024

Conference

Conference	Joint International Conference on Computational Linguistics, Language Resources and Evaluation
Abbreviated title	LREC-COLING 2024
Country/Territory	Italy
City	Torino
Period	20/05/24 → 25/05/24

Access to Document

1298_PaperFinal published version, 175 KBLicence: CC BY-NC

Cite this

@inproceedings{4e3eeb7009cf495d8ca70178abda3ff4,

title = "Meta-Evaluation of Sentence Simplification Metrics",

abstract = "Automatic Text Simplification (ATS) is one of the major Natural Language Processing (NLP) tasks, which aims to help people understand text that is above their reading abilities and comprehension. ATS models reconstruct the text into a simpler format by deletion, substitution, addition or splitting, while preserving the original meaning and maintaining correct grammar. Simplified sentences are usually evaluated by human experts based on three main factors: simplicity, adequacy and fluency or by calculating automatic evaluation metrics. In this paper, we conduct a meta-evaluation of reference-based automatic metrics for English sentence simplification using high-quality, human-annotated dataset, NEWSELA-LIKERT. We study the behavior of several evaluation metrics at sentence level across four different sentence simplification models. All the models were trained on the NEWSELA-AUTO dataset. The correlation between the metrics{\textquoteright} scores and human judgements was analyzed and the results used to recommend the most appropriate metrics for this task.",

author = "Noof Alfear and Kazakov, {Dimitar Lubomirov} and Hend Al-Khalifa",

year = "2024",

month = may,

day = "25",

language = "English",

booktitle = "Proceedings of LREC-COLING 2024",

note = "Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC-COLING 2024 ; Conference date: 20-05-2024 Through 25-05-2024",

}

Alfear, N, Kazakov, DL & Al-Khalifa, H 2024, Meta-Evaluation of Sentence Simplification Metrics. in Proceedings of LREC-COLING 2024: The 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. Turin, Italy, Joint International Conference on Computational Linguistics, Language Resources and Evaluation, Torino, Italy, 20/05/24.

TY - GEN

T1 - Meta-Evaluation of Sentence Simplification Metrics

AU - Alfear, Noof

AU - Kazakov, Dimitar Lubomirov

AU - Al-Khalifa, Hend

PY - 2024/5/25

Y1 - 2024/5/25

N2 - Automatic Text Simplification (ATS) is one of the major Natural Language Processing (NLP) tasks, which aims to help people understand text that is above their reading abilities and comprehension. ATS models reconstruct the text into a simpler format by deletion, substitution, addition or splitting, while preserving the original meaning and maintaining correct grammar. Simplified sentences are usually evaluated by human experts based on three main factors: simplicity, adequacy and fluency or by calculating automatic evaluation metrics. In this paper, we conduct a meta-evaluation of reference-based automatic metrics for English sentence simplification using high-quality, human-annotated dataset, NEWSELA-LIKERT. We study the behavior of several evaluation metrics at sentence level across four different sentence simplification models. All the models were trained on the NEWSELA-AUTO dataset. The correlation between the metrics’ scores and human judgements was analyzed and the results used to recommend the most appropriate metrics for this task.

AB - Automatic Text Simplification (ATS) is one of the major Natural Language Processing (NLP) tasks, which aims to help people understand text that is above their reading abilities and comprehension. ATS models reconstruct the text into a simpler format by deletion, substitution, addition or splitting, while preserving the original meaning and maintaining correct grammar. Simplified sentences are usually evaluated by human experts based on three main factors: simplicity, adequacy and fluency or by calculating automatic evaluation metrics. In this paper, we conduct a meta-evaluation of reference-based automatic metrics for English sentence simplification using high-quality, human-annotated dataset, NEWSELA-LIKERT. We study the behavior of several evaluation metrics at sentence level across four different sentence simplification models. All the models were trained on the NEWSELA-AUTO dataset. The correlation between the metrics’ scores and human judgements was analyzed and the results used to recommend the most appropriate metrics for this task.

M3 - Conference contribution

BT - Proceedings of LREC-COLING 2024

CY - Turin, Italy

T2 - Joint International Conference on Computational Linguistics, Language Resources and Evaluation

Y2 - 20 May 2024 through 25 May 2024

ER -