A Dataset for Evaluating Contextualized Representation of Biomedical Concepts in Language Models - Laboratoire Interdisciplinaire des Sciences du Numérique Access content directly
Journal Articles Scientific Data Year : 2024

A Dataset for Evaluating Contextualized Representation of Biomedical Concepts in Language Models

Hossein Rouhizadeh
  • Function : Author
Irina Nikishina
  • Function : Author
Anthony Yazdani
Alban Bornet
  • Function : Author
Boya Zhang
  • Function : Author
Julien Ehrsam
  • Function : Author
Christophe Gaudet-Blavignac
Douglas Teodoro

Abstract

Abstract Due to the complexity of the biomedical domain, the ability to capture semantically meaningful representations of terms in context is a long-standing challenge. Despite important progress in the past years, no evaluation benchmark has been developed to evaluate how well language models represent biomedical concepts according to their corresponding context. Inspired by the Word-in-Context (WiC) benchmark, in which word sense disambiguation is reformulated as a binary classification task, we propose a novel dataset, BioWiC, to evaluate the ability of language models to encode biomedical terms in context. BioWiC comprises 20’156 instances, covering over 7’400 unique biomedical terms, making it the largest WiC dataset in the biomedical domain. We evaluate BioWiC both intrinsically and extrinsically and show that it could be used as a reliable benchmark for evaluating context-dependent embeddings in biomedical corpora. In addition, we conduct several experiments using a variety of discriminative and generative large language models to establish robust baselines that can serve as a foundation for future research.

Dates and versions

hal-04574786 , version 1 (14-05-2024)

Identifiers

Cite

Hossein Rouhizadeh, Irina Nikishina, Anthony Yazdani, Alban Bornet, Boya Zhang, et al.. A Dataset for Evaluating Contextualized Representation of Biomedical Concepts in Language Models. Scientific Data , 2024, 11 (1), pp.455. ⟨10.1038/s41597-024-03317-w⟩. ⟨hal-04574786⟩
20 View
0 Download

Altmetric

Share

Gmail Mastodon Facebook X LinkedIn More