Integrating biomedical knowledge information like concept name a.k.a concept mention during transfer learning is not a common fashion as it needs an alignment of large-scale concepts with unstructured biomedical text during pretraining. Existing learning objectives on biomedical text mainly rely on masked language modeling or random span denoising where grafting knowledge like integrating medical concepts has never been explored nor been discussed to facilitate the model pretraining. A prevailing assumption is that such concept-aware pretraining on biomedical text can be effective in biomedical task-specific language models. Based on this assumption, we introduce BioCAT5, a biomedical domain-specific text-to-text transfer transformer that integrates concept related knowledge from the unified medical language system (UMLS) during pretraining. BioCAT5 leverages concept-aware denoising -- an out-of-the-box language model learning objective that is trained to reconstruct text by masking out the concept spans of the input sequence. To facilitate the BioCAT5 pretraining, large-scale concept mentions from UMLS are considered to align with PubMed abstract text to create the pretraining dataset.
Our scripts h-job-abci3-pretraining-biocat5.sh, and scripts/finetune-*.sh are mainly prepared for launching multi-node training and fine-tuning on the ABCI 3.0 computation cluster.
One sample pretraining JSON input out of 1219 JSON files is included in the data/sample_pretraining_data/
Mohammad Golam Sohrab (2026). BioCAT5: Concept Mention-Aware Text-to-Text Transfer Transformer for Biomedical Literature. (Accepted as a Finding in AACL-2026)
This work has been partially supported by JSPS KAKENHI Grant Number JP24K15097.