18 / 2025-03-21 10:56:24

ViT-driven Visual Extraction in R2GenCMN for medical report generation

vision transformer, radiology report generation, CMN

Medical imaging is a critical tool for diagnosing diseases. Its textual interpretation is vital for analysis and treatment planning. Automating report generation eases radiologists' workload and boosts clinical automation. This field has gained attention with the development of AI in healthcare. Prior studies often used pre-trained CNNs like VGG and ResNet for feature extraction. However, despite their effectiveness, their local receptive fields limit the ability to capture global context. To overcome this limitation, this study proposes a novel approach, ViT-R2GenCMN, which employs the Vision Transformer (ViT) to replace the traditional CNN-based visual extractor. ViT processes images by segmenting them into patch sequences and utilizes a self-attention mechanism to directly model global dependencies. In this work, we integrate ViT into the Cross-modal Memory Networks (CMN) framework to improve the alignment between visual and textual information, thereby enhancing the quality of generated radiology reports. This paper examines the applicability of ViT in the task of radiology report generation and evaluates its specific impact on cross-modal interaction performance. Experimental results demonstrate the effectiveness of our proposed model, achieving state-of-the-art performance on the widely recognized IU X-Ray benchmark dataset.