Abstract

Automated reporting of chest radiographs is an emerging task at the intersection of medical image analysis and natural language generation. In this problem, a model receives a chest X-ray and produces a clinically meaningful textual description, including the presence or absence of respiratory diseases. Conventional systems rely on an encoder–decoder pipeline in which a convolutional neural network (CNN) encodes the image and a recurrent neural network (RNN) decodes the representation into a report word by word. Recent work has shown that reinforcement learning can further align generated reports with sequence-level objectives.To overcome this limitation, the proposed CXR-CapsNet model adapts deep reinforcement learning with embedded rewards to the domain of chest radiographic report generation by modifying the underlying CNN for medical imaging and by introducing an enhanced reward mechanism based on clinical and linguistic evaluation metrics. The policy network provides local guidance for predicting the next token in the report, while the value network provides global guidance over possible continuations of the partially generated sequence. A reward network, built on visual–semantic embeddings, combines similarity in the joint image– text space with a linear combination of metrics such as BLEU, CIDEr, ROUGE, and METEOR to better reflect the quality of the report. The policy and value networks are first pre-trained separately, with the reward network trained independently. In a benchmark chest radiographic data set, the proposed CXR-CapsNet achieves performance comparable to or better than strong baselines, while offering a substantial improvement over previous reinforcement-learning-based approaches for the generation of medical reports.