基于多模态医学影像报告的自动生成研究与应用

AbstractWith the continuous development of scientific technology and medical standards,a large amount ofmedical imaging data is generated every day,which increases the burden of diagnosis and imaging reportwriting for radiologists.In recent years,the research of the automatic generation of image reports is mostlybased on the model design of image caption.Although some results have been achieved,there are stillcertain limitations.There is no sufficient analysis of medical image features and medical semantics.Thetraining of the model forcibly aligns the image features with the text features in the report,resulting in alow-quality medical image report and limited clinical significance.Based on this,this study proposes a medical imaging report automatic generation model based onTopic attention mechanism named TAMRGM.The model can fully consider the medical semantic featuresof image reports and the advantages of deep leaming training models,and can automatically generate moreaccurate image reports.It first extracts the features of medical images,and fully integrates the medicalsemantic features of the image report.By adding attention mechanism to multi-modal fusion of imageimages and text features,the topic generator is used to complete the qualitative analysis of the image,andthen generate the detailed information of the image report sentence by sentence to complete the generationof high-quality image report.Finally,the TAMRGM model is trained and evaluated based on the OpenIdata set,and the image reports generated by the model are evaluated using BELU Score,METEOR,ROUGE,and CIDEr evaluation indicators.The experimental results show that the effect of the TAMRGMmodel is better than the CNN-RNN,Hierarchical Generation,and Co_Attention models,proving theeffectiveness of the TAMRGM model for generating high-quality medical image reports.In summary,on the basis of the research and investigation of the automatic generation of medicalimage reports,the author proposes a model for automatic generation of medical image reports based on theTopic attention mechanism,TAMRGM,and the construction and effectiveness of the model on the chestX-ray data set OpenI verification.The research results are hopefully assist the radiologist to quicklycomplete the writing of the image report and provide better diagnosis and treatment services for thepatients.Keywords:Deep Leaming,Medical Report Generation,Attention Mechanism,Multimodal LeamingⅡ