基于双分支特征分解的红外与可见光融合网络

Infrared-Visible Image Fusion Network Based on Dual-Branch Feature Decomposition

导出

摘要多模态图像融合将来自不同传感器的信息相互融合,以获取互补的模态特征。红外与可见光图像融合是多模态任务中的热门话题。但现有方法在有效整合这些不同模态特征和生成全面特征表示方面存在困难。为此,提出双分支特征分解(DBDFuse)网络,引入双分支特征提取结构,其中OAT(Outlook Attention Transformer)块,用于提取高频局部特征,而ST(Stoken Transformer)中新设计了Fold和Unfold模块,用于高效捕获低频全局依赖关系。ST将原始全局注意力分解为稀疏关联映射和低维注意力的乘积,以此来达到捕获低频全局特征的目的。实验证明,DBDFuse网络在红外与可见光图像融合中超越了最先进的(SOTA)方法,生成的融合图像在视觉效果上具有更高的清晰度和细节保留能力,同时提升了模态间的互补性。此外,还提高了红外与可见光融合图像在下游任务中的性能,在M3FD目标检测任务中融合图像平均精度均值达到80.98%,在LLVIP语义分割任务中平均交并比达到63.9%。 Multimodal image fusion involves the integration of information from different sensors to obtain complementary modal features.Infrared-visible image fusion is a popular topic in multimodal tasks.However,the existing methods face challenges in effectively integrating these different modal features and generating comprehensive feature representations.To address this issue,we propose a dual-branch feature-decomposition(DBDFuse)network.A dual-branch feature extraction structure is introduced,in which the Outlook Attention Transformer(OAT)block is used to extract high-frequency local features,whereas newly designed fold-and-unfold modules in the Stoken Transformer(ST)efficiently capture low-frequency global dependencies.The ST decomposes the original global attention into a product of sparse correlation maps and lowdimensional attention to capture low-frequency global features.Experimental results demonstrate that the DBDFuse network outperforms state-of-the-art(SOTA)methods for infrared-visible image fusion.The fused images exhibit higher clarity and detail retention in visual effects,while also enhancing the complementarity between modalities.In addition,the performance of infrared and visible light fusion images in downstream tasks has been improved,with mean average accuracy of 80.98%in the M3FD object detection task and mean intersection to union ratio of 63.9%in the LLVIP semantic segmentation task.

作者高训东陈辉姚亚宁张成铖 Gao Xundong;Chen Hui;Yao Yaning;Zhang Chengcheng(School of Information and Communication,Guilin University of Electronic Technology,Guilin 541004,Guangxi,China;Guangxi University Key Laboratory of Microwave and Optical Wave Application Technology,Guilin 541004,Guangxi,China)

机构地区桂林电子科技大学信息与通信学院广西高校微波光波应用技术重点实验室

出处《激光与光电子学进展》北大核心 2025年第14期472-482,共11页 Laser & Optoelectronics Progress

基金广西自然科学基金(2021JJA170177)。

关键词图像融合特征分解双分支网络红外图像可见光图像 image fusion feature decomposition dual branch network infrared image visible image

分类号 TP751 [自动化与计算机技术—检测技术与自动化装置]

作者简介通信作者:高训东,xundonggao@guet.edu.cn;通信作者:陈辉,Chenhui02@guet.edu.cn。