DAMA: Data- and Model-aware Alignment of Multi-modal LLMs

Abstract

Direct Preference Optimization (DPO) has shown effectiveness in aligning multi-modal large language models (MLLM) with human preferences. However, existing methods exhibit an imbalanced responsiveness to the data of varying hardness, tending to overfit on the easy-to-distinguish data while underfitting on the hard-to-distinguish data. In this paper, we propose Data- and Model-aware DPO (DAMA) to dynamically adjust the optimization process from two key aspects: (1) a data-aware strategy that incorporates data hardness, and (2) a model-aware strategy that integrates real-time model responses. By combining the two strategies, DAMA enables the model to effectively adapt to data with varying levels of hardness. Extensive experiments on five benchmarks demonstrate that DAMA not only significantly enhances the trustworthiness, but also improves the effectiveness over general tasks. For instance, on the Object-HalBench, our DAMA-7B reduces response-level and mentioned-level hallucination by 90.0% and 95.3%, respectively, surpassing the performance of GPT-4V.

@misc{lu2025damadatamodelawarealignment,
      title={DAMA: Data- and Model-aware Alignment of Multi-modal LLMs}, 
      author={Jinda Lu and Junkang Wu and Jinghan Li and Xiaojun Jia and Shuo Wang and YiFan Zhang and Junfeng Fang and Xiang Wang and Xiangnan He},
      year={2025},
      eprint={2502.01943},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2502.01943}, 
}
Copy

Jinda Lu

卢金达

Junkang Wu

吴俊康

Jinghan Li

李静涵

Shuo Wang

王硕 副研究员

Xiang Wang

王翔 教授

Xiangnan He

何向南 教授