Customizing Language Models with Instance-wise LoRA for Sequential Recommendation

Abstract

Sequential recommendation systems predict a user’s next item of interest by analyzing past interactions, aligning recommendations with individual preferences. Leveraging the strengths of Large Language Models (LLMs) in knowledge comprehension and reasoning, recent approaches have applied LLMs to sequential recommendation through language generation paradigms. These methods convert user behavior sequences into prompts for LLM fine-tuning, utilizing Low-Rank Adaptation (LoRA) modules to refine recommendations. However, the uniform application of LoRA across diverse user behaviors sometimes fails to capture individual variability, leading to suboptimal performance and negative transfer between disparate sequences. To address these challenges, we propose Instance-wise LoRA (iLoRA), integrating LoRA with the Mixture of Experts (MoE) framework. iLoRA creates a diverse array of experts, each capturing specific aspects of user preferences, and introduces a sequence representation guided gate function. This gate function processes historical interaction sequences to generate enriched representations, guiding the gating network to output customized expert participation weights. This tailored approach mitigates negative transfer and dynamically adjusts to diverse behavior patterns. Extensive experiments on two benchmark datasets demonstrate the effectiveness of iLoRA, highlighting its superior performance compared to existing methods in capturing user-specific preferences and improving recommendation accuracy.

Publication
In NeurIPS 2024

Citation:

@article{ilora,
  author       = {Xiaoyu Kong and
                  Jiancan Wu and
                  An Zhang and
                  Leheng Sheng and
                  Hui Lin and
                  Xiang Wang and
                  Xiangnan He},
  title        = {Customizing Language Models with Instance-wise LoRA for Sequential
                  Recommendation},
  booktitle    = {NeurIPS},
  year         = {2024}
}
Xiaoyu Kong
Xiaoyu Kong
孔晓宇
Jiancan Wu
Jiancan Wu
吴剑灿 博士后
Xiang Wang
Xiang Wang
王翔 教授
Xiangnan He
Xiangnan He
何向南 教授