This is a collection of research papers for Federated Learning for Large Language Models (FedLLM). The repository will be continuously updated to track the frontier of FedLLM.
In this section, we will list recent FedLLM papers accepted by top tier AI/ML/Networking conferences and journals.
format:
- [title](paper link) [Venue]
- authors
- datasets
- models
- [code](code link) [slide](slide link)
- Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Models [ICLR 2025]
- authors: Linh Tran, Wei Sun, Stacy Patterson, Ana Milanova
- datasets: Caltech101, OxfordPets, OxfordFlowers, Food101, CIFAR-100
- models:
- code and slide
-
Fisher Information-based Efficient Curriculum Federated Learning with Large Language Models [EMNLP 2024]
- authors: Ji Liu, Jiaxiang Ren, Ruoming Jin, Zijie Zhang, Yang Zhou, Patrick Valduriez, Dejing Dou
- datasets: QNLI, SST-2, CoLA, MRPC, RTE, BoolQ, MPQA, Subj, Trec, MR
- models: RoBERTa LARGE and LLaMA
- code and slide
-
FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations [NeurIPS 2024]
- authors: Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, Ang Li
- datasets: Databricks-dolly-15k, Alpaca, and Wizard dataset
- models: Llama-7B, LLaMA-2-7B
- code and slide
-
FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Model [KDD 2024]
- authors: Feijie Wu, Zitao Li, Yaliang Li, Bolin Ding, Jing Gao
- datasets: GSM-8K, HumanEvalX, dolly-15K
- models: LLaMA-2-7B
- code and slide
-
PrE-Text: Training Language Models on Private Federated Data in the Age of LLMs [ICML 2024]
- authors: Charlie Hou, Akshat Shrivastava, Hongyuan Zhan, Rylan Conway, Trang Le, Adithya Sagar, Giulia Fanti, Daniel Lazar
- datasets: C4
- models: DistilGPT2
- code and slide
-
Analysis of Privacy Leakage in Federated Large Language Models [AISTATS 2024]
- authors: Minh N. Vu, Truc Nguyen, Tre' R. Jeter, My T. Thai
- datasets: IMDB review, Yelp review, Twitter-emotion, and Finance
- models: BERT, RoBERTa , distilBERT, GPT1, GPT2
- code and slide
-
Improving LoRA in Privacy-preserving Federated Learning [ICLR 2024]
- authors: Youbang Sun, Zitao Li, Yaliang Li, Bolin Ding .
- datasets: MNLI, SST2, QNLI , QQP and GSM-8K
- models: RoBERTa, LLaMA 7B
- code and slide
-
Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization [EMNLP 2023]
- authors: Tianshi Che, Ji Liu, Yang Zhou, Jiaxiang Ren, Jiwen Zhou, Victor S. Sheng, Huaiyu Dai, Dejing Dou.
- datasets: QNLI, SST-2, CoLA, MRPC, RTE, and BoolQ, MPQA, Subj, TREC, and MR
- models: RoBERTa, GPT2, LLaMA 7B
- code and slide
-
- authors: Zhuo Zhang, Yuanhang Yang, Yong Dai, Qifan Wang, Yue Yu, Lizhen Qu, Zenglin Xu.
- datasets: RTE, MRPC, SST-2, QNLI, QQP, MNLI
- models: RoBERTa
- code and slide
-
Petals: Collaborative Inference and Fine-tuning of Large Models [ACL 2023]
In this section, we will list high-quality FedLLM preprints that have been uploaded to open-access repositories like ArXiv.
-
Federated Sketching LoRA: On-Device Collaborative Fine-Tuning of Large Language Models [Arxiv 2025]
- authors: Wenzhi Fang, Dong-Jun Han, Liangqi Yuan, Seyyedali Hosseinalipour, Christopher G. Brinton
- datasets: QNLI, MRPC, CoLA, MNLI, RTE, SST-2, QQP
- models: RoBERTa, LLaMA-3.2-3B
- code and slide
-
Photon: Federated LLM Pre-Training [Arxiv 2024]
- authors: Lorenzo Sani, Alex Iacob, Zeyu Cao, Royson Lee, Bill Marino, Yan Gao, Dongqi Cai, Zexi Li, Wanru Zhao, Xinchi Qiu, Nicholas D. Lane
- datasets: C4
- models: 3B, 7B models
- code and slide
-
Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clients [Arxiv 2024]
- authors: Jabin Koo, Minwoo Jang, Jungseul Ok
- datasets: BANKING77 and 20 Newsgroup
- models: RoBERTa
- code and slide
-
MIRA: A Method of Federated MultI-Task Learning for LaRge LAnguage Models [Arxiv 2024]
- authors: Ahmed Elbakary, Chaouki Ben Issaid, Tamer ElBatt, Karim Seddik, Mehdi Bennis
- datasets: Dolly-15k and Natural Instruction
- models: Llama and GPT-2-large
- code and slide
-
FedSpaLLM: Federated Pruning of Large Language Models [Arxiv 2024]
- authors: Guangji Bai, Yijiang Li, Zilinghan Li, Liang Zhao, Kibaek Kim
- datasets: WikiText2, PTB, C4
- models: OPT-125m, OPT-1.3b, and LlaMA-2 7b
- code and slide
-
Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Models [Arxiv 2024]
- authors: Sajjad Ghiasvand, Yifan Yang, Zhiyu Xue, Mahnoosh Alizadeh, Zheng Zhang, Ramtin Pedarsani
- datasets: SST-2, QNLI, QQP, MNLI
- models: BERT and LLaMA
- code and slide
-
Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models [Arxiv 2024]
- authors: Yao Shu, Wenyang Hu, See-Kiong Ng, Bryan Kian Hsiang Low, Fei Richard Yu
- datasets: Dolly-15K
- models: LLaMA-3B
- code and slide
-
On the Client Preference of LLM Fine-tuning in Federated Learning [Arxiv 2024]
- authors: Feijie Wu, Xiaoze Liu, Haoyu Wang, Xingchen Wang, Jing Gao
- datasets: Summarization
- models: LLaMA-2-7B, Alpaca-7B
-
SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Models [Arxiv 2024]
- authors: Zheng Lin, Xuanjie Hu, Yuxin Zhang, Zhe Chen, Zihan Fang, Xianhao Chen, Ang Li, Praneeth Vepakomma, Yue Gao
- datasets: E2E
- models: GPT2
-
- authors: Haifeng Wen, Hong Xing, Osvaldo Simeone
- datasets:
- models:
-
Save It All: Enabling Full Parameter Tuning for Federated Large Language Models via Cycle Black Gradient Descent [Arxiv 2024]
- authors: Lin Wang, Zhichao Wang, Xiaoying Tang
- datasets: GLUE
- models: GPT2-small, BLOOM, RoBERTa-base, ChatGLM3-6B and LLaMA2-7B
-
Thinking Forward: Memory-Efficient Federated Finetuning of Language Models [Arxiv 2024]
- authors: Kunjal Panchal, Nisarg Parikh, Sunav Choudhary, Lijun Zhang, Yuriy Brun, Hui Guan
- datasets: AG News, SST2, Yelp, SQuADv2,...
- models: Llama2-7B
-
Personalized Wireless Federated Learning for Large Language Models [Arxiv 2024]
- authors: Feibo Jiang, Li Dong, Siwei Tu, Yubo Peng, Kezhi Wang, Kun Yang, Cunhua Pan, Dusit Niyato
- datasets:
- models: GPT-2
-
Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Models [Arxiv 2024]
- authors: Zihan Fang, Zheng Lin, Zhe Chen, Xianhao Chen, Yue Gao, Yuguang Fang
- datasets: 20NEWS, E2E
- models: Bert and GPT-2
-
Dual-Personalizing Adapter for Federated Foundation Models [Arxiv 2024]
- authors: Yiyuan Yang, Guodong Long, Tao Shen, Jing Jiang, Michael Blumenstein
- datasets: Flan
- models: LLaMA-7B
-
FedRDMA: Communication-Efficient Cross-Silo Federated LLM via Chunked RDMA Transmission [Arxiv 2024]
- authors: Zeling Zhang, Dongqi Cai, Yiran Zhang, Mengwei Xu, Shangguang Wang, Ao Zhou
- datasets: AdvertiseGen
- models: GPT2
-
Privacy-Aware Semantic Cache for Large Language Models [Arxiv 2024]
- authors: Waris Gill, Mohamed Elidrisi, Pallavi Kalapatapu, Ali Anwar, Muhammad Ali Gulzar
- datasets: GPTCache dataset
- models: Llama 2, MPNet, Albert
-
Analysis of Privacy Leakage in Federated Large Language Models [Arxiv 2024]
- authors: Minh N. Vu, Truc Nguyen, Tre' R. Jeter, My T. Thai.
- datasets: IMDB, Yelp
- models: RoBERTa, DistilBERT, GPT, GPT2
-
Federated Fine-tuning of Large Language Models under Heterogeneous Language Tasks and Client Resources [ArXiv 2024]
- authors: Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, Yaliang Li.
- datasets: AllenAI natural instruction dataset v2.
- models: LLaMA-1.3B
- code and slide
-
OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning [ArXiv 2024]
- authors: Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, Siheng Chen.
- datasets: Alpaca, Alpaca-GPT4, FinGPT, MedAlpaca, CodeMathInstruct, UltraFeedback, HH-RLHF
- models: LLaMA-7B
- code and slide
-
On the Convergence of Zeroth-Order Federated Tuning for Large Language Models [ArXiv 2024]
- authors: Zhenqing Ling, Daoyuan Chen, Liuyi Yao, Yaliang Li, Ying Shen.
- datasets: Databricks-dolly-15k, GSM8K, CodeAlpaca, Alpaca
- models: LLaMA-3B
- code and slide
-
Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes [ArXiv 2024]
- authors: Zhen Qin, Daoyuan Chen, Bingchen Qian, Bolin Ding, Yaliang Li, Shuiguang Deng.
- datasets: Natural Instructions and Dolly-15K.
- models: DataJuicer-1.3B, LLaMA-3B.
- code and slide
-
Towards Building the Federated GPT: Federated Instruction Tuning [ArXiv 2024]
- authors: Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Yufan Zhou, Guoyin Wang, Yiran Chen.
- datasets: Dolly-15K.
- models: Shepherd-7B.
- code and slide
-
Asynchronous Local-SGD Training for Language Modeling [ArXiv 2024]
- authors: Bo Liu, Rachita Chhaparia, Arthur Douillard, Satyen Kale, Andrei A. Rusu, Jiajun Shen, Arthur Szlam, Marc'Aurelio Ranzato.
- datasets: C4 dataset.
- models: 150M Model
-
DiLoCo: Distributed Low-Communication Training of Language Models [ArXiv 2023]
- authors: Arthur Douillard, Qixuan Feng, Andrei A. Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc'Aurelio Ranzato, Arthur Szlam, Jiajun Shen.
- datasets: C4 dataset.
- models: 150M Model
-
Federated Generative Learning with Foundation Models [OpenReview 2023]
- authors: Jie Zhang, Xiao hua Qi, Shengyuan Pang, Siyuan Pan, Xiaobing Tu, Pengfei Wan, Bo Zhao.
- datasets: ImageNet and DomainNet .
- models:
-
FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning [ArXiv 2023]
- authors: Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, Jingren Zhou.
- datasets: Databricks-dolly-15k, GSM8K.
- models: LLaMA-7B
- code and slide
-
FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Models [ArXiv 2023]
- authors: Tao Fan, Yan Kang, Guoqiang Ma, Weijing Chen, Wenbin Wei, Lixin Fan, Qiang Yang.
- datasets: AdvertiseGen.
- models: BERT, GPTs, ChatGLM-6B, LLaMA, BLOOM, Baichuan
- code and slide
- Federated Fine-Tuning of LLMs: Framework Comparison and Research Directions [Arxiv 2024]
- authors: Na Yan, Yang Su, Yansha Deng, Robert Schober
- Federated Large Language Models: Current Progress and Future Directions [Arxiv 2024]
- authors: Yuhang Yao, Jianyi Zhang, Junda Wu, Chengkai Huang, Yu Xia, Tong Yu, Ruiyi Zhang, Sungchul Kim, Ryan Rossi, Ang Li, Lina Yao, Julian McAuley, Yiran Chen, Carlee Joe-Wong
-
Prompt Public Large Language Models to Synthesize Data for Private On-device Applications [Arxiv 2024]
- authors: Shanshan Wu, Zheng Xu, Yanxiang Zhang, Yuanbo Zhang, Daniel Ramage
- datasets: LLM-mix-166G
- models:
-
FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learning [ICASSP 2023]
- authors: Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, Gongshen Liu
-
Visual Prompt Based Personalized Federated Learning [TMLR 2023]
- authors: Guanghao Li, Wansen Wu, Yan Sun, Li Shen, Baoyuan Wu, Dacheng Tao
-
Prompt Federated Learning for Weather Forecasting: Toward Foundation Models on Meteorological Data [IJCAI 2023]
- authors: Shengchao Chen, Guodong Long, Tao Shen, Jing Jiang
-
Large Language Models Empowered Autonomous Edge AI for Connected Intelligence [IEEE Communication Magazine]
- authors: Yifei Shen, Jiawei Shao, Xinjie Zhang, Zehong Lin, Hao Pan, Dongsheng Li, Jun Zhang, Khaled B. Letaief.