Bayesian Parameter-Efficient Fine-Tuning for Overcoming Catastrophic Forgetting

Authors

Haolin Chen (Idiap Research Institute, EPFL) haolin.chen@idiap.ch
Philip N. Garner (Idiap Research Institute) phil.garner@idiap.ch

Abstract

Although motivated by the adaptation of text-to-speech synthesis models, we argue that more generic parameter-efficient fine-tuning (PEFT) is an appropriate framework to do such adaptation. However, catastrophic forgetting remains an issue with PEFT, damaging the pre-trained model's inherent capabilities. We demonstrate that existing Bayesian learning techniques can be applied to PEFT to prevent catastrophic forgetting as long as the parameter shift of the fine-tuned layers can be calculated differentiably. In a principled series of experiments on language modeling and speech synthesis tasks, we utilize established Laplace approximations, including diagonal and Kronecker-factored approaches, to regularize PEFT with the low-rank adaptation (LoRA) and compare their performance in pre-training knowledge preservation. Our results demonstrate that catastrophic forgetting can be overcome by our methods without degrading the fine-tuning performance, and using the Kronecker-factored approximations produces a better preservation of the pre-training knowledge than the diagonal ones.

Source Code

GitHub: https://github.com/idiap/bayesian-peft

Audio Samples: Main Results

Target speaker: p248

p248_031: Because it's a waste of time for both sides.

Ground Truth	Reference	Pre-trained	Full	Linear	LoRA	LoRA+EWC	LoRA+KFAC

p248_142: He sees a difference in the style of the two teams.

Ground Truth	Reference	Pre-trained	Full	Linear	LoRA	LoRA+EWC	LoRA+KFAC

p248_232: Police later said the scheme would end in November.

Ground Truth	Reference	Pre-trained	Full	Linear	LoRA	LoRA+EWC	LoRA+KFAC

p248_116: Among them was Gary Robertson from Dundee.

Ground Truth	Reference	Pre-trained	Full	Linear	LoRA	LoRA+EWC	LoRA+KFAC

p248_155: They were later discharged from hospital.

Ground Truth	Reference	Pre-trained	Full	Linear	LoRA	LoRA+EWC	LoRA+KFAC

OOD speaker: p225

p225_201: The early goal was a shock to the system for Hearts.

Ground Truth	Reference	Pre-trained	Full	Linear	LoRA	LoRA+EWC	LoRA+KFAC

p225_290: Scotch beef is badly missed.

Ground Truth	Reference	Pre-trained	Full	Linear	LoRA	LoRA+EWC	LoRA+KFAC

p225_173: But the story of the play is worth a play in itself.

Ground Truth	Reference	Pre-trained	Full	Linear	LoRA	LoRA+EWC	LoRA+KFAC

p225_220: He would discuss the peace process in Northern Ireland.

Ground Truth	Reference	Pre-trained	Full	Linear	LoRA	LoRA+EWC	LoRA+KFAC

p225_318: The problem was he thought you were a big man.

Ground Truth	Reference	Pre-trained	Full	Linear	LoRA	LoRA+EWC	LoRA+KFAC

OOD speaker: p245

p245_257: However, the force has not yet received a formal complaint.

Ground Truth	Reference	Pre-trained	Full	Linear	LoRA	LoRA+EWC	LoRA+KFAC

p245_315: They had a confession.

Ground Truth	Reference	Pre-trained	Full	Linear	LoRA	LoRA+EWC	LoRA+KFAC

p245_284: It is a vote of confidence in the skills in Scotland.

Ground Truth	Reference	Pre-trained	Full	Linear	LoRA	LoRA+EWC	LoRA+KFAC

p245_233: It would appear that it has not been a problem.

Ground Truth	Reference	Pre-trained	Full	Linear	LoRA	LoRA+EWC	LoRA+KFAC

p245_068: The First Minister is scheduled to be elected by the Parliament tomorrow.

Ground Truth	Reference	Pre-trained	Full	Linear	LoRA	LoRA+EWC	LoRA+KFAC

OOD speaker: p261

p261_278: They are interior designers and architects.

Ground Truth	Reference	Pre-trained	Full	Linear	LoRA	LoRA+EWC	LoRA+KFAC

p261_130: He is a great addition to our team.

Ground Truth	Reference	Pre-trained	Full	Linear	LoRA	LoRA+EWC	LoRA+KFAC

p261_325: Disruption will be kept to a minimum.

Ground Truth	Reference	Pre-trained	Full	Linear	LoRA	LoRA+EWC	LoRA+KFAC

p261_113: He thought she was amazing.

Ground Truth	Reference	Pre-trained	Full	Linear	LoRA	LoRA+EWC	LoRA+KFAC

p261_133: This would indicate a surge in inflation was unlikely.

Ground Truth	Reference	Pre-trained	Full	Linear	LoRA	LoRA+EWC	LoRA+KFAC

OOD speaker: p302

p302_116: I can assure you, the new Augusta National is exactly that.

Ground Truth	Reference	Pre-trained	Full	Linear	LoRA	LoRA+EWC	LoRA+KFAC

p302_067: Treatment is not an issue with these people.

Ground Truth	Reference	Pre-trained	Full	Linear	LoRA	LoRA+EWC	LoRA+KFAC

p302_198: This must be kept in total perspective.

Ground Truth	Reference	Pre-trained	Full	Linear	LoRA	LoRA+EWC	LoRA+KFAC

p302_167: But even without either, Glasgow will be outstanding.

Ground Truth	Reference	Pre-trained	Full	Linear	LoRA	LoRA+EWC	LoRA+KFAC

p302_079: I was in jail for five years.

Ground Truth	Reference	Pre-trained	Full	Linear	LoRA	LoRA+EWC	LoRA+KFAC

Audio Samples: Varying Regularization Strength