论文标题
用于参数有效微调的审慎语言模型启动的系统分析
Systematic Analysis for Pretrained Language Model Priming for Parameter-Efficient Fine-tuning
论文作者
论文摘要
最近，参数效率（PE）方法（例如提示或适配器），用于调整预训练的语言模型（PLM）对下游任务已很受欢迎。但是，障碍仍然阻止这些方法发挥全部潜力。例如，两个重大的挑战是射击适应性和交叉任务概括。为了解决这些问题，我们提出了一个一般的PE启动框架，以增强和探索PE方法的几种适应和概括能力。在此框架中，PLM用PE方法进行了启动，以快速适应各种目标任务。为了评估这些PE方法的概括能力，我们对包含160个不同NLP任务的几杆跨域基准进行了实验。我们的实验不仅揭示了最佳的启动策略，而且还验证了启动启动促进对目标任务的适应。
Parameter-efficient (PE) methods (like Prompts or Adapters) for adapting pre-trained language models (PLM) to downstream tasks have been popular recently. However, hindrances still prevent these methods from reaching their full potential. For example, two significant challenges are few-shot adaptation and cross-task generalization. To tackle these issues, we propose a general PE priming framework to enhance and explore the few-shot adaptation and generalization ability of PE methods. In this framework, PLMs are primed with PE methods for rapidly adapting to various target tasks. To evaluate the generalization ability of these PE methods, we conduct experiments on a few-shot cross-domain benchmark containing 160 diverse NLP tasks. Our experiment not only reveals the best priming strategy but also verifies that priming facilitates the adaptation to target tasks.
