Efficient transfer learning of large models with limited resources: A survey

In recent years, the fast-evolving deep learning techniques have dominated critical fields such as natural language understanding, computer vision, multimodal processing and data mining, therefore greatly advancing the development of artificial intelligence (Al) technology. Among these advancements, transfer learning (TL) has emerged as a pivotal technigue aimed at effectively reusing and sharing knowledge across multiple related models. This approach not only reduces the substantial costs associated with data collection and annotation, but also contributes to enhanced generalizability and capability of deep models. However, the exponential growth in the size, complexity, and depth of deep large models has presented serious challenges to traditional training and transfer algorithms, particularly in terms of computational and storage requirements. Such high computational complexity poses significant obstacles to effective knowledge transfer in resource-constrained scenarios, incuding but not limited to wearable technology, military applications, and healthcare systems. To address these challenges, efficient transfer learning algorithms have recently emerged as a promising solution, enabling agile adaptation and deployment of large models with minimal resource overhead. These algorithms are expected to become a key technological driver in the future development of Al. This paper stands out as the first comprehensive survey on the field of efficient transfer learning, aiming to systematically summarize research progress in this thriving research field over the past five years. Concretely, this paper investigates efficient transfer learning across three primary application fields: natural language processing, computer vision, and multimodal models. Among each application field, this paper further identifies and elaborates on five representative technical approaches that have gained prominence in recent research: modifying model structures, adjusting pre-training parameters, adapting original inputs (outputs), injecting adaptive parameters, and introducing adaptive modules, Each of these approaches is subjected to a comprehensive and thorough review, analyzing their respective strengths, limitations, and potential applications. This critical evaluation provides readers with a nuanced understanding of the current state-of-the-art in efficient transfer learning. The primary contributions of this survey are threefold: (l) This survey presents the first systematic review of efficient transfer learning, offering invaluable technical insights and guidance for future research endeavors in this rapidly evolving field. (2) This survey proposes a novel technique-based framework that provides a clear and systematic research guideline, enabling readers to navigate the complex landscape of efficient transfer learning methodologies. (3)This survey conducts an in-depth analysis of the shortcomings and limitations of current methods, thereby identifying critical research gaps and providing insightful directions for future investigations. Efficient transfer learning serves as a crucial bridge between cutting-edge Al technologies and their practical applications in everyday life. It holds the potential to enable easier and cheaper access to the power of large models, benefiting a wide range of enterprises and individuals across various sectors. By providing comprehensive overviews of the current state of the art, solid theoretical foundations, practical guidance, along with critical insights into future research directions, this survey contributes significantly to the development of efficient transfer learning, and is hope to inspire researchers and practitioners to push the boundaries of the research field.

Recommended citation: X. Li, J. Li, L. Zhu, and H. T. Shen. (2024). "Efficient transfer learning of large models with limited resources: A survey." Chinese Journal of Computers. 47(11).
Download Paper