In parallel with the rapid adoption of Artificial Intelligence (AI) empowered by advances in AI research, there have been growing awareness and concerns of data privacy. Recent significant developments in the data regulation landscape have prompted a seismic shift in interest towards privacy-preserving AI. This has contributed to the popularity of Federated Learning (FL), the leading paradigm for the training of machine learning models on data silos in a privacy-preserving manner. In this survey, we explore the domain of Personalized FL (PFL) to address the fundamental challenges of FL on heterogeneous data, a universal characteristic inherent in all real-world datasets. We analyze the key motivations for PFL and present a unique taxonomy of PFL techniques categorized according to the key challenges and personalization strategies in PFL. We highlight their key ideas, challenges and opportunities and envision promising future trajectories of research towards new PFL architectural design, realistic PFL benchmarking, and trustworthy PFL approaches.

—federated learning, personalized federated learning, non-IID data, statistical heterogeneity, privacy preservation, edge computing.

HE pervasiveness of edge devices in modern society,such as mobile phones and wearable devices, has led to the rapid growth of private data originating from distributed sources. In this digital age, organizations are using big data and artificial intelligence (AI) to optimize their processes and performance. While the wealth of data offers tremendous opportunities for AI applications, most of these data are highly-sensitive in nature and they exist in the form of isolated islands. This is especially relevant in the healthcare industry where medical data are highly-sensitive and they are often collected and reside across different healthcare institutions [

]. Such circumstances pose huge challenges for AI adoption as data privacy issues are not well addressed by conventional AI approaches. With the recent introduction of data privacy preservation laws such as the General Data Protection Regulation (GDPR) [

], there is an increasing demand for privacy-preserving AI [

] in order to meet regulatory compliance.

In view of these data privacy challenges, Federated Learning (FL) [

] has seen growing popularity in recent years. FL

Alysa Ziying Tan is with the School of Computer Science and Engineering, Nanyang Technological University, Singapore; Alibaba-NTU Singapore Joint Research Institute, NTU, Singapore; and Alibaba Group, Hangzhou, China.

Han Yu is with the School of Computer Science and Engineering, Nanyang Technological University, Singapore.

Lizhen Cui is with the School of Software, Shandong University (SDU), Jinan, China; and the Joint SDU-NTU Centre for Artificial Intelligence Research (C-FAIR), SDU, Jinan, China.

Qiang Yang is with the Department of Computer Science and Engineering, Hong Kong University of Science and Technology, Hong Kong; and WeBank, Shenzhen, China.

is a learning paradigm that enables collaborative training of machine learning models involving multiple data silos in a privacy-preserving manner. The prevailing FL setting assumes a federation of data owners (a.k.a. clients), which may be as small as individual mobile devices to as large as entire organizations, that collaboratively train a model under the orchestration of a central parameter server (a.k.a. the FL server) [

]. The training data are stored locally and are not directly shared during the training process. Most of the existing FL training approaches are derived from the Federated Averaging (FedAvg) algorithm introduced in [

]. The goal is to train a global model that performs well on most FL clients.

FL can be categorized into horizontal FL (HFL), vertical FL (VFL) and federated transfer learning (FTL), according to how data are distributed in terms of feature and sample spaces among participating entities [

]. HFL refers to scenarios whereby participants share the same feature space but have different data samples. It is the most commonly adopted FL setting popularized by Google, which applied HFL to train language models in mobile devices [

]. In VFL, participants have overlapping data samples, but differ in the feature space. A typical application scenario would involve the collaboration of multiple organizations from different industry sectors (e.g., a bank and an e-commerce company) which have different data features but may have a large number of shared users. FTL is applicable when participants have little overlap in both the feature space and the sample space. For example, organizations from different industry sectors serving markets in different regions can leverage FTL to collaboratively build models. Existing PFL works mainly focus on the HFL setting which makes up the majority of the FL application scenarios [

]. The HFL setting is the focus of this paper. For brevity, we use the terms HFL and FL interchangeably in the rest of this survey.

B. Motivations for Personalized Federated Learning

illustrates the key concepts and motivations for centralized machine learning (CML) [

], FL and PFL. We consider a cloud-based CML setting where data are pooled together in the cloud server to train an ML model. In this setting, the CML model achieves good generalization from the rich amount of data. However, CML faces bandwidth and latency challenges due to the sheer amount of data transferred to the cloud. It also does not preserve data privacy or not personalize well.

The FL setting assumes a federation of distributed clients, each with its own private local dataset. As these clients face data scarcity that limit their capacities to train effective

Fig. 1: Concept, Motivations & Proposed Taxonomy for Personalized Federated learning. 

. Centralized machine learning (CML) which pools data together to train a central ML model. 

. Federated learning (FL) which trains a global model under the orchestration of a central parameter server. Data resides in different data silos. 

. Personalized federated learning (PFL) which addresses the limitations of FL through global model personalization and personalized models learning. 

local models, they are motivated to join the FL process to obtain a better performing model. FL enables collaborative model training on data silos in a privacy-preserving manner, which sets it apart from the CML setting. Additionally, FL is communication-efficient as it only transfers model parameters which are a fraction in size compared to transferring raw data. By considering privacy and communication constraints, FL is applicable to support a wide range of application scenarios such as Internet of Things (IoT), that entails privacy, connectivity, bandwidth and latency challenges in varying edge computing environments [

However, the general FL approach faces several fundamental challenges: (i) poor convergence on highly heterogeneous data, and (ii) lack of solution personalization. These issues deteriorate the performance of the global FL model on individual clients in the presence of heterogeneous local data distributions, and may even disincentivize affected clients from joining the FL process. Compared to traditional FL, PFL research seeks to address these two challenges.

1) Poor Convergence on Heterogeneous Data: 

When learning on non-independent and identically distributed (non-IID) data, the accuracy of FedAvg is significantly reduced. This performance degradation is attributed to the phenomenon of client drift [

], as a result of the rounds of local training and synchronization on local data distributions that are non-IID. Fig. 

illustrates the effect of client drift on IID and non-IID data. In FedAvg, the server updates move toward the average of client optima. When data are IID, the averaged model is close to the global optimum 

as it is equidistant to both local optima 

is not equidistant to the local optima. In this illustration, 

will therefore be far from the global optimum 

, and the global model does not converge to its true global optimum. As the FedAvg algorithm experiences convergence issues on non-IID data, careful tuning of hyperparameters (e.g., learning rate decay) is required to improve learning stability [

In the vanilla FL setting, a single globally-shared model is trained to fit the “average client”. As a result, the global model will not generalize well for a local distribution that is very different from the global distribution. Having a single model is often insufficient for practical applications which often face non-IID local datasets. Taking the example of applying FL to develop language models for mobile keyboards, users from different demographics are likely to have divergent usage patterns due to diverse generational, linguistic and cultural nuances. Certain words or emojis are likely be used predominantly by specific groups of users. For such a scenario, a more tailored prediction pattern is needed for each individual user in order for the word

Fig. 2: Illustration of client drift in FedAvg for 2 clients with 2 local steps. 

There are several surveys on the general concepts, methods and applications of FL [

]. Others review FL from the perspectives of privacy [

]. Our survey focuses on PFL, which studies the problem of learning personalized models to handle statistical heterogeneity under the FL setting. There is a shortage of a comprehensive survey on PFL that provides a systematic perspective on this important topic for new researchers. In this paper, we bridge this gap in the current FL literature. Our main contributions are summarized as follows:

We provide a succinct overview of FL and its categorization. A detailed analysis of the key motivations for PFL in the current FL settings is also included.

We identify personalization strategies to address key FL challenges, and offer a unique data-based, model-based, architecture-based and similarity-based perspective for guiding the review of the PFL literature. Based on this perspective, we propose a hierarchical taxonomy to present existing works on PFL, highlighting the challenges they face, their main ideas and assumptions they made which could introduce potential limitations.

We discuss commonly adopted public datasets and evaluation metrics in the current literature for PFL benchmarking, and offer suggestions on enhancing PFL experimental evaluation techniques.

We envision promising future trajectories of research towards new architectural design, realistic benchmarking, and trustworthy approaches towards building personalized federated learning systems.

In this section, we provide an overview of the PFL strategies which are the basis for our systematic and comprehensive review of existing PFL approaches. We organize the literature around the proposed taxonomy (Fig. 

c) that divides PFL methods according to the key challenges and personalization strategies involved.

The first strategy addresses the performance issues in training a globally-shared FL model on heterogeneous data. When learning on non-IID data, the accuracy of FedAvgbased approaches is significantly reduced due to client drift. Under global model personalization, the PFL setup closely follows the general FL training procedure where a single global FL model is trained. The trained global FL model is then personalized for each FL client through a local adaptation step that involves additional training on each local dataset. This two-step “FL training + local adaptation” approach is commonly regarded as an FL personalization strategy by the FL community [

]. As personalization performance directly depends on the generalization performance of the global model, many PFL approaches aim to improve the performance of the global model under data heterogeneity in order to improve the performance of subsequent personalization on local data. Personalization techniques for this category are classified into data-based and model-based approaches. Databased approaches aim to mitigate the client drift problem by reducing the statistical heterogeneity among the clients’ datasets, while model-based approaches aim to learn a strong global model for future personalization on individual clients or improve the adaptation performance of the local model.

Strategy II: Learning Personalized Models 

The second strategy addresses the challenge of solution personalization. In contrast to the global model personalization strategy which trains a single global model, approaches in this category train individual personalized FL models. The goal is to build personalized models by modifying the FL model aggregation process. This is achieved through applying different learning paradigms in the FL setting. Personalization techniques are classified into architecture-based and similarity-based approaches. Architecture-based approaches aim to provide a personalized model architecture tailored to each client, while similarity-based approaches aim to leverage client relationships to improve personalized model performance where similar personalized models are built for related clients.

In personalized FL model training, the optimization objective is formulated differently from the vanilla FL setting, as an individual personalized model is learned for each client. Here, we provide formulations of the optimization objectives under the FL setting and the local learning setting in order to highlight the positioning of PFL approaches. The standard FL objective is given as

encodes the parameters of the global model and

represents the expected loss over the data distribution 

. The prevailing FL formulation minimizes the aggregation of local functions and entails a common output for all clients using the global model without any personalization. In the presence of data heterogeneity (i.e., the underlying data distributions across the clients are not identical), simply minimizing the average local loss with no personalization will result in poor performance.

At the opposite end of the spectrum, we consider a local learning setting where each client 

locally without any communication with other clients. The objective is given as

encodes the parameters of the local model of client 

. In this setting, the resulting models may not achieve good generalization performance as the number of training examples that the local models are exposed to are limited. Stronger generalization guarantees can be obtained with more collaboration amongst clients to exploit the pool of knowledge for model training.

Comparing the formulations of the standard FL and local learning settings, standard FL facilitates collaboration and knowledge sharing amongst clients but does not entail personalized outputs as it relies on a shared global model for client inference. On the other hand, local learning entails a fully personalized model for each client, but fails to leverage potential performance gains from inter-client collaboration. Given the need to achieve a balance between generalization and personalization performance, PFL approaches fall between the standard FL setting and the local learning setting.

In this section, we survey PFL approaches following the global model personalization strategy. The main setup and configurations for these approaches are illustrated in Fig. 

. Based on our proposed taxonomy, they are divided into 

Motivated by the client drift problem arising from federated training on heterogeneous data, data-based approaches aim to reduce the statistical heterogeneity of client data distributions. This helps to improve the generalization performance of the global FL model.

As the IID property of training data is a fundamental assumption in statistical learning theory, data augmentation methods to enhance the statistical homogeneity of the data have been extensively studied in the field of machine learning. Over-sampling techniques involving synthetic data generation (e.g., SMOTE [

]), and under-sampling techniques (e.g., Tomek links [

]) have been proposed to reduce data imbalance. These techniques, however, cannot be directly applied under the FL setting, where data residing at the clients in the federation are distributed and private.

a) is highly challenging as it often requires some form of data sharing or relies on the availability of a proxy dataset that is representative of the overall data distribution. In [

], the authors proposed a data sharing strategy that distributes a small amount of global data balanced by classes to each client. Their experiments show that there is potential for significant accuracy gains (

30%) with the addition of a small amount of data. In [

], the authors proposed FAug, a federated augmentation approach that involves training a Generative Adversarial Network (GAN) model in the FL server. Some data samples of the minority classes are uploaded to the server to train the GAN model. The trained GAN model is then distributed to each client to generate additional data to augment its local data to produce an IID dataset. In [

], the authors proposed Astraea, a self-balancing FL framework to handle class imbalance by using Z-score based data augmentation and down-sampling of local data. The FL server requires statistical information about clients’ local data distributions (e.g., class sizes, mean and standard deviation values). In [

], the authors proposed the FedHome algorithm that trains a Generative Convolutional Autoencoder (GCAE) model using FL. At the end of the FL procedure, each client performs further personalization on a locally augmented class-balanced dataset. This dataset is generated by executing the SMOTE algorithm on the low dimensional features of the encoder network based on the local data.

Another line of work focuses on designing FL client selection mechanisms to enable sampling from a more homogeneous data distribution, with the aim of improving model generalization performance (Fig. 

], the authors proposed FAVOR which selects a subset of participating clients for each training round in order to mitigate the bias introduced by nonIID data. A deep Q-learning formulation for client selection was designed with the objective of maximizing accuracy, while minimizing the number of communication rounds. In a similar approach, a client selection algorithm based on the MultiArmed Bandit formulation was proposed in [

] to select the subset of clients with minimal class imbalance. The local class distributions are estimated by comparing the similarity between the local gradient updates submitted to the FL server with the gradients inferred from a balanced proxy dataset residing on the server.

Recently, there is an emerging line of work that focuses on developing client selection strategies to tackle data and resource heterogeneity challenges that are prevalent in edge computing applications. For cross-device FL, there is often significant variability in hardware capabilities in terms of computation and communication capacities. Heterogeneity also exists in data, whereby the quantity and distribution of data differ among clients. Such diversity exacerbates challenges such as communication costs, stragglers and model accuracy. In [

], the authors proposed a tier-based FL system (TiFL) that groups clients into tiers based on training performance. The algorithm adaptively selects participating clients from the same tier for each training round by optimizing both accuracy and training time. This helps alleviate the performance issues caused by data and resource heterogeneity. In [

], the authors proposed FedSAE, a self-adaptive FL system that adaptively

Fig. 3: The setup & configurations of approaches that fall under 

) regularized local loss; regularization can be performed 

) between historical local model snapshots, (

selects clients with larger local training loss in each training round to accelerate the convergence of the global model. A prediction mechanism of the affordable workload of each client is also proposed to enable the dynamic adjustment of the number of local training epochs for each client in order to improve device reliability.

Although data-based approaches improve the convergence of the global FL model by mitigating the client drift problem, they generally need to modify the local data distributions. This may result in loss of valuable information associated with the inherent diversity of client behaviors. Such information can be useful for personalizing the global model for each client. In this section, we cover model-based global model personalization FL approaches. The objective is either to learn a strong global FL model for future personalization on each individual client, or to improve the adaptation performance of the local model.

Model regularization is a common strategy for preventing overfitting and improving convergence when training machine learning models. In FL, regularization techniques can be

applied to limit the impact of local updates. This improves convergence stability and the generalization of the global model, which in turn, can be used to produce better personalized models. Instead of just minimizing the local function 

is the regularization loss, which is generally formulated as a function of the global model 

. Regularization can be applied in the following ways as illustrated in Fig. 

Several works implement regularization between the global and local models to tackle the client drift problem that is prevalent in FL due to statistical data heterogeneity. FedProx [

] introduced a proximal term to the local sub-problem which considers the dissimilarity between the global FL model and local models to adjust the impact of local updates. Along with model dissimilarity, FedCL [

] further considers parameter importance in the regularized local loss function using Elastic Weight Consolidation (EWC) [

] from the field of continual learning. The importance of the weights to the global model is estimated on a proxy dataset in the FL server. They are then transferred to the clients where penalization steps are carried out to prevent important parameters of the global model from being changed when adapting the global model to clients’ local data. Doing so alleviates the weight divergence between the local and global models, while preserving the knowledge of the global model to improve generalization. Recently, SCAFFOLD [

] uses variance reduction to alleviate the effect of client drifting that causes weight divergence between the local and global models. The update directions of the global (

, is added as a component of the local loss function to correct local updates.

2) Between Historical Local Model Snapshots: 

Recently, a contrastive learning-based FL – MOON [

] has been proposed. The goal of MOON is to reduce the distance between the representations learned by the local models and the global model (i.e., to alleviate weight divergence), and increase the distance between the representations learned between a given local model and its previous local model (i.e., to speed up convergence). This emerging approach enables each client to learn a representation close to the global model to minimize local model divergence. It also speeds up learning by encouraging the local model to improve from its previous version.

Commonly known as “learning to learn”, meta-learning aims to improve the learning algorithm through exposure to a variety of tasks (i.e., datasets) [

]. This enables the model to learn a new task quickly and effectively. Optimizationbased meta-learning algorithms, like Model-Agnostic MetaLearning (MAML) [

], are known for their good generalization and fast adaptation on new heterogeneous tasks. They are also model-agnostic and can be applied to any gradient descent-based approaches, enabling applications in supervised learning and reinforcement learning.

], the authors drew parallels between meta-learning and FL. Meta-learning algorithms run in two phases: metatraining and meta-testing. The authors mapped the metatraining step in MAML to the FL global model training process, and the meta-testing step to the FL personalization process in which a few steps of gradient descent are performed on local data during local adaptation. They also show that FedAvg is analogous to the Reptile algorithm, and are in fact equivalent when all clients possess equal amounts of local data. Given the similarities in the formulations of meta-learning and FL algorithms, meta-learning techniques can be applied to improve the global FL model, while achieving fast personalization on the clients (Fig. 

], which is a variant of FedAvg built on top of the MAML formulation, has also been proposed.

is the step size. The cost function can be written as the average of meta-functions 

is the meta-function associated with client 

. In contrast to the optimization objective of FedAvg in Eq. (

) which aims to learn a global model that performs well on most participating clients, the new goal is transformed to learn a good initial global model that performs well on a new heterogeneous task after it is updated with a few steps of gradient descent. This problem formulation is suitable for learning an improved global model initialization for stronger personalization on local data silos with heterogeneous distributions. However, this approach is computationally expensive due to the need to compute second-order gradients. To reduce computational overhead, the authors evaluated 2 forms of gradient approximations: (i) FO-MAML [

], which replaces the gradient estimate with its first-order approximation where the Hessian term is ignored; and (ii) HF-MAML [

], which replaces the Hessian-vector product with gradient differences. It has been found that HF-MAML achieves better gradient approximation.

The idea of Per-FedAvg has been extended in [

] to propose a federated meta-learning formulation using Moreau envelopes (pFedMe). It incorporates an 

-norm regularization loss which can control the balance between personalization and generalization performance. It has achieved improved accuracy and convergence over FedAvg and Per-FedAvg.

] proposed the ARUBA framework. It is based on online learning to achieve adaptive meta-learning under FL settings. When combined with FedAvg, it improves model generalization performance and eliminates the need for hyperparameter optimization during personalization.

Transfer learning (TL) is commonly used for model personalization in non-federated settings [

]. It aims to transfer knowledge from a source domain to a target domain, where both domains are often different but related. TL is an efficient approach that leverages knowledge transfer from a pre-trained model, thereby avoiding the need to build models from scratch. TL-based PFL approaches have also emerged. FedMD [

] is an FL framework based on TL and knowledge distillation for clients to design independent models using their own private data. Before the FL training and knowledge distillation phases, TL is first carried out using a model pre-trained on a public dataset. Each client then fine-tunes this model on its private data.

Domain adaptation TL techniques are commonly adopted to achieve PFL. These techniques aim to reduce the domain discrepancy between the trained global FL model (i.e., the source domain) and a given local model (i.e., the target domain) for improved personalization. There are several studies in FL that uses TL in the healthcare domain for model personalization (e.g., FedHealth [

]). The training procedure generally involves three steps: (i) training a global model via FL; (ii) training local models by adapting the global model on local data; and (iii) training personalized models by refining the local model using the global model via transfer learning. In order to enable domain adaptation, an alignment layer, such as the correlation alignment (CORAL) layer [

], is often added before the softmax layer for adaptation of the second-order statistics of the source and target domains (Fig. 

To reduce training overhead in deep neural networks, the lower layers of the global model are often transferred and

TABLE I: Summary of personalization techniques in 

reused directly in the local models as low level generic features are learned. Other layers of the local model are fine-tuned with the local data to learn task-specific features for personalization.

. We now summarize and compare the personalization techniques in terms of their advantages and disadvantages (as shown in Table 

Data-based approaches aim to reduce the statistical heterogeneity of client data distributions to tackle the problem of client drift. Data augmentation methods are easy to implement in the general FL training procedure. However, the applicability of these data augmentation methods is limited to some extent as the possibility of privacy leakage from data sharing has not been adequately addressed in existing designs. Data samples [

] or data statistics about the clients’ data distributions [

] are often shared during the training process. Client selection methods improve the model generalization performance by optimizing the subset of participating clients for each FL communication round. As this requires computationally intensive algorithms such as deep Q-learning [

], it incurs higher computational overhead than FedAvg. Additionally, many of these data-based approaches assume the availability of a proxy dataset that is representative of the global data distribution [

]. In order to construct such a proxy dataset, it is necessary to understand the global data distribution, which is challenging under FL settings due to privacy-preservation concerns.

Model-based approaches closely follow the general FL training procedure in which a single global model is trained. Regularization methods such as FedProx [

] are easy to implement and they only require a slight mod-ification to the FedAvg algorithm. Meta-learning optimizes the global model for fast personalization. However, gradient approximations are needed as it is expensive to compute second-order gradients [

]. Transfer learning improves personalization by reducing the domain discrepancy between the global and local models. As the above approaches assume a single global model setup where a single global model is learnt over heterogeneous data silos, they are not well-suited for solution personalization when there are significant differences among the client data distributions. Additionally, model-based approaches generally assume that all clients and the FL server share a common model architecture. This assumption requires all clients to have sufficient computation and communication resources. However, edge computing FL clients are often resource-constrained [

In this section, we survey PFL approaches following the strategy of learning personalized models. The main setup and configurations for these approaches are illustrated in Fig. 

achieve personalization through a customized model design that is tailored to each client. Parameter decoupling methods implement personalization layers for each client, while knowledge distillation methods support personalized model architectures for each client.

Parameter decoupling aims to achieve PFL by decoupling the local private model parameters from the global FL model parameters. Private parameters are trained locally on the clients, and not shared with the FL server. This enables task specific representations to be learned for enhanced personalization.

The division between private and federated model parameters is an architectural design decision. There are generally two configurations used in parameter decoupling for deep feed-forward neural networks (Fig. 

a). The first is a “base layers + personalized layers” design proposed by [

]. In this setting, personalized deep layers are kept private by the clients for local training to learn personalized task-specific

Fig. 4: The setup & configurations of approaches that fall under 

Strategy II: Learning Personalized Models

) parameter decoupling; parameter privatization designs include 

) personalized feature representations, (

) knowledge distillation; knowledge can be distilled 

representations, while the base layers are shared with the FL server to learn low-level generic features.

The second design considers personalized feature representations for each client. In [

], a document classification model using a Bidirectional LSTM architecture is trained via FL by treating user embeddings as the private model parameters, and character embeddings (i.e., LSTM and MLP layers) as the FL model parameters. In [

], Local Global Federated Averaging (LG-FedAvg) has been proposed to combine local representation learning and global federated training. Learning lower dimensional local representations improves communication and computational efficiency for federated global model training. It also offers flexibility as specialized encoders can be designed based on the source data modality (e.g., images, texts). The authors also demonstrated how fair and unbiased representations that are invariant to protected attributes (e.g., race, gender) can be learned by incorporating adversarial learning into FL model training.

As the idea of parameter decoupling has some similarities to split learning (SL) [

], a distributed and private machine learning paradigm, we briefly discuss their differences in this section. In SL, the deep network is split layer-wise between the server and the clients. Unlike parameter decoupling, the server model in SL is not transferred to the client for model training. Instead, only the weights of the split layer of the client model are shared during forward propagation and the gradients from the split layer are shared with the client during backpropagation. SL therefore has a privacy advantage over FL as the server and clients do not have full access to the global and local models [

]. However, training is less efficient due to the sequential client training process. SL also performs worse than FL on non-IID data and has higher communication overheads [

In server-based horizontal federated learning (HFL) [

], the same model architecture is adopted by both the FL server and the FL clients. The underlying assumption is that there is sufficient communication bandwidth and computation capacity at the clients. However for practical applications with a large number of edge devices as FL clients, they are often resource-constrained. Clients may also choose to have different model architectures due to different training objectives. The key motivation for knowledge distillation in FL is to enable a greater degree of flexibility to accommodate personalized model architectures for the clients. At the same time, it

also seeks to tackle communication and computation capacity challenges by reducing resource requirement.

Knowledge Distillation (KD) for neural networks was introduced by [

] as a paradigm for transferring the knowledge from an ensemble of teacher models to a lightweight student model. Knowledge is commonly represented as class scores or logit outputs in existing FL distillation approaches. In general, there are four main types for distillation-based FL architectures: (i) distillation of knowledge to each FL client to learn stronger personalized models, (ii) distillation of knowledge to the FL server to learn stronger server models, (iii) bi-directional distillation to both the FL clients and the FL server, and (iv) distillation amongst clients (Fig. 

], the authors proposed FedMD, a distillation-based FL framework which allows clients to design diverse models using their own private data via KD. Learning occurs through a consensus computed using the average class scores on a public dataset. For every communication round, each client trains its model using the public dataset based on the updated consensus, and fine-tunes its model on its private dataset thereafter. This enables each client to obtain its own personalized model while leveraging knowledge from other clients. In [

], the authors proposed FedGen, a data-free distillation framework that distills knowledge to the FL clients. A generative model is trained in the FL server and broadcast to the clients. Each client then generates augmented representations over the feature space using the learned knowledge as the inductive bias to regulate its local learning.

], the authors proposed the FedDF algorithm. It assumes a setting in which the edge clients require different model architectures due to diverse computational capabilities. The FL server constructs 

distinct prototype models, each representing clients with identical model architectures (e.g., ResNet, MobileNet). For each communication round, FedAvg is first performed among clients from the same prototype group to initialize a student model. Cross-architecture learning is then performed via ensemble distillation, in which the client (teacher) model parameters are evaluated on an unlabelled public dataset to generate logit outputs that are used to train each student model in the FL server.

Knowledge may also be distilled in a bi-directional manner between the FL client and FL server within the same FL training procedure. In [

], the authors proposed Group Knowledge Transfer (FedGKT) to improve model personalization performance for resource-constrained edge devices. It uses alternating minimization to train small edge models and a large server model through a bi-directional distillation approach. The large server model takes extracted features from the local models as inputs, and uses the KL-divergence loss to minimize the difference between the ground truth and soft labels predicted by the local models. By doing so, the server model absorbs the knowledge transferred from the local models. Similarly, each local model calculates the KLdivergence loss using its private dataset and the predicted soft labels transferred from the server. This facilitates knowledge transfer from the server model to the local models. Using this bi-directional distillation framework, computation burden is shifted from the edge clients to the more powerful FL server. However, there is potential privacy risk as the ground truth labels from each client are uploaded to the FL server.

KD-based PFL may also be carried out in distributed settings where knowledge is transferred amongst neighboring clients in a network. In [

], the authors proposed an architecture agnostic distributed algorithm for on-device learning – D-Distillation. It assumes an IoT edge FL setting in which every edge device is connected to only a few neighboring devices. Only connected devices can communicate with each other. The learning algorithm is semi-supervised, with local training performed on private data and federated training on an unlabeled public dataset. For each communication round, each client broadcasts its soft decisions to its neighbors, while receiving their soft decision broadcasts. Each client then updates its soft decisions based on its neighbors’ soft decisions via a consensus algorithm. The updated soft decisions are then used to update the client’s model weights by regularizing its local loss. This procedure facilitates model learning via knowledge transfer amongst neighboring FL clients in a network.

Similarity-based approaches aim to achieve personalization by modeling client relationships. A personalized model is learned for each client, with related clients learning similar models. Different types of client relationships have been studied in PFL. Multi-task learning and model interpolation consider 

clustering considers group level client relationships.

The goal of multi-task learning (MTL) is to train a model that jointly performs several related tasks. This improves generalization by leveraging domain-specific knowledge across the learning tasks. By treating each FL client as a task in MTL, there is potential to learn and capture relationships among the clients exhibited by their heterogeneous local data (Fig. 

] has been proposed to extend distributed MTL into the FL settings. MOCHA uses a primal-dual formulation to optimize the learned models. The algorithm addresses communication and system challenges prevalent in FL which are not considered in the field of MTL. Unlike the conventional FL design which learns a single global model, MOCHA learns a personalized model for each FL client. While MOCHA improves personalization, it is not suitable for cross-device FL applications as all clients are required to participate in every round of FL model training. Another drawback of MOCHA is that it is only applicable to convex models, and is thus unsuitable for deep learning implementations. This motivated [

] to propose the VIRTUAL federated MTL algorithm that performs variational inference using a Bayesian approach. Although it can handle non-convex models, it is computationally expensive for large-scale FL networks.

], the authors proposed FedAMP, an attention-based mechanism that enforces stronger pair-wise collaboration amongst FL clients with similar data distributions. In contrast to the standard FL framework in which a single global model is maintained by the server, FedAMP maintains a personalized cloud model 

for each client in the server. The personalized cloud model 

is the linear combination of the local client models 

to perform local training on its own dataset. The local weights are computed as:

] uses EWC to prevent catastrophic forgetting when moving across learning tasks. Parameter importance is estimated using the Fisher information matrix and penalization steps are carried out to preserve important parameters. At the end of each communication round, each client sends its updated local parameters and the diagonal of its Fisher information matrix to the server. These parameters will be shared among all clients to perform local training in the next round.

], a new formulation that learns personalized models using a mixture of global and local models has been proposed to balance generalization with personalization (Fig. 

d). Each FL client learns an individual local model. A penalty parameter 

is used to discourage the local models from being too dissimilar from the mean model. Pure local model learning occurs when 

is set to zero. This is equivalent to the fully personalized FL setting in Eq. (

) where each client trains its own model locally without any communication with other clients. As 

increases, mixed model learning occurs and the local models become increasingly similar to each other. The setting approximates global model learning in which all local models are forced to be identical when 

approaches infinity. In this way, the degree of personalization can be controlled. Additionally, the authors proposed a communication-efficient variant of SGD known as the Loopless Local Gradient Descent (L2GD). Through a probabilistic framework that determines whether a local GD step or a model aggregation step is to be performed, the number of communication rounds is reduced significantly.

] proposed the APFL algorithm with the goal of finding the optimal combination of global and local models in a communication-efficient manner. They introduced a mixing parameter for each client which is adaptively learned during the FL training process to control the weights of the global and local models. This enables the optimal degree of personalization for each client to be learned. The weighting factor on a particular local model is expected to be larger if the local and global data distributions are not well-aligned, and vice versa. A similar formulation involving the joint optimization of local and global models to determine the optimal interpolation weight has been proposed in [

] proposed the HeteroFL framework which trains local models with diverse computational complexities, based on a single global model. By adaptively allocating local models of different complexity levels according to the computation and communication capabilities of each client, it achieves PFL to address system heterogeneity in edge computing scenarios.

For applications in which there are inherent partitions among clients or data distributions that are significantly different, adopting a client-server FL architecture to train a shared global model is not optimal. A multi-model approach in which an FL model is trained for each homogeneous group of clients is more suitable (Fig. 

e). Several recent works focus on clustering for FL personalization. The underlying assumption of clustering-based FL is the existence of a natural grouping of clients based on their local data distributions.

], hierarchical clustering has been incorporated into FL as a post-processing step. An optimal bi-partitioning algorithm based on cosine similarity of the gradient updates from the clients is used divide the FL clients into clusters. As multiple communication rounds are needed to separate all incongruent clients, the recursive bi-partitioning clustering framework incurs high computation and communication costs that limit practical feasibility for large-scale settings. Another hierarchical clustering framework for FL has been proposed in [

]. It uses an agglomerative hierarchical clustering formulation that reduces clustering to a single step to lower computation and communication loads. The procedure begins by first training a global FL model for 

communication rounds. The global model is then fine-tuned on the private datasets of all clients to determine the difference 

values for all clients are used as inputs to the agglomerative hierarchical clustering algorithm to generate multiple client clusters. FL training is then performed independently for each client cluster to produce multiple federated models. This approach is designed for a wider range of non-IID settings and allows training on a subset of clients during each round of FL model training. However, computing the pairwise distance between all clients in agglomerative clustering can be computationally intensive when there are a large number of clients.

Other clustering approaches require a fixed number of clusters to be set at the beginning of FL training. In [

], the authors proposed the Iterative Federated Clustering Algorithm (IFCA). Instead of a single global model, the server constructs 

global models and broadcasts these models to all clients for local loss computation. Each client is assigned to one of the 

clusters the global model of which achieves the lowest loss value on the client’s data. Cluster-based FL model aggregation within the cluster partition is then performed by the server. Compared to FedAvg, the communication overhead of IFCA is 

times higher as the server needs to broadcast 

cluster models to all clients in every communication round.

], the authors proposed community-based FL (CBFL) to predict patient hospitalization time and mortality. They trained a denoising autoencoder and performed K-means clustering with a pre-determined number of clusters to cluster patients based on the encoded features of their private data. An FL model is then trained for each cluster.

TABLE II: Summary of personalization techniques in 

], the authors proposed FedGroup, an FL clustering framework that implements a static client clustering strategy and a newcomer client cold start mechanism. FedGroup performs clustering on the local client updates using the Kmeans++ algorithm [

] based on the Euclidean distance of the Decomposed Cosine similarity (EDC).

], the authors proposed a multi-center formulation that learns multiple global models. It introduces a new distance-based multi-center loss function:

. Expectation Maximization is used to solve the distance-based objective clustering problem and derive the optimal matching of clients to each cluster center. In the E-step, the cluster assignment 

. Otherwise, it is set to 0. In the M-step, the cluster centers 

to perform fine-tuning of the local model parameters 

on its private training data. The above steps are repeated until convergence.