Data privacy is a significant concern in today’s world, with many countries enacting laws just like the EU’s General Data Protection Regulation (GDPR) to guard personal information. In the sector of machine learning, a key issue arises when clients want to leverage pre-trained models by transferring them to their data. Sharing extracted data features with model providers can potentially expose sensitive client information through feature inversion attacks.

Previous approaches to privacy-preserving transfer learning have relied on techniques like secure multi-party computation (SMPC), differential privacy (DP), and homomorphic encryption (HE). While SMPC requires significant communication overhead and DP can reduce accuracy, HE-based methods have shown promise but suffer from computational challenges. 

A team of researchers has now developed HETAL, an efficient HE-based algorithm (shown in Figure 1) for privacy-preserving transfer learning. Their method allows clients to encrypt data features and send them to a server for fine-tuning without compromising data privacy.

On the core of HETAL is an optimized process for encrypted matrix multiplications, a dominant operation in neural network training. The researchers propose novel algorithms, and , that significantly reduce the computational costs in comparison with previous methods. Moreover, HETAL introduces a brand new approximation algorithm for the softmax function, a critical component in neural networks. Unlike prior approaches with limited approximation ranges, HETAL’s algorithm can handle input values spanning exponentially large intervals, enabling accurate training over many epochs.

The researchers demonstrated HETAL’s effectiveness through experiments on five benchmark datasets, including MNIST, CIFAR-10, and DermaMNIST (results shown in Table 1).

HETAL addresses an important challenge in privacy-preserving machine learning by enabling efficient, encrypted transfer learning. The proposed method protects client data privacy through homomorphic encryption while allowing model fine-tuning on the server side. Furthermore, HETAL’s novel matrix multiplication algorithms and softmax approximation technique can potentially profit other applications involving neural networks and encrypted computations. While limitations may exist, this work represents a major step towards practical, privacy-preserving solutions for machine learning as a service.

Try the Paper and Github. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group.

In the event you like our work, you’ll love our newsletter..

Don’t Forget to affix our 39k+ ML SubReddit