Daniele Romanini et al. PyVertical

The paper introduces PyVertical, a framework for vertical federated learning with split neural networks (SplitNNs). It trains a network on data split across several owners, and the raw data never leaves each owner's device. "Vertical" means different owners hold different features of the same dataset. Private Set Intersection (PSI) finds and links the data points those datasets share without giving away anything else. To show it works, the authors trained a split network on MNIST, with each sample split between two owners and a third party (the data scientist) running the training.

Introduction

• Data stuck in separate silos is hard to use for ML, especially when it's sensitive or legally protected.
• The paper looks at Vertical Federated Learning (VFL), which is different from the usual horizontal kind. Horizontal partitioning spreads the same features across owners. Vertical partitioning spreads different features of the same records. Their example is one patient's medical data held by different hospitals. (Qiang Yang et al. lay out both kinds.)

Contributions

• They build on earlier work to use SplitNNs and PSI for VFL.
• They release PyVertical, a new open source framework for training on vertically partitioned data.

Background and related work

PSI (Private Set Intersection)

A cryptographic technique that lets two parties find what their datasets have in common without revealing anything else.

Split neural networks

The network is trained in pieces, and different parties hold different pieces.

Vertical federated learning

Training one model together on a dataset whose features are spread across several stakeholders.

Framework

• PyVertical is a Python framework that uses SplitNNs and PSI for VFL.
• There are one or more data owners and one data scientist. The scientist sets up the split model and sends a segment to each data owner to train.

Experiment

• They split MNIST vertically, cutting each image in half and giving each half to a different data owner, then trained the network on that.

Evaluation and conclusion

• PyVertical is out as an open source project for more VFL research.
• They trained a model on vertically partitioned data, which shows the framework works.

Limitations and future work

• The experiment assumes every party is honest. They call out the need for identity management, making sure parties follow the PSI protocol, and agreeing on data ID schemas.
• It assumes data points are split evenly and every owner gets the same model segment. Asymmetric models and datasets are future work.
• Single-headed.
  • Training is limited to two data owners, and doesn't yet add privacy techniques like decentralized identities or differential privacy.

PyVertical protocol

Figures and implementation

• Figure 2 shows the protocol on MNIST with just one data owner, based on the idea from Angelou et al. (2020).
• Figure 3 shows dual-headed PSI data linkage.
  1. The data scientist finds the intersection with Data Owner 1.
  2. Then the same with Data Owner 2.
  3. The data scientist computes the overall intersection.
  4. So the data scientist runs one PSI with each data owner, one after another or at the same time.

Experimental setup

Data owner model

• Turns a 392-length input into a 64-length intermediate vector.
• ReLU activation, giving an abstract representation of the data.

Data scientist model

• Takes a 128-length vector (the data owners' outputs joined together).
• Turns it into a 10-class vector (softmax) for the possible MNIST digits.
• One 500-length hidden layer with ReLU.
• Every layer is fully connected.

Training parameters

• Data owner model learning rate: 0.01.
• Data scientist model learning rate: 0.1.
• Batch size 128.
• Only the first 20,000 MNIST training images.
• 30 epochs.