Project Fiesole
Warning
The question
Can we VFL MNIST?
Project Fiesole tries to show that a CNN can work inside Vertical Federated Learning, so models improve from more data without anyone handing their data over. The data lives in islands here. Different organizations hold different features about the same people. Instead of pooling it all in one place, VFL lets them train one model together, each side bringing its own features. Data stays private, and the model still sees inputs that no single dataset would have. Fiesole is a subproject of Project Florence.
Method
In VFL each island holds different "input parameters" for the same "subject". So in this demo we split the original image (here, 9) into four distinct pieces. Each will necessarily work as a "distinct input."
I made four training sets, one per quadrant, with masking.
Control Group
Ran all ten epochs on the original dataset.
- MNIST CNN: 99.16 %
One by One
I ran ten epochs on one dataset without coming back to it, then moved to the next one.
- TL(10): 26.82 %
- TL(10), TR(10): 36.44 %
- TL(10), TR(10), BL(10): 30.66 %
- TL(10), TR(10), BL(10), BR(10): 47.24 %
Round Robin
Round-robin
One epoch on TL, then TR, BL, and BR, then back around. Four cycles, 16 epochs total.
- Cycle 1
- TL(1): 59.37 %
- TL(1), TR(1): 79.6 %
- TL(1), TR(1), BL(1): 88.51 %
- TL(1), TR(1), BL(1), BR(1): 79.88 %
- Cycle 2
- TL(1): 85.68 %
- TL(1), TR(1): 87.69 %
- TL(1), TR(1), BL(1): 91.4 %
- TL(1), TR(1), BL(1), BR(1): 88.52 %
- Cycle 3
- TL(1): 82.21 %
- TL(1), TR(1): 86.56 %
- TL(1), TR(1), BL(1): 90.98 %
- TL(1), TR(1), BL(1), BR(1): 89.07 %
- Cycle 4
- TL(1): 85.24 %
- TL(1), TR(1): 86.33 %
- TL(1), TR(1), BL(1): 91.99 %
- TL(1), TR(1), BL(1), BR(1): 90.12 %
Round Robin, 10 cycles
Still around 92%, max. Never goes above 95%.
Federated Learning
Results
- The averaged model maxes out around 10-15 federations. Before each one, each learner trains ten epochs on its own dataset.
- It never gets over 95% accuracy.
- The full CNN, by contrast, hits 98% quickly.
- I think this comes from the lack of understanding "across quadrants."
- I didn't implement optimizing features (dropout, reducing LR, etc.)
Adjustments
- Instead of just blacking out three quadrants and leaving one, I'll try adding some noise (a random number from 0-255) to the blacked-out three to see any differences. I hypothesize this can mimic or simulate knowledge across quadrants, slightly improving performance.
- 4
- 5
- Update. Ambient noise drops accuracy a lot, both with one fixed pattern reused across federations and with a fresh pattern generated each time.
Next Steps
- I'll start working on implementing CIFAR-10 with a similar architecture.
Summary
- Performance. The VFL model peaks around 93% accuracy within 10-15 epochs. A full CNN trained on the whole dataset gets to 98% quickly.
- Patterns across quadrants. The big worry is whether the network can pick up patterns that span different quadrants.
- Record linkage. The current setup ignores record linkage, the step that tells which pieces across quadrants belong to the same sample. Adding it might be what gets us past 95%.
Potential Solutions
- Image IDs. Give each image and its quadrants a unique numeric ID. The hope is that the network then learns how the quadrants relate.
- Training variations. Use those IDs in different ways, from one example at a time to batches. Speed is the worry here.
- Noise. Fill the blacked-out parts of the image with random noise. This might fake some knowledge across quadrants and push accuracy up.
Other Considerations
- Testing. There is a separate test set. Every quadrant learner and the averaged learner use it only for evaluation.
- Code review. I shared the relevant code on GitHub so others can dig in and help tune it.
Feedback and Recommendations
- Careful with noise. Use a much smaller noise range instead of 0-255, so the network doesn't overfit to it.
- Talk it through. This needs in-person discussion or longer meetings before we settle on an approach.
Status
✅ SUCCESS
Further Actions
Superseded by anaclumos/malmantile Project Malmantile