Hypernetwork (machine learning)

In machine learning, a hypernetwork is a neural network whose output supplies some or all of the parameters of other neural networks, called target networks. Its inputs can include information associated with a particular task. During its training, gradients can propagate through the generated parameters to update the hypernetwork, and once trained it can generate parameters for different conditioning inputs through a forward pass. Hypernetworks were described in a 2016 paper that demonstrated weight generation for convolutional and recurrent networks. Later research applied hypernetworks to continual learning and personalized federated learning.

See also

References

  1. 1 2 Chauhan, Vinod Kumar; Zhou, Jiandong; Lu, Ping; Molaei, Soheila; Clifton, David A. (2024). "A brief review of hypernetworks in deep learning". Artificial Intelligence Review. 57: 250. doi:10.1007/s10462-024-10862-8.
  2. ↑ Ha, David; Dai, Andrew; Le, Quoc V. (2016). "HyperNetworks". arXiv:1609.09106 [cs.LG].
  3. ↑ von Oswald, Johannes; Henning, Christian; Grewe, Benjamin F.; Sacramento, João (2020). "Continual learning with hypernetworks". International Conference on Learning Representations. arXiv:1906.00695.
  4. ↑ Shamsian, Aviv; Navon, Aviv; Fetaya, Ethan; Chechik, Gal (2021). "Personalized Federated Learning using Hypernetworks". Proceedings of the 38th International Conference on Machine Learning. Proceedings of Machine Learning Research. Vol. 139. pp. 9489–9502.