Skip to content

DPsurvey.md

Latest commit

 

History

History
45 lines (39 loc) · 5.34 KB

DPsurvey.md

File metadata and controls

45 lines (39 loc) · 5.34 KB

Survey of differential programming

This is a preliminary survey of the literature on differential programming, by which we mean a system which couples together some kind of deep neural network controller with a differentiable memory module. Some of the papers below are not directly about differential programming in this sense, but they are relevant to our approach to the subject. Our purpose here is to list the models used and the benchmark tasks considered.

The papers

  1. Alex Graves, Greg Wayne, Ivo Danihelka "Neural Turing machines" 2014 link.
  2. Armand Joulin, Tomas Mikolov "Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets" 2015 link.
  3. Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, Phil Blunsom "Learning to Transduce with Unbounded Memory" 2015 link.
  4. Jason Weston, Sumit Chopra, Antoine Bordes "Memory Networks" 2014 link.
  5. Scott Reed, Nando de Freitas "Neural Programmer-Interpreters" 2015 link, video.
  6. Wojciech Zaremba, Tomas Mikolov, Armand Joulin, Rob Fergus "Learning simple algorithms from examples" 2015 link, video.
  7. Arvind Neelakantan, Quoc V. Le, Ilya Sutskever "Neural Programmer: Inducing Latent Programs with Gradient Descent" 2015 link.
  8. Wojciech Zaremba, Ilya Sutskever "Learning to Execute" 2014 link
  9. Wojciech Zaremba, Ilya Sutskever "Reinforcement Learning Neural Turing Machines - Revised" 2015 link.
  10. Rudy Bunel, Alban Desmaison, Pushmeet Kohli, Philip H.S. Torr, M. Pawan Kumar "Adaptive Neural Compilation" 2016 link, GitHub, supplementary materials (includes task details).
  11. Caglar Gulcehre, Sarath Chandar, Kyunghyun Cho, Yoshua Bengio "Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes" 2016 link.
  12. Łukasz Kaiser, Ilya Sutskever "Neural GPUs Learn Algorithms" 2015 link, video.
  13. Nal Kalchbrenner, Ivo Danihelka, Alex Graves "Grid Long Short-Term Memory" link.
  14. Karol Kurach, Marcin Andrychowicz, Ilya Sutskever "Neural Random-Access Machines" 2015 link.
  15. Sebastian Riedel, Matko Bošnjak, Tim Rocktäschel "Programming with a Differentiable Forth Interpreter" 2016 link.
  16. Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, Timothy Lillicrap "One-shot Learning with Memory-Augmented Neural Networks" 2016 link.
  17. Yuhuai Wu, Saizheng Zhang, Ying Zhang, Yoshua Bengio, Ruslan Salakhutdinov "On Multiplicative Integration with Recurrent Neural Networks" 2016 link.
  18. Ozan İrsoy, Claire Cardie "Modeling Compositionality with Multiplicative Recurrent Neural Networks" 2014 link.
  19. Wojciech Zaremba's thesis "Learning algorithms from data" 2016 link.
  20. Marcin Andrychowicz, Karol Kurach "Learning Efficient Algorithms with Hierarchical Attentive Memory" 2016 link.

The tasks

Generally the sequence tasks are run either on sequences of binary numbers, sequences of vectors of binary numbers, or sequences of integers 0-9.

  • Copy (given a sequence, return the same sequence): [1], [3], [6], [9], [14].
  • Repeat copy (given a sequence S and integer N, return the sequence S repeated N times): [1], [9].
  • Reverse (given a sequence, return the reversed sequence): [3], [6], [9], [14], [20].
  • Addition (given a pair of sequences representing integers in some base, return the sequence representing their sum): [2], [5], [6], [12], [20].
  • Multiplication (given a pair of sequences representing integers in some base, return the sequence representing their product): [6], [12].
  • Search (given a sequence and a symbol, return the first index of the symbol in the sequence): [14], [20]
  • Merge (given a pair of sequences, merge them): [14], [20]
  • Sort (given a sequence of integers, return it in ascending order): [1], [5], [20].
  • Bigram flipping (given a sequence of even length, decompose it linearly into pairs and transpose each pair): [3].
  • De-duplicate (given a sequence in which every symbol is repeated consecutively N times, return the sequence with all N duplicates identified): [9].
  • Array (given an index and a sequence, return the element of the sequence at that index): [14].
  • Increment (given a sequence of integers, increment each element): [14].
  • Swap (given a sequence and two indices, swap the symbols at those indices): [14].
  • Permutation (given a sequence of integers representing a permutation, and an arbitrary sequence of the same length, permute the second sequence according to the first): [14].