Day 2 - Friedemann Zenke, Matthew Cook on Neural Computation



After the coffee break, Matthew Cook kicked off the 2nd part of the morning asking, "What is computation"? It is NOT the computer science view of algorithm with a halting condition. Is a bucket of water? Yes and no, in any case this argument about definitions is not fruitful but considering different ways to compute is useful.

This generated an immense amount of discussion about the meaning of "computation" in the brain and in computer science. After this discussion was used up, Matt pointed out the continuing need (last made in his great 2019 talk at CCNW) for increasing the models of computation. There is always room to at least consider more models; most will not work for AI or not explain anything new, but there will occasionally be new insights from new ways of looking at computing.

Among the models considered, Turing machines, cellular automata, neural networks, synchronous RTL logic circuits, various computer architectures, cortex.

One definition of a computation is "a useful reduction of information", e.g. 4+2=6, the 4 and 2 are lost but the sum is maintained. Another is a spiking neuron: It integrates its input and occasionally deems it worthy to spike. Another is a transformation of representation according to Barbara Webb, correlation by itself is not useful but prediction based on it is?XXX

There were many points of view presented; too many to note here in real time.

Then Friedemann took over and continued the discussion about the nature of neural computation. He brought up the question of what AI and DNN community has brought in to enable their impressive results. From his list the audience contributed the following list of things that AI has effectively exploited:

  • A big variety of clever architectures like CNN, resnets, soft WTA layers, RNNs, transformers, attention and multi-headed attention, some of which have proven very effective but seem to have only vague connection with known neuroscience.
  • Clever hand crafting of loss functions and objectives like cross entropy
  • Lots of data, labeled and unlabeled
  • Reinforcement learning that can discover optimal loss functions and control policies from trial and error with reward
  • Self supervision to predict the future or context in unlabeled data
  • Training/optimization algorithms like backprop and ADAM that can solve the credit assignment problem for changing weights and escaping local minima and flat regions, like LSTMs and GRUs do for RNNs that are trained with BPTT.
  • Activation functions like ReLU and leaky ReLU, and custom neural units, e.g. as in RNNs or max pooling units.
  • Transfer learning in particular as a way to augment limited real data for pretraining DNNs
This discussion, although often devolving to arguments of the definition of "computation" seemed very interesting to many in the audience who contributed to a lively discussion which brought us to before lunch walks or swimming or snorkeling.









Comments

Post a Comment

Popular posts from this blog

Day 9 - Ben Grewe and Pau Aceituno - Cortical neaning rules and dynamical states

Image
 This morning at 930 we heard very interesting session on the beach from Ben Grewe and Pau Vicente, both from Zurich. The slowly increasing number of positive COVID cases forced cancellation of all indoor groupings, and so the beach event took place under brilliant blue sky and shimmering sea. Benni started off by making the bold statement that learning rules did not matter. What matters is what the cell wants to do, via a cost function that the cells try to optimize. In the end, via calcium second messengers, the cells ultimately seems to have two attractor states, resting and active, and cellular feedback mechanisms always make neurons move to these stable down and up states. After the coffee break (where participants were exhorted not to blow hot air at each other), Pau Aceituno took over. He looks at the problem in terms of Shannon information. The S sensor B brain and A action represent an agent (animal or artificial) that must process information. Just like in a Morse code where
Read more

Day 3 on Navigation: Barbara Webb, Julien Serres and Pavan Ramdya

Image
The day started with an introduction from Florian Engert who posed the question " What problems are animal solving or evolved to solve?". Then he posed three specific questions that are interesting to ask: First: Identify what is the problem the animal is solving.   Second: how does it solve it? What is the nature of the input? What is the dynamics and what is the precise nature of the output? What is the algorithm through which the animal solve it?   Third: How is it implemented?   He mentioned that the day reovolves around these questions in the context of navigation. Naviagation in terms of Today is insect-heavy.     Baraba mentioned that you can never be insect heavy (she loves insect brain) :)   Although today's discussion is mostly about insects, Florian mentioned that he believes the same principles apply to all animals.     Barbara started by asking only students (and not faculty and navigation experts) to say what comes to their minds when they think about the na
Read more