|
Home | Return to Collected Papers and Articles
NN Discussion on Chimera's Cradle - March 2003
(Ted W.)
Setting aside the philosophical implications, there are fairly well documented 'functional regions' of the brain, which relate to sensory and motor operations. A 'natural neural net' within the brain would have two characteristics which are pertinent to my idea:
1. They are 3 dimensional networks within the brain.
2. The physical location of components of the network may fall within 'functional regions' of the brain.
Accepting these characteristics we would have to consider that shape of the network as well as location of each component is descriptive in relationship to interaction with these 'regions', even ignoring what we typically think of as 'content'. I suspect that the distinction we tend to make between 'processing hardware' and 'storage hardware' is biased by our computing metaphor.
If we assume that a network within the brain somehow represents a memory, then this implies that relation of memory is a function of the physical structure and location of the network specific to the memory. Which if true would mean that descriptive processes or constructs are unnecessary because the physical structure itself defines the relationships. If part of the network touches the "blueness" region of the brain within the visual center, chances are the remembered object is blue (as a weak example). You think about blue and that network in brain becomes active, which raises the activity level of all networks which share axial connections (or physical proximity). Therefore thinking of blue lowers the threshold to activate the structure representing our memory of the sky. I am not implying that there is a direct physical symbolic representation for 'blue' within the brain, but perhaps a network that touches on all of the components that comprise the 'bluish' impression.
To quote Dr. Minsky from a discussion of human reasoning: "..perhaps the most important kind of learning would be in how we refine that 'matching' procedure..", and in another: "Because I feel that most of the generalization is already hidden in the description processes". Should this speculation on my part be true, the implications would be very significant in regards to memory retrieval and human reasoning processes.
I think the uniqueness of the theory is that there would be much less 'processing' occurring in the Von Neumann sense. Meaning there is not a linear search process to determine relations, the relations are already described in the process of creating the physical structure of the memory. This would be one view of possible hardware to provide 'reasoning by analogy'.
I believe that this line of investigation is not a waste of time, given that most 'Artificial Neural Networks' seem to fail to actually represent neural structures in the brain. The ANN seems played out, as it is currently conceived.
(Sam F.)
Hi Ted,
I think that there are some good lines of thought in there. While interesting networks can be made with only a few neurons and fewer layers, they are not going to get us very far, and eventually some architecture is going to have to appear.
You seem to be advocating hard wiring the architecture of the network, if I've understood you correctly. I intuitively feel that the network should be able to evolve its own architecture, but maybe this was what you had in mind.
A 'natural neural net' within the brain would have two characteristics which are pertinent to my idea:
1. They are 3 dimensional networks within the brain.
Apart from ease of use, as far as I can tell there isn't actually any difference between 2 and 3 dimensions as far as the network cares. So long as every neuron is able to reach every neuron that it needs, how does a 3D architecture help? On the other hand, it may well be useful to model the network in 3D if only to give constraints to any structural evolution that the network would go. On the other other hand, is there any reason to constrain the network in any way?
I suspect that the distinction we tend to make between 'processing hardware' and 'storage hardware' is biased by our computing metaphor.
I think that you are certainly right that the distinction isn't as clear in our brains as it is in computers, but actually in most ANNs memory and processing are linked, I think. Thus, if you have a huge neural network, the network might "memorize" its inputs, even if there is no explicit storage space. It might be interesting to try to model the way our brains remember things in an ANN. You'd need to have some kind of echoing around in short-term memory, and Hebbian learning slowly converts the short-term into long term. The system would have to have some way of only keeping the memories it needs.
You think about blue and that network in brain becomes active, which raises the activity level of all networks which share axial connections (or physical proximity). Therefore thinking of blue lowers the threshold to activate the structure representing our memory of the sky.
I think that this relates very closely to what I was getting at in the other forum with my categories thread. How it is that our brains can make these sorts of connections is still a great mystery, as far as I can tell. After all, will you have a dedicated network to "blue" that acts in the way you posit? Will you then also have dedicated networks to all possible features, and maybe all abstract features? Obviously not, yet somehow our brains can still form connection through abstract features.
I believe that this line of investigation is not a waste of time, given that most 'Artificial Neural Networks' seem to fail to actually represent neural structures in the brain.
Indeed. I am in full agreement that ANN's need to improve beyond where they've been since the 40s. We need to be able to handle larger sizes of networks that have the ability to create their own architecture, through cell survival and evolution, in such a way that various areas of the network become functionally distinct. Maybe then we'd start to get somewhere.
(Ted W.)
Thanks Sam, it is something that has been bugging me for several months now, starting from doubts that we reason using formal logic.
You seem to be advocating hard wiring the architecture of the network, if I've understood you correctly. I intuitively feel that the network should be able to evolve its own architecture, but maybe this was what you had in mind.
I have come to the opinion that we are born with certain hard-wired architectural features, at several "levels" of the brain. After that time other structures develop within the basic framework provided. It is how the structures (or sub-networks) develop that is extremely important I think. I am starting to believe that we missed the mark in this regard with our ANNs to date. There is a stage in infant brain development in which a massive increase in connections occurs, and then a development period following in which most of the connections "die off", as if a process of natural selection is occurring. That supports some of my thoughts on reasoning development being a process of reduction more than one of generalization. The "learning by analogy" process as Minsky calls it.
Apart from ease of use, as far as I can tell there isn't actually any difference between 2 and 3 dimensions as far as the network cares. So long as every neuron is able to reach every neuron that it needs, how does a 3D architecture help? On the other hand, it may well be useful to model the network in 3D if only to give constraints to any structural evolution that the network would go. On the other other hand, is there any reason to constrain the network in any way?
I think that the "shape" of the structure, as well as the location of the components are important to the function/information it represents. The physical proximity to other structures may be significant. We are just beginning to discover how little we know about how NNNs grow, but it seems clear that there is a lot more happening than connections. I suspect that proximity is important, even when axial connections don't exist. I recall that one of the speculative functions of glial cells was as insulation.
It might be interesting to try to model the way our brains remember things in an ANN. You'd need to have some kind of echoing around in short-term memory, and Hebbian learning slowly converts the short-term into long term. The system would have to have some way of only keeping the memories it needs.
I agree, and in fact have been working myself towards proposing a project to define a system of ANN which more closely represents what we have learned about neurology. I didn't bother applying the argument to "starchy" on the other forum, but there are clear physical constraints that indicate a distinction between short-term and long-term memory. Clearly a short-term memory can not be represented as a new system of connections between neurons, simply because we remember things faster than connections can grow. I am also intrigued by the significance of a neuron having one output and N inputs (or so we now think). That seems to indicate hierarchical organization, which makes me think the majority of "intelligence" happens at a lower level, and is just observed (and directed to some degree) by our erroneously homogenous seeming "I".
I think that this relates very closely to what I was getting at in the other forum with my categories thread. How it is that our brains can make these sorts of connections is still a great mystery, as far as I can tell. After all, will you have a dedicated network to "blue" that acts in the way you posit? Will you then also have dedicated networks to all possible features, and maybe all abstract features? Obviously not, yet somehow our brains can still form connection through abstract features.
Your comment illuminates one of the areas that most disturbs me about most of the neurological papers I have read. Just as the distinction between process and data is probably a computational metaphor bias, so might our state oriented modeling of dynamic processes reflect our bias towards reductive scientific analysis. By which I mean that we like to create some kind of quantified states for complex systems, thereby allowing us to theorize on the cause of changes between two states. One of the reasons that this can discourage understanding is that it creates an artificial impression of synchronicity in asynchronous systems. Given that the two "snapshots" of our complex system are arbitrary, there is no accurate way to relate temporality to changes in the components of the system. For example, consider two satellite photos of a busy city, separated by some arbitrary amount of time. Without prior understanding, could we accurately extrapolate from this where people were going, let alone why? Brownian Motion is another analogy, prediction of a particles motion is possible, but not by creating an arbitrary sequence of "state" snapshots and trying to produce a linear function to explain the behavior.
I suspect that the brain is made up of an extremely large number of asynchronous interacting systems. Perhaps "blue" does not live within a network, but rather lives within the interactions between networks. There have been sufficient studies to show that there is not a clear physical symbolic representation of images (by repeated viewing of an object during brain scans). Said studies being one of the reasons behind the thoughts in the preceding paragraph...
The idea that we reason by logic still manages to hold on to a number of die-hard fans. It seems that a large proportion of cognitive scientists believe that categories are arranged in our brains in a hierarchical fashion, much like Cyc's system. Thus, if we want to know that my pet dog breathes, we go up the chain from Rex through dog through mammal and up to animal (possibly passing through vertebrates along the way - who knows how fine these levels can be), note that a characteristic of animals is that they breathe, and thus work out that since all animals breathe, Rex breathes. They claim that this idea makes sense because it "saves space," i.e. we don't have to explicitly store in memory that Rex breathes. This system seems ludicrous to me. I think that people are too easily influenced by computers.
(Sam F.)
I have come to the opinion that we are born with certain hard-wired architectural features...
You may have misunderstood what I meant - I didn't put it very well. What I had in mind was not that a single network starts from scratch and slowly forms an architecturally sound brain. Rather, the general shape of the brain would be evolved through generations of networks, each of which would "grow" their architecture based on the combination of the evolved code, and on survival of individual neurons (like the cell death in a two year old).
Cell death may not actually be well enough understood for us to model it aptly. We know that simulation has a large effect on which cells survive, so it seems to be that the cells that are placed such that they carry the best information live, and the dead wood is trimmed off. We need a method of determining which cells to keep.
Interestingly, when I used to build networks to solve silly little puzzles, I would find that I could delete a good third of the connections and not lose any fitness. Sometimes, though, I would delete a single connection and the network could no longer do anything. What I don't know is whether after I trimmed off the irrelevant and redundant neurons I had actually made the network better in any way. It could be that the network was better able to generalize after being made smaller. It could be the opposite. Maybe I should experiment a little.
reasoning development being a process of reduction more than one of generalization. The "learning by analogy" process as Minsky calls it.
Hmm, I should probably take a look at The Society of Mind again. Could you explain what you mean?
I think that the "shape" of the structure, as well as the location of the components are important to the function/information it represents.
Yes, you're perfectly right. I was stuck in the neural networks mode and not in the brain mode. This relates to the glial-cell questions that were being raised in the other forum. Also, the chemical environment that the cells are in must affect how they function.
I am also intrigued by the significance of a neuron having one output and N inputs (or so we now think).
Hmmm, this seems a little sketchy. Let's say you had 100 neurons at the sensory input layer. Then you'd only be able to have a maximum of 50 neurons at the next layer, since there would be no point in having a neuron with only one input. 25 on the next and 12 on the next. This would mean that you could have no more neurons in your entire brain than twice the number of input neurons. If you throw in the fact that our brains really aren't feed-forward networks and there are numerous loops and such, you seriously limit the number of neurons and connections that you can have.
However, this brings me to another point that most ANNs today are stuck in the feed-forward mentality. Even if they have small recursion loops imbedded within them, they basically take an input and give a single output. If the nets were allowed to choose their own architecture we would hopefully see much more information going forward, backwards and around in circles, getting modified by new information all the time. Perhaps some neurons could even take more time to pass on information than others. The result would be a completely non-linear way of dealing with information, much like our own brains.
(Ted W.)
...It seems that a large proportion of cognitive scientists believe that categories are arranged in our brains in a hierarchical fashion, much like Cyc's system. They claim that this idea makes sense because it "saves space," i.e. we don't have to explicitly store in memory that Rex breathes. This system seems ludicrous to me. I think that people are too easily influenced by computers.
I couldn't agree more. I am continually astounded at the assumptive nature of the metaphors used to explain brain phenomena. Even by some of AI's founders! Dr. Minsky has even stated that he thinks we should quit trying to produce homogenous brain models (which I agree with), and that we should model subsystems similar to that in computers, such as cache, memory, mass storage, etc. (which I object to). More specifically though, I think that formal logic is an expression of intelligence, and not an expression of the hardware upon which intelligence operates.
Cell death may not actually be well enough understood for us to model it aptly. We know that stimulation has a large effect on which cells survive, so it seems to be that the cells that are placed such that they carry the best information live, and the dead wood is trimmed off. We need a method of determining which cells to keep.
I agree. I think that this is one of the most critical areas of simulating a real NN, yet has been fairly neglected in research. But then this is one of the most critical aspects to any evolving system: how do you determine the "success" of constituent components to reinforce them? I would lean towards approaching from a "use it or lose it" mentality.
What I don't know is whether after I trimmed off the irrelevant and redundant neurons I had actually made the network better in any way. It could be that the network was better able to generalize after being made smaller. It could be the opposite. Maybe I should experiment a little.
I would be extremely interested in the results!
"reasoning development being a process of reduction more than one of generalization. The 'learning by analogy' process as Minsky calls it. "
Hmm, I should probably take a look at The Society of Mind again. Could you explain what you mean?
Basically the idea is that beliefs/expectations are built from analogy, rather than deductive or inductive process. So characteristics of a concept are projected from the characteristics to similar concepts. My version was a little more assumptive than his, but he still found amusement in the idea that deduction has very little to do with thought under my version. Here is a snippet from our correspondence on the subject:
-------------snip----------
(TW) The alternative theory then being that infants' perceive everything in completely (and inappropriately) generalized terms, and go through a process of refinement by removing assumed relationships (characteristics) when specific experience contradicts them. That is an upside down model of how the process is typically perceived... But what if AI researches having been trying to solve the problem backwards?
(MM) That where I think I agree again, if I understand you. Because I feel that most of the generalization is already hidden in the description processes. Which, in the case of Vision, involve fully half of the human brain! If so, then the role of 'deduction", etc., if I understand you, must surely be vastly less than most of those thinkers seem to assume!
-------------snip----------
Also, the chemical environment that the cells are in must affect how they function.
Absolutely! That is another of the areas that really bother me in previous NN work. The lure of the logic gate analogy has caused far too much emphasis upon the "firing" behavior in my opinion. To the degree that the majority of the physical processes have been ignored in ANN. Perhaps "firing" is related to reinforcement or development, and is not even part of "processing"...
"I am also intrigued by the significance of a neuron having one output and N inputs (or so we now think). "
Hmmm, this seems a little sketchy. Let's say you had 100 neurons at the sensory input layer. Then you'd only be able to have a maximum of 50 neurons at the next layer, since there would be no point in having a neuron with only one input. 25 on the next and 12 on the next.
Yes, you are quite correct, my thoughts in this area are very sketchy. I would describe them as more of a vague intuition at best. But I think we should not get caught in the "layers" metaphor. I would imagine that a network could have inputs connected at any level in other networks, as well as being "loopy". My vague thought is more towards the concept that there are networks which function on the behavior of sub-networks. Rather like abstractions, instead of all dealing with raw sensory data.
However, this brings me to another point that most ANNs today are stuck in the feed-forward mentality. Even if they have small recursion loops imbedded within them, they basically take an input and give a single output. If the nets were allowed to choose their own architecture we would hopefully see much more information going forward, backwards and around in circles, getting modified by new information all the time. Perhaps some neurons could even take more time to pass on information than others.
The result would be a completely non-linear way of dealing with information, much like our own brains.
Again, I agree completely. I would further expect a "successful" such network to operate asynchronously, and be extremely "lumpy". To use a silly example, have you ever made a mistake, then later started to make the same mistake but "catch yourself"? That would seem to indicate that we have behaviors over-riding each other, rather than being an always refined behavioral process...
I very strongly believe that there is a lot to be gained from pursuing a more reality based NN model.
Home | Return to Collected Papers and Articles
Contact info@artificialingenuity.com
Copyright © 2005 Artificial Ingenuity, LLC
Last modified: June 11, 2005
Initial design by Webinizer, LLC