"A hello could lead to a million things" - Anonymous
This is an update to my ill-advised forays into building an AGI. Somehow, I have managed to make this somewhat work I have an AI currently reading a book and being tested on whether it gets any whole words right by predicting the next word. I concluded that the linear algebra methods I was using to build the network were not sufficient to carry on the types of experiments I wanted to do.
Please be forewarned this is a bit of a weird post... it is more about what does not work in neural networks and the reason why I think this is interesting is for what it tells us about philosophy and how thinking itself might work.
Hello World - A AI Speaks | A Logic Called Joe (webador.co.uk)
Hello World - Measures Of Language Skill | A Logic Called Joe (webador.co.uk)
The best version I have currently got uses a very specific architecture and err it's pretty good with that being the likelihood that this could randomly happen by 1/10^7000. Yes, that is a 0 followed by 7000 zeroes and then a 1. So I have set myself the task of teaching an AI English or at least to detect if it was reasonably possible.
This is a blog post on what experiments around this have taught me about intelligence and why that is interesting for philosophical reasons. This blog post is about a giant blob of intelligence I built to show a point of philosophy.
Building A Different Library - A Memory Library
So in the beginning, I was doing everything using linear algebra and Numpy but I have concluded that what I would need was a library that would let me add or remove neurones from it.
The maths this one is based on is very different net being somewhere halfway between a Turing Tape and a Neural Network and questionable on which one you would define it as. I tested its functioning by building Xor gates and being able to control any connections across the network.
The work I did was to build a class that would take command to connect two neurones anywhere in the network and would proceed to do it.
This is because something that has become clear to me is how recurrent neural networks work. A recurrent neural network includes neurones that connect to neurones after them and therefore creates memory by recycling the previous states of neurones as inputs,
This then allows the network to have memory; it should be obvious why when doing a stupid thing like trying to look to build an AGI why I would want the memory. Lots of memory. This library I built was intended to build so much memory that every Neurone in its network was connected to both its past self, all the previous neurones representing the decisions already taken and all the future Neurones representing the previous state.
If intelligence was about pure numeric calculation then this should work...
Despite sounding cool it is wholly impractical, A neural network that connects to all aspects of itself in past and future sounds like it should be able to ascend to godhood. But it doesn't work. I have some experiments left to complete about removing connections but I have done 4-5 tests and it appears to not work; you cannot take a ridiculous amount of neurones and connect them to everything not nailed down and expect results.
It's admittedly something like a Hopfield Network trying to learn a human language and our brains are neither like big blobs of neurones nor circles which would probably be the shape if such intelligence was an effective way of building a neural network... If this worked then slime mould should be the smartest organism in existence. Somewhat terrifyingly the average error per page is shown below.
I don't think I will continue to experiment with the blob intelligence but see below it did on average get less than 0.5 on an average error which means it is sometimes responding and getting closer to the right answer. That is a little scary as it suggests the giant mass of neurones might understand a little but did not converge because there is simply too much pull from all the different directions. I think the network is not learning a decision tree for the next letter but projecting the average sound/output from one letter to another.
In a word where a brain might try to speak along the blob is just humming along to the book...
The previous architecture works better. Though this brings in an interesting insight that borders on the philosophical, It is not just the connections you make that matter it is the connections that are left out. I have tried some experiments where neurones are removed or added and I am at the infant stages of running these. Interestingly this does not disrupt the network when removing the smallest weights noticeably. So I have been experimenting with something which in my head is like how I think Neurogenesis might work in the human brain to kill off weak connections in the network and keep adding new ones.
Neither has allowed the bob brain to think any better at least in a way I can detect any improved performance...
Fig 1 the average error of the blob seems a bit better than taking the average
Gaussian Distribution
This distribution problem fascinates me as I think it sets an upper limit to how a network can be built. You cannot just add neurones and connections till it gets smarter. Now I have hope that this might be overcome and to this end, I have been thinking of running some experiments all of which rely on the library I have been working on.
- Try a genetic algorithm to build a better AGI. I have a map to a minimum functioning AI that taught itself to follow a book. Well, could artificial evolution help iteratively improve that mind map to make the AI smarter? Seems worth a go I have done it before and has the added benefit I can look people in the eye and talk about my self evolving AI. Seems like a conversation starter.
- Look for an algorithm on how to connect up a network that overcomes these issues and does not result in the blob intelligence.
- Try and figure out the right approach to building memory in the network.
Memory Creates Momentum
Now, this is an induction I have not figured out how to show this perfectly but it makes sense that tying a network to its previous outputs as is suggested to create memory ala recurrent neural networks would create momentum in a neural network in that it will no longer be able to output anything possible but will be tied to itself.
I, therefore, think the problem with the blob AI is that it is too heavily connected to its past self. I have a belief that therefore the current problem in looking for an AGI would be
The below is an experiment in a momentum neural network where each neurone is tied to a momentum of previous output. You can see that surprisingly it works somewhat. But it is 10th of effectiveness as my original architecture and the momentum should create memory but instead impairs the network and requires a much larger network to get the results.
The second version shows the same data and shows that I have tried to take it to its extreme and as you can see around 62 (the number of symbols it speaks in) its performance drops off a cliff. I am currently experimenting to find where that cliff is and how steep it is as I assume it is a Gaussian distribution.
I am also experimenting with just adding more depth as you can see and that will take some time.
This has given me the hypothesis that too much memory in a neural network is bad it leads to a network tied too heavily to the past and not making changed decisions but averaging the output of all past values. There might be some work to do on how to architect adding new neurones to the network, there is probably something about how much and where recurrent networks should work.
So I have a hypothesis that memory might be a big thing for getting AGI right but even when we find solutions it will be important that we avoid creating a network trapped in the momentum of its memories. When adding too many neurones performance drops to 0.
Now we can see from the above the neural network that I got to "speak" that its performance also drops when too many neurones are added as inputs and this appears to follow the theory that performance in neural networks looks like a Gaussian distribution rising and then falling off when too many added.
Memory Or Gaussian - Either way no blob
If memory causes momentum in the network or there is a relationship between inputs and performance as a gaussian distribution then it is clear we cannot just keep adding inputs to a neurone. Either would act as an upper limit to how big we could build an AI. The below would indicate some soft evidence that the issue is more with the gaussian relationship between inputs and performance than memory.
Therefore I would propose the only way around this hard limit would be to figure out intelligent ways in which we start connecting up and building these AI. There is a final option that maybe with a big enough neural network maybe it can overcome these issues the issue is a mirage and a singularity will see the performance improvements.
The problem with that is it will become a wild goose chase we will keep adding neurones without finding it if it is about sheer computational power. We do not know if we need to keep adding depth or breadth to build AGI. Completing the below would give some more information but not much.
It seems likely I am missing something in how to connect these neurones in more intelligent ways.
What I mean By Architecture
It is clear from the blob we cannot just connect everything to everything and expect to get intellect. Therefore specific choices will need to be made about what is an AI gets connected to what. By this I mean architecture but maybe there is a better word like brain plan, connection map, neural highway, or whatever. Though clearly intelligence is not just about calculating lots of numbers.
Brain Surgery On The Blob Intelligence
When I got up this morning I did not expect to write the above title but here we are,
So the blob is proof of the idea that it is not enough to just throw computation at a problem. We cannot just create huge amounts of Neurones (presently) and hope they figure it out. I took this a step further and created the blob but after reading a single page the weakest weight in the whole network is removed. So this is a blob of intelligence slowly being compressed.
The intent here is to show that the bob intelligence will not architect itself. Whatever an AGI is it is not a mess of numbers that will magically sort themselves out and be expected to kill the bad neurones so removing neurones might be expected to eventually help the thing become intelligent.
I have shown a much smaller AI above that outperforms the blob intelligence. If intelligence == computation then the blob intelligence should be able to outperform the smaller network at some point. It doesn't therefore a key to getting the best performance in neural networks and perhaps the key to AGI itself will have to be how we construct these things.
Something that this shows is that it does not seem to be we can connect everything and then expect the blob to start removing the right stuff; at least in any reasonable time frame...
Maybe We Should Grow The AGI's
I liked writing this code it is called things like Trepanning for deciding if to add new neurones and connections and brain damage for removing them. Though the purpose was for the AI to remove parts of itself that were not working while running thereby compressing itself. It then adds new neurones to itself by adding values.
This did not help the blob but it surprisingly did not disrupt it; it has become an interesting idea for me on how to decide to add and remove parts of an AI while it is running is one idea I have to9 how to solve my current problems.
You only have to look up how neurogenesis works in biological organisms and get a sense that we are missing something about how brains grow and decide to build connections between neurones.
Hypothesis Time
So I have some interesting hypotheses about the nature of the mind from doing these experiments. Some are obvious but one of the reasons for doing this post is to work through my thinking about them.
Some Insights from the Blob
- Intelligence is not computation we can connect everything to everything and the network will not converge. This should feel common sense being smart is not the same as being useful. The blob intelligence should conceivably have a connection representing the mapping of every input to every output and should eventually converge to something. It doesn't mean that to be effective a neural network will need to be architecture to achieve performance and this makes sense that for neurones to effectively work and act as discriminators it would be helpful if they did not connect to everything (the opposite of being a discriminator).
- There are evolution algorithms used for building neural networks that decide what parts of the network will connect to where and outperform normal networks both in size and performance. We can show from our blob intelligence that biology either by evolution or a superior patterning algorithm knows how to connect neurones to form intelligence that we do not yet have. Look up NEAT and Hyper NEAT algorithms as an example of evolutionary approaches to building a network this way.
- If all parts are connected then it tends to output the average of what it should output. Too many connections do not create intelligence it creates mediocrity.
- This seems or is possibly related to the amount of memory and architecting or for architecting better to allow for optimal amounts of inputs and outputs in a given part of the network as shown from the momentum testing.
Conclusions
There is a fascinating piece of philosophy that I think gets completely glossed over with Neural networks it is unclear when we say intelligence what we mean as we use it as if to imply a commodity or amount. The blob intelligence should be smarter than other networks I have built but it fails to manage to show any evidence it understood language at all. A smaller network manages to show some evidence it does understand but is especially architecture to do so.
Therefore we have to conclude that architecture matters and understanding that will be key to learning how to build better AI.
Hypotheses I have formed
- Intelligence is not a reliable value it is unclear when the blob intelligence is dumb and a smaller network can manage the task why we ought to talk about intelligence as a value like IQ.
- Biology has some better ways of building neural networks than we do. It might be genetic related or might be an iterative method.
- Architecting problems can make an AI better and brains and this comes with the added suggestion that different brains might be built in different ways or with different architecting algorithms.
It is the last one I find fascinating as it suggests to me that when or if we find such a method we might be required to either evolve or grow these brain plans to architect the design correctly or spend ages improving models of AI by tiresome trial or error. Would it still be AI and artificial if it had to evolve over long periods? AI makes us ask all sorts of philosophical questions about the nature of thought and we might never know the answers if every intelligence we create evolves from stochastic processes.
The blob intelligence is interesting as a thought experiment. It proves that neurones do not result in intelligence but the underlying "design" matters. It raises questions about how best to decide that design be it evolution or algorithmically and begs questions about how maybe it works in biological organisms.
Add comment
Comments