New Years Resolutions

"Cheers to the new year and another chance to get it right" - Oprah Windfrey 

This is my learning blog, I write this as the new year is upon us and thinking on what I want to learn for the next year.

Build a transformer

I built a neural network library a long time ago but the transformer has always been a bit of a oddity to me. I have  a bad habit of simplifying it to a series of Relu linear algebra matrixes with a SoftMax head on which the crown of attention (of which supposedly is all you need) gently rests. Therefore I think I want to right a whole transformer architecture in C++ just for the hell of it.

I at one point wrote a code line for a evolutionary neural network system that looked for stock trading processes the idea was this evolutionary thing that could try and find the optimised model and then run that model. I never really ended up trading period but quite liked building AI that do that. This model implemented gradient descent, linear algebra and it evolved SoftMax and Relu combination of interactions prior to learning about the transformer so I have largely in my head kept simplifying my understanding of the transformer to those two things the Relu and attention head and never driven myself to write a whole large language model from scratch at a compiler math centric level of interpretability and therefore I want to set myself that challenge. 

I think it is one thing to build something using someone else's maths and systems I think it really lets you clearly think about it if you can build one from scratch. Admittedly being a mechanic who can build a car does not make you drive the car better but I think you should try these things.

Therefore I want to write something end to end from training to output something that looks like a LLM so I know I understand what that looks like. I am admittedly jealous of anyone with AI in their title I do not want to be that moving forward.

Become a "proper" Data Engineer

I have built "things" but I believe to be called a engineer is to adopt a theme a crown of some value. I want to set myself the challenge of finding some project that it is building the backend of a website or something that involves what I would call "advanced SQL" i.e. something that relies on indexes or partitions in the memory and is very runtime dependant.

This would be the sort of stuff if you wanted to build a real time event driven database architecture. 

I mean being predominantly a data analyst I can build and provision databases the thing is I have never done that "in anger" where need to be very controlled. I have worked on projects where aspects of this interact with the wider stack and pipelines and there are more moving pieces but it tends to remain simple. The problem with SQL is often you can turn everything into some form of select and its hard to push yourself to really have to  make yourself use all the advanced features all at once; especially when you also have AWS/ADF and python that can all prepare data.

I feel presently I am very much a data analyst and while I have done data pipelines I think these are both currently and will be very likely moving forward be dominated by the low code field (sorry to anyone who disagrees) unless I can push through into something that is very time dependant and design specific niche I think I will be at a disadvantage compared to the competition.

I do not really know what I will need to build but probably need to concoct a reason to experiment with something like this.

I also think this is part of my belief that until you built it and maintained it in a muddy field while being shelled you do not really know how it works; you only think you do. I want to try and put myself in the position of building a whole database from scratch is the route to feeling that type of pain.

Build AI

I do not want to walk away from my series on Hello World. It really does appear interesting to me and I have a soft spot for building AI and especially AI that more closely matches the Hodgkin and Huxley equations that describe the differential equations related to neurones. I have a long-term goal to build a simulation that very precisely works.

The system that I am currently working uses evolutionary and genetic algorithms to describe the chemical properties of the neurone cell. Therefore I have a database built from ground up setup that can run trial and tests for it and I am quietly confident that a certain chemical simulation will work. Such that I think happens is that what makes our brain cells special is they fire and this takes place over time and that you only need to get the chemical changes right in the neurone cell and it would work.

I have initial results to point at and say different chemical change rates does result in different behaviours and performance and a long-term evolutionary algorithm to do the testing.

I had a bunch of initial perverse incentives where maybe a line of code made it centre on some outcome which was not what I wanted but I am resolutely working through those problems. This was like I built the testing processes but I would forget the model might be able to die off and therefore not do anything but because it did not choose it would be counted as a zero and that might be better than doing the wrong thing etc.

But it is now working and I have comparisons and a quite sizeable 50+ parameterised system that can change the way the chemical simulation takes place and I can just run changes on whether you get a better AI from using human neurone simulations by changing those parameters. It is also now fairly easy to add or remove chemicals and do A and B testing on different neurone designs.

I want to at some point test a theory on affect of chemical simulation controlling learning rates and it already has a large number of tests done.

After that I have the mates that can build a brain computer interface and as sure as bobs your uncle fanny is your aunt that is brain simulation done my son...

I really do also believe this is the next stage for AI after the LLM because I feel that what built us the LLM is Bayesian mathematics (its basically SoftMax) and the Nash equilibrium from Von Neuman. The issue I feel both those miss is when you look at a snooker table and try and figure out outcomes and calculate Nash equilibrium well every move can completely rearrange that table and change the subsequent game and its so swingy. I think life is a bit more like snooker where all the balls are constantly moving than the type of Bayesian maths that produced the LLMs and allow for machine learning; I suspect its best described as a set of differential equations and not linear algebra and good look to you with figuring out how to standardise the differential equations and get them efficient well then you can have a AI more like us and less like a LLM.

I have the prototype I can just leave it running to build up testing on what various changes to the chemical simulations cause. 

I think what I am saying here is Von Neuman and what a lot of our AI is built on is game theory and Bayesian estimation of outcomes. The problem like what I experienced with the snooker game was how do you estimate the Nash equilibrium for a snooker game. Every action moves most but not all balls if you try harder you can move more of the balls. Snooker is a bit more like life than chess in than its much more uncertain every action by a opponent radically changes the possible actions and the whole game cannot be described as a series of optimal actions (much like life; albeit my alumni who went to better colleges than mine might disagree).  

Life is a odd thing, you can do the right thing at every time step and end up in hell. You can be a fool and be forgiven. I think therefore the missing issue issue unexplored by von Newman (etc) is that well how do you resolve to represent that probability as a simultaneously malleable object that represents a risk reward outcome in the classical sense but is time dependant and learns its expressions over time. I think that's the missing bridge for AI.

It already has neuroplasticity and I would like to do testing on various different ways I could re-engineer that system so it could more closely match our brain growth processes and be self designing. I mean it already prunes and connects up on its own but making that system rigorously tested might be worth while.

Therefore I think there's a missing step for neural networks to better utilise differential equations and more closely resemble the brain.

Its fun its weird, I am largely confident one day it will work forgive me if it doesn't.... someone needed to have a go didn't they....

Old You Tube post - its visual, I have graphs and real time FX to show the basic premise... What's more the base maths is human neuron for Hodgkin and Huxley equations!

Brain Emulation Chemical Simulator Visual 

I think in a couple of months I can run another visualisation demo on how far I have got.

I think a final thing I might try after built both is try seeing what combining the transformer and the chemical simulations does. It stands to reason you do not want to build a whole human brain simulation at scale as we do not have the computer that would fit on but maybe a experiment in using transformer architecture for simplified doing bit and a more granular simulation of just the scaled down amygdala or the executive functions might be a test worth doing.

I am not sure if that would work I have some limited experience to say it would but it would involve error propagation between a very static transformer system and a very dynamic and different system based on differential equations. Why I think it might work well together is the transformer architecture adds in these Tanh and Cos type fixes to the maths according to the original paper attention is all you need. Therefore I think the fake amygdala would work as a sort of proxy for those in the transformer while the chemical simulator would be a fake higher brain function system type thing... (maybe I need the chemical simulator to get better to try it).

LLMs are not that imaginative and definitely not very independent and maybe something that combined both would be a cool system to build and might actually be those things so maybe aim is to finish test get the C++ written for a transformer and then next year I can see if can build a independent and functioning AI system. The only thing that would be scary about that would be I would guess if it works it would not really have any baby steps it would either be 

I think I also at one point wanted to look if you could map human brain waves onto the chemical simulation but I think it needs to perform much better before you would try that. I have been reading beginner books on the brain for this purpose to see how feasible it would be to bridge that gap using the chemical simulator above but I really did just find the way brain waves work is very hard to map into mathematical terms and reading up the brain is very big; you'd probably need a lot of sensors and the sort of brain computer interfaces that the universities use is around 30k.

I did look at building my own with some friends and that might be a project I look into but probably next year as I think I need the evolutionary algorithm to get the chemical simulation as right as possible before trying that. 

But then it might all just come together the thing is in optimising something that learns like waves and was more like the snooker system talked about above hopefully you have something that learns on the go. Then why wouldn't a system like that be able to just slowly emulate the human brain waves. It is all at some point is just a matter of maths if you get the initial system in place. So lets hope the evolutionary system delivers a optimised and fine tuned chemical simulation for the neurone learning mechanisms; And I know it sounds mad but if its possible at all surely it probably would work that way is my thinking. Figure out how to fine tune the initial simulation then go and apply in all different places. 

The plan...Slow slow slow...then download the mind into a laptop.... probably over ambitious but worth a try....

PLay gAmEs

I had a kid and I had been building this traditional roguelike; it is funny it is just the level of the old time ROGUE but he recognises red dots as enemies and will respond. I am a little captivated by that design.

The design goal has been to have a totally randomised game development it already has a language generation system and history system that generates a world I think I want that to go further.

I have a basic magic system in my game that is derived from everything having materials and reactions. I would like it to in the future have a wholly randomly generated skill tree with wholly randomised magic systems. That sound ambitious but random classes, already exist, with random 1st level skills, but based on material reactants. Such that fire melts ice and electricity damages water, it already has the start of randomly generated world, history and those reactions can be setup slightly differently. This means in setup it can build a world where in a book it will tell you that the gods smote the demons with lightning or the fairies where driven out by wood and if you equip the right armour and equipment then you are better at those foes and they are less likely to get through your armour.

I also want to get it to the point where it randomly generates a whole spell list so currently it can go get a material say textiles in the game and build a magic class around that; so you end up with sewing mage and leather paladin alongside the more normal fire and ice classes. I would want it at some point randomly generate their whole spell list and I think that would be fun,

It is a small game but it is my cosy game where I get a lunchtime I auto generate a new world and just go and play usually a distraction for 10 minutes seeing what odd combinations it creates. 

Be a better leader

I would like to be a better leader and communicator. I am what they call a dyslexic, I went to university they taught me stuff about leadership. I kind of grew up a air cadet then a officer cadet who fascinated by military life but chickened out of becoming a first lieutenant and somehow ended up  as a data analyst.

I mean in hindsight I could always see the strengths and weaknesses that led me here. Often my thinking about leadership was well well you have not given me the data how could you expect me to lead you and why would I expect you to follow, but with a bit of time and age id like to be better at that. 

Building a game and showing it to my little boy and saying; this is code this builds this and look now daddy is the green dot running away from the red dots sounds twee and he is admittedly only 3 (then 2) but it makes you understand what leadership is about when I explain all that complexity and when I simplify and he understand and responds to the choices. Such that he can tell me where to go in the game that I have built for him. 

I think id like to give the leadership stuff another go. I am technical but leadership curious because I think I am aware as a dyslexic communication is not my biggest strengths.  The truly persuasive have always been good at rhetoric or writing I am good at writing in Python. I think it would be good for me to find a project that stretched me in that direction.

I think what my plan is maybe try combining all those options build the C++ implementation of the transformer, hopefully by then the human neurone simulation would be significantly tested maybe look at could you combine that in some real time service and launch it online as a data engineering project.

Hopefully that service will not involve downloading minds into laptops but I am open to suggestions on what such a service should include.

Happy new year to you. Stay curious.