"Cheers to the new year and another chance to get it right" - Oprah Windfrey
This is my learning blog, I write this as the new year is upon us and thinking on what I want to learn for the next year.
Build a transformer
I built a neural network library a long time ago but the transformer has always been a bit of a oddity to me. I have a bad habit of simplifying it to a series of Relu linear algebra matrixes with a SoftMax head on which the crown of attention (of which supposedly is all you need) gently rests. Therefore I think I want to right a whole transformer architecture in C++ just for the hell of it.
I at one point wrote a code line for a evolutionary neural network system that implemented gradient descent, linear algebra and it evolved SoftMax and Relu combination of interactions prior to learning about the transformer so I have largely played make believe and refused to believe I did not understand the significance but I think in all honestly I have never written a whole large language model from scratch at a compiler math centric level of interpretability and therefore I want to set myself that challenge.
Therefore I want to write something end to end from training to output something that looks like a LLM so I know I understand what that looks like. I am admittedly jealous of anyone with AI in their title I do not want to be that moving forward.
Become a proper Data Engineer
I have built "things" but I believe to be called a engineer is to adopt a theme a crown of some value. I want to set myself the challenge of finding some project that it is building the backend of a website or something that involves what I would call "advanced SQL" i.e. something that relies on indexes or partitions in the memory and is very runtime dependant.
I feel presently I am very much a data analyst and while I have done data pipelines I think these are both currently and will be very likely moving forward be dominated by the low code field (sorry to anyone who disagrees) unless I can push through into something that is very time dependant and design specific niche I think I will be at a disadvantage compared to the competition.
I also think this is part of my belief that until you built it and maintained it in a muddy field while being shelled you do not really know how it works; you only think you do. I want to try and put myself in the position of building a whole database from scratch is the route to feeling that type of pain.
Build AI
I do not want to walk away from my series on Hello World. It really does appear interesting to me and I have a soft spot for building AI and especially AI that more closely matches the Hodgkin and Huxley equations that describe the differential equations related to neurones. I have a long-term goal to build a simulation that very precisely works.
The system that I am currently working uses evolutionary and genetic algorithms to describe the chemical properties of the neurone cell. Therefore I have a database built from ground up setup that can run trial and tests for it and I am quietly confident that a certain chemical simulation will work. Such that I think happens is that what makes our brain cells special is they fire and this takes place over time and that you only need to get the chemical changes right in the neurone cell and it would work.
I have initial results to point at and say different chemical change rates does result in different behaviours and performance and a long-term evolutionary algorithm to do the testing.
I had a bunch of initial perverse incentives where maybe a line of code made it centre on some outcome which was not what I wanted but I am resolutely working through those problems.
After that I have the mates that can build a brain computer interface and as sure as bobs your uncle fanny is your aunt that is brain simulation done my son...
I really do also believe this is the next stage for AI after the LLM because I feel that what built us the LLM is Bayesian mathematics (its basically SoftMax) and the Nash equilibrium from Von Neuman. The issue I feel both those miss is when you look at a snooker table and try and figure out outcomes I suspect its best described as a set of differential equations and not linear algebra and good look to you with figuring out how to standardise the differential equations. I think that is metaphorically my issue with present day AI research of which I accept I might be among a small minority that gets what I am saying here.
I think what I am saying here is Von Neuman and what a lot of our AI is built on is game theory and Bayesian estimation of outcomes. The problem like what I experienced with the snooker game was how do you estimate the Nash equilibrium for a snooker game. Every action moves most but not all balls if you try harder you can move more of the balls. Snooker is a bit more like life than chess in than its much more uncertain every action by a opponent radically changes the possible actions and the whole game cannot be described as a series of optimal actions (much like life; albeit my alumni who went to better colleges than mine might disagree).
Life is a odd thing, you can do the right thing at every time step and end up in hell. You can be a fool and be forgiven. I think therefore the missing issue issue unexplored by von Newman (etc) is that well how do you resolve to represent that probability as a simultaneously malleable object that represents a risk reward outcome in the classical sense but is time dependant and learns its expressions over time. I think that's the missing bridge for AI.
Therefore I think there's a missing step for neural networks to better utilise differential equations and more closely resemble the brain.
Its fun its weird, I am largely confident one day it will work forgive me if it doesn't.... someone needed to have a go didn't they....
Old You Tube post - its visual, I have graphs and real time FX to show the basic premise... What's more the base maths is human neuron for Hodgkin and Huxley equations!
Brain Emulation Chemical Simulator Visual
PLay gAmEs
I had a kid and I had been building this traditional roguelike; it is funny it is just the level of the old time ROGUE but he recognises red dots as enemies and will respond. I am a little captivated by that design.
The design goal has been to have a totally randomised game development it already has a language generation system and history system that generates a world I think I want that to go further.
I have a basic magic system in my game that is derived from everything having materials and reactions. I would like it to in the future have a wholly randomly generated skill tree with wholly randomised magic systems. That sound ambitious but random classes, already exist, with random 1st level skills, but based on material reactants. Such that fire melts ice and electricity damages water, it already has the start of randomly
Be a better leader
I would like to be a better leader and communicator. I am what they call a dyslexic, I went to university they taught me stuff about leadership. I kind of grew up a air cadet then a officer cadet who fascinated by military life but chickened out of becoming a first lieutenant and somehow ended up as a data analyst.
I mean in hindsight I could always see the strengths and weaknesses that led me here. Often my thinking about leadership was well well you have not given me the data how could you expect me to lead you and why would I expect you to follow, but with a bit of time and age id like to be better at that.
Building a game and showing it to my little boy and saying; this is code this builds this and look now daddy is the green dot running away from the red dots sounds twee and he is admittedly only 3 (then 2) but it makes you understand what leadership is about when I explain all that complexity and when I simplify and he understand and responds to the choices. Such that he can tell me where to go in the game that I have built for him.
I think id like to give the leadership stuff another go. I am technical but leadership curious.
Happy new year to you. Stay curious.
Add comment
Comments