Building A Transformer AI From Scratch - Part 5 Its a debug party

It did not break the PC either. No memory leaks, it chugs away because the code I wrote has yet to be optimised. Can chug through 100s of thousands of paragraphs in a day and on my old, broken, decrepit PC.

A small dataset would still be 10,000 to 100,000 samples. I manufactured 60 just to test does error decrease period. 

A more composed start

It's been a hard week; I will probably be a bit saner next week. But about mid-week, late at night, I was still debugging, discovering huge errors, most made by the AI used in vibe coding. Much of this is my debugging documents written in the midst of that, so there is a little bit of crazy energy going on. I won't lie, I had a despair moment where the vibe coding AI system pulled the rug out from me, and I had to remind myself it does not have much context, and I was pushing it a bit.

So I am going to start again and split this up. First, I will split it out and criticise my tools. I will speak about how I got a lot of the base code from AI itself and vibe coding. I will tell you why I hate it, then why I tolerate it, and ultimatley it did the job. 

AI helped write another AI! 

Then I will go through the implementation and naming I am using.

Criticism of vibe coding as an idea

This is a 3rd/4th C++ project, maybe a 6/7 project with AI in it. Probably 1st time with vibe coding.

When I have done something like this in the past, I always did a scrappy version in Python first, and then, when I had that right, I ported it into C++ if the build would benefit from substantial speed. 

The AI does not seem to appreciate the time and cost of making objects in memory. C++ is a very elegant language with very fine control over memory, and a lot of the rules are meant to help you make and destroy things correctly and keep C++ fast because it does not delete anything under garbage collection like Python. The AI I have noticed follows the rules that people get taught about C++, but then makes and destroys big memory objects willy-nilly, often hidden by how it returns from functions.

By using vibe coding, I kind of have missed the normal draft prototype I would have done in Python, but I will definitely still need to go through and work out what the speed up will be, as this initial draft is going to be slower than it would be. Given that I definitely will need to do two drafts, there is part of me that is uncertain if I saved time.

The other thing I have noticed is that it might be assumed that generative AI would generate something like code from the Platonic realm of the forms, that is, in some sense, the best saddle point between each of the competing designs in its training data. There are a few mistakes where it built stuff that was either not the most efficient or had a bug if changed one thing, or felt like it was multiple good templates badly welded together. 

I have a hypothesis that the AI dread is around this belief that AI has access to some Platonic state of real maths of things; that it is "better". I think this is wrong, it is merely faster for the "better" brain, i.e. us to offshore some of our thinking to its more rough but better recall of factual information from its weights. I think we should embrace it, but I also get the angst that it is not a perfect system, and I think much of the problem is that it is a new tool.

The impact is that AI follows all the rules but without any of the spirit of them. I am glad I purposefully did a software project and did it via Vibe coding, but on a lot of reflection, I am really uncertain if I genuinely think I would want to use it again. I think it definitely worked here, as I struggled to find many existing examples of Transformer implementations, let alone implementations of the backward pass in C++; so generating them seems like the right call, but I think there is a lesson here that vibe coding use has to be moderated. I think I would definitely use it in certain circumstances, but I think my take on it is as another great tool and not a replacement.

For example, I have ended up transcribing from the AI and not copy-pasting, and I think if I had copy-pasted, I would have had no confidence to let this run. Therefore, does this really save me time?

My take on this is that AI is a grade A research force multiplier. A lot of the AI architecture I built, I could not find examples to learn from. Most initial coding to build a minimum viable project is research, but optimisation problems, silent bugs, and technical debt would balloon if relying on it.

Issues with the vibe coding

The way it praises you all the time. It's affirming at the start, but by the end, I am paranoid it has ulterior motives. Why did you not tell me I was perfect, only good AI!

When you get more than one AI involved, it does he;p find issues, but they quickly start adding gimmicks. i.e. they add something, and it is almost certainly talked about online, but it's poorly applied there and now in what you're doing. It's like having a guy in the room randomly shout make it red cars go faster if they are red.

Ask for the same code again and again; there are small changes. I think it's inferring small editorial checks and changes, but this is unhelpful when testing, and you do a double check, and you see the same parameter slightly changed. 

At one point, I was looking time and time again at a line that had d_token_embeddings_.transpose() I kept prompting the AI about this. Next time I re-asked for the code, it changed it to token_embeddings_.transpose() and radically rewritten the function.

If this were a junior developer, this would be them rewriting code behind your back to avoid a telling off! But why is the AI scared of being told off? It doesn’t even have the pain receptors for that...

I hate attention masking. Once it has added, it is hard to get it to drop, and I worried that if I asked it to, it would change the whole design.

I built assertions into each of the objects. A lot of those assertions worked with the first draft of building a transformer and were built using the AI input. When moving forward, I found myself testing the same functions again and changing them towards more robust checks.

I put this under the heading of what AI produces is easy to think falls within a “platonic ideal”, i.e. the AI will produce the centre of the Venn diagram of all coding functions and therefore is or is likely to be better than what you produce yourself. I often do not do this, but it is in many senses, lazy, producing the minimum viable solution to answer the users' questions. 

In my case, the matrix in the input into layers of transformer blocks should be test y== rows and columns of x, but the multiplication process means y.rows && y.columns will now equal x.columns. Therefore, the AI set me up with code that was sufficient to answer my current query, but not in any sense ideal and then failed when further changes were made. This might seem like a very small matter, but I do not have problems like this when coding in a more planned manner.

When fixing something because of the above slight changes, there is a deja vu of whether I introduced this bug or if the AI introduced it. Therefore, there is this, I guess, seduction pushing you to blame the AI more and yourself less, which I think is dangerous. I think some of the problems I am blaming the AI for above might be due to me, but I have genuinely checked, and I have at least one instance where the changes the AI made were drastic, meaning I think it sometimes does “improve” and gaslight the user.

The AI seemed to never properly understand proper architecture. It was meant to create a function called forward, softmax, cross entropy, step, and backwards. Not only did a few times did some of these functions disappear a good few times, but things that should happen in step would move backwards. i.e. it liked zeroing d_token_embeddings_ in cross entropy calculations. This completely threw me for a while as I wondered if it knew stuff I did not. But then, how does the gradient accrue? I feel this underlines the risks of a tool that produces text from stitched-together text rather than with intent, and while I do mental somersaults over the meaning of AI’s capability with one-shot meaning, I have an abiding belief it would struggle with intent.

Masking, everything keeps telling me masking is really important because the AI cannot know what is happening in the future of the sentence. The issue isthat you do not load anything from the target IDs into the AI input. Also, using cross-entropy only lets me pick one output. In BERT-style models, you randomly assign some of the data to be blank so the AI has to infer more. But I haven't started testing different ideas like that. When using more than one, did was decide masking was a real object that needed a vector of booleans around it and then proceeded to change the whole architecture around masking. The issue is surely that I just created a 

I think what it is in GPU acceleration, the blocker is passing memory about,t so setting up the full sentence and cycling through it is possibly faster to load up one sentencthanen just cycle through one matrix of memory changes, as you're not passing as much data intoGPU's VRAM. But I have elsewhere been explicitly told that it creates a bunch of saddle points in the learning, so you should randomise the order, so I struggle to get the circumstances where masking is better.

Me in a multi-way argument between the AI over this, which led to some strange code which I had to fix through testing. I have a suspicion that AI has a tendency to stitch this code together from various sources and templates, and so some aspects that make perfect sense somewhere else ended up in my architecture. Then the AI optimises around it, even though I cannot for the life of me figure out why that design choice matters here and now. 

I strongly suspect that AI did not build a transformer; it built 80-95% of one and then just walked off to leave me to figure out that gap. The other deeply unsatisfying thing is I probably had just the minimum skills to fix this issue, which means the more skilled you are to fix the risks that AI create,s the more likely it would just be better to go ahead and do it yourself. 

Though you can only see all that after you've written it, gone through i,t and systematically debugged, it because most of its big errors are architectural. Therefore at the beginning you feel great thinking this is helpful but all the pain is back loaded and all the joy is front loaded with the kicker is the architectural mistake it will make are only apparent at the end meaning if your using this to learn you are taking a gamble with not knowing if you will understand well enough at the end to fix the mess the AI has built. 

For me, this set off rounds of refactoring and rereading code that ultimately took more time thanif I  had written the code from nothing.

What’s more, there is no story or project mechanism that you can put in place to mitigate this risk.

This feels like a life lesson, no knowledge can be gained for free,e you pay in time, humiliation or the vexation of the problem. This is to say, AI vibe coding is neither programmatically nor pedagogically a sound choice. It has aspects of some of this, and it may still be of interest to some people, but it does not look to me like a sound or complete replacement.

There was a later point where I found it definitely had a size mismatch in a piece of code long ago moved on from. Asked for layer norm again and got given code that had a new line that just resized that output, so there was no issue. No subtley just outright created a new resize function after gaslighting me that the shape always should have been X.

Then, after fixing all the layers and norms, it quietly corrected the whole initial set of matrix sizes it confidently produced at the start.  There is an element here where it feels that we are doing a mutual debugging of your code, plus the AI at the same time.

An example of this is I used another AI to criticise the output and provide fixes, and I had to stop it because it really ended up going into the weeds of unnecessary changes, and the AIs started asking each other questions about the rest of the project, which were irrelevant. One of the suggestions was to split the functions up into a separate softmax function, which it has eventually reverted to quietly. It's this forward function that I ended up rewriting myself and cutting out the otherAIi and only using the primary one that I started the project with.

Name of variables over time might vary. Therefore, it becomes a liability on a large enough project that it will change names 100 versions later.

There is an element that I feel this makes me thinthat, kthat,  for best results, a human should sit in and listen and manage the AI agents, but if you do that, I think it does start to add value. Though then the more you lean out and let the AI automate themselves the more it feels they get carried away. All the silly problems where on the areas most redrafted by the different Ai and I have slowly been removing things as I go, not adding them. I have a suspicion that the code I am using is probably very much a copy of something that the AI I am using ingested on a website, and likely not unique, because when you start criticising it and adding to it, that feels like where the risk is that it hallucinates will grow. 

There is also the matter of ego and responsibility. What has already been produced was produced under a lot of prompting, and I have already done a lot of testing. I do not think I made a mistake, but I note when using the AI its easy to blame it and use it as a crutch to build the object and then a scapegoat where it fails. This could create the worst form of design by committee. 

All in all, it works, but it's not a magic bullet; it still has skills involved. The exaggeration about AI agents doing all the coding seems a flawed approach from what I can see. You need to at least read and test the code, and that traditionally takes the most time, and if you are struggling with the testing of the code, you will start reading it more and writing more of it till you stop using the AI. It's a bit catch 22 on,e like that it's good till it's not, at which point you struggle to know why you bothered. But while vibe coding worked, it will work very well for me, and I was the clever clogs solving all the problems when it didn't, it was the AI's fault and you have to learn to take responsibility for both the good and the bad if you use any v(there is a lesson in humility in all this somewhere).

What I like about vibe coding

If I tell the AI I want things changed, it tends to remember. It does feel like having a pair programmer on the same project in this regard. It does feel like, barring the criticisms above,e that you can iterate with it and work through the problems. 

I genuinely feel like it's a genuine pair programming experience and gives those types of benefits.

I can treat it like I use Stack Overflow, just cycling through different examples, asking questions and trying to get advice on the problem. 

When creating unit tests, it did seem to always know,w given some set of values and a rough expected val,ue which is very useful for devising unit tests and building tables of what the parameters should be.

AI is very good at one-shot learning soit'ss very good at explaining its own codede which makes it a powerful learning tool as I can flip between the code, formulas and explanation back and forth. That might sound like the ungrateful whining that my coding AI gf is not considering my feelings when writing me code, but I think a point of critical failure for vibe coding is any mismatches between the AI and user and some sort of check-in to ensure alignment would complete the pair programming simulation. A small bit that says run the code output should be X, might just join those final dots. 

Above, I strongly complained about masking and stuff. There was a moment aha moment, where I learned about teacher forcing training and realised the model does need to learn the whole output sequence (up to sequence length). I had a bit of a floored moment for me. There is this thing called exposure bias. There is a mathematical level that this weirds me out because, for accuracy the generate one token at a time, but a lot of the methods try to learn long-range dependencies.  

Therefore, where the element that I most strugglewith d and the AI most struggled with was where I learned the most from. There seems to be a powerful reason to consider that I should take ownership of the interaction and not assume it was because vibe coding was the issue. 

I think I should be suspicious that the areas I struggled the most with and asked the most questions on are the areas that the Au might struggle most. I also think the application of masking was wrong, but broadly, everything it did had a point. Seeing less “culpable” and an “agent” anan a interactive experiential archive might be useful.  It is probably the case I am just over scrutinising this section and repeated questions hint to the AI different directions it might make different changes. 

I still feel suspicious, as it was going OK even at that section until I used more AI agents.

Though all of that is best used in very short sprints with expected project support, if you were already this sophisticated,d I feel that you would not need the AI. 

I still strongly feel this process of vibe coding is more of a replacement for what you do to learn to code, in finding code challenges and doing them and just persevering. I can automate making those, and AI is fantastic because AI can just generate these massive code bases where the bugs are, hidden like where Wally. 

Things I'd Change

I think the AI could do more to stop and suggest unit tests and just stop the user and say now please run this checklist and give me the answers so we can e confirm we are on track. I think live coding could be good for pair programming, but it needs a structure to be added.

I feel that my resistance to it would be lower if I felt it was better at stopping and making me do things that I should, including creating a learning process. 

Things fixed:

Was still getting my head around all the naming systems. When the AI learns, it runs forwardcreatings and stores up error which becomes applied when calls step. Having not started running end-to-end, missed the zero grad process, which is meant to see that the gradient has been applied, and zero all the matrices, missed some matrices used for gradient in the transformer model. Without doing this would have built up errors and would have applied the sum(error) on each step. This probably would have been un- optimal and likely would not have converged, but I know from some toy tests that AI can sometimes work like this. 

Had not properly initialised d_ matrices in the transformer model. What the d_ matrices should do is accumulate gradients in forward, but they were getting multiplied. 

Check d_logits_.zero();

d_token_embeddings_ need to look into its use in forward. 

Vibe code included the creation of a const vector of pairs. This is not allowed in C++ v17 because it makes no sense. Seriously, how did I miss that a data container I cannot add things to or change? I just typed the code out and got confused when it caused an error. I think this shows a critical issue with vibe coding, that it causes people not take responsibility. 

During vibe coding, there was a fair bit of me changing things. Tokens size was being used to size the Matrix where I embeddedthe token. This seemed to be a no-brainer timer saving not to do that and return when I hit a pad token, but then the length of the token was no longer always going to be the size of the maximum size, so it could create a malformed matrix. I had to fix this. It is a small piece, but it goes to show an issue with vibe coding is that the AI makes shortcuts and those are fine in itscoded,e but they are not “platonic” or “archetypical”

Matrix Model::embed_tokens(const std::vector<int>& tokens)

{

Matrix embeds(max_seq_len_, d_model_);

Rewrote the forward function: Forward originally looked like the following.

float Model::forward(const std::vector<int>& input_ids, const std::vector<int>& target_ids,float tempreture)

{

check_sequence_length(input_ids.size());

check_sequence_length(target_ids.size());

last_input_ids_ = input_ids;

last_target_ids_ = target_ids;

//token and position embeddings

Matrix x = embed_tokens(input_ids);

//forward through transformer layers

for (int i = 0; i < num_layers_; ++i)

{

x = layers_[i].forward(x);//cycle along layers outputs 

}

//final layer norm

x = final_ln_.forward(x);

//output projection

logits_ = x * token_embeddings_.transpose();

//compute loss and get gradients

d_logits_.zero();

Matrix softmax_out = logits_;

//create mask for pad tokens

//would put a mask here, but zeroes all the inputs 

return sofmax(softmax_out, target_ids, tempreture);

}

Now I did a double-take when I read this and wen,,t I really sure that is missing something; not the masking that was a silly idea, the AI critic added, and I do not think I am actually ready to use it, but I sort of blinked anAIAIhe Ai had started to go off topic. But if you notice, it's missing proper cross-entropy

Silly mistakes put the smaller as greater than. This started labelling all the names in any text as .a Unk