A and B testing

"Economics runs the world" - Machine Gun Kelly

Introduction

So a really quick update with me playing around at building a large financial model and running it in simulated trades for fun, but currently not profit. Its a bit of a placeholder but I did the optimisation work. I had a few highs and lows.

An Auspicious Start

I have managed to get the AI to trade more but it results in it acting like a crazy high frequency trader and then slowing down. I'm running a longer test to both confirm it will slow down as presently it trades minute to minute.

Interestingly, the accuracy and ROI did not suffer.  It just buys in out a lot. It makes smaller but more numerous profits.

The strength of making it trade hyper fast is more decisions mean more chance to learn. It does have a initial impact in it making more mistakes though.

It started getting better on the compounding values but not on accuracy and ROI in the long-term. It sort of plateaued in learning and sat where it was so the values compounded a little more but it was not getting exponentially better. 

So in summary it wasn't trying to do lots of quick trades that made it good.

All in all great first prototype and got a baseline now to optimise everything that I could...

I did some market research

I spoke to a few businesses. Some seemed rude,  some seemed curious. I got some great advice that they would consider what I am building a EA or electronic advisor. It actually seems really common. Turns out most people build a model hand crafted by a Quant run it till it stops running and then rebuild the model.

But it does not seem many people have autonomous self directed AI doing their trading that rewrites itself continuously... Oh ok so it seems like it's a bit of a novelty what I am doing...

I then got to see what traders called interfaces again no AI...

I think mostly what exists is some form of decision trees and maybe things like logistic regression. I have mixed feelings about this because I am unsure why more experiments are not taking place (probably they are but why would they tell me). But also might be good for this being a unique selling point.

So yeah, I feel that I at least might have something to offer if I am allowed to offer anything so I asked the FCA some questions about what rules there are. I do not want to even share data with the crypto trader I have ready to review the AI performance before I know legal stuff. They also confirmed what I was doing was a little non-standard....

Great!

Then the power cut

Then after several weeks of work I got a power cut... timing is everything hey?! So I restarted with a new version and made some minor tweaks in the direction I wanted and found this one. Did not seem to make money so I went into A and B testing mode with different

I wrote to the FCA as looking to find out what is  the legal position on if can sell AI signals at all. There are rules that can be quite strict but I think I'm ok so I think it requires due diligence. My reasoning there is that It got me some knowledge that generally yes they have never had to really talk to someone proposing using a AI to give signals precisely like this; and yes they wanted to talk to them and I wanted to talk to them to find out what theoretically I could do with the work.

I have a crypto trader that I have been talking to and what my AIM with them is to find is this working on a single strategy and just reapplying it repeatedly or is multiple trading strategies evident. So I will see what can be shared and move from there.

It would seem obvious to me that it would do scalping i.e. high frequency trading. What would be a big deal is if multiple strategies where evidenced and if it shifted between these strategies. If I can evidence it does that then I can sell as a service as while I am not a financial expert I believe that the moral and economic issue is that if too many traders deploy the same or highly similar trading strategies then the market may have a major crash when that strategy fails and logically the more people trading a single strategy the more likely that failure will be.

What I would ultimately want is a product that trains itself, adapts to the market and takes the market signals and reverse engineers a strategy. It would be a big deal if that strategy was at least somewhat personalised but I think it posses a problem in that in chess there is more board states than their are atoms in the universe because there is a range of moves you can make in trading there is only buy or sell. 

A Ghost In The Market - Or I go off on a tangent about philosophy...

This is where my scepticism of certain schools of economics play a part in my approach there is either a Nash equilibrium (optimal play) for the game of trading or there isn't; if there is and you can apply AI to it and that AI finds that game to be easy or easier than a human then surely progressively the idea we are making rationale market choices diminishes.

The game does not seem that complicated with only buy and sell for it to have a wealth of Nash Equilibriums and the standard Neo Liberal stuff seems to focus on these strategies are hard to discern you need lots of Quants and banking specialists. But what happens if you say here is a AI that can do this really well and outplays any average human. In chess you may continue playing for fun but if that AI now trading on various assets affects the cost of your morning coffee in playing your game it seems different.

What you actually end up saying is that the AI plays a game of buy and sell and the relative power of these AI in these price games decide the final prices and who makes profits. The assumption of certain economic philosophies is that we all trade on the market and that generates signals to the market. Does it matter if the signals are now just AI generated or did the human element matter to deciding what that price should be?

If you get to the point that AI say run the market and therefore theoretically decide the value of a given asset is that fair to base the price of a thing based on the Nash equilibrium of a game unrelated to the underlying value systems of those that actually use a given good?

If the market hit over 50% AI usage (its at 60-75% in America); is the market run by AI and is the market therefore also artificial I guess is my concern.

That's what makes me wonder about AI in banking as I can see it is effective and trying to work out if there is a product to sell here but also a little bewildered by the "good" in the goods; so to speak...

A and B testing

So despite how well it worked and got the 1% I have then been running tests on tweaks to architecture of the model, tweaks to the maths, additions and subtractions. Boom! it hit 7% in 24 hours instead of 1% in 3 days. 

This version utilised a lot of different skills in setting it up and the first three iterations of it failed like not failed to coalesce just stopped trading until I figured out underlying causes and fixed. 

The 7% is actually unheard of and if I was you I would not believe me but I do keep checking the code and rereading just raw data to check logics by hand! I think I would put it down to a bit of luck there but the maths I put into this AI also worked in other places previously so while it is surprising it is that effective it was not surprising for me that it was effective period as I expected an increase.

I guess it was a series of A and B tests and maybe this was just "luck" and representative of multiple runs but I had implemented a few upgrades.

Also slightly mad it was an improvement I thought I had already rolled into this code for the previous blog post but actually it was a old code line and it was only when I found another improvement and realised I had not rolled in an previous one into the code so it got two upgrades at once.

It hovers exactly around 50% accuracy sometimes dipping down a bit and then shooting back up. I think what it is especially good at is dropping the buy or sell in sudden scalp and seems to sort of more capable in sort of making sudden changes of gear in its strategies. I tend to focus on bitcoin as I roughly know the price and see when the AI in its simulation trades on it. What I am happy about is that this version does seem to jump in and out of trades.

This is why I am hopeful with this one to send to the cryptocurrency trades in that it seems to change strategy depending on the market because it does seem to be inconsistent with how long holds bitcoin.

I also figured out how to split up my data to make better graphs to show you... its the little things you know.....

Graphs for new output