I Wrote A Programming Language - Part 2

“Programs must be written for people to read, and only incidentally for machines to execute” – Hal Abelson

I am developing a programming language to migrate and better structure my existing code, while experimenting with unconventional approaches to writing software. At its core, this is a creative exercise: an opportunity to build my ideal scripting language.

Learning memory management has been a steep curve, and I made a few architectural mistakes that needed to be cleaned up. The good news is that it is now robust, and the code has been free of pointer-related issues for about a week.

The graph database components are now functioning, and I am partway through building a file management system. This is where I had hoped to be by this point last week.

Working title is Goblin. Needless to say it's a work in progress.

Key feature call out

The graph feature is all setup in the above you can start with a record called ghoul and using there connections hop across multiple files using only a few reads.

This language uses two syntaxes: ~ and .. The tilde (~) represents a journey or traversal, shifting your context to a new location. The full stop (.) acts as a getter, retrieving data from that location.

This approach removes joins from the language entirely. Nested joins can have N × M time complexity, which quickly becomes impractical when a primary dataset contains millions of customers. Once joins span multiple tables with millions of rows, performance can fail altogether.

Traditionally, this is difficult to achieve because each connected object would need some form of ownership or reference to the others, which is not usually permitted. Through careful memory management, however, these objects do not store keys to related objects as they would in SQL. Instead, they retain a copy of the related object's memory address. When an object needs data, it bypasses a lookup and accesses the related object directly, as though it were part of its own dataset.

Data engineering often assumes that designers will use a star or snowflake schema. These models help prevent join complexity from becoming explosive, but they also discourage wide database designs. The underlying reason for this limitation is the reliance on keys and joins. With direct memory ownership, traversing three or more related tables, as shown below, is computationally inexpensive. This remains true even with millions of rows, making wider data models practical.

One frustration I have with SQL is that you must define a table before inserting data into it. Even when you know everything about an event you want to capture, you still cannot add it to the database in a single script. In a real-world application, this often means writing multiple INSERT statements across several tables whenever new data is added.

Each INSERT statement also requires you to explicitly list the fields and provide a corresponding value for each one:

Like

INSERT INTO [Data] (

[Field 1],

[Field 2],

[Field 3]

)

VALUES (Value1, Value2, Value3)

I do not like the ammount of grammar, its wordy and it will fail if I send data in field 1 that is bigger than the VARCHAR written earlier maybe days earlier.

I have already told the database which fields exist when I defined the table, yet I have to specify them again during the insert. Then I must list every value in the correct order; if anything does not match, the operation fails.

This is very simple in goblin being less script.

The example above creates two new tables, inserts the data, and links them in a single command. In SQL, the equivalent statement would be many times larger. This approach also becomes especially useful when building API access: a remote application can log a call and capture customer, agent, and device data in one operation. Even when the data spans multiple datasets or tables, I can continue using the connect command to branch out into additional data and observations.

Nothing prevents me from building an entire record set from a single API call while also embedding the relationships between all of the associated data.

You will spot that presently this is not ACID compliant and is a odd mix of traditions from query and scripting languages.

I plan to build table variables that can collect these records while ensuring they remain ACID compliant, although I am not there yet. I have only recently reached the point of building what I originally envisioned. The idea is that these datasets can be used like notepads, allowing data to be added freely, while the underlying tables can enforce data-quality requirements, such as requiring specific variables.

For scripting I want to develop ways of structuring a language around functions that allow you to build something you'd usually use another language for inside the query language.

I also began developing functions without initially realising it. My original idea was to use subqueries {} and calculations (), with much of the work divided between them. After thinking it through, I decided that calculations should be permeable and inherit the current filters, whereas subqueries should not. This makes it easier to use the subquery folding described earlier to combine datasets without repeatedly writing ALL to remove filters before passing the results to calculations.

Longer term to meet all these goals I think I need to do the following.

Because a calculation inherits the current context from the parent query that calls it, it can be used to create nested case statements within the data. It works much like an if/then statement in C++ or Python, but can be written on a single line. I plan for functions to be saved as named text scripts within tables. This would make it possible to assess a table’s functions and recommend minimum data-hygiene standards, including rules that prevent incompatible data from being added.

In object-oriented programming, the goal is to have a set of objects that own both their data and their functions. The tables described above meet that requirement, effectively bringing object-oriented programming into a query language. It does not need to be more complicated than that.

Mathematical operations follow the same contextual principle, similar to linear algebra. When you call an argument and request its data, it calculates the total based on the current context. For example, if you perform a calculation and then call the won value, it sums all won values before multiplying the result by 0.5. Although this may seem counterintuitive, it is necessary in some cases. The alternative is to use the for command, which iterates through the context and treats each data value much like a row in Python.

As shown at the bottom, sending a top-level won calculation into a global parameter—a value enclosed in square brackets []—makes that value available inside the inner for calculation. This allows you to perform operations such as calculating an average. It is computationally efficient and reflects what Python and SQL do behind the scenes: calculate a value once at the top level rather than repeatedly calling a function from within a process that must reach outside its current context to perform the calculation.

Project Planning

My storyboard looks like this.

Do a while statements I am seeing this being a while and a calculation/subquery twice. First one filters the data if there is one or more results hey run the second. Same with if. Do a if statement. I might also look at until statements I sort o do not know why they fell out of favour.

Adapt current connect command to create complex data structures from current primitive nodes. If you think about it like this python lists, data tables etc. are all just data connected sequentially to each other. A mesh network are just nodes connected to everything in their network. Trees are just branches and edges of nodes. Virtually all data structures can be rebuilt from the primitive data structures of a node and edges.

You will note the language enforces naming these relationships and this means that because you can only go down one path you cannot cause an explosion in the amount of paths open to the ~ and other path systems. I am thinking at some point id like it to be highly expressive allowing to write crawlers that can treat data like a network and alongside the top level query should be great for discovery.

Write a file management system (started) if I can dump these files out and bring them back then RAM stops being an issue pretty much everything I want to do from SQL

Need a ? value; oddity of the language is because these are all dictionaries and not going to be acid compliant until cleaned into their tables need a way for finding data based on column names.

normalise command? I think it would be cool if could just run commands to take data and using the relationships split out into 6NF or denormalise just using their relationships and commands. It would make my life easier.

add back in translator to control the language

add in ? process for wildcard rely on translator for

Fuzzy testing: Lots and lots of fuzzy testing.

Implement design patterns for normal memory patterns (linked lists, dictionaries) and marry up design philosophy. I would like this to be something like imports in Python as this limits the language sprawl.

Do implementation of variety of test cases of using goblin in actual analytics

Do optimisation B trees etc on data structure for the file management. i.e. I have at this point my dream programming language

Look at comparisons with Goblin and SQL, PostResSql etc; Use to optimise the language for speed.

Setup concurrency: I want to call a AI on my computer and then check in with it.

Setup api: I want to talk to my gobli8n instance from a computer in another country.

Implement goblin DB

Implement visual solutions for Goblin DB

Sail into the sunset on the yacht goblin DB has bought me after everyone decided it was bestest language. Buy that bank from Harry Potter for marketing synergy. I mean I said dream language

Current Syntax

3

3+7

3-2

PRINT 3

3

PRINT 3+7

10

PRINT 3-2

1

PRINT 3^2

9

PRINT3-2

print 3

3

print 3+7

10

print 3-2

1

print 3.5

3.5

print 3+7.7

10.7

print 3-2.5

0.5

print 3-2.5*10

-22

print (3-2.5)*120

60

PRINT 'Hello world'

Hello world

PRINT 'Hello'+' world'

Hello world

PRINT (UPPER 'Hello world')

HELLO WORLD

PRINT (lower 'Hello world')

hello world

PRINT '#'*50

###################################################

create name='alice' alias='ronda'

Record added

Added name set to 'alice' affected 1 records

Added alias set to 'ronda' affected 1 records

create name='bob' age=34.6

Record added

Added name set to 'bob' affected 1 records

Added age set to 34.6 affected 1 records

create name='CHARLIE' DOB=DATE 2019 2 7

Record added

Added name set to 'CHARLIE' affected 1 records

Added dob date 2019 year 2 months 7 days 0 hours 1 records affected

all print key name alias age dob

0 alice ronda NULL NULL

1 bob NULL 34.60 NULL

2 CHARLIE NULL NULL 2019-02-07

name == 'alice' print alias

ronda

age == 34.6 print name

bob

name == 'bob' print age

34.60

age > 5.6 print name

bob

age < 3235.6 print name

bob

name == 'alice' print alias print alias

ronda NULL ronda

ronda

name == 'alice' print alias "print alias"

ronda

name == 'alice' UPDATE alias 'bond girl'

name == 'alice' insert familyname 'Jackson'

name == 'alice' print familyname

NULL

name == 'alice' delete alias

name == 'alice' print alias

NULL

{name == 'alice'} + {age < 3235.6} print name

alice

bob

{name == 'alice'} (age < 3235.6 print name)

bob

(name == 'alice') (age < 3235.6 print name)

bob

name == 'alice' (age < 3235.6 print name)

name == 'alice' (age < 3235.6) print name

alice

(name == 'alice') + (age < 3235.6 print name)

bob

create name='eric' alias='loverboy' connect married name='fran'

Record added

Added name set to 'eric' affected 1 records

Added alias set to 'loverboy' affected 1 records

Record added

Added name set to 'fran' affected 1 records

name=='eric' ~married print name

name == 'alice' connect justfriends {age < 3235.6 print name}

bob

name=='alice' ~justfriends print name

bob

create name='ghoul' alias='a real monster' all name ! 'ghoul' connect enemy {name=='ghoul'}

Record added

Added name set to 'ghoul' affected 1 records

Added alias set to 'a real monster' affected 1 records

name=='alice' ~enemy print name

ghoul

create name='buffy' alias='a real monster slayer' all name=='ghoul' connect hates {name=='buffy'}

Record added

Added name set to 'buffy' affected 1 records

Added alias set to 'a real monster slayer' affected 1 records

name=='ghoul' ~hates print name

buffy

name=='alice' print .enemy .hates .name name

buffy alice

name=='alice' ~enemy print name

ghoul

name=='alice' ~enemy delete

name=='alice' ~enemy print name

all won=5*6 print won

Added won set to 5 affected 6 records

30.00

all won=(0.5*won) print won

Added won set to 0.5 affected 6 records

90.00

all for (won=(0.5*won)) print won

Added won set to 0.5 affected 1 records

45.00

all [total_won]=won for (AVG_WON=(won/[total_won])) print AVG_WON

Added avg_won set to won affected 1 records

0.17

all total_won=won for (AVG_WON=(won/total_won)) print AVG_WON

Added avg_won set to won affected 1 records

0.17

Update on Omen and Hello World

I am still working on this its split into 4 projects to research 4 different aspects of the problem. There is a language learning version that takes the human brain emulation of project Elihu and tries to adapt it to be a LLM. This is slow going and I am really trying to pick the nuts out of the problem.

This being randomly getting 3 symbols in a row is highly unlikely within 8000 characters so it shows understanding and its a human brain emulation not a transformer model.

Then I am looking at a generalised trading bot. Multiples of these show a ROI of > 1 even when statistically aggregated (but I think taxes in the UK are larger than that); mumbles something about fundamental versus technical analysis.

Though from the perspective its short exposure times and the reinforcement learning is self determined so the assumption is anything > 1 ROI learned by self play and so would readapt in event of a trading system death and it gave the user 5-10 minutes to trade on the signal. Also where > ROI is statistics so is probably generalised.

statistically significant but fake money and lower than the tax rate. Which reminds me I mentioned the no no and need a disclaimer: investments in securities, stocks, or fiat are subject to market investments the value of investments can go up or down, be taken by the tax man, money printer or acts of god. This content is for informational purposes only and does not constitute financial, tax, legal and or marital advice. I do not know what that means when they print money like they do but its a legal requirement that I remind you.

Then I have an attempt and simplifying project Elihu into a more basic version that treats everything as liquid cells rather than trying to work ack from human based data and would sort of be more generalised. So far this is not working but it felt worth trialling other approaches as this is where the money is at the moment is bringing down the cost of transformers by using biologically inspired patterns.

I liked the idea of evolving a whole human brain emulation but this is lightning fast and if it works it works. I read the Navier Stokes papers they put out and it gave me a bit of a brain wave so trialling a simplification to the algorithm.

I tried getting funding for running a variant of the LLM one alongside human EEG and having about 20k of results thought if I could show any significant improvement against that 20k results could get the net claim of information had to have been transferred out of the human EEG. I have made a few attempts at getting funds from accelerators to this end but I think I need to double down and try a lot more.

I probably cannot afford to just throw compute at the issue so I think if I can just exhaust the accelerators it will tell me whether need to just build myself. I also think have the problem as after reading up on the law for patenting a ANN its really the process that uses the ANN so really you have nothing until you build the end product.

Therefore there is this problem with venture capital that it sounds like they invest in future tech but bizarrely they will only really be interested if you build it yourself and really if patent your only protecting your method and there really is more than one way to skin the cat.

I had graphs for the closing Omen part but forgot to save them so great new feature for when next post!!!

I Wrote A Programming Language - Part 2

Add comment

Comments

Create Your Own Website With Webador