I Wrote A Programming Language - Part 1

“No research will answer all queries that the future may raise. It is wiser to praise the work for what it has accomplished and then to formulate the problems still to be solved.” – Theobald Smith

I am still not entirely sure how I feel about this project. It is the lowest-level work I have ever undertaken, and I have had genuine moments of pain—particularly when I discovered a dangling pointer somewhere in a morass of several thousand lines of C++.

I can, however, feel my C++ improving. I am reaching the point where I no longer have to stop and search for the most basic idioms. I have also run into real memory-management problems, several times coming close to building my own templated objects to manage memory. I may yet do that before the project closes.

The project began because I wanted to learn about indexes. From there, I kept moving further down the stack. At some point, I started asking myself: do I really want to rewrite this? Surely I can improve on Python’s garbage collection or SQL’s memory heap? The answer was no—you probably do not want to rewrite SQL. But you learn a great deal by trying. I would still recommend it: ten out of ten for learning, even if it takes three weeks to produce half a week’s worth of practical work.

Another goal is to scale up Hello World. To do that, I would eventually need to move more of it into compiled code. There are also language-level issues I would like to clean up to make Hello World faster. In an ideal version, this language could become the foundation for how the AI interfaces with the other projects: something lower-level and closer to the command prompt.

That is how I convinced myself to write an entire programming language in low-level C++. Somewhere along the way, it became a query language—or at least query-language-like—because that would eventually lead me to B-trees and SQL-style indexes. In doing so, I would tick off a long-standing bucket-list item.

I also want it to become a graph database, because I think graph structures are genuinely useful. Eventually, it needs functional programming features too, so that I can use it for the scripting I have in mind. I have a rough idea of how it could become both a query language and a scripting environment.

I like giving projects names. For now, this one is Project Goblin. After choosing the name, I searched the web and found two unfinished Rust languages called Goblin. They do not appear to be trademarked, and neither has much documentation, but I will sort out a proper name later. If necessary, this can be the Goblin Query Project—not GQL, as that is already trademarked.

Under the Hood

I built a custom lexer for this project—what AI enthusiasts would now call a tokenizer. It splits the language into categories and assigns meaning to each part. Single quotes (‘) denote strings, which the language calls Datum, the Latin singular of data. Double quotes (“) introduce comments, which the lexer ignores. Numbers are converted as they are encountered.

Parentheses, (), contain calculations. When the lexer identifies them, they are passed to a sub-parser so that mathematical precedence is respected. They can also be used for more unusual calculations, which I still need to explore further. Curly braces, {}, denote subqueries. They represent more complex changes of context and will probably become the foundation for functions and scripting. Any unquoted word is treated as an argument and is later processed by the parser into internal commands.

The language is built around a left–symbol–right pattern. Symbols such as !=, /, ^, *, <, and > determine an outcome based on the values on either side. It is not strongly typed yet, so it is possible to combine values in odd ways. There is some error handling, but it currently relies on standard C++ errors; a dedicated exception and warning system is still to come.

I built a parser using an abstract syntax tree. The approach is slightly unusual: it mixes bottom-up parsing with a syntax tree. Most queries are read from left to right, but mathematics has its own rules. Parentheses, exponents, multiplication, and division must be resolved before addition and subtraction.

The interpreter uses a symbol table. Once the parser has resolved numeric expressions, the interpreter reads the remaining query from left to right. Symbols and arguments are cross-referenced against a command graph, which determines which database operation to run.

In simple terms, split the input into words and identify special commands based on '' or {}. Then mark the tokens in the order they should be processed, allowing mathematical expressions to be handled. Use a three-value pattern: one value represents an argument symbol, while the other two represent the command and its argument.

Because the language follows an overall pattern of “filter until complete, then run a command and pass it a defined set of arguments,” I have found it difficult to write code that does not resemble English.

Everyone hates everyone else’s syntax

These are the language’s overall goals. It is important to stress that, at this stage, I am simply experimenting and seeing what works for me. People can become very passionate about their programming languages, but I want to approach this as a philosophical and learning exercise: what would my own language look like, and where would its technical capabilities begin to break down? Treat this as blue-sky thinking.

I believe there is a gap in the market. One of my pet peeves is that early project discovery is often assumed to happen in SQL. However, SQL has constraints and design choices intended to enforce the final output, rather than make the intermediate stages as agile as possible.

Minimalism—everyone says it, but a language should reduce as much as possible. Capital letters will be ignored everywhere to make schemas more consistent and controlled.

Somewhere, someone will read this and be disgusted by a piece of syntax I have chosen. I intend to lean into that. If there is an approach that is common today or considered standard, I will try not to follow it where doing so creates an opportunity to explore something meaningfully different.

I am using CRUD (Create, Read, Update, and Delete) as the foundation for the overall structure, building enough around each function to create a working demonstration. The basics are now in place for graph-database-style pointers, which should reduce reliance on joins and make data discovery more accessible for junior analysts.

Rather than relying solely on traditional joins, the approach uses subquery unions, which I plan to optimise, alongside graph pointers. These pointers will make related records in other tables feel effectively adjacent to the main table, providing a more efficient alternative to a SQL join. Subquery unions can support cases where joins are used as filters, while pointers make discovery easier. The goal is not to create a data-storage tool, but a top-down query tool that lets users pull board-level data into RAM and apply a series of expressive filters.

I have also been playfully considering a broader question: AI has learned from the vast amount of code available through Git, allowing it to work across nearly every programming language and understand publicly documented vulnerabilities. What happens when I start creating entirely new languages?

The work so far

This is the call order. The actual commands are in red and right of helper= and enclosed in “”. I have not finished it as a command line system yet.

This is the output: I still have a bit more to do on date too string conversaion. Deletes currently flags records but because I need to make decisions on exactly how graph and keys will work in the language and any deletion of a record itself will need to handle that its not fully implemented. I have put input in normal and output in bold.

3

3+7

3-2

PRINT 3

3

PRINT 3+7

10

PRINT 3-2

1

PRINT 3^2

9

PRINT3-2

print 3

3

print 3+7

10

print 3-2

1

print 3.5

3.5

print 3+7.7

10.7

print 3-2.5

0.5

print 3-2.5*10

-22

print (3-2.5)*120

60

PRINT 'Hello world'

Hello world

PRINT 'Hello'+' world'

Hello world

PRINT (UPPER 'Hello world')

HELLO WORLD

PRINT (lower 'Hello world')

hello world

PRINT '#'*50

###################################################

create name='alice' alias='ronda'

Record added

Added name set to 'alice' affected  1 records

Added alias set to 'ronda' affected  1 records

create name='bob' age=34.6

Record added

Added name set to 'bob' affected  1 records

Added age set to 34.6 affected  1 records

create name='CHARLIE' DOB=DATE 2019 2 7

Record added

Added name set to 'CHARLIE' affected  1 records

Added dob date 2019 year 2 months 7 days 0 hours 1 records affected

all print key name alias age dob

0 alice ronda NULL NULL

1 bob NULL 34.60 NULL

2 CHARLIE NULL NULL NULL

name == 'alice' print alias

ronda

age == 34.6 print name

bob

name == 'bob' print age

34.60

age > 5.6 print name

bob

age < 3235.6 print name

bob

name == 'alice' print alias print alias

ronda NULL ronda

ronda

name == 'alice' print alias "print alias"

ronda

name == 'alice' UPDATE alias 'bond girl'

name == 'alice' print alias

bond girl

name == 'alice' delete alias

name == 'alice' print alias

NULL

{name == 'alice'} + {age < 3235.6} print name

alice

bob

But you have CRUD in principle as the core of the whole language.

I pulled the ontology

It originally included a complete semantic model for automatic compression, essentially keeping everything in sixth normal form. While it worked, I eventually removed it after a bug left me confused and led me to suspect the translation systems. Building it took considerable time, and removing it took just as long because it was tightly integrated with the rest of the system.

The approach replaced every string in the database with a key-value pair. I later realised this became difficult in a database where values may need to be deleted, so I built a complex mechanism to deprecate values and schedule their eventual deletion.

The problem was that deleting a word required updating every key-value pair added after the deletion was recorded in the memory-tracking system. This meant tracking deletions across every system and node. To account for those deletions, each node needed both local and global generation numbers to determine whether it had been created before or after the most recent deletion.

Queries also needed to account for those offsets by comparing object generations. That quickly became impractical: everything in the system was a record, and each record would need to track its own deletion state until garbage collection could reduce its generation.

Lesson learned: keep it simple.

Conclusion

The basic structure is in place, but there is still a long way to go.

The syntax is largely the reverse of SQL: it processes left to right, whereas SQL is read from the bottom up. It also borrows elements from Lisp. In SQL, filtering appears at the end, followed by the table, the list of variables to retrieve, and then the SELECT statement at the front—along with curly braces and additional grammar. This language removes much of that grammar, reuses many familiar commands in different ways, and generally follows a filter > command > arguments structure. Symbols, data, and patterns are interspersed to provide context for each command.

To simplify the work, each record is managed as a dictionary into which named fields can be added. To make the behaviour clear and safe, there are three operations: update (the record should already exist; change this field), insert (the record does not yet exist; add it), and upsert (represented by an equals sign), which simply ensures the data is stored.

Next, I need to create a process for extracting measures and roll-ups—such as counts, averages, and other aggregations—so that top-level data is easy to retrieve. I also need a way to calculate values using a defined filtering method, ideally as a per-record process. These are the next two stories I plan to work on.

Nesting subqueries makes the language more expressive than SQL joins. As the chart below shows, it can support all the main join types, while also allowing those operations to be nested.

At this stage, everything is expressed as filters. However, I think the model could remain simple: an unprocessed subquery, {}, on its own could calculate measures across filtered data and return counts or similar values. A subquery assigned to a named value could instead run at the row level, adding a new field. This would establish a separate process for defining functions and, potentially, variables.

I expect the subquery process to become a major part of the language. The system already runs similar, but distinct, code depending on the symbol applied to a query. That may be one of the language's key innovations.

I still need to implement keys and linking, along with a way to perform calculations at the row level. From there, I can expand into scripting use cases. I expect those scripts to use the subquery design for definition and calculations to invoke them, following a Lisp-like pattern. I need to revisit the evaluation order for calculations and determine how easily an expression can be recontextualised as a function call before it is handled as a sub-calculation. The graph aspect should also be interesting: it may require its own interaction model, and I still have symbols available to represent graph traversals.