“Languages must change thinking / the most damaging phrase in programming is it has always been done this way” – Alan J. Perlis.
Introduction
I’m building a programming language called Goblin, with an unusual fascination for set theory. It’s largely an excuse to explore and build a wide range of code, but I hope to use it for a variety of projects in the future.
The project began as a bucket-list goal: build a high-speed database and implement SQL with index support. Eventually, that felt too straightforward, so I jettisoned SQL and decided to create my own language instead.
I built a semantic model that previously broke and went from there. Added in a lot of small functions and that let me build trees and what I call as trains to navigate and search the data in a user defined way (analogous to a web crawler but in your own code).
But to start with was my excuse to build this was to build and learn about indexes at the most bottom up level. So this might be the story how I gave up on indexes in SQL and wrote my own language to do that.
What Is an Index for our use here?
An index is a data structure that acts like a bookmark for a table or database, helping the database locate data more quickly.
This improvement in retrieval speed typically comes from an organised structure, such as a tree or a set of bookmarks. A complete tree structure can consume significant memory, while bookmarks without an organising principle become little more than a series of pointers.
SQL terminology can also be unhelpful, as it often uses labels such as clustered, non-clustered, or hashed rather than identifying the specific underlying data structures and explaining how they interact with the data.
Criticism of Indexes
As I started pushing SQL in more demanding ways, I found indexes and partitioning surprisingly difficult to get to grips with. In theory, these should be bread-and-butter concepts for most data engineers, but in practice I have found that they are rarely used.
My personal explanation why used less today is that the issues we have now have shifted and when you get really close to understanding what indexes are you see that database technology might still be rooted in old problems.
Back to the origin story for Goblin; I wanted to broaden my SQL skills, so I added them to my learning list for the year. Over time, though, I kept redefining that goal because I realised I did not really like indexes...
My view is that these tools were developed when hard disks were the primary bottleneck. In a post-SSD world, it is possible to keep “hot” data in storage that performs much closer to RAM, even when it is not held in memory. Meanwhile, “cold” data can be stored on cheaper HDDs or in the cloud. This has shifted the technical challenge towards how we use RAM, especially as its price has increased.
Indexes were developed when databases likely contained far fewer rows. Their advantage is that they can reduce lookup times to roughly log(N), rather than requiring a scan of every row. By organising data around a balanced central point, with lower values on one side and higher values on the other, a database can search through a tree much faster than it can search the entire dataset.
However, if you are comfortable with the maths, you will see how quickly the memory requirements can grow. In the past, this trade-off made sense because disk access and CPU capacity were the main bottlenecks. Index structures also need to be maintained as data is inserted, updated, and deleted. For very large tables, that maintenance can consume significant RAM and reduce performance. And for tables supporting live, web-style workloads, an index may not always be the right choice: if writes and deletes are as frequent as reads, the cost of maintaining it can outweigh the benefit.
Now that RAM prices are through the roof maybe should reevaluate how index data is my thought. Maybe a more human designed approach with fine level manipulation would be useful.
B++ tree definition
Modern SQL engines do not typically build a complete tree all at once. Instead, they maintain a small number of nodes, each with a fixed pointer capacity. When a node exceeds that capacity, the engine rebalances the structure and splits the contents into smaller groups.
Alternative tree structures can store a fixed number of nodes in an array, with a linked list beneath them. Values can be laid out sequentially, then searched or aggregated to partition the data into searchable sectors.
Benefit: this approach uses a single node with X pointers, plus N additional pointers when the list has not yet been built around the target data.
It provides simple memory limits, but it does not guarantee O(log n) time complexity; performance is closer to n/x, plus overhead.
Its main strength is that it is lightweight, much like a Python dictionary.
A full B+ tree, by contrast, requires n heavier nodes, typically with two pointers per node and a stored copy of the comparison value. Used carelessly, this can become memory-intensive.
It also requires a traversal algorithm to navigate down the tree. The algorithm should be straightforward, although this is my first time implementing one in this language.
Automatic rebalancing can be used, but it may result in a tree that is not perfectly balanced, with branches containing different numbers of levels before reaching their leaves.
Requirements: a function to retrieve values by address rather than by name, for speed.
A rebalance operation that splits an overfull node further during insertion.
My impression of SQL was that indexes were created with a few commands, while their size, RAM usage, benefits, and trade-offs were often obscured. Indexing felt like a dark art: it could speed up queries, but the associated memory costs and the impact on inserts and deletes were not always clearly signposted.
I decided that, in Goblin, I would build a full B+ tree in human-readable code.
Building A B+ Tree In Goblin
I started with pseudocode to outline how I thought it would run.
“Initial filter sets out what is going to be built on top of”
name==’alice’ to_apply=1
[row]=1
while{to_apply==[row] count > 1}
{
while{to_apply==[row]}
{
create connect graph_alice {to_apply==[row] alias .max 1 to_apply=0 [helper]=alias}
connect graph_alice {to_apply==[row] alias .max 1 to_apply=0}
test=[helper]
to_apply=1+[row]
}
[row]=1+[row]
}
all delete to_apply
Trains Choo Choo
To traverse that tree, you would build a train—my current name for this design idiom. In its generalised form, it uses a while command, with the first step defining what it is trying to find. There may also be room for a while loop here. The process then examines the available paths and selects one, continuing until it finds the desired result.
The generalised form would look like this:
[Traverse tree]=(to_find field)(while (field ! to_find)(if(field > to_find)( ~right_path)(~left_path)))
If I need to use calculations, I should ensure that the code handles them correctly. My understanding is that using a while loop with a calculation should recurse while preserving the current selection. I need to think this through further to make sure it does not introduce any issues, but I believe that approach is correct.
Logically, I see only two approaches: recursion and a standard while loop.
Although the example above uses Goblin to take a data set and build a tree from the bottom up, trees are typically built from the top down.
For this, I briefly explored →relations, which is a vec_out command in Goblin terminology. It retrieves the relatives of the current context and should make it easier to create train-style designs across an entire network. The code is designed to avoid duplicate records: if two nodes share the same relative, that relative is included only once.
However, finding everyone exactly six steps from the current data would require →relations →relations →relations →relations →relations →relations. Finding everyone within two steps—that is, one- and two-step relationships—would use (→relations) + (→relations →relations). My thought is that this should make it easy to build trains.
Additions to the language to make this work
-Pop lets you just grab a value by index; pop 0 gets the lowest key of that set by standard.
-? is a question and lets you check if data exists within a data set;t an argument command looks if it exists in data, global checks if it exists in namespace, and look-in a (.) checks if it appears as a relationship.
-?! confirms that the value does not exist.
-I might add in? num ?str ?date to do type checking as well, as that would be simple with the semantic model.
-added =.look-in to allow a different way to reassign relationships.
-Changed the way a global group or function is defined and no longer uses a = but uses a colon: as this leaves you able to then do [global]=() to quickly reassign to current context. A colon usipulates athat ist or definition is to follow so this improves the language's similarity to English, and I feel that using a different symbol so you get [global]:()() hopefully looks more distinct.
Levelled Up Tree
The options below use the rows themselves to store the tree pointers, making them slightly more compact than the full index described above.
name==’alice’
[head_of_tree]=( pop 0)
popout “removes value by index; if no number follows, removes first in context”
for
(
[helper]=alias
[alice_temp]=()
with [head_of_tree]
“At each while stage, it checks the relationship and splits based on alias”
while (? .right_path ? .left_path)
(
if([helper] > alias)
(
~right_path
)
(
~left_path
)
)
“If not, the new value will have itself added ”
if(? .right_path)
(
if([helper] > alias)([temp]=.left_path=.right_path=[alice_temp]=[temp])
(.left_path=[alice_temp])
)
(
alias=[helper]
.right_path=[alice_temp]
)
)
This is what that looks like in goblin currently.
name=='alice' [head_o_tree]=(pop 0)
popout 0
for ([helper] = alias [alice_temp] = () with[head_o_tree]
while (? .right_path ? .left_path)
( if ([helper] > alias)
(~right_path)
(~left_path)
if (? .right_path)
(if ([helper] > alias)
([temp] =.left_path =.right_path = [alice_temp] = [temp])
(.left_path = [alice_temp]))
(.right_path = [alice_temp])))";
It looks better on my other computer webador tends to flatten it all left but on mine it's very indented and clearly readable.
Build a list, pop from it, traverse the tree, and add to the bottom of the tree. In Python, you would need to model this with two classes: one for nodes and one for the tree.
And there we have it: a B+ tree. I could have simply learned about SQL indexes, but for some reason, I felt compelled to understand the concept at such a fundamental level that I developed my own programming language to critique it—and then implemented it in fewer lines of code.
I also think that structuring it as shown above makes it very readable—at least in a style similar to Python or C++.
How Could You Make This Better - Choo Choo, Chuga Chugga
Next, I could extend this so that, as it traverses, it swaps old nodes for new ones. I do not plan to take it any further, but I have built the functions needed to remove nodes as they are found and store them in groups. I think this “train” concept could become a core design pattern for the language.
I was surprised by what worked...
I want to call this the centipede. Because it is a query language that shuffles variables through a tree, I initially thought I needed to move values around in a circle. Then I realised the language already allows me to do the following. It works sequentially, from left to right: temp becomes the left path; leftie becomes right-ey; right-ey becomes Alice; and temp inherits the original left path. The final step is unnecessary, but the carousel works. As a way to build trains that crawl across a web of data—shuffling data and databases according to rules—the approach below works.
I also implemented a swap command that exchanges the values of two variables, but it is not used here.
[temp]=.left_path=.right_path=[alice_temp]=[temp]
The example above is a train, but it can also shuffle data. It moves through the data and rebalances values using two global variables to support the reshuffling. Conventional wisdom says that globals are a bad idea, and I do not disagree. However, following that logic too strictly would push me toward recreating many features of existing languages. So, for now, Goblin makes heavy use of globals out of pure stubbornness.
The strongest argument against globals is that another part of the code could interact with them and change their values, making bugs difficult to track down. My response is: why are you repeatedly calling different variables by the same name?
I think globals can be useful when they are used carefully. In the example above, there is a clear rule: when you change contexts and move around the database, global values allow you to break out of those boundaries. As shown above, trees often have a previous stage where the relevant value can be captured in a variable.
The difficult part is that the current context filters the data, and there is only one way to turn that filtered state into a variable.
I will continue to watch and see how this develops. I may eventually adopt local variables; I have not used the @ symbol yet, which is the conventional SQL notation for a local variable that is cleared between queries. If you need local variables, plenty of other approaches can produce the same result. But only a global variable is truly global.
I made it work
I realised I needed my while to be able to pass its current context back into itself. I changed calculations when used with while so that it folds its current context back on itself. This allows you to use while in two ways.
The normal way test do while{}{} or while{}() or while(){} will do this and they will check
But while()() will work much more like an easy-to-create train, as here h, the second.
While(?! steps)(steps=steps+1 →relations )
I think that would get you something like across this network: how many different ways are there to get here.
While(?! legnth)([steps]=[steps]+1 ([steps] < legnth legnth=[steps]) →relations ) min legnth
I think that would do A*.
None of what I built runs on recursion either, so this would be effective at places where Python would have memory errors first.
My groups,s I ordered them so now.
I liked the database origami idea la ot,t t but it seems excessive and is not computationally optimised for big data sets
Therefore, I use the → symbol, which I refer to as “vector in” or “vector out” depending on the direction it points: right is out, and left is in. It is a mathematical operator, so you can think of it as typically appearing on the left-hand side of an equation.
I also adapted the code order when joining a global. This means that if you place sort commands after the setup code for a global, they will run each time new values are added. Because it uses a bubble sort, it should remain computationally inexpensive, provided the sorts themselves are small.
Currently, it does not support sorting by multiple values. I consider this somewhat unnecessary and find it unusual when people want it, as only the first value really matters.
I had a real mouth-drop moment when I realised that a group like this can only change the nodes given to it—unless you specifically use the all command or involve another group. However, this also allows a single table to reach out and set or create data in other groups.
So when I said “factory-oriented” rather than “object-oriented,” I meant factories in the plural.
Weird Things I Learned About My Own Language
I had a delete command. While writing the tests, I tried to write them from memory so I could learn and master my own language—a strange concept, and perhaps a lesson in metacognition in itself.
One of the odd things I discovered was that I instinctively wrote delete, set the context to everything, and used tlook-in with the relationship name prefixed by a full stop (.). This attempted to delete the entire database and wipe out all its information. The reason is that delete checks for an argument afterwards; if there is none, it assumes you want to delete the full context, then checks the groups and clears any invalid pointers.
It was looking for an argument, and I had supplied a look-in value instead. Because of the rules I had written, it fell back to its default behaviour: deleting the nodes themselves.
ALL OF IT, BECAUSE I HAD SAID ALL. Obvious in hindsight, really.
I have now changed it so that you can use either a look-in value or an argument.
When I originally wrote it, I considered that syntax a little terse. It was almost inevitable that someone would trip over it—but it was readable, it was technically correct, and surely I would not be the person to perform the equivalent of DROP TABLE on my own database in my own language.
Now, to delete a node or group of nodes, you must write delete and follow it with a calculation or subquery that defines the context of what you want to remove.
All delete () “- Works because All says context is all,l and the full context gets passed into the calculation”
delete (all) “- Works because All says context is all and is called after the calculation is done”
delete {name==’alice’}“- Works but deletes just where named alice ”
name==’bob’ delete {name==’alice’}“- Works but deletes just where named Alice, not Bob”
delete {name==’bob’ name==’alice’}“- does not work because no one is named bob and alice”
all delete “No longer works”
all delete .next_alice “Never works”
This is what I learned: no command should run directly on the database; preferably, three safeguards should be in place. I had thought
Appendix
New colon method for building function examples.
Creates a function
[Increment_steps]:(steps)(steps_walked=steps_walked+steps)
Creates a shelf or group (I call them groups generally, but in code it's called shelf fn; I guess in maths theory it would be a set)
[alice_table]:{name == 'alice'}
[mutant_factory]:{mutant = 1}{delete mutant}
To make groups more flexible in general and allow using them like variables n,,ot everything a class or factory added these capabilities. If a factory or classifier has already been set up
[global]=() “will write current context to global, will create it if it doesn't set up query; will be blank s, so use a variable that just stores data without describing it; will run existing factory systems on the data. Quick way to save current context into a global that can have factory settings preset to clean and manage the data”
[global]=( number > 5) “will write current context to global where number greater than 5, will create if does not exist but setup query will be blank so used to create a variable that just stores data without describing it, will run existing factory systems on the data. Quick way to save current context into a global that can have factory settings preset to clean and manage the data”
[global]={} “will write subquery into global, will create if does not exist and set the setup to subquery, can be used to quickly create a factory or classifier, if the object has already defined query and deletion rules will update the query, any future additions will have factory definitions in this query run against them. Will not change deletion rules; will error if it tries to overwrite a formally written group with deletion rules to stop creating a situation where a group is formally created and then income and outgoing rules are changed. Becomes a quick way to create a factory or classifier or overwrite an old one wholly.”
insert [global] () “will insert current context so if the global does not exist will create it but will not add a query so will not create a factory”
insert [global] {} “will insert current context so if the global does not exist will create it and will add a query, a quicker way to create and safe if exists, any future additions will have factory definitions in this query run against them”
update [global] () “will change the global to the current context but not create it if it does not exist. Will run existing query if exists within the group on the new data”
update [global] {} “will change the global to the subquery but not create it if it does not exist; the subquery will become the creation subquery; any future additions will have factory definitions in this query run against them. Will not change deletion rules will error if tries to overwrite a formally written group with deletion rules to stop creating situation where formally created a group and changed income and outgoing rules. Allows overwriting of existing groups.”
[global]←() “add current context into a group, will not run change existing query rule. Will have current factory rules applied to themselves. Useful as can have a train (see above) move through data and to a group for other systems to use and look at”
[global]←{} “add subquery into a group, will not run change existing query rule. Will have current factory rules applied to themselves. This is due to the logic that unless you overwrite and provide a full new definition you are required to follow existing rules.”
[global]:{}{} “Way to create a group with strict deletion rules and entrance rules. Is protected against tampering if the group is formally created like this”
I should then be able to expand the colon way of building a group. It would be good if you could create the full object, as currently you create the group and add functions. You will notice all of this lets you rapidly work with variables or define them as factors; you can never redefine the variable in a way that then leaves it with data inside that no longer matches a written definition given to it,t such that the data inside should always match the label.
Special consideration was given to running empty calculations becauseit'ss easier to just run, but everywhere else in code
Things You Learn When You Code Step By Step
I like this feature
[global]=.lookin
You read it and think all it does it takes a global makes a group and puts all the lookin giving the related values but that is all the related in data from data that’s already filtered. That I could do and squirrel away that data mid-processing when I have it at hand; it's a query language, but you can get the data and manipulate it like it's a procedural language.
Conclusion
I added a lot of “stuff” to get this to work, and this felt like the first full script I wrote in Goblin. I can tick off the bucket list of building an index.
It feels like a full language now. I am sort of surprised I could write in it at all for a long time. I felt I was just stringing commands and cool ideas at a wall, but I actually enjoyed writing Goblin code, and it felt elegant.
The use of subqueries or calculations doing different things in different places keeps cropping up. Despite being very fuzzy, logi,f {} does things that ignore what is outside itself, and () does things and takes its context from what comes before itself, creating some interesting symbolic logic, like with the while loops, where while()() and while{}() are very different commands and have specific meaning in context.
I like that it really does not read like any other language I have learned before this. I read the code written for Goblin, and if I had not written it myself, I would have gone, huh, what does that do? The fact it’s a query language implementing a procedure creates a weirdness to reading it; if you were to try and do that in Python, you would end up creating a series of other classes and writing functions (you could do it the same way in Goblin). You could not write like that in SQL. But no object was explicitly created, and if someone came along, they could interact with that data structure using the →relations command and follow your tree and inspect it.
Add comment
Comments