"I do not need a hard disk in my computer if I can get to the server faster, carrying around these non-connected computers is byzantine in comparison" - Steve Jobs
Introduction
I was writing a blog post about designing my dream programming language when my computer suddenly died because of a power issue—possibly related to the fan. I opened it up, confirmed that the hard drives were fine, and removed the RAM.
I was forced to replace my old machine. Fortunately, I lost nothing but time. The experience left me feeling unexpectedly philosophical, and I spent some time rethinking my backup protocols for my code. I have routinely backed up my machines for years, but this time, despite having two copies, I discovered that one was missing.
Ahhh….
This led me to take a brief hiatus from programming and reflect on what I was doing. I had reached the point where I needed to plan more carefully: the language fundamentals and core processes existed in a rough first draft, but they needed to be revisited.
So, once again, I set out to create a programming language. Its working name is Goblin.
What Do With A Programming Language With Only One User
First and foremost, my long-term goal is to adapt the language for large-scale simulations, allowing data to be stored and analysed in one place while reducing my current reliance on multiple languages.
To support everything I want the language to do, including its current capabilities, I need to make several improvements. One major missing component is a serialiser for the language's data structures.
The immediate challenge is that the language currently uses an in-memory graph database. It also needs optimisation to make better use of a CPU's L1 cache.
The next step is to begin writing data to disk. The simplest approach is to use write-ahead logs: the program would maintain a log file and periodically save its position to disk, continually appending the latest instructions. If power is lost, I can replay the instructions in the log file to restore the database to its most recent state.
Left unchecked, however, the write-ahead log would grow indefinitely. A second process is therefore needed to periodically write the current state of the data to persistent storage. This process, known as serialisation, would save the data in a binary format on disk.
A thing about doing this properly is it takes a bit of thinking as
The Different Ways To Write A File
Your OS spoils you, in my opinion. The hard disk is in reality just a space in which to write 0s and 1s, a large physical area that can be addressed and used. Everyone knows this, but I think most people have not thought about the exact implications when they use their PC in this way.
At the top level, the Kernel tracks the individual hard disks it is connected to as device drivers. A structured file system then tracks ownership, locations, and usage on the hard disk. You largely never see this level of abstraction because all the commands are not done on bare chrome, but passed to the OSs system, and that manages the whole process for you.
This creates a system where you could just give a filename and allows the OS to go and pass that file into a location on the hard disk. We all sort of intrinsically understand this concept because the various OS we use present it as the default and make it seem like the natural way to work.
The other choice would be to try to bypass the OS and write as bare chrome. This can cause corruptions to the OS if it overwrites anything important, and the OS will typically stop any program ran at a level below administrator from doing this and accessing the hard disk in this fashion. The data would be there and presumably you could retrieve it, but the OS would not know it was there, might overwrite it later, and so no file would be there in the filesystem.
A mix of these processes though creates a Raw-Block Database engine. The database messages the OS and creates a file to a large chunk of continuous data. The alternative is to create a hard disk partition and close off whole chunks of the hard disk for this purpose. The end result is that the OS now knows to leave that data area alone and the database has privileged access over its own data and can bypass the OS, which yields faster read and write times.
Though this creates another issue that this continuous area of hard disk is now open to use but the management and interpretation is done by the database programme. In order to be effective the programme must understand exactly where its data is, how it is organised, and how to recover it when something goes wrong.
Hard Disk Management Patterns
We already encountered one method in the previous discussion on B trees and indexes. In the current build the programme uses these structures and points at memory space within RAM rather than on the hard disk itself.
Let us try to over simplify this by thinking of the hard disk like the tools and structures we use in other programming languages.
The tables we use in databases are probably most alike the lists we use in programming. In RAM you can, as discussed, reserve a continuous area of the hard disk and write to it in sequence. As you read through the data you might read from the start where you began the file and continue onwards. Though in the language I am writing there are a number of items that add data to its lists and use strings, which are famously variable sized. The issue you immediately run into is that if you do not know the length of what you are reading and writing to and from memory then you do not know where, when, or how far to skip ahead in the database.
Ideally you might want something like this pointer paradigm we have in RAM where you can just dereference to the location of the value in RAM, but that causes a few issues. The first one is that you would need to ensure that any tampering with that file that you are trying this with will be picked up and handled correctly.
One solution is to use a value to represent a split in the data. In that case, it does not matter how much data you write to disk: as long as the next value is signalled by a sequence of bits that marks the split point, the reader can identify its boundary. The drawback is that, although the data is now divided into distinct sections, you still need to begin at the start of the file and scan for those split points unless you explicitly record their locations.
With careful design, however, you can store the offsets for these split points in the nodes held in memory. This allows you to look up data directly by its offset instead of scanning the entire file each time.
All of this relies on the data remaining intact. If those marker points are lost or corrupted, your program may attempt to load everything on disk after a given offset into memory. Because a hard disk is usually larger than available memory, that quickly becomes a problem. Controls are therefore essential. Reserving memory addresses through the operating system is an important first step, and it is also why multiple overlapping administrative-level programs should not be given access to the same memory space.
This approach works well when data can simply be appended to the end of the system while offsets are used to retrieve records from the middle. It becomes more complicated when data must be deleted, as frequent deletions can quickly fragment the disk and waste storage space.
For this reason, some database systems avoid deleting data altogether. Instead, they continually write to the end of an ever-growing write-ahead log. Most systems eventually need to remove older data, however, and handle this in different ways. One option is a circular structure, where data is written until the end is reached and then new data begins filling available space from the start. These approaches often have sophisticated names, but they are variations on the same underlying idea: managing storage efficiently while preserving fast access to data.
Your RAM cannot eat the whole databases
This started as a in memory graph database language but I would like it to have both options.
As mentioned earlier, a hard disk can store far more data than memory. As data is loaded into RAM, the system must also determine which data to remove when space becomes limited.
This can be as simple as monitoring the available RAM and removing the oldest files from memory. By keeping recently used data in order of access, the system can discard older items while retaining the data most likely to be needed.
More advanced approaches balance a file’s age against how frequently it has been read. For example, a counter can be increased each time a file is accessed, helping the system decide whether to remove the next oldest file or keep it in memory.
When RAM is managed effectively, the system begins removing data as memory fills up. It should do so in an order that keeps the data most likely to be requested again available in memory.
Concurrent Systems
Although I have not demonstrated it yet, all nodes are reentrant-ready. Each system includes an equivalent of a “Do Not Disturb” sign to prevent conflicts while it is in use.
All current functions are coroutine-ready, and none should individually consume excessive computer resources. However, I am not yet certain how this will scale when all database options are included. Because the design already uses a graph database, it is built around structures intended to handle recursive code, which I believe presents a similar challenge.
The next step should be to create a thread pool. This should be relatively straightforward: the functions can remain largely the same, but before a task is passed to the language system, it would move through a thread-management layer and be assigned to an available worker thread.
This would make it possible to run multiple programs in the new language simultaneously. From there, it could eventually support distributed execution, where a thread or CPU is located not on the user’s own PC, but on another machine—effectively creating a cloud-based system.
Distributed Systems
I want a system that can run scalable simulations effectively. The ultimate goal would be to make the database distributed.
Building a distributed system is somewhat beyond my current capabilities, but I believe it would be a worthwhile challenge.
Internet connections can be unavailable, and the computers you expect to communicate with may go offline unexpectedly. These failures are non-deterministic, meaning they can occur without warning. The system would therefore need a way for its different components to reach consensus.
My initial approach may be overly simple: each node should operate independently and manage its own data, but…
When a node needs data held elsewhere, it could follow stored pointers to the next node, using signposts to locate the required information. One challenge is that IP addresses can change, so a heartbeat mechanism or similar process would be needed to keep the mesh network synchronised.
This is the area that lies beyond my current capabilities, but I see it as an ambitious, over-the-horizon goal.
Next Steps
This is likely to be a long-term project, primarily intended for learning. However, it has already reached a point where, if I set it up on your computer, it could handle most of what standard Python can do—excluding libraries such as requests and packages installed through pip, like pygame.
The next step is to implement a simple write-ahead saving method. The program will save everything you write to a file, so if it crashes, you can run a command that replays that file, restores your most recent actions, and returns you to where you were.
After that, I will implement a straightforward file-saving process using the operating system’s native storage system. Even if I build everything on this list, the underlying goal is for the architecture to represent everything as a series of pointers. This removes RAM as a limiting factor because data can be stored on a relatively fast SSD and, eventually, scaled out across networked storage.
To make this work, you first need persistent locations that represent the starting points for your database. From there, relationships can be used to move efficiently through the graph. This means dedicated save points will be essential.
Once basic saving and concurrency are in place, I can begin rewriting the tools I use in other languages in Goblin.
The next step is to implement a way to save nodes in a database using low-level Chrome interactions with the hard disk, creating the database model discussed above. Ultimately, file-start offsets will be added as another node type. This will make it possible to write Goblin scripts that operate across the entire hard disk, treating it as one large graph database.
Before that, I created another game as a learning exercise to gain more experience with low-level Chrome interactions with the CPU and hard disk.
I will also need to implement enough commands to search this data heap. Initially, the system should support both node and graph database operations and full-disk scans.
At some point, I may need to pause to implement tamper checks, disk defragmentation, and other cleanup and disk-management processes.
It will probably need an installer by this stage.
Then I’ll explore whether I can turn it into a distributed system. I think security can remain relatively lightweight, because you should not trust data from a computer you do not directly supervise. When one goblin reaches out to another, would you really trust that other goblin?
Seriously, though, security would need much more consideration at that point. A distributed process could allow an attacker, through insertion or impersonation, to take over a node in the network and feed it incorrect data. Currently, Goblin has no mechanism for interpreting code: you give it data, and it remains data. It cannot retrieve code from one of these sources and use it to escalate its permissions, reinterpret it as a website, enable cross-site scripting, or anything similar. I believe it should stay that way. I have long held the view that security becomes far more manageable when data remains data—never interpreted as code or passed to the operating system. Even so, I will need to take a long, hard look at security at this stage. I suspect it may become a significant rabbit hole for the project.
I love the idea that, eventually, I could sit at a desk, type commands, and have a system retrieve data from a vast network—or allow code to run against a database. That may sound contradictory to the point above, but Goblin should naturally function as a coding sandbox. It should operate within a defined, bounded area of the hard drive, with only a small number of fixed interactions with the operating system. Those interactions should be limited to precompiled, unmodifiable functions. If designed correctly, those safeguards could make Goblin preferable for certain data-discovery tasks, even in an enterprise environment.
I am not completely certain about this final point, but I believe Goblin could be naturally secure in this way.
I probably will not get there but that’s the point of a weird coding learning blog its to challenge oneself.
This would make it easy for the hello world type tests because I can run the brain emulation tests on a distributed system and never leave the same language or terminal.
Finally breathe and start to look at optimisation, I am a big believer in write it dirty first write it right and fast after but with projects like this where its at the edge or beyond your level of competency this can cause pressure.
Finally I am going to treat this as another bucket list stretch challenge to try and get all this done in a year. Last year I wrote I wanted to learn a bunch of stuff and I have systematically done all that (I did the totally random game stuff and it now has random mage types and random spells but have not blogged about it).
I think I am going to fail this year. But at least I will learn a lot.
Also remember one day your PC will die you will be sat with it with a screw driver unsure if the backups worked and taking it apart to get the hard disks out thinking I really should have done more on the infrastructure and backing up. I did and I still thought that and I still panicked and I still was taking a computer apart after losing some of it.
As a play on memento mori... i.e. remember you must die.
Memento computarumtuum moriturum esse.
Remember the computer will die... the Roman senate paid someone to walk behind Caesar whispering remember you will die to him is the myth to try and make him less arrogant. Maybe you could say to your local 10 x programmer or CTO in your life... I'm not brave enough to try it but its there as a suggestion.
Add comment
Comments