The Design and Structure of the Domainatrix Operating System
I bought a PC in July 1996 at the age of 20. I'd started computing way before that, starting when I was 12, on a Sinclair ZX Spectrum - a 8-bit machine with a 3.5 MHz processor, eight colours and 48K of RAM, though I had the extended, 128K version! I'm quite certain, especially given the high percentage of European people over here, that quite a few of you will be familiar with it, if you're not, please feel free to check out http://void.jump.org for part of the CURRENT scene (yes! :)) still associated with it. That was the machine which I cut my teeth on and learned to program and hack on (oh, the feel of interpreted BASIC - typing in PRINT p and getting "2" on the screen - there's nothing quite like that, some people over here must be knowing what I'm talking about, right guys? :)), and the one I played my very first computer game (!!!) on!
By age 15 or so, I was master of the Speccy, knowing every little detail about it, from the insides of its ROM to practically each and every one of its system variables (BIOS data area to you PC lamers ;)) by name and address. All the while we were banging around on it (and each other - the video game swapping scene at school had always been quite vigorous, and at least on one occasion led to my getting beat up by an older guy), the PC was always the "other machine" for us, the one that grown-ups used, the ones that were in offices (International Business Machines, hmmm, how enticing is that for a 14-year-old?), so when I finally DID move to the PC, well, suddenly given 24bpp colours, 44Khz 16-bit-sound, 14.4k access to a worldwide network known as the Internet, my immediate reaction to it was I guess the only one that could have been: I wanted to be thoroughly, thoroughly acquainted with its insides, how it worked, functioned and did its stuff at a very low level, so I always knew exactly what was going on inside it. I guess there was only one way that could be done and that was: to write an OS.
There were a couple of features, while programming on the Speccy, that I'd always had the idea that big machines had (such as they never crashed!! :)) that I wanted to put in, plus by this time I'd used Windows and Linux and seen couple of things that I liked and disliked about both of them, and wanted to just take the best out of both (and everything else that was out there) and put that into my own system. Now, I understood that this would be mostly for fun and hobby's sake, however I also wanted the OS to have a particular use, i.e. not just another common or garden desktop OS. It was this very thought that I was thinking when, well it just hit me: if it's not desktop, what else could it be but server? This, together with the fact that I'd just been reading a bit about TCP/IP which just fitted so nicely into the Intel 32-bit architecture, made me decide that this was going to be its purpose. The name just came to me one day out of nowhere, practically every other cool name even vaguely associated with the Internet had been taken up. The basic idea was to get TCP/IP performance up to the maximum level that the chip would allow, with nothing standing in the way. This meant that the OS would have to be structured so that re-assembly, right from IP fragment to data stream level used the fastest possible path with the minimum number of copies, as well as high-speed encapsulation, i.e. repeated addition of various headers to some given data, before it was finally OUT()putted to some device.
Given the purpose though, a couple of other features too, would have to be put into the system to help it achieve maximum performance - a fast filesystem, graphical terminals, so we could finally kiss goodbye to the 1982 VGA ROM font, and what I believe are the three killer features that Domainatrix will have - extensibility, so that, for example, the webserver process could actually determine the caching scheme used by the OS for that page, the ability to redirect STDIO to sockets, to facilitate CGI programming, and the big one - a hot-swappable kernel!! It was needed so that I could (theoretically) ensure a 100% successful hit rate.
I got about programming all this in February or March 1999, deciding first of all what the memory map would be. I decided to keep the IDT right at the beginning, just like DOS in real-mode ;), and the GDT right after that. To find out their occupancies, I used a simple bitmap, so that I use the bsf instruction to scan here and there to see which entries were filled. The kernel came after that, followed by the data structures it was going to need. The first part of the kernel would obviously consist of the hot-swap code itself, since it would be overlaid by code from the new image, and therefore would need to be the same across all kernels. My original idea for the hotswap was to make a copy of the File System Driver code for the device from which the kernel was being loaded (more about this later), after which the internal OS entry for that device would be changed to point to the copy, and a fread() would be done to read in the new image. At the moment we are talking about actual filesystems, and the layering needed for the drivers, and I'm unsure which way we are going to go, but it's going to be pretty similar to what I originally planned.
My next task was to decide on how to prioritise processes, and after thinking about it for a bit I came up with the very simple scheme of having a counter for every proc equal to its priority which the OS would simply count down on every context switch, and upon reaching 0 would simply allow the next timer interrupt to cause a jump to the next task (i.e. an actual context switch! :)).
My next thought was about memory management, and after reading a lot about paging, and various algorithms, I decided to go for a pure segmented system only, because, remember, as this was a 100% asm OS, I'd be expecting pretty much all apps for it to be programmed in asm as well, which meant small programs only. I simply assigned a data structure to keep a list of all procs in memory, i.e. their FULL pathname (so I could tell when something was already there, upon a new proc. being created), and another one with a list of all the segments in memory, because we'd be having lots of instances of the server processes in memory, and we'd obviously want them to share code segments. Each entry consisted of the start of that segment in memory and its end, a pointer to the next suspended one which we'd need for virtual memory, and a fourth d-word containing: 1) which LDT entry in the GDT that segment was, 2) which no. entry IN that LDT, and 3) numbers of that no. entry in the LDT (only if 2 pointed to the Code Segment entry of that LDT, of course).
My next thought was about I/O. Now every OS textbook goes on for pages about streams and theory etc., but communicating over serial lines was not our objective here (though we would be Telnetted too of course). Now I/O has been traditionally redirected, in practically every operating system there is, to Files, some of which maybe handles to the system in some way, like CON in DOS. When thinking about the purpose of my OS, I came up with the idea of I/O being redirectable towards SOCKETS!!! As far as I know, this has never been done before. This would make a webserver's job running CGI programs tremendously easier (and FASTER!!) than it is currently, rather than having to pick up whatever the program was outputting to STDOUT itself. Now this obviously meant that the webserver program would have to exec() a program, however there would have to be some mechanism for it to output to a socket that was held by the webserver. For a solution to this, I came up with the Call() scheme. A Call() (A call() and exec() btw are the only two ways in which a process can be created, there is no fork(), or anything of the like) syscall in Domainatrix causes the code for a program to be loaded into memory, just like an exec(), and the current CS entry in the LDT of the process switched to THAT area of memory, WITH NO OTHER CHANGES TAKING PLACE. I.e. the new program that's just started to execute shares all other LDT entries with (for example) the webserver process, including the one that stores the Socket data for that process.
Going further along with this thought, I decided to allocate two data segments for a proc, one "write" one, where it could put any data which would be used as parameters or the like, and a "read" one, where results of any kind would be received. So, it was very easy - the webserver program would call() the CGI program to be executed, while setting the Output Redirected flag in the parameters to call(), which would happily write to STDOUT, with the result being everything getting sent over the network. Pretty nifty eh? :) Not just that, it made pipes tremendously easy, cause that would also of course be a very similar mechanism (the "read" and "write" segments aiding a lot in the process).
The next step was to think about device drivers, and the model I would use. I spent a LOT of time thinking about this, and reading up on how various OSes were doing it, in the end, considering the slightly different low-level paradigm that 100% assembly offered, I came up with the following - a d.d. has an Init section which gets run immediately upon loading into memory, a Send section, which would be executed when a proc. wanted to output something to that device, and a Recv() section, containing the IRQ handler for that device (note that this would assume that every device, at least every device for the purposes of the OS anyway HAS only one IRQ). Both of the latter execute in the context of a process, i.e. the Send() and Recv() sections would be simply JMPed to, with IRQs simply being shut off by the driver writer around critical sections. This also made IRQ chaining for PCI devices very easy, because the OS could simply keep track of what IRQs had been registered (i.e. already being handled) by a driver (by device NAME, very unconventional, but it would work - very similar to Linux's "eth0" and "cua3" scheme).
As an update, Domainatrix has decided to adopt the Uniform Driver Interface, mainly because I like the idea of a standard model, common across all OSes, plus there's the (promised) big-time vendor support, which is what we'd really need, and is very tempting. :)
So, that's just a little bit of what's happened till now, I finally put up a webpage on the Internet and started trying and getting developers for it only recently. We have 7 people so far, after posting to the newsgroups, and a talented bunch they are too - in our team is a 14(!!!)-year-old who already knows a little bit about protected mode programming!!! We're having a whole lot of fun, and getting some work done. Things we have on the agenda are: a port to IA-64 - very ambitious, yes I know, but that just tempts me more ;), SMP capabilities, RT scheduling maybe, so we can serve things like video, well, a whole lot more...
Do you want to join us? If you can code demos, I'd say you're perfect to join our team. We need people who share my low-level performance-is-No.-1 attitude, but who are not afraid to go deep into theory and algorithms before starting to code. Maybe you could do our graphics subsystem, which actually needs starting right away, maybe you could help us perfect our memory management system, which is at the moment under severe discussion, or maybe you can just hang in there and do the code for the optimized bubble-sort that we really need at the moment. You're not a Java "programmer" are you? You'd rather die than use an API wouldn't you? Your 3D engine's rasterizer doesn't have any pipeline stalls in its inner loop does it? Well let's put that talent to some goooooood use... Mail me and we can get started! Do it now! See ya! :)