~vdupras/duskos

ref: 5d294d8c81cb95765fc96ad7c9ad681db18d3d76 duskos/README.md -rw-r--r-- 11.6 KiB
5d294d8cVirgil Dupras posix/vm: fix compilation under cygwin a month ago

#Dusk OS

Dusk OS is a 32-bit Forth and big brother to Collapse OS. It does everything Collapse OS does, has the same laser sharp focus on simplicity, but has a widened scope:

  • Runs on as many modern devices as possible.
  • Access a wide variety of filesystems.
  • Access a wide variety of storage devices, including modern ones.
  • Access, visualize and print images and PDFs.
  • Compress and decompress files.
  • Edit and compile C code. Possibly other languages too.
  • No focus on binary size (unlike Collapse OS, we have plenty here).

Additionally, it's worth mentioning that as a consequence of its laser sharp focus on simplicity, Dusk OS is also defined by what it doesn't do:

  • Not POSIX
  • No security (single user)
  • No focus on speed
  • No concurrency other than interrupt handlers

Depending on the simplicity tradeoffs, it's also possible that it has no networking. To be determined.

#Why build this OS?

This operating system's goal is to allow its users to maximally operate the machines they have during the first stage of civilizational collapse.

Most modern operating systems can do all of the above and are self-contained, self-hosting. Why do we need another one? Simplicity.

It's difficult to predict post-collapse conditions, but we can suppose that many users will need to use their machines in novel and creative ways. Hackability of the operating system then becomes paramount. Open source modern operating systems all can be modified to fit its user's needs, but their complexity limits the likelihood that the user is able to do so. A simpler OS increases this likelihood.

But we can't have our cake and eat it too, right? Either you have a simple toy OS or a complex one. Well, maybe not?

Its authors believe that in the history of computing, Forth has been under-explored. Its approach to simplicity is, we think, revolutionary. It has significant shortcomings when system specifications become more complex (Forth hates complexity and doesn't manage it well), but we believe it possible to elegantly marry it with languages that like complexity better.

This mix, we believe, could provide a creative user with computing powers rarely seen with other approaches. We've got to try it.

To be clear: this is a research project, we don't know what it will yield beforehand. We have the intuition that it might lead to a big "ah ah!" moment and reveal a breathtaking combination of power and simplicity.

#A whole OS built from source on boot

One thing that makes Dusk OS special is that it boots from a very tiny core (1000 lines of x86 assembly). From this tiny core, on boot, it builds its way up to a system that has a functional C compiler, which then allows it to bootstrap itself some more.

With regards to "source bootstrapping", it's even more extreme than Collapse OS because modern machines allows this process to run very fast and the whole process is not much slower than a regular Linux boot. On Collapse OS target machines, this process would be prohibitive, so a bigger part of the OS is cross-compiled into the kernel.

This peculiarity of Dusk OS has interesting properties. The nicest one, in my humble opinion, is that this allows us to sidestep the entire problems of binary compatibility and relocation and only deal with source compatibility. So, no ELF, no binutils, only code that is designed to run from where it was written in the first place. This is so much simpler!

Object files? Global symbols? Nah. C functions that don't have a static storage type are simple Forth words.

#Who is Dusk for?

The target Dusk user is someone who's creative, close to hardware, can read a datasheet. Dusk shines (well, will shine) when one wants to poke around the hardware without limit.

It compares favorably to other more complete OSes because there's no concurrent process to mess with your poking and the driver structure is (well, will hopefully be) more approachable, hackable due to its stricter scope and savvier target audience.

Let's use an example. Let's say you're on a notebook that runs on a chipset of Intel's ICHn family. You read the datasheet and see "oh, nice, there's an SPI interface in there. Maybe that it's not hooked to anything on the notebook, let's play with it."

Now, that chipset is very, very central to the computer. There are good chances, on a BSD or Linux system, that if you begin poking around its registers, you'll step on someone else toes and crash the system because, for example, of some other process that needed to read from disk at the same time.

In Dusk, you could completely break the SATA controller, you'll still be golden as long as you don't access mass storage. Because Dusk doesn't have concurrency, you have tight control over what happen or doesn't happen on the machine, so all you need to do is to avoid words that access mass storage. That give you ample wiggling space for your hacking session.

To be clear: this is also possible with a custom made BSD or Linux, but you're going to have to strip a lot of pieces from your distro before you get there and some of those pieces might be useful debugging tools which will be difficult to retrofit because they need a wider system. You'll also need a higher cognitive space to fit BSD/Linux wider abstractions in your mind.

#Roadmap

The high level roadmap for Dusk OS is:

  1. Have a 32-bit Forth written in x86 run on top of of a Linux system.
  2. Create a pseudo C compiler that is partly written in Forth and has the peculiarity of bootstrapping itself from it. No binary except Dusk's core.
  3. Target a machine (probably legacy PC) and have it run bare metal on it.
  4. Self-host.
  5. Steal drivers from Linux and/or BSD kernels to widen hardware support.
  6. Port exiting POSIX applications so that Dusk OS fulfills its stated goals.
  7. Add support for more machines.

#Status

Current roadmap status is discussed on a mailing list thread, but here's the summary: the C compiler is advancing well, bare metal too. You can see the kind of code that Dusk is capable of compiling and executing at fs/tests/cc/test.c and you can read about its technical details. The system is capable of booting from a FAT16 volume which is currently read from a RAM drive.

The roadmap advances with a series of "milestones" subprojects requiring Dusk OS to advance in multiple areas at the same time. The current milestone is to port Collapse OS' "blkpack" tool to Dusk OS C (with the broader milestone, of course, to port the whole of Collapse OS' C code to Dusk). For this goal to be reached, the subitems that are still needed are:

  • Implement scanf().
  • Implement getline().
  • Copy/paste blkpack.c and fire it up.
  • Probably fix many compiler bugs.

In a general way, here's the list of things missing from Dusk C for it to be consider a "real" C compiler:

  • ops width depending on type (everything is dword)
  • struct, union
  • typedefs
  • global variables (partial support now)
  • while, switch, goto
  • floats
  • a few ops here and there
  • preprocessing
  • the check phase (anything that is currently understood by the parser is compiled no matter how nonsensical).
  • the stdlib (but it's going to be significantly different than a regular C stdlib, see fs/doc/cc.txt)
  • probably countless bugs

So, there's a fair chunk of work left, but there's also a lot that's already done.

Development happens on sourcehut.

Unlike Collapse OS which is a personal effort and doesn't lend itself well to collaboration, Dusk OS' wider scope makes it fitting for a collective effort. Let's discuss this on its public mailing list.

#Collective effort

Collapse OS is specifically a personal effort and I don't accept contributions to it because its narrow scope makes the project very opinionated. I'm not willing to make compromises on those opinions, so I prefer to go alone. This works only because of the narrow scope.

Dusk OS has a wider scope and is likely to stay incomplete without a collective effort. Therefore, my stance is different here, I'm open to compromise.

I began accepting patches, but faced a problem: I apply double standards to patch reviews. On the one hand, I don't want to accept patches that I consider sloppy, but on the other hand, I have no clear guidelines for external contributions and, worst of all, I do commit, sometimes, some sloppy code myself because I sometimes want to move fast and I'm unsure whether that code will stay as is.

So, when comes the time to review, I am in a weak position to require high code quality and I can hardly reject code that works: people have been pouring work in this. Each time I accept such patch, I weaken the code and I don't like it.

All of this to say: please wait a bit. I'm trying to reach "liftoff" conditions for the project and I'm too excited to reach it so I don't document and I allow myself some sloppiness from time to time. When this rocket has lifted off, (when it's self-hosting, bare metal and has a C compiler that can begin to take a part of the "complexity load" of the system), then I'll go with a consolidation phase and the project will be ready to really become collective.

Meanwhile, I'm very open to discussions, comments, debates, questions. I would even say that I'm not closed to patches, as long as you're aware of the fact that I apply double standards until liftoff.

#Build and run Dusk

Dusk is designed to run on bare metal and to build itself from itself (it's almost self-hosting, but not quite yet. Soon). However, it's also possible to build Dusk from any POSIX platform using Dusk's C VM from posix/vm.c.

This VM implements a Forth that can interpret the whole of Dusk's Forth code, but this VM is CPU-agnostic and has its own simplistic bytecode. This means that this VM cannot do fancy stuff like run native code generated by its assemblers and compilers. That makes running Dusk from this VM less than exciting: it's just a boring Forth.

However, that boring Forth is enough to generate bare metal images for any of its target platforms, so that's why it exists. To build this VM, you need:

  • GNU Make
  • A C compiler
  • mtools

Running make will yield a ./dusk binary which if opened, provides an interactive prompt.

Documentation is lacking for now, you'll have to look at the source to have an idea of available vocabulary. Type bye to quit.

There is some documentation in /fs/doc.

Dusk OS expects a raw input. With a regular TTY, your input will be buffered and echoed twice. To avoid that, you can invoke it like this:

(stty -icanon -echo; ./dusk; stty icanon echo)

make run does this for you.

But if you really want to play with Dusk, you shouldn't play with this prompt, but rather play with a bare metal build which can be ran under QEMU. See below.

#Building bare metal binaries

make run builds and runs a binary designed to work on a POSIX platform, but if you want to build binaries designed to run on bare metal, it's another matter.

For now, only the x86 PC platform is supported and can be built with make pc.img. If you have QEMU installed, you can run it right away with make pcrun.

To run Dusk on an actual machine, what you can do is to write it directly to a USB key (dd if=pc.img of=/dev/sdX) and then boot from that key in "Legacy BIOS" mode.