Self-hosting (compilers)

An operating system is self-hosted when the toolchain to build the operating system runs on that same operating system. For example, Windows can be built on a computer running Windows.

Before a system can become self-hosted, another system is needed to develop it until it reaches a stage where self-hosting is possible. When developing for a new computer or operating system, a system to run the development software is needed, but development software used to write and build the operating system is also necessary. This is called a bootstrapping problem or, more generically, a chicken or the egg dilemma.

A solution to this problem is the cross compiler (or cross assembler when working with assembly language). A cross compiler allows source code on one platform to be compiled for a different machine or operating system, making it possible to create an operating system for a machine for which a self-hosting compiler does not yet exist. Once written, software can be deployed to the target system using means such as an EPROM, floppy diskette, flash memory (such as a USB thumb drive), or JTAG device. This is similar to the method used to write software for gaming consoles or for handheld devices like cellular phones or tablets, which do not host their own development tools.

Once the system is mature enough to compile its own code, the cross-development dependency ends. At this point, an operating system is said to be self-hosted.

Software development using compiler or interpreters can also be self hosted when the compiler is capable of compiling itself.[1]

Since self-hosted compilers suffer from the same bootstrap problems as operating systems, a compiler for a new programming language needs to be written in an existing language. So the developer may use something like assembly language, C/C++, or even a scripting language like Python or Lua to build the first version of the compiler. Once the language is mature enough, development of the compiler can shift to the compiler's native language, allowing the compiler to build itself.

The first self-hosting compiler (excluding assemblers) was written for Lisp by Hart and Levin at MIT in 1962. They wrote a Lisp compiler in Lisp, testing it inside an existing Lisp Interpreter. Once they had improved the compiler to the point where it could compile its own source code, it was self-hosting.[2]

The compiler as it exists on the standard compiler tape is a machine language program that was obtained by having the S-expression definition of the compiler work on itself through the interpreter.

— AI Memo 39[2]

This technique is usually only practicable when an interpreter already exists for the very same language that is to be compiled; though possible, it is extremely uncommon to humanly compile a compiler with itself.[3] The concept borrows directly from and is an example of the broader notion of running a program on itself as input, used also in various proofs in theoretical computer science, such as the proof that the halting problem is undecidable.

Ken Thompson started development on Unix in 1968 by writing and compiling programs on the GE-635 and carrying them over to the PDP-7 for testing. After the initial Unix kernel, a command interpreter, an editor, an assembler, and a few utilities were completed, the Unix operating system was self-hosting - programs could be written and tested on the PDP-7 itself.[4]

Douglas McIlroy wrote TMG (a compiler-compiler) in TMG on a piece of paper and "decided to give his piece of paper to his piece of paper," doing the computation himself, thus compiling a TMG compiler into assembly, which he typed up and assembled on Ken Thompson's PDP-7.[3]

Development of the GNU system relies largely on GCC (the GNU Compiler Collection) and GNU Emacs (a popular editor), making possible the self contained, maintained and sustained development of free software for the GNU Project.

Many programming languages have self-hosted implementations: compilers that are both in and for the same language. An approach is bootstrapping, where a core version of the language is initially implemented using another high-level language, assembler, or even machine language; the resulting compiler is then used to start building successive expanded versions of itself.

The following programming languages have self-hosting compilers:

• Bootstrapping (compilers)
• Compiler-compiler
• Cross-compiler
• Dogfooding
• Futamura projection
• Self-interpreter
• Self-reference
• Indirect self-modification