~pixelherodev/zyg

ref: 27187f8391f97f3e6f69df47ee5ad9337d9975a0 zyg/design/legacy -rw-r--r-- 6.7 KiB
27187f83Noam Preil fixes 5 months ago
                                                                                
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
# Why? Why make this?

There are very different design goals for this and for the zig compiler.

## Simplicity. 

The defining feature of zyg.

Stage2 has a bunch of nice features like incremental compilation, hotswapping, writes machine code directly, LSP, the parser is error-ignoring, and so on - these add complexity to the compiler in a way that can't be removed at build time. 

I intend to build the tcc or cproc of the zig world.

## Compilation speed

I'm one of the heretics that considers compilations speed as *more* important than runtime speed.

The faster the compilation, the more time you can spend debugging, which means more bugs get fixed, more time is taken on the actual work of developing and less time spent fencing with toy swords.

Increased compilation time means more time spent working, which means more time spent on proper optimizations - tweaking data structures and algorithms, for instance - and less CPU time spent on minor 5% optimizations.

# Notes

Hmm...

Upstream needed incremental compilation + parallelization. Why?

With C, parallelized compilation means multi*processing* - a compiler instance per source file. Zig can't do that the same way, since we need the root source file - hence, multithreading. Lazy eval somewhat reduces how much work needs to be done, but not by enough to match the compilation speed of e.g. cproc / tcc, which is zyg's goal.

Maybe there's a way to get multiprocessing back? What if each instance is spawned by the single invocation - thus allowing zyg to be used as a drop-in Zig replacement - with a --root argument? It'd require each instance to have shared understanding of the package location - fork() should allow this in a very simple manner, as long as the root information is effectively read-only after the initial startup.

Would this be worthwhile, though? Compilation performance is an important design goal, and this shouldn't be complex if done properly. 

What would this look like, if implemented?

Lexing and parsing multiprocessing architecture has been implemented and tested, and is good!

Current design:

Notation:
- "D" indicates Done, fully implemented!
- "M" indicates "mostly" done; a little bit more work is needed for 100% compliance
- "A" indicates *architecturally* implemented; the overall logic is there but it's missing the details
- "N" indicates total absence; it hasn't been accounted for at all yet.
- "T" indicates that it's not yet present but is trivial to add
	- This is so that I can look back and laugh at how stupid I was for thinking anything would be easy
- "C" indicates that it is currently being worked on, and there's no point try to precisely define how finished it is yet
- "P" indicates that it's been planned for and designed, but isn't implemented at all
	- this is to remind me that it needs to be tested before more implementatino is done
- there's nothing else because this notation is already needlessly complicated

* Process CLI args - A
* Determine package paths and dependencies - A
* Spawn parsing subprocess for root source file - M
	* Also spawn subprocess per package root - T
* Wait for child processes, spawning new ones as needed - D
	* Use a queue for parscesses, don't spawn them immediately - T
	* Read in finished ASTs when a parscess completes - C
* AST Resolution - P
	* This is handled by the parscess monitor after all exit - it builds up the list of all ASTS, then merges them to hand back to the main compiler bits
	* Replace imports, merge ASTs (e.g. `@import("a.zig")` gets replaced with the parsed struct that is the root of a.zig - T

Everything after this hasn't even been designed thoroughly yet, and needs to be tested architecturally before implementation can proceed.

* Typification
	* Evaluate comptime functions where necessary
	* Propagate known information, build up dependency list
* Semantic analysis
	* Executes any comptime functions inside of runtime code; handles comptime+runtime interleaving
* Produce IR
* Backend

Comptime logic:
		* Use full sema->ir pipeline, then interpret (JIT?) IR
		* Static storage for named comptime values, erase the rest after a 'comptime engine instance' ends
			* Spawn a comptime engine instance for each comptime "root;" the call site at which comptime execution begins

* Parser Subprocesses ("Parscesses")
	* If already parsed or in process, abort
	* Lex
		* Inform parent of all encountered imports ASAP
			* Resolve root,std,etc
	* Parse entire file
		* Deduplicate *all* references within the AST
			* Reduce memory consumption
			* Makes type comparisons a single integer comparison instead of requiring complex logic!
	* Pipe results to parent
		* Piping is slower than shm, but removes the need for any means of shared memory or synchronization
			* Slower != slow.
		* Need a simple protocol to differentiate imports, types, etc. Could be as simple as a "type byte" (import,expr,type), where imports are read and parscesses spawned, and others are read straight from the pipe into list (after ensuring capacity)
		
Okay, new problem: I want to support other languages in the frontend (current public list: Zig, C, DragonFruit, LOST, and Coyote). This requires a major change in the design. Bright side, my old project name (Cimple Compiler Components; Project Tricarbon) is very well-suited to this.

Fortunately, there are *some* constraints to guide this design process. Languages like C++ are absolutely not under consideration. 

Each component should be distinct. The parser subprocess should probably be a dedicated binary. Challenge: semantics differ between languages, heavily. Should sema be run in subprocesses? That removes a major advantage (the comptime engine could otherwise be used for implicit execution for *any language*); comptime is too tightly integrated into semantic analysis (probably rightly, though I should reexamine that) to keep it separate. It also means a large duplication of work, since a lot of semantics *don't* differ! It's impossible to cleanly split those that differ from those that don't.

Except... maybe it's *not*. Specific stages of sema can be kept generalized, most likely; typification should be language-independent (though unneeded for e.g. LOST).

Typification also includes part of comptime, though for anything other than Zig that would be unused at this stage.

After typification is general semantic analysis. This *will* be harder to split. e.g. wrapping arithmetic differs in Zig and C. Can this be generalized sanely without explicit language checks?

In this *specific* case, yes; simply keep the operations different (Wrapping addition vs unwrapping addition). This does not prove the general case.

What about differing operator precedence? Well, that could be solved by storing precedence explicitly.

Hmm...

Volatility? Nope, identical behavior :)