Release 0.10.0 NEWS With 2621 commits (and almost three years of development) since 0.9.0 this release is not only the biggest (103% more commits) but the longest in terms of calendar time so far. Read about the changes in the Changelog file. -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEE8vXXNG287MqW+cRngmaFFkBlu0IFAl5j+zgACgkQgmaFFkBl u0LJoRAAmS/TRUfDhNI6PP1/PtLD9ycIWbWHhnQU1vi6nFYYZ+fRCVQmELNReZhN J0YFT4n+iKv4uTsQ/rcg+V1b2KaqrS6/jV29yRnqwbqPU7Ur4IPWiAGXyQ6vE08M CQDIbn1YIhvw7oD+EUgZjHFAMQHML71cF2ypb/JjN9usIgqngYH1vWMUte80MEeS P7nwxfnbi+T0plro7pfYL1YGIxwGcgcA2ifAOosoSbozF0DlL5jQ+/jMwaebtPGk OlyE43wPuzS2XqbVLZlAHeViI+GgxBaDXFSbv7X/sHRgGnJLPuQeYPf47luMzZvU /pYqfzkvSqIbyt4eop5t25U4/6UtJOD7ld/1jz+GGSLtyU1Y5vUrrdyp8IBGcG0j XoZm2pElS5aVIKYz6UnXwoRN9pO62G9MaqQ0WkC8XWaMUGu0helIcxD4DtPdGqkL lV2kG4DUM+v+meF/o/Rn9OCcnanxb/+PvSF6UcvooiWU/RVd+HYyVM0Lrp1sZCoe jOFT4plbKmm2rrVFLtQxtAaHWVlziXxaE9pmitPsy8xdMPFl/XGSP5UBFb78TiO2 jLotjZoS/b0377pY7J5DPqeEi3wcHp/In+0t8bKPNY1ascRLu5LAbu4LCKLTDi0f kiTCuZUZ/NhaH8iswWuM6SXyizG0dvhsOOjOIcSsOW5UkD0oXro= =16Ck -----END PGP SIGNATURE-----
Release 0.9.0 NEWS With 1287 commits since 0.8.4 this release is the biggest one in the history of Viua so far. And not only in terms of commit count, but also functionality. CLOSURES Closures were reworkerd. The `enclose*` family of instructions was renamed to `capture*`, and closures got their own assembly directive - `.closure:`. Functions declared as closures are not directly callable and must be instantiated first using `closure` instruction. CONCURRENCY Concurrency-related aspects of the VM also got some love. Blocking operations (`join` and `receive`) can now specify a timeout for how long they should block a process. The timeout may be specified in seconds, milliseconds, or as the `infinity` token. DEFAULT VALUES Some instructions now can be written in a shortened form and the assembler will inset default values for omitted tokens. These include for example `join`, `receive`, `istore`, and `fstore`. DEFAULT COMPILE-TIME KEYWORD The new compile-time `default` keyword can be used wherever it is legal to omit a token. The assembler will change the `default` keyword into the default value for that place. It is useful when usage of the default value should be stated explicitly. IOTA COMPILE-TIME KEYWORD The new compile-time `iota` keyword generated an ever-increasing integer starting from 1. It can be used to automatically assign register indexes: .name: %iota answer istore %answer local 42 print %answer local It is especially useful when code is changed, as the assembler will reindex the registers automatically and free the programmer from this task. WATCHDOG PROCESSES CANNIBALISE TIME OF CRASHED PROCESSES Previously, when a process crashed, the VM spawned a watchdog and run it to completion. This is now fixed, and crashed processes become their own watchdogs on failure - so watchdogs are now on the same level as all other processes, and may be preempted to prevent starvation of "normal" processes. Before the crashed process becomes a watchdog its stack is unwound. VOID TARGET REGISTER The new `void` keyword can be used as a target register. Using void register as the target register will drop the value that would by normally produced by the instruction. Some examples: ; drop the result of function call (if any) call void foo/0 ; delete the value move void %1 local EXPLICIT REGISTER SET SPECIFIERS The `tmpri` and `tmpro` instructions are no longer needed as the VM supports explicit register set specifiers for register operands. This means that to move a value from local register set to the static register set the following instruction can be used: ; move <target> <source> move %1 static %1 local ...instead of this sequence: ress local tmpri %1 ress static tmpro %1 Explicit register set specifiers make code shorted and more efficient. ATOMS Atoms are unique values whose only property is that they may be checked for equality. Useful as tags. Supported in Viua by `atom` and `atomeq` instructions. TEXT AND UTF-8 Starting with this release, Viua uses UTF-8 as its internal character set for text values. What is more, a special text type was added to the VM's list of primitive types. Text values must always be valid UTF-8. A `text*` family of instructions was introduced to distinguish text values from a "string of bytes" values produced by `strstore` instruction. As an additional feature, all values can be casted to text using the `text` instruction: istore %1 local 42 text %2 local %1 local ; register 2 will contain text "42" DEFERRED CALLS A very useful feature. Deferred calls may be registered to be called when the frame they were registered in is popped off the stack (during unwinding, normal returns, or tail calls). They are useful as a debugging aid, and can be used to implement resource management schemes. More information about deferred calls can be found on the weekly.viuavm.org blog. ADDITIONAL NOTES This release also introduces one very useful and important feature: processes can now contain many stacks, and switch execution between them. Such an additionl will make implementation of several new functionalities much easier and more intuitive. For example: - interrupts: they must be run in the context of a specified process, but are in no relation to what the process is currently executing so they are a prime candidate to be run on a different stack - message filtering: a function could be provided to filter messages available in a process' mailbox, and as this function's execution is is only a tool to achieve some other goal, it would be a good idea to run the function on another stack - stackful coroutines: coroutines that can yield from frames at arbitrary depth of the stack, not only from the top-most frame and possibly more. A style guide has been introduced with the help of the `clang-format` tool. VM got a basic static analyser that is able to catch errors at compile time.
Release 0.8.4 NEWS With 75 commits since 0.8.3 this relase brings few, but important improvements to the core of the virtual machine. SMP Viua now support simultaneous multiprocessing for virtual processes. The maximum number of virtual process that can be run in parallel depends on the settings the machine is started with, and is limited by capabilities of the underlying hardware (for example - the VM will not run 4 processes in parallel if only two CPU cores are available, assuming each core can run only one thread). INSTANT DETACHING Processes can be spawned as immediately detached, by using 0 as the target register index. The VM will interpret this as "spawn this process and execute it, and I am not interested in communicating with it". The VM will give parent process neither the possibility of joining spawned process, nor will it return a PID for the process. LAUNCH-TIME CONFIGURABLE SCHEDULER NUMBERS Viua can be instructed to spawn a certain number of VP and FFI schedulers at launch-time, the limits are no longer hard-set at compile time. The number of spawned schedulers can not be changed at runtime. Two environment variables control the number of schedulers spawned: - VIUA_VP_SCHEDULERS: for number of VP schedulers - VIUA_FFI_SCHEDULERS: for number of FFI schedulers The CPU frontend provides the `--info` option; among other things, it provides information about scheduler numbers the VM would spawn in current evironment. Use `--json` option to get `--info` output in JSON format.
Release 0.8.3 NEWS With 141 commits since 0.8.2 this release is mostly a gradual Release 0.8.3 is mostly a gradual improvement over 0.8.2, the 141 commits pushed to the repository introduce some new but not revolutionary features, and a bunch of minor fixes. The big things release brings are vector packing, multiple FFI schedulers, and possibility of embedding metadata in Viua VM bytecode files. NEW COMMENT SYNTAX The assembler supports Lua-style comments beginning with `--` and running till the end of line. Semicolon comments are not deprecated and can still be used. EXTERNAL FUNCTION AND BLOCK SIGNATURES INCLUDED IN BYTECODE DISASSEMBLY Before 0.8.3 the disassembler did not emit signatures for functions and blocks linked from external to the analysed translation units. This proved to be extremely inconvenient and was fixed in this release. Viua VM bytecode files now be freely disassembled and reassembled without worrying about missing signatures during reassembling. EMBEDDING METADATA IN BYTECODE FILES Starting with this release Viua VM bytecode format supports embedding metadata in bytecode files by employing `.info:` assembler directive. Example line: .info: license "GNU GPL v3" Metadata is stored in a simple map where keys are restricted ASCII identifiers, and values are strings. String is currently the only embeddable value type. COMPILE-TIME JUMP TARGET VERIFICATION Jump targets are now verified at compile-time and assembler refuses to generate bytecode when it detects jump errors in input source code. Some absolute jumps cannot currently be verified by assembler and their verification is perfoemed at runtime (e.g. absolute forward jumps). COMPILE-TIME BLOCK CHECKING Assembler checks if signatures for blocks used in a translation unit are present, and refuses to compile code when input source code contains references to undefined blocks. VECTOR PACKING `vec` instruction can now be used to pack objects during vector creation. Until this release vectors had to be created by spawning a vector object, and then pushing objects one by one to the newly created container. vec 1 vpush 1 (istore 2 40) vpush 1 (istore 2 41) vpush 1 (istore 2 42) This proved to be cumbersome, so `vec` instruction was augmented to support packing. Now, vec instruction receives three operands instead of one (the second and the third operands are optional, and default to zero). istore 2 40 istore 3 41 istore 4 42 vec 1 2 3 Second operand to the vec instruction is the index of the first register to be packed, the third is the number of registers to be packed. Above code creates a vector in register 1, and packs three objects starting from register 2. Packed objects are moved inside the vector, leaving input registers empty. By placing objects into registers carefully vector packing can be used to speed up multiple-value returns from functions. REMOVE RACE CONDITION WHEN REGISTERING EXCEPTIONS THROWN BY FFI CALLS This release removes a race condition from the code that could cause the machine to lose exceptions that were thrown by FFI calls. The race condition was triggering a bug when an exception was registered in a process by the FFI scheduler while VP scheduler was inspecting the process's state after executing it. If the exception was registered between checks for terminated-status, and stopped-status the exception has been quietly dropped and the process removed from the pool; the precise order of events that triggered the bug was: - exception has been thrown in FFI call, but the FFI scheduler did not yet register it in the process that requested the call - VP scheduler did not mark the process as terminated (and so the exception handling routines have not been invoked) - FFI scheduler registered an exception (which pushed the process into a terminated-stopped state) - VP scheduler checked if the process stopped and removed it from the list of running processes MULTIPLE FFI SCHEDULERS As of 0.8.3 Viua VM is now able to spawn and utilise multiple FFI schedulers (their number configurable at compile time, by default the VM spawns two). This means that several FFI calls can be serviced in parallel, so the machine blocks less and is able to execute FFI-intensive programs faster. By default Viua spawns three C++ threads now, which causes overscheduling on systems with single or dual-core CPUs. VM configuration can be adjusted if this is undesirable.
Release 0.8.2 NEWS With 171 commits since 0.8.1, this release brings improvements to machine's core code, and some long overdue fixes. The changes in this utterly uninteresting release are mostly internal and will not (*should* not) affect user code much. EXTRACTED VIRTUAL PROCESS SCHEDULER OUT OF CPU Virtual process scheduler was extracted out of the CPU code. This means that the CPU is now only the source of static information about running program - what modules are loaded, where a function begins, what methods a class responds to, etc. It also holds the queue of FFI call requests (it has not changed since the last release). The *dynamic*, process-related information that are prone to change during lifetime of a program (e.g. frames on call stack of process, exceptions mid-flight, messeges in transfer) are now handled by "virtual process scheduler", which is a special class concerned only with managing VM processes. For now, the VM spawns only one VPS (virtual process scheduler) but in future releases it will be changed, and machine will spawn a number of VPSs - each in its own host thread. Introduction of multiple active VPSs will mark the move from just concurrency, to true parallelism of virtual processes inside the machine (i.e. they will not just run one-after-another round-robin style, but some of them might actually be running at *the same* time, only under different scheduler). FEATURE OF THE DAY: std::unique_ptr<> Machine now employs `std::unique_ptr<>` in some places in code to manage lifetime of dynamically allocated objects. The `std::unique_ptr<>` is now heavily used only in the `VirtualProcessScheduler` code to manage stack frames, exceptions, and global and static register sets, but in later releases it will gradualy replace naked pointers in other parts of the VM. LESS CRASHING, MORE STACK TRACES In previous releaes, when one virtual process crashed, and the crash was not handled by the watchdog process, it brought down the whole VM, and produced a stack trace. This behaviour is now altered: when a process crashes and there is no watchdog to catch the escaped exception the VM will print a stack trace for the crashed process but will continue running. This change is intended to enhance reliability, errors will always happend, they just should be isolated from the "healthy" parts of the system and serviced, instead of bringing the whole system down with them. What if the `main/` function crashes? If there is a watchdog process to handle the failure, the exception will be serviced and the VM will continue execution, to finish with exit code 0. If there is no watchdog process, machine will continue running and when all other processes finally terminate it will close with exit code 1. WATCHDOG PROCESS SPAWNED PER-VPS, NOT PER VM Each VPS manages its own watchdog; thus, virtual-process-crashing errors are localised to the next nearest point after the process, instead of being propagated further (to the CPU) and clogging up the metaphorical pipes. Currently, there is only one active VPS so this move does not change much, but this will pay off when machine starts spawning multiple schedulers. FUTURE According to release schedule, next release should introduce multiple FFI schedulers. This will provide speed improvements due to greater parallelism on machines with more cores to utilise (as each scheduler will run in parallel on its own thread). Machines with one and two cores will not notice any change; and there are no plans to "overschedule", i.e. to spawn more threads than there are real cores available to utilise (if the hardware provides only two cores, only one FFI scheduler will be spawned).
Release 0.8.1 NEWS With 238 commits since 0.8.0, this release brings in another set of improvements the code of the machine; stronger compile-time checks, improved entry function generation in assembler, different flavours of main function and fixed FFI scheduling. STRONGER COMPILE-TIME CHECKS Assembler is now able to better verify function calls; a whole new verification stage was introduced into the assembler's code which is able to detect arity mismatches between function calls and declarations, frames with gaps (i.e. when not all parameter slots in the frame are filled), and double-passing parameters. Also, all errors are now enabled by default - this change was suggested by Harald Eilertsen (https://github.com/snake66), who also provided valuable insight on the topic of multithreading in C++, which is highly relevant to another part of this release. FIXED FFI SCHEDULING Viua VM spawns two threads when it launches: one runs virtual processes of the machine, and the other runs functions executed directly on the host CPU (the foreign functions). Release 0.8.0 shipped with a race-condition which sometimes could make the machine hang after the FFI call was scheduled. IMPROVED EXPIRED POINTER HANDLING When an expired pointer is encountered, its type may be safely inspected and its string representation may be safely generated. In previous releases this would throw an exception. This is particularly important when an expired pointer is present in stack trace after a virtual process crash as it may be printed without any problems. FATAL VM EXCEPTION WHEN WATCHDOG PROCESS EXITS Watchdog process cannot finish execution normally. If the machine detects it did, it throws a fatal exception and shuts itself down. IMPROVED SAMPLE CODE Sample code must have been updated for current release due to stricter compiler checks. As a bonus, old non-testing code has been cleaned up and rewritten in up-to-date VM assembly. FUTURE Next release will be mostly a refactoring one, so should arrive in less than a month if everything goes smooth and as planned (fingers crossed). It will bring a new virtual process scheduler, extracted from the main CPU code; the CPU will become more static - only providing information about entry points locations, type hierarchy status and routing calls between virtual and FFI schedulers.
Release 0.8.0 NEWS This is another release introducing substantial changes to machine's code and capabilities. CLOSURES Viua 0.8.0 sports much better support for closures than previous releases by giving programmers way to choose which parts of the environment (which values) are captured and how they are captured (by copy, by move, or by reference). The cost for this feature is lost backwards compatibility. VIRTUAL PROCESSES Threads are renamed to processes. A unit of execution in Viua VM is a "virtual process". Each virtual process is separated from every other one. Processes communicate only by means of message packages. That does not sound very much like a definition of a "thread". INTER-FUNCTION TAILCALLS Viua 0.8.0 supports inter-function tailcalls. Looping can be now be elegantly implemented using recursion instead of assembler-like `branch` and `jump` instructions. IMMEDIATE COPY ON PASS-BY-COPY Starting with this version, machine immediately copies parameters passed by copy. FOREIGN CALL OFFLOADING Before this version FFI calls were blocking whole machine, i.e. if one process called a foreign function, no other process could run until foreign function returned. This was caused by the fact that machine was single-threaded. Viua 0.8.0 spawns two threads: one for running native Viua code, and a second one for running foreign code. The process that called a foreign function is suspended until the foreign function returns but all other VM processes are not. STANDARD LIBRARY IMPROVEMENTS Some additions and improvements were made to standard library modules `std::misc`, `std::vector` and `std::functional`.
Release 0.7.0 NEWS This release introduces some substantial changes to the machine, and that is the reason for jumping from 0.6.1 straight to 0.7.0. Some backwards incompatible changes: - `throw` no longer leaves thrown objects in their registers, thrown objects are moved out of their stack frame, - `free` instruction was renamed to `delete`, Some more backwards incompatible changes, this time due to introduction of pointers. Using pointers if cheaper than using references (by a large margin) so a few standard library functions were rewritten to accept pointers where they used to accept references. Pointers are cheaper because they do not escape machine's basic memory management model. Then, a fix. If a detached thread generates uncaught exception the machine will print out trace of the correct stack. Previously always the stack of `main()` function was printed. One feature implemented in this release is a "pass-by-move", a very efficient method of passing parameters to functions. Last, but not least, come the enhancements in multithreading. First, return values can be extracted from threads when they are joined. Second, threads can be suspended and woken up; this is important because the CPU can easily skip suspended threads without looking at their state (other than the `suspended` status). Last feature introduced by the 0.7.0 is also one of the more important ones. Machine provides a way for the programmers to specify a function to be run as a "watchdog thread". If a thread dies when a watchdog is active the machine passes a message to the watchdog telling it what thread died, and why. This enables the watchdog to restart the deceased thread if necessary. If the watchdog itself dies it is restarted automatically by the machine. In my opinion, this is a great win for the reliability of software written to run on the machine. Instead of worrying about every edge case the programmer can let the software crash, log the error, restart the part of the program that failed, and carry on like if nothing at all has happened. As always, there are also numerours small improvements and bug fixes.
Release 0.6.1 NEWS This release introduces basic multithreading support to the machine. The threads are green, i.e. they are implemented inside the machine and are not system threads. During the first (very simple and probably useless) benchmark, the machine has been switching between 510,000+ threads. Each thread had more than 1,000,000 opcodes to execute. Next releases will focus on expanding theading functionality. Features to implement include message passing, return value extraction, and priority modifications.
Release 0.6.0 550 commits since last relase. NEWS This relase amounts to two months and a week of work. The immediate jump from v0.5.0 to v0.6.0 is justified, because not only the time period between releases is unusually long (in terms of Viua relases) but also because the change scope is unsusually broad. To sum up the relase in six words: > Refactoring, bugfixes, features. In large quantities. CHANGELOG 1) Added `Prototype` class for user-defined types User-defined types can be created with `class`, `derive`, `attach` and `register` instructions. Instances of these types are created with `new` instruction. Read sample code and documentation to learn more. 2) Better error messages after crash Long traces do not prevent users from discovering source of the crash - exception details are printed *after* the stack trace. 3) Documented and tested function fall-through behaviour Functions support falling-through to the next one. This could lead to endless loops should the "next" function call the "falling" one. Such bugs are prevented by limiting stack size to 8192 frames. The limit may be changed at compile time. An `Exception` is thrown when the limit is exceeded. 4) EXCALL opcode is no more The `excall` instruction has been removed from the bytecode definition, and `call` instructions should be used instead. This provides a uniform call interface for both native and foreign functions. 5) EXIMPORT opcode is no more The `eximport` instruction has been renamed to `import`. 6) `VIUAPATH` and `VIUAAFTERPATH` used by `link` instruction It is now possible to put Viua libraries and tell the machine where to look for them. 7) Added `VIUAPRELINK` and `VIUAPREIMPORT` environment variables These environment variables may be used to tell the machine to always import and link specified modules. Note that the machine must be able to find requested modules on its `PATH` variables. 8) Support for relative jumps in assembler Users can now write `jump -9` or `branch 4 +8 +1` to tell the assembler to generate jumps relative to the current instruction index. Note: jumps are recalculated as absolute in final executables. 9) Merged pull-requests fixing build issues from @saper A big "Thank you" for @saper for sending pull requests that fixed some build issues on FreeBSD. 10) More language constructs added to assembler For example, instruction counting using `[ (instr0) (instr1) ... (instrN) ]` syntax or instruction nesting using `instr2 (instr1 (instr0 op0))` syntax. Again, read sample code and documentation to learn more. 11) Dynamic dispatch using `msg` instructions Viua supports dynamic dispatch of methods and all types that have registered at least one method. 12) User-defined types written in C++ Viua provides support for nearly-direct usage of classes defined in C++ from machine's bytecode. This is done using black magic, pointers to members and virtual functions. The most important thing to know is that a C++ class must derive from `Type` and provide methods with correct signatures to be loadable into machine's typesystem. Read sample code and documentation to learn more. 13) Better implementation of references References should not leak memory now. 14) Altered catch-throw mechanism instructions Renamed `try` instruction to `enter`, and `tryframe` to `try` in order to have code read in a more intuitive way. 15) Added `std::io` module Very basic I/O capabilities. 16) Added `std::random` module Very basic random-related functions using standard Linux interfaces. 17) Bytecode size and jumps are encoded using `uint64_t` This makes the bytecode size bigger, but allows creating programs much bigger than 65 kilobytes.