Historical note.
Cache loaded module.
Add a very basic test.
The name is provisional. To be clear, this is not about the recursive lexical binding form.
An isolated content-addressable package loading system.
Requires Fennel version 0.5.0-dev.
To contribute, please send patches to myself or the
Fennel mailing list. Discussion
happens largely on the #fennel channel on Freenode or Matrix.
This repository represents my attempts to build a content-addressable package system for Fennel/Lua. It's not particularly sophisticated, but for pure Fennel/Lua code, it doesn't need to be.
The problem is that extending the system to cover modules implemented in C rather than Fennel/Lua changes it from an easy, fun side project to a sisyphean nightmare, because C code tends to be packaged and built in ways that vary both wildly and pointlessly. Without the ability to cover code that uses C code, this system is of limited value.
The process starts with the importer. When you want to use a module,
it first has to be imported into the store along with all the modules
it requires. We analyze the module to identify its dependencies by
using containment to replace require with a shim which records the
dependency modules. In the future we could do this by parsing the
source instead since not all requires happen at the top level; the
current approach means that importing only works on well-behaved modules.
Once we determine a module's dependency tree, we walk it down to the leaves and for each leaf module we checksum it. The checksum is based on several factors: the contents of the file, the checksum of letrec itself, and the versions of Fennel and Lua used. The leaf module is copied into the store using a key based on the checksum. Once the leaves are imported, we walk back down the tree, but for non-leaf modules we incorporate the checksums of their dependencies into their checksums. We also store metadata for each module indicating how the module names for each dependency map to checksums.
Rather than loading modules by name, you load them based on their key which ensures you're always getting the exact same thing you asked for.
After a module and its dependencies have been imported, it can be
loaded. We add a package searcher which knows how to look in the store
for modules. It loads it in a context which has require replaced
with a letrec-aware equivalent. This version of require looks at the
metadata tree that was generated during import, and whenever a module
needs to load one of its dependencies, it ensures that it always gets
the right one that was locked in at import time.
Right now we just use a local filesystem directory is the store, but it's easy to imagine replacing it with a networked store such as IPFS, git, DAT, etc.
Right now, the method we use to detect requires is unusual; it runs
the file in an isolated environment which records all calls to the
require function and aborts if an error is encountered. This means
it only works on modules whose dependencies are loaded at the top of
the file.
We don't even attempt to handle C modules at this time.
Copyright © 2020 Phil Hagelberg and contributors
Released under the same license as Fennel.