761c5c60173ab11e81f2667b5176e9b4d3d58648 — Phil Hagelberg 14 days ago 64aeabb
Add support for fennel.macro-searchers.

We now have a fennel.macro-searchers table which functions as a
compile-time equivalent to the package.searchers table; anyone can
insert their own searcher functions to change how macros can be found.

The main thing I don't like about this is that it exposes our little
internal trick of passing :env :_COMPILER in the options table to load
things in compiler scope. We have supported this for the environment
for internal reasons for a while, but now we both expand it (to also
affect :scope) and document it as part of the interface, in order to
simplify the implementation of the default macro searcher function.

I don't love that trick; it was originally added in order to allow us
to load plugins in compiler scope without reaching into the guts of
the compiler module, but it feels like a bit of a hack.

We could maybe make it cleaner by exposing make-compiler-scope and
make-compiler-env as part of the public API, but I'm not sure it's
necessary. If this shorthand gets the same job done as adding two more
functions to our already-big API, maybe that's fine. Or we can add the
shorthand now and introduce the other functions later if we determine
they are useful.
5 files changed, 63 insertions(+), 22 deletions(-)

M api.md
M changelog.md
M reference.md
M src/fennel.fnl
M src/fennel/specials.fnl
M api.md => api.md +5 -0
@@ 29,6 29,11 @@ usually accept these fields:
  disable sandboxing.
* `unfriendly`: disable friendly compiler/parser error messages.

You can pass the string `"_COMPILER"` as the value for `env`; it will
cause the code to be run/compiled in a context which has all
compiler-scoped values available. This can be useful for macro modules
or compiler plugins.

Note that only the `fennel` module is part of the public API. The
other modules (`fennel.utils`, `fennel.compiler`, etc) should be
considered compiler internals subject to change.

M changelog.md => changelog.md +1 -0
@@ 2,6 2,7 @@

## 0.9.0 / ???

* Add `macro-searchers` table for finding macros similarly to `package.searchers`
* Support `&as` inside pattern matches
* Include stack trace for errors during macroexpansion
* The `sym` function in compile scope now takes a source table second argument

M reference.md => reference.md +31 -1
@@ 1049,7 1049,37 @@ inside compiler scope which macros run in.

The `require-macros` form is like `import-macros`, except it does not
give you any control over the naming of the macros being
imported. Consider using `import-macros` instead of `require-macros`.
imported. It is strongly recommended to use `import-macros` instead.

### Macro module searching

By default, Fennel will search for macro modules using the same logic
it uses to search for normal runtime modules: by walking thru entries
on `fennel.path` and checking the filesystem for matches. However, in
some cases this might not be suitable, for instance if your Fennel
program is packaged in some kind of archive file and the modules do
not exist as distinct files on disk.

To support this case you can add your own searcher function to the
`fennel.macro-searchers` table. For example, assuming `find-in-archive`
is a function which can look up strings from the archive given a path:

(local fennel (require :fennel))

(fn my-searcher [module-name]
  (let [filename (.. "src/" module-name ".fnl")]
    (match (find-in-archive filename)
      code (values (partial fennel.eval code {:env :_COMPILER})

(table.insert fennel.macro-searchers my-searcher)

The searcher function should take a module name as a string and return
two values if it can find the macro module: a loader function which will
return the macro table when called, and an optional filename. The
loader function will receive the module name and the filename as arguments.

### `macros` define several macros

M src/fennel.fnl => src/fennel.fnl +3 -0
@@ 50,6 50,8 @@
    ;; to provide targeted error messages.
    (when (and (not opts.filename) (not opts.source))
      (set opts.source str))
    (when (= opts.env :_COMPILER)
      (set opts.scope (compiler.make-scope compiler.scopes.compiler)))

(fn eval [str options ...]

@@ 98,6 100,7 @@
            :gensym compiler.gensym
            :load-code specials.load-code
            :macro-loaded specials.macro-loaded
            :macro-searchers specials.macro-searchers
            :search-module specials.search-module
            :make-searcher specials.make-searcher
            :makeSearcher specials.make-searcher

M src/fennel/specials.fnl => src/fennel/specials.fnl +23 -21
@@ 1044,17 1044,18 @@ table.insert(package.loaders, fennel.searcher)"
      (table.insert allowed k))

(fn compiler-env-domodule [modname env ?ast ?scope]
  (let [filename (compiler.assert (search-module modname)
                                  (.. modname " module not found.") ?ast)
        globals (macro-globals env (current-global-names))
        scope (or ?scope (compiler.make-scope compiler.scopes.compiler))]
    (utils.fennel-module.dofile filename
                                {:allowedGlobals globals
                                 :useMetadata utils.root.options.useMetadata
                                 : env
                                 : scope}
                                modname filename)))
(fn default-macro-searcher [module-name]
  (match (search-module module-name)
    filename (values (partial utils.fennel-module.dofile filename
                              {:env :_COMPILER}) filename)))

(local macro-searchers [default-macro-searcher])

(fn search-macro-module [modname n]
  (match (. macro-searchers n)
    f (match (f modname)
        (loader ?filename) (values loader ?filename)
        _ (search-macro-module modname (+ n 1)))))

;; This is the compile-env equivalent of package.loaded. It's used by
;; require-macros and import-macros, but also by require when used from within

@@ 1074,11 1075,10 @@ table.insert(package.loaders, fennel.searcher)"
It ensures that compile-scoped modules are loaded differently from regular
modules in the compiler environment."
                    (or (. macro-loaded modname) (metadata-only-fennel modname)
                        (let [scope (compiler.make-scope compiler.scopes.compiler)
                              env (make-compiler-env nil scope nil)
                              mod (compiler-env-domodule modname env nil scope)]
                          (tset macro-loaded modname mod)
                        (let [(loader filename) (search-macro-module modname 1)]
                          (compiler.assert loader (.. modname " module not found."))
                          (tset macro-loaded modname (loader modname filename))
                          (. macro-loaded modname)))))

(fn add-macros [macros* ast scope]
  (compiler.assert (utils.table? macros*) "expected macros to be table" ast)

@@ 1092,14 1092,15 @@ modules in the compiler environment."
                   (or real-ast ast)) ; real-ast comes from import-macros
  ;; don't require modname to be string literal; it just needs to compile to one
  (let [filename (or (. ast 2 :filename) ast.filename)
        modname-code (compiler.compile (. ast 2))
        modname ((load-code modname-code nil filename) utils.root.options.module-name
        modname-chunk (load-code (compiler.compile (. ast 2)) nil filename)
        modname (modname-chunk utils.root.options.module-name filename)]
    (compiler.assert (= (type modname) :string)
                     "module name must compile to string" (or real-ast ast))
    (when (not (. macro-loaded modname))
      (let [env (make-compiler-env ast scope parent)]
        (tset macro-loaded modname (compiler-env-domodule modname env ast))))
      (let [env (make-compiler-env ast scope parent)
            (loader filename) (search-macro-module modname 1)]
        (compiler.assert loader (.. modname " module not found.") ast)
        (tset macro-loaded modname (loader modname filename))))
    (add-macros (. macro-loaded modname) ast scope parent)))

(doc-special :require-macros [:macro-module-name]

@@ 1209,6 1210,7 @@ Lua output. The module must be a string literal and resolvable at compile time."
 : current-global-names
 : load-code
 : macro-loaded
 : macro-searchers
 : make-compiler-env
 : search-module
 : make-searcher