~technomancy/fennel

526e5e61649266c11d677bf00d619b5bcfaea72e — Phil Hagelberg a month ago 452a1ae + e84a8d4
Merge branch 'plugin2' into main

# Conflicts:
#	changelog.md
#	src/fennel/specials.fnl
#	src/launcher.fnl
M api.md => api.md +38 -0
@@ 248,4 248,42 @@ that environment, in a way that's portable across any Lua 5.1+ version.
local f = fennel.loadCode(luaCode, { x = y }, "myfile.lua")
```

## Plugins

Fennel's plugin system is extremely experimental and exposes internals of
the compiler in ways that no other part of the compiler does. It should be
considered unstable; changes to the compiler in future versions are likely
to break plugins, and each plugin should only be assumed to work with
specific versions of the compiler that they're tested against. The
backwards-compatibility guarantees of the rest of Fennel **do not apply** to
plugins.

Compiler plugins allow the functionality of the compiler to be extended in
various ways. A plugin is a module containing various functions in fields
named after different compiler extension points. When the compiler hits an
extension point, it will call each plugin's function for that extension
point, if provided, with various arguments; usually the AST in question and
the scope table.

* `symbol-to-expression`
* `call`
* `do`
* `fn`
* `destructure`

The `destructure` extension point is different because instead of just
taking `ast` and `scope` it takes a `from` which is the AST for the value
being destructured and a `to` AST which is the AST for the form being
destructured to. This is most commonly a symbol but can be a list or a table.

The `scope` argument is a table containing all the compiler's information
about the current scope. Most of the tables here look up values in their
parent scopes if they do not contain a key.

Plugins are activated by passing the `--plugin` argument on the command line,
which should be a path to a Fennel file containing a module that has some of
the functions listed above. If you're using the compiler programmatically,
you can include a `:plugins` table in the `options` table to most compiler
entry point functions.

[1]: https://github.com/rxi/lume#lumehotswapmodname

M changelog.md => changelog.md +26 -2
@@ 1,7 1,13 @@
# Summary of user-visible changes

## 0.5.1 / ???
## 0.6.0 / ???

This release introduces the plugin system as well as starting to
sandbox the compiler environment for safer code loading. Nothing is
blocked yet, but it emits warnings when macros use functionality that
is not considered safe; future versions will prevent this.

* Add plugin system.
* Sandbox compiler environment and emit a warning when it leaks.
* Fix a bug where repls would fail when provided with an overridden env.
* Expose `list?` and `sym?` in compiler API.


@@ 11,15 17,20 @@

## 0.5.0 / 2020-08-08

This release features a version of the Fennel compiler that is
self-hosted and written entirely in Fennel!

* Fix a bug where lambdas with no body would return true instead of nil.
* Fix a bug where global mangling would break when used with an environment.
* Fix a bug where globals tracking would lose track of allowed list.
* Fix a bug where top-level expressions in `include` would get skipped.
* The "fennelfriend" module is now incorporated into the compiler, not separate.
* The Fennel compiler is now self-hosted and written entirely in Fennel!

## 0.4.2 / 2020-07-11

This release mostly includes small bug fixes but also adds the
`with-open` macro for automating closing file handles, etc.

* Fix a bug where multiple `include` calls would splice locals incorrectly
* Support varargs in hashfn with `$...` (#298)
* Add `with-open` macro for auto-closing file handles (#295)


@@ 32,6 43,9 @@

## 0.4.1 / 2020-05-25

This release mostly includes small bug fixes, but also introduces a very
experimental command for compiling standalone executables.

* Experimental `--compile-binary` command (#281)
* Support shebang in all contexts, not just dofile
* Pinpoint source in compile errors even when loading from a string


@@ 41,6 55,12 @@

## 0.4.0 / 2020-05-12

This release adds support for Lua 5.3's bitwise operators as well as a
new way of importing macro modules. It also adds `pick-values` and
`pick-args` for a little more flexibility around function args and
return values. The compiler now tries to emit friendlier errors that
suggest fixes for problems.

* Add `import-macros` for more flexible macro module loading (#269)
* Ensure deterministic compiler output (#257)
* Add bit-wise operators `rshift`, `lshift`, `bor`, `band`, `bnot`, and `bxor`


@@ 55,6 75,8 @@

## 0.3.2 / 2020-01-14

This release mostly contains small bug fixes.

* Fix a bug where `include` could not be nested without repetition (#214)
* Fix a bug where globals checking would mistakenly flag locals (#213)
* Fix a bug that would cause incorrect filenames in error messages (#208)


@@ 63,6 85,8 @@

## 0.3.1 / 2019-12-17

This release mostly contains small bug fixes.

* Look for init file for repl in XDG config dirs as well as ~/.fennelrc (#193)
* Add support for `--load FILE` argument to command-line launcher (#193)
* Fix `each` to work with raw iterator values (#201)

M reference.md => reference.md +4 -1
@@ 993,7 993,7 @@ Example:

This prints all the functions available in compiler scope.

### Compiler API
### Compiler Environment

Inside `eval-compiler`, `macros`, or `macro` blocks, as well as
`import-macros` modules, these functions are visible to your code.


@@ 1022,6 1022,9 @@ and a metatable that the compiler uses to distinguish them. You can use
* `varg?` - is this a `...` symbol which indicates var args?
* `multi-sym?` - a multi-sym is a dotted symbol which refers to a table's field

* `assert-compile` - works like `assert` but takes a list/symbol as its third
  argument in order to provide pinpointed error messages.

These functions can be used from within macros only, not from any
`eval-compiler` call:


M src/fennel.fnl => src/fennel.fnl +6 -2
@@ 35,7 35,11 @@
        _ (when (and (= opts.allowedGlobals nil)
                     (not (getmetatable opts.env)))
            (set opts.allowedGlobals (specials.current-global-names opts.env)))
        env (and opts.env (specials.wrap-env opts.env))
        ;; This is ... not great. Should we expose make-compiler-env in the API?
        env (if (= opts.env :_COMPILER)
                (specials.wrap-env (specials.make-compiler-env
                                    nil compiler.scopes.compiler {}))
                (and opts.env (specials.wrap-env opts.env)))
        lua-source (compiler.compile-string str opts)
        loader (specials.load-code lua-source env
                                  (if opts.filename


@@ 91,7 95,7 @@

            :eval eval
            :dofile dofile*
            :version "0.5.1-dev"
            :version "0.6.0-dev"

            :repl repl})


M src/fennel/compiler.fnl => src/fennel/compiler.fnl +3 -0
@@ 187,6 187,7 @@ rather than generating new one."
(fn symbol-to-expression [symbol scope reference?]
  "Convert symbol to Lua code. Will only work for local symbols
if they have already been declared via declare-local"
  (utils.hook :symbol-to-expression symbol scope reference?)
  (let [name (. symbol 1)
        multi-sym-parts (utils.multi-sym? name)
        name (or (hashfn-arg-name name multi-sym-parts scope) name)]


@@ 412,6 413,7 @@ if opts contains the nval option."
        exprs)))

(fn compile-call [ast scope parent opts compile1]
  (utils.hook :call ast scope)
  (let [len (# ast)
        first (. ast 1)
        multi-sym-parts (utils.multi-sym? first)


@@ 678,6 680,7 @@ which we have to do if we don't know."
        {:returned true}))

    (let [ret (destructure1 to nil ast true)]
      (utils.hook :destructure from to scope)
      (apply-manglings scope new-manglings ast)
      ret)))


M src/fennel/specials.fnl => src/fennel/specials.fnl +3 -0
@@ 120,6 120,7 @@ By default, start is 2."
              fargs (if scope.vararg "..." "")]
          (compiler.emit parent (string.format "local function %s(%s)"
                                               fname fargs) ast)
          (utils.hook :do ast sub-scope)
          (compile-body nil true
                        (utils.expr (.. fname "(" fargs ")") :statement))))))



@@ 227,6 228,7 @@ the number of expected arguments."
      (compiler.emit parent f-chunk ast)
      (compiler.emit parent "end" ast)
      (set-fn-metadata arg-list docstring parent fn-name))
    (utils.hook :fn ast f-scope)
    (utils.expr fn-name "sym")))

(doc-special "fn" ["name?" "args" "docstring?" "..."]


@@ 847,6 849,7 @@ Method name doesn't have to be known at compile-time; if it is, use
                 :_VARARG (utils.varg)

                 :unpack unpack ; compatibilty alias
                 :assert-compile compiler.assert

                 ;; AST functions
                 :list utils.list

M src/fennel/utils.fnl => src/fennel/utils.fnl +7 -1
@@ 236,6 236,12 @@ has options calls down into compile."
    (set (root.chunk root.scope root.options root.reset)
         (values chunk scope options reset))))

(fn hook [event ...]
  (when (and root.options root.options.plugins)
    (each [_ plugin (ipairs root.options.plugins)]
      (match (. plugin event)
        f (f ...)))))

{;; general table functions
 : allpairs : stablepairs : copy : kvmap : map : walk-tree : member?



@@ 244,6 250,6 @@ has options calls down into compile."
 : expr? : list? : multi-sym? : sequence? : sym? : table? : varg? : quoted?

 ;; other
 : valid-lua-identifier? : lua-keywords
 : valid-lua-identifier? : lua-keywords : hook
 : propagate-options : root : debug-on?
 :path (table.concat ["./?.fnl" "./?/init.fnl" (getenv "FENNEL_PATH")] ";")}

M src/launcher.fnl => src/launcher.fnl +11 -3
@@ 9,7 9,7 @@ Usage: fennel [FLAG] [FILE]
Run fennel, a lisp programming language for the Lua runtime.

  --repl                  : Command to launch an interactive repl session
  --compile FILES         : Command to compile files and write Lua to stdout
  --compile FILES         : Command to AOT compile files, writing Lua to stdout
  --eval SOURCE (-e)      : Command to evaluate source code and print the result

  --no-searcher           : Skip installing package.searchers entry


@@ 25,6 25,7 @@ Run fennel, a lisp programming language for the Lua runtime.
  --load FILE (-l)        : Load the specified FILE before executing the command
  --lua LUA_EXE           : Run in a child process with LUA_EXE (experimental)
  --no-fennelrc           : Skip loading ~/.fennelrc when launching repl
  --plugin FILE           : Activate the compiler plugin in FILE
  --compile-binary FILE
      OUT LUA_LIB LUA_DIR : Compile FILE to standalone binary OUT (experimental)
  --compile-binary --help : Display further help for compiling binaries


@@ 33,6 34,9 @@ Run fennel, a lisp programming language for the Lua runtime.
  --help (-h)             : Display this text
  --version (-v)          : Show version

  Globals are not checked when doing AOT (ahead-of-time) compilation unless
  the --globals-only flag is provided.

  Metadata is typically considered a development feature and is not recommended
  for production. It is used for docstrings and enabled by default in the REPL.



@@ 41,7 45,7 @@ Run fennel, a lisp programming language for the Lua runtime.

  If ~/.fennelrc exists, loads it before launching a repl.")

(local options [])
(local options {:plugins []})

(fn dosafely [f ...]
  (let [args [...]


@@ 109,7 113,11 @@ Run fennel, a lisp programming language for the Lua runtime.
    "--no-metadata" (do (set options.useMetadata false)
                        (table.remove arg i))
    "--no-compiler-sandbox" (do (set options.compiler-env _G)
                                (table.remove arg i))))
                                (table.remove arg i))
    "--plugin" (let [plugin (fennel.dofile (table.remove arg (+ i 1))
                                           {:env :_COMPILER})]
                 (table.insert options.plugins 1 plugin)
                 (table.remove arg i))))

(when (not options.no_searcher)
  (let [opts []]

A src/linter.fnl => src/linter.fnl +74 -0
@@ 0,0 1,74 @@
;; An example of some possible linters using Fennel's --plugin option.

;; The linters here can only function on static module use. For instance, this
;; code can be checked because they use static field access on a local directly
;; bound to a require call:

;; (local m (require :mymodule))
;; (print m.field) ; fails if mymodule lacks a :field field
;; (print (m.function 1 2 3)) ; fails unless mymodule.function takes 3 args

;; However, these cannot:

;; (local m (do (require :mymodule)) ; m is not directly bound
;; (print (. m field)) ; not a static field reference
;; (let [f m.function]
;;   (print (f 1 2 3)) ; intermediate local, not a static field call on m

;; Still, pretty neat, huh?

;; This file is provided as an example and is not part of Fennel's public API.

(fn save-require-meta [from to scope]
  "When destructuring, save module name if local is bound to a `require' call.
Doesn't do any linting on its own; just saves the data for other linters."
  (when (and (sym? to) (not (multi-sym? to)) (list? from)
             (sym? (. from 1)) (= :require (tostring (. from 1)))
             (= :string (type (. from 2))))
    (let [meta (. scope.symmeta (tostring to))]
      (set meta.required (tostring (. from 2))))))

(fn check-module-fields [symbol scope]
  "When referring to a field in a local that's a module, make sure it exists."
  (let [[module-local field] (or (multi-sym? symbol) [])
        module-name (and module-local (. scope.symmeta
                                         (tostring module-local) :required))
        module (and module-name (require module-name))]
    (assert-compile (or (= module nil) (not= (. module field) nil))
                    (string.format "Missing field %s in module %s"
                                   field module-name) symbol)))

(fn arity-check? [module] (-?> module getmetatable (. :arity-check?)))

(fn arity-check-call [[f & args] scope]
  "Perform static arity checks on static function calls in a module."
  (let [arity (# args)
        last-arg (. args arity)
        [f-local field] (or (multi-sym? f) [])
        module-name (and f-local (. scope.symmeta (tostring f-local) :required))
        module (and module-name (require module-name))]
    (when (and (arity-check? module) debug debug.getinfo
               (not (varg? last-arg)) (not (list? last-arg)))
      (assert-compile (= (type (. module field)) :function)
                      (string.format "Missing function %s in module %s"
                                     field module-name) f)
      (match (debug.getinfo (. module field))
        {: nparams :what "Lua" :isvararg true}
        (assert-compile (<= nparams (# args))
                        (: "Called %s.%s with %s arguments, expected %s+"
                           :format f-local field arity nparams) f)
        {: nparams :what "Lua" :isvararg false}
        (assert-compile (= nparams (# args))
                        (: "Called %s.%s with %s arguments, expected %s"
                           :format f-local field arity nparams) f)))))

(fn check-unused [ast scope]
  (each [symname (pairs scope.symmeta)]
    (assert-compile (or (. scope.symmeta symname :used) (symname:find "^_"))
                    (string.format "unused local %s" symname) ast)))

{:destructure save-require-meta
 :symbol-to-expression check-module-fields
 :call arity-check-call
 :fn check-unused
 :do check-unused}