#Faith

It's been a long road...
Getting from there to here.

The Fennel Advanced Interactive Test Helper.

To use Faith, create a test runner file which calls the run function with a list of module names. The modules should export functions whose names start with test- and which call the assertion functions in the faith module.

#Usage

The file faith.fnl can be used as a script or as a library:

$ fennel faith.fnl --tests test.start test.optimize # as a script

Using it as a library requires writing a loader script.

(local t (require :faith))

(local default-modules [:test.one-thing :test.other :test.third])

(t.run (if (= 0 (length arg)) default-modules arg))

You can run the t.run function from the REPL as well after reloading your test modules.

Tests are just functions in test modules which call assertion functions.

(local t (require :faith))

;; A setup-all function can load files from disk; connect to a server, etc
(fn setup-all []
  (with-open [f (io.open "test/data.txt")]
    (let [contents (f:read :*all)]
      ;; whatever the setup-all function returns will be passed as
      ;; an argument to every test function.
      {: contents :length (length contents) :status "initialized"})))

(fn test-add [_data]
  (t.= 2 (+ 1 1))
  ;; assert= tests for deep equality, not just table identity
  (t.= [1 99] [1 (+ 45 44)]))

(fn test-check [data]
  (t.= 0 (- 2 2)))

{: setup-all
 : test-add
 : test-check}

You can provide setup and teardown functions to run before and after each test, as well as, setup-all and teardown-all to run before and after each test module. Whatever values setup-all returns are passed into each of the test functions and also the teardown-all function.

Note that in a language like Fennel that has tail-call optimization, it's possible for an assertion on the last line of a function to fail in a way that obscures the line number of the failure. If this is a concern, you can put a nil or (values) on the last line of each test function.

This is an issue for any test framework; it is not specific to Faith.

For assertion failures where the data being compared is multiple lines, Faith will shell out to diff to display the differences between the expected and actual. To customize the diff output, you can export FAITH_DIFF="diff -y %s %s" for example, to get a side-by-side diff.

Faith supports PUC Lua ("regular" Lua) 5.1 to 5.4 as well as LuaJIT.

If the luasocket or luaposix libraries are installed, Faith will use them to calculate the total runtime of the test run. Without these libraries, Lua is unable to track elapsed time with granularity of under a second, so approximate times will be displayed instead.

#Assertions

All assertions take an optional message string as their last argument.

• is: checks truthiness (anything other than false or nil)

All these assertions take the expected value first, then the actual.

• =: equality checks using deep equality (works on tables too)
• not=: checks the opposite of =
• <: checks that the arguments are in increasing order
• <=: checks that the arguments are in increasing or equal order
• almost=: is the number given within a tolerance of expected?
• identical: checks that its two table arguments are rawequal to each other
• match: checks that the actual string matches an expected pattern
• not-match: checks the opposite
• error: checks that a function errors out and the error matches an expected pattern

You can call skip in a test to indicate that the test is incomplete without triggering a failure.

Note that in normal Fennel, regular = checks for identity on tables; it does not perform "deep equality", which can often trip people up. Faith's = assertion ensures that all the values in a table are equal.

(faith.is (= [1 2 3] [1 2 3])) ; fails using Fennel's =
(faith.= [1 2 3] [1 2 3]) ; passes using Faith's =
(faith.identical [1 2 3] [1 2 3]) ; fails
(let [t [1 2 3]] (faith.is (= t t))) ; passes
(let [t [1 2 3]] (faith.identical t t)) ; passes

#Configuration

Faith features several configuration options that can be used to customize functionality.

To configure your test execution, pass a configuration table to faith.run:

(local t (require :faith))
(local tests [:module1 :module2])

(local all-options
  {:diff-cmd "diff -u --color=always %s %s"
   :hooks {:begin (fn [report module-names] ...)
           :done (fn [] ...)
           :begin-module (fn [] ...)
           :end-module (fn [] ...)
           :begin-test (fn [] ...)
           :end-test (fn [] ...)}

(t.run tests all-options)

#:diff-cmd

A string that contains a shell command for the purpose of displaying the difference between expected data and actual data for tests. When tests fail, this string is interpolated by Lua's string.format function, passing in the expected data first, then the actual data.

NOTE: This overrides all diff tools established by environment variables, e.g. FAITH_DIFF, DIFF, etc.

{:diff-cmd "your-diff-command -expected %s -actual %s"}

#Hooks

Faith's primary functionality can be extended with hooks, i.e. functions that are called throughout faith.run.

Some hooks receive the same data structures throughout, namely:

• report:

  {:module-name "<name of relevant module>"
   :started-at "<timestamp>"
   :ended-at "<timestamp>"
   :results []}
  

• result:

  {:char "<character>" ;; . - Success
                       ;; E - Error
                       ;; F - Failure
                       ;; S - Skip
   :msg "<result context>"
   :reason "<if failed, failure reason>"
   :tostring (fn [result name] ...)
   :type "<type of result>" ;; :err  - Error
                            ;; :fail - Failure
                            ;; :pass - Success
                            ;; :skip - Skip
   :where "<if failed, line of code where failure occurred>"}
  

Some hooks have default values that generate output for Faith, so replacing the following hooks will override that functionality:

{:begin-module
 :done
 :end-test}

#:begin

:begin is called before running all tests.

Arguments:

• report - Table containing metadata about the tests.
• module-names - A sequential table of the names of every module that will be run.

Example:

(local begin-hook [report _module-names]
  (print (.. (string.format "Running module %q. Start-time: %s | Results: "
                            report.module-name report.started-at)
             (fennel.view report.results))))

#:done

:done is called after all tests have completed.

NOTE: This overrides default functionality that prints test results at the end.

Arguments:

• report - Table containing metadata about the tests.

Example:

(local done-hook [report]
  (print (.. "You did it! Here's your report: " (fennel.view report))))

#:begin-module

:begin-module is called before running all tests for a module.

NOTE: This overrides default functionality that prints module information prior to running tests.

Arguments:

• report - Table containing metadata about the tests.
• tests - Table of test functions returned from the module itself.

Example:

(local begin-module-hook [report tests]
  (print (string.format "\nStarting module %s with %d test(s)"
                        report.module-name
                        (accumulate [count 0 _ (pairs tests)] (+ count 1)))))

#:end-module

:end-module is called after all tests have been run for a module.

Arguments:

• report - Table containing metadata about the tests.

Example:

(local end-module-hook [report]
  (print (string.format "\nCompleted module %q" report.module-name)))

#:begin-test

:begin-test is called immediately before running a test.

Arguments:

• name - String containing the name of the next test function.

Example:

(local begin-test-hook [name]
  (print (string.format "\n* Running test function %q" name)))

#:end-test

:end-test is called immediately after running a test.

NOTE: This overrides default functionality that prints the characters of every test result in the report.

Arguments:

• name - String containing the name of the test function that was just invoked.
• result - A table containing metadata about the outcome of the test function.
• total-count - Total number of assertions that were run.

Example:

(local end-test [name result total-count]
  (print (string.format "\n* Running test function %q" name)))

#Developing Faith

Run make testall to run the full suite against all supported Lua versions. Currently the Makefile assumes that there is a checkout of Fennel itself in the same directory as your checkout of Faith, but you can override this with, e.g., make test FENNEL=/usr/local/bin/fennel.

Discussion happens on the Fennel mailing list and on the #fennel channel on Libera chat and matrix.org.

#TODO

• [ ] detailed/colored diffs for failed equality assertions?

#License

Faith was based on lunatest originally but has evolved significantly since its beginning.

Copyright © 2009-2024 Scott Vokes, Phil Hagelberg, and contributors

All files in this repository released under the MIT License.