~jomco/proof-specs

Automates testing clojure.spec data generators 
Version 0.1.8-SNAPSHOT
Version 0.1.7
Always report problems as data.

refs

master
browse  log 
v0.1.7
release notes 

clone

read-only
https://git.sr.ht/~jomco/proof-specs
read/write
git@git.sr.ht:~jomco/proof-specs

You can also use your local clone with git send-email.

#Proof Specs

Automatic testing of clojure.spec data generators.

#Dependency coordinates

Clojars Project

#Documentation

API documentation is available inline and at cljdoc.

#RATIONALE

clojure.spec generators can fail for unexpected reasons. It requires expertise to see where a complex spec fails to generate.

A practical way to work around the problem is to build and test specs incrementally; the clojure.spec design encourages building nested specs from components; speccing collections independently of their components, building keysets (maps) out of individually specced keys allows testing spec compoments independently. It still takes dicipline to test incrementally and backtracing where a definition went wrong can take much longer than is desirable.

#RESOLUTION

Test component specs automatically using proof-specs. This adds the requirement that all specs registered in a particular set of namespaces must have a working generator. proof-spes will attempt to generate data for each spec in the selected namespaces and reports on every failure.

#CONSEQUENSES

Specs that are only used as return specs for functions must have a generator, even though they won't be used in any other test.

#NON-GOALS

proof-specs will not try to figure out the reason for a spec/generator not working. It will just list all failing generators and the given errors.

#USAGE

#Running from clojure code

(require '[nl.jomco.proof-specs :refer [proof-specs]])

(proof-specs
  :num-vals 10
  :verbose true
  :limit-ms 500
  :include [#"nl.jomco.*"])

Produces something like:

(proof-specs {:limit-ms 1000
                                             :num-vals 10
                                             :include #{#"nl\.jomco\.proof-specs-test"}})
{:problems
 #:nl.jomco.proof-specs-test{:non-spec
                             {:nl.jomco.proof-specs/message
                              "Unable to construct gen at: [] for: :nl.jomco.proof-specs-test/non-spec",
                              :nl.jomco.proof-specs/type
                              clojure.lang.ExceptionInfo,
                              :clojure.spec.alpha/path [],
                              :clojure.spec.alpha/form
                              :nl.jomco.proof-specs-test/non-spec,
                              :clojure.spec.alpha/failure :no-gen},
                             :missing-deps
                             #:nl.jomco.proof-specs{:message
                                                    "Unable to resolve spec: :nl.jomco.proof-specs-test/missing",
                                                    :type
                                                    java.lang.Exception},
                             :non-spec-dep
                             {:nl.jomco.proof-specs/message
                              "Unable to construct gen at: [:nl.jomco.proof-specs-test/non-spec] for: :nl.jomco.proof-specs-test/non-spec",
                              :nl.jomco.proof-specs/type
                              clojure.lang.ExceptionInfo,
                              :clojure.spec.alpha/path
                              [:nl.jomco.proof-specs-test/non-spec],
                              :clojure.spec.alpha/form
                              :nl.jomco.proof-specs-test/non-spec,
                              :clojure.spec.alpha/failure :no-gen}},
 :specs
 #{:nl.jomco.proof-specs-test/non-spec
   :nl.jomco.proof-specs-test/missing-deps
   :nl.jomco.proof-specs-test/non-spec-dep}}

The return value will not contain a :problems key when no problems were found.

#Running as a leiningen alias

Add a "proof-specs" alias to your project.clj - also make sure you include proof-specs as a development dependency.

...
  profiles {:dev {:dependencies [[nl.jomco/proof-specs "<VERSION>"]]}}
  ...
  :aliases {"proof-specs" ["run" "-m" "nl.jomco.proof-specs"
                           "--include" ".*jomco.*"
                           "--require" "nl.jomco.proof-specs,nl.jomco.blerk"]}
  ...

And run using:

lein proof-specs

Produces something like:

Problems generating data for 1 out of 12 specs:
:nl.jomco.blerk/yelp

#:nl.jomco.blerk{:yelp #error {
 :cause "Couldn't satisfy such-that predicate after 100 tries."
 :data {...}, :max-tries 100}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message "Couldn't satisfy such-that predicate after 100 tries."
   :data {...}, :max-tries 100}
   :at [...]}]
 :trace
 [...]}}

The exit status will be 0 when no problems were found, otherwise 1.

#Interpreting output

When there are multiple failing generators, a good strategy is to focus on and fix the simplest generators first, since that also fixes complex generators that only fail because of failing components.