9b32216f981848505cc60539e7a0c55efab390ab — Sam Whited 1 year, 9 months ago 804efe9 integration_design
design: adds a design doc for integration testing

See #42

Signed-off-by: Sam Whited <sam@samwhited.com>
1 files changed, 158 insertions(+), 0 deletions(-)

A design/42_integration_testing.md
A design/42_integration_testing.md => design/42_integration_testing.md +158 -0
@@ 0,0 1,158 @@
# Proposal: Create integration testing package

**Author(s):** Sam Whited  
**Last updated:** 2020-04-05  
**Status:** thinking  
**Discussion:** https://mellium.im/issue/42

## Abstract

An API is proposed for running and configuring XMPP servers for use in
integration tests.

## Background

The nature of an XMPP library requires integration tests.
Not only because a network protocol naturally needs external resources (like a
network and a server), but because the public Jabber network is built on
interoperability, and any XMPP library must be able to integrate with it.
Robust integration testing will require running various servers such as
[Prosody] and [Ejabberd] which will require different configuration,
certificates, and other resources for different tests.
To facilitate this a package should be written to create a temporary directory
with various config files and other resources and then run commands such as
`ejabberdctl` pointing to this directory.

[Prosody]: https://prosody.im/
[Ejabberd]: https://www.ejabberd.im/

## Requirements

- Ability to run integration tests locally without manually configuring servers
  or other tools
- Must be able to run one set of tests against multiple servers
- Ability to run multiple tests against the same server before shutting down the
  child process

## Proposal

The proposed API adds three new types, seven functions, and three methods that
would have to be maintained if we reached version 1.0 and began observing the Go
compatibility promise:

// Cmd is an external command being prepared or run.
type Cmd struct {

	// New creates a new, unstarted, command.
	func New(ctx context.Context, name string, opts ...Option) (*Cmd, error)

	// Close kills the command if it is still running and cleans up any
	// temporary resources that were created.
	func (cmd *Cmd) Close() error

	// ConfigDir returns the temporary directory used to store config files.
	func (cmd *Cmd) ConfigDir() string

	// Dial attempts to connect to the server by dialing localhost and then
	// negotiating a stream with the location set to the domainpart of j and the
	// origin set to j.
	func (cmd *Cmd) Dial(ctx context.Context, j jid.JID, t *testing.T, features ...xmpp.StreamFeature) (*xmpp.Session, error)

// Option is used to configure a Cmd.
type Option func(cmd *Cmd) error

	// Args sets additional command line args to be passed to the command.
	func Args(f ...string) Option

	// Cert creates a private key and certificate with the given name.
	func Cert(name string) Option

	// Log configures the command to log output to the current testing.T.
	func Log() Option

	// LogXML configures the command to log sent and received XML to the current
	// testing.T.
	func LogXML() Option

	// TempFile creates a file in the commands temporary working directory.
	// After all configuration is complete it then calls f to populate the
	// config files.
	func TempFile(cfgFileName string, f func(*Cmd, io.Writer) error) Option

// SubtestRunner is the signature of a function that can be used to start
// subtests.
type SubtestRunner func(func(context.Context, *testing.T, *Cmd)) bool

	// Test starts a command and returns a function that runs f as a subtest
	// using t.Run. Multiple calls to the returned function will result in
	// uniquely named subtests. When all subtests have completed, the daemon is
	// stopped.
	func Test(ctx context.Context, name string, t *testing.T, opts ...Option) SubtestRunner

The main downside to this proposal is that `Test` is a higher-order function
that also returns a higher-order function.
This is confusing and results in some odd syntax such as:

	context.Background(), t,
)(func(ctx context.Context, t *testing.T, cmd *integration.Cmd) {

Unfortunately, this can't be helped.
If we were to put everything in one function (including the subtest function and
the variadic options) it becomes even worse:

	context.Background(), t,
	func(ctx context.Context, t *testing.T, cmd *integration.Cmd) {

Regardless of this rather minor downside, this API allows us to create
subdirectories that act as more specific packages for launching and configuring
specific servers, for instance, an `integration/prosody` package might have an
API like the following:

// ConfigFile is an option that can be used to write a temporary Prosody config file.
func ConfigFile(cfg Config) integration.Option

// New creates a new, unstarted, Prosody daemon.
func New(ctx context.Context, opts ...integration.Option) (*integration.Cmd, error)

// Test starts a Prosody instance and returns a function that runs f as a
// subtest using t.Run.
func Test(ctx context.Context, t *testing.T, opts ...integration.Option) integration.SubtestRunner

// Config contains options that can be written to a Prosody config file.
type Config {
	Admins []string
	VHosts []string

Internally they would use the `integration` package an its various options.
For example, `prosody.ConfigFile` could be implemented in terms of