feat: builder: Impl exec:nullop
fix .env.toml example (#1019)
docker generic builder (#988)
Testground is a platform for testing, benchmarking, and simulating distributed and p2p systems at scale. It's designed to be multi-lingual and runtime-agnostic, scaling gracefully from 2 to 10k instances, only when needed.
NOTE: currently, we don't distribute binaries, so you will have to build from source.
Prerequisites: Go 1.14+, Docker daemon running.
$ git clone https://github.com/testground/testground.git $ cd testground $ make install # builds testground and the Docker image, used by the local:docker runner. $ testground daemon # will start the daemon listening on localhost:8042 by default. # => open a different console (client-side), in the same directory (testground/testground repo checkout) # import the network test plan from this repo into $TESTGROUND_HOME/plans # read more about this path at https://docs.testground.ai/getting-started#running-testground $ testground plan import --from ./plans/network # run two instances of the `ping-pong` test case from the `network` plan, # building with docker:go, running with local:docker $ testground run single --plan=network --testcase=ping-pong \ --builder=docker:go --runner=local:docker \ --instances=2
See Getting started and the rest of the docs on our docs website for more info! 🚀
This README is just the tip of the iceberg! Check out our full documentation site at docs.testground.ai.
There you will find a conceptual system walkthrough, tips on writing test plans, instructions on running test plans, configuring runners and builders, deploying Kubernetes clusters, and a lot more.
Please report any problems or inaccuracies by opening a docs issue on this repo.
You develop distributed test plans as if you were writing unit tests against local APIs.
Your test plan calls out to the coordination API to:
There is no special "conductor" node telling instances what to do when. The choreography and sequencing emerges from within the test plan itself.
You decide what versions of the upstream software you want to exercise your test against.
Inside your test plan:
Via a TOML-based composition file, you instruct Testground to:
You collect the outputs of the test plan with a single command, and use data processing scripts and platforms (such as the upcoming Jupyter notebooks integration) to draw conclusions.
(🌕 = fully supported // 🌑 = planned)
A test plan is a blackbox with a formal contract. Testground promises to inject a set of env variables, and the test plan promises to emit events on stdout, and assets on the output directory.
For running test plans written in different languages, targeted for different runtimes, and levels of scale:
docker:gobuilders: compile test plans written in Go into executables or containers.
cluster:k8srunners: run executables or containers locally (suitable for 2-300 instances), or in a Kubernetes cloud environment (300-10k instances).
Got some spare cycles and would like to add support for writing test plans Rust, Python or X? It's easy! Open an issue, and the community will guide you!
Redis-backed lightweight API offering synchronisation primitives to coordinate and choreograph distributed test workloads across a fleet of nodes.
Test instances are able to set connectedness, latency, jitter, bandwidth, duplication, packet corruption, etc. to simulate a variety of network conditions.
Create a k8s cluster ready to run Testground jobs on AWS by following the instructions at
Compiling test plans against specific versions of upstream dependencies (e.g. moduleX v0.3, or commit 1a2b3c).
So that a single test plan can work with a range of versions of the components under test, as these evolve over time.
Diagnostics: Automatic diagnostics via pprof (for Go test plans), with metrics emitted to InfluxDB in real-time. Metrics can be raw data points or aggregated measurements, such as histograms, counters, gauges, moving averages, etc.
Results: When the test plan concludes, all results are pushed in batch to InfluxDB for later exploration, analysis, and visualization.
Create tailored test runs by composing scenarios declaratively, with different groups, cohorts, upstream deps, test params, etc.
Emit and collect/export/download test outputs (logs, assets, event trails, run events, etc.) from all participants in a run.
There are some basic, project-agnostic Testground test plans in the
We use these plans to validate the functionality of Testground itself.
To link them under
$TESTGROUND_HOME/plans, if you're using default paths, these commands should do the trick
(assuming you have already run the Testground daemon once, and therefore the
$TESTGROUND_HOME layout has been created
$ # from the root of this repo, run the following; it will symlink all test plans under $TESTGROUND_HOME/plans $ ln -s $PWD/plans/* $HOME/testground/plans $ testground run single --plan network --testcase ping-pong --builder=docker:go --runner=local:docker --instances=2
For project-specific test plans, check out these repos:
To use them, import them into
$TESTGROUND_HOME/plans using the following testground commands:
$ testground plan import --git --from https://github.com/libp2p/test-plans.git --name libp2p $ testground plan import --git --from https://github.com/ipfs/test-plans.git --name ipfs $ # to run the find-peers test case from the libp2p/dht test plan (this is not a complete command!) $ testground run single --plan libp2p/dht --testcase find-peers --builder docker:go --runner local:docker <options>
Please read our CONTRIBUTING Guidelines before making a contribution.
@daviddias, @stebalien, @hacdias, @jimpick, @aschmahmann, @dirkmc, @yusefnapora.
You can find notes from the Testground team meetings at github.com/testground/pm
Dual-licensed: MIT, Apache Software License v2, by way of the Permissive License Stack.