Stateful Testing¶
Stateful testing is a more advanced method of property-based testing used to test complex systems. In a stateful test you define a number of actions that can be combined together in different ways, and Hypothesis attempts to find a sequence of those actions that results in a failure. This is useful for testing complex contracts or contract-to-contract interactions where there are many possible states.
Brownie utilizes the hypothesis
framework to allow for stateful testing.
Much of the content in this section is based on the official hypothesis.works website. To learn more about stateful testing, you may wish to read the following articles:
- Rule Based Stateful Testing by David R. MacIver
- Solving the Water Jug Problem from Die Hard 3 with TLA+ and Hypothesis by Nicholas Chammas
- Hypothesis Documentation on stateful testing
Warning
This functionality is still under development and should be considered experimental. Use common sense when evaluating the results, and if you encounter any problems please open an issue on Github.
Rule-based State Machines¶
A state machine is a class used within stateful testing. It defines the initial test state, a number of actions outlining the structure that the test will execute in, and invariants that should not be violated during execution.
Note
Unlike regular Hypothesis state machines, Brownie state machines should not subclass RuleBasedStateMachine
.
Rules¶
At the core of every state machine are one or more rules. Rules are class methods that are very similar to @given
based tests; they receive values drawn from strategies and pass them to a user defined test function. The key difference is that where @given
based tests run independently, rules can be chained together - a single stateful test run may involve multiple rule invocations, which may interact in various ways.
Any state machine method named rule
or beginning with rule_
is treated as a rule.
class StateMachine:
def rule_one(self):
# performs a test action
def rule_two(self):
# performs another, different test action
Initializers¶
There is also a special type of rule known as an initializer. These are rules that are guaranteed to be executed at most one time at the beginning of a run (i.e. before any normal rule is called). They may be called in any order, or not at all, and the order will vary from run to run.
Any state machine method named initialize
or beginning with initialize_
is treated as an initializer.
class StateMachine:
def initialize(self):
# this method may or may not be called prior to rule_two
def rule(self):
# once this method is called, initialize will not be called during the test run
Strategies¶
A state machine should contain one or more strategies, in order to provide data to it’s rules.
Strategies must be defined at the class level, typically before the first function. They can be given any name.
Similar to how fixtures work within pytest tests, state machine rules receive strategies by referencing them within their arguments. This is shown in the following example:
class StateMachine:
st_uint = strategy('uint256')
st_bytes32 = strategy('bytes32')
def initialize(self, st_uint):
# this method draws from the uint256 strategy
def rule(self, st_uint, st_bytes32):
# this method draws from both strategies
def rule_two(self, value="st_uint", othervalue="st_uint"):
# this method draws from the same strategy twice
Invariants¶
Along with rules, a state machine often defines invariants. These are properties that should remain unchanged, regardless of any actions performed by the rules. After each rule is executed, every invariant method is always called to ensure that the test has not failed.
Any state machine method named invariant
or beginning with invariant_
is treated as an invariant. Invariants are meant for verifying correctness of state; they cannot receive strategies.
class StateMachine:
def rule_one(self):
pass
def rule_two(self):
pass
def invariant(self):
# assertions in this method should always pass regardless
# of actions in both rule_one and rule_two
Setup and Teardown¶
A state machine may optionally include setup and teardown procedures. Similar to pytest fixtures, setup and teardown methods are available to execute logic on a per-test and per-run basis.
-
classmethod
StateMachine.
__init__
(cls, *args)¶ This method is called once, prior to the chain snapshot taken before the first test run. It is run as a class method - changes made to the state machine will persist through every run of the test.
__init__
is the only method that can be used to pass external data into the state machine. In the following example, we use it to pass the accounts fixture, and a deployed instance of a token contract:class StateMachine: def __init__(cls, accounts, token): cls.accounts = accounts cls.token = token def test_stateful(Token, accounts, state_machine): token = Token.deploy("Test Token", "TST", 18, 1e23, {'from': accounts[0]}) # state_machine forwards all the arguments to StateMachine.__init__ state_machine(StateMachine, accounts, token)
-
classmethod
StateMachine.
setup
(self)¶ This method is called at the beginning of each test run, immediately after chain is reverted to the snapshot. Changes applied during
setup
will only have an effect for the upcoming run.
-
classmethod
StateMachine.
teardown
(self)¶ This method is called at the end of each successful test run, prior to the chain revert.
teardown
is not called if the run fails.
-
classmethod
StateMachine.
teardown_final
(cls)¶ This method is called after the final test run has completed and the chain has been reverted.
teardown_final
is called regardless of whether the test passed or failed.
Test Execution Sequence¶
A Brownie stateful test executes in the following sequence:
- The setup phase of all pytest fixtures are executed in their regular order.
- If present, the
StateMachine.__init__
method is called.- A snapshot of the current chain state is taken.
- If present, the
StateMachine.setup
method is called.- Zero or more
StateMachine
initialize methods are called, in no particular order.- One or more
StateMachine
rule methods are called, in no particular order.- After each initialize and rule, every
StateMachine
invariant method is called.- If present, the
StateMachine.teardown
method is called.- The chain is reverted to the snapshot taken in step 3.
- Steps 4-9 are repeated 50 times, or until the test fails.
- If present, the
StateMachine.teardown_final
method is called.- The teardown phase of all pytest fixtures are executed in their normal order.
Writing Stateful Tests¶
To write a stateful test:
- Create a state machine class.
- Create a regular pytest-style test that includes the
state_machine
fixture. - Within the test, call
state_machine
with the state machine as the first argument.
-
brownie.test.stateful.
state_machine
(state_machine_class, *args, settings=None)¶ Executes a stateful test.
state_machine_class
: A state machine class to be used in the test. Be sure to pass the class itself, not an instance of the class.*args
: Any arguments given here will be passed to the state machine’s__init__
method.settings
: An optionaldict
of Hypothesis settings that will replace the defaults for this test only.
This method is available as a pytest fixture
state_machine
.
Basic Example¶
As a basic example, we will create a state machine to test the following Vyper Depositer
contract. This is very simple contract with two functions and a public mapping. Anyone can deposit ether for another account using the deposit_for
method, or withdraw deposited ether using withdraw_from
.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | deposited: public(HashMap[address, uint256])
@external
@payable
def deposit_for(_receiver: address) -> bool:
self.deposited[_receiver] += msg.value
return True
@external
def withdraw_from(_value: uint256) -> bool:
assert self.deposited[msg.sender] >= _value, "Insufficient balance"
self.deposited[msg.sender] = _value
send(msg.sender, _value)
return True
|
If you looked closely you may have noticed a major issue in the contract code. If not, don’t worry! We’re going to find it using our test.
Here is a state machine and test function we can use to test the contract.
import brownie
from brownie.test import strategy
class StateMachine:
value = strategy('uint256', max_value="1 ether")
address = strategy('address')
def __init__(cls, accounts, Depositer):
# deploy the contract at the start of the test
cls.accounts = accounts
cls.contract = Depositer.deploy({'from': accounts[0]})
def setup(self):
# zero the deposit amounts at the start of each test run
self.deposits = {i: 0 for i in self.accounts}
def rule_deposit(self, address, value):
# make a deposit and adjust the local record
self.contract.deposit_for(address, {'from': self.accounts[0], 'value': value})
self.deposits[address] += value
def rule_withdraw(self, address, value):
if self.deposits[address] >= value:
# make a withdrawal and adjust the local record
self.contract.withdraw_from(value, {'from': address})
self.deposits[address] -= value
else:
# attempting to withdraw beyond your balance should revert
with brownie.reverts("Insufficient balance"):
self.contract.withdraw_from(value, {'from': address})
def invariant(self):
# compare the contract deposit amounts with the local record
for address, amount in self.deposits.items():
assert self.contract.deposited(address) == amount
def test_stateful(Depositer, accounts, state_machine):
state_machine(StateMachine, accounts, Depositer)
When this test is executed, it will call rule_deposit
and rule_withdraw
using random data from the given strategies until it encounters a state which violates one of the assertions. If this happens, it repeats the test in an attempt to find the shortest path and smallest data set possible that reproduces the error. Finally it saves the failing conditions to be used in future tests, and then delivers the following output:
def invariant(self):
for address, amount in self.deposits.items():
> assert self.contract.deposited(address) == amount
E AssertionError: assert 0 == 1
Falsifying example:
state = BrownieStateMachine()
state.rule_deposit(address=<Account '0x33A4622B82D4c04a53e170c638B944ce27cffce3'>, value=1)
state.rule_withdraw(address=<Account '0x33A4622B82D4c04a53e170c638B944ce27cffce3'>, value=0)
state.teardown()
From this we can see the sequence of calls leading up to the error, and that the failed assertion is that self.contract.deposited(address)
is zero, when we expected it to be one. We can infer that the contract is incorrectly adjusting balances within the withdraw function. Looking at that function:
9 10 11 12 13 14 | @external
def withdraw_from(_value: uint256) -> bool:
assert self.deposited[msg.sender] >= _value, "Insufficient balance"
self.deposited[msg.sender] = _value
send(msg.sender, _value)
return True
|
On line 12, rather than subtracting _value
, the balance is being set to _value
. We found the bug!
More Examples¶
Here are some links to repositories that make use of stateful testing. If you have a project that you would like included here, feel free to edit this document and open a pull request, or let us know about it on Gitter.
- iamdefinitelyahuman/NFToken: A non-fungible implementation of the ERC20 standard.
- apguerrera/DreamFrames: Buy and sell frames in movies.
- curvefi/curve-dao-contracts: Vyper contracts used by Curve DAO
Running Stateful Tests¶
By default, stateful tests are included when you run your test suite. There is no special action required to invoke them.
You can choose to exclude stateful tests, or to only run stateful tests, with the --stateful
flag. This can be useful to split the test suite when setting up continuous integration.
To only run stateful tests:
$ brownie test --stateful true
To skip stateful tests:
$ brownie test --stateful false
When a stateful test is active the console shows a spinner that rotates each time a run of the test has finished. If the color changes from yellow to red, it means the test has failed and hypothesis is now searching for the shortest path to the failure.