Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
TestBox is a xUnit style and behavior-driven development (BDD) testing framework for CFML (ColdFusion). It does not depend on any other framework for operation and it has a clean and descriptive syntax that will make you giggle when writing your tests. It ships with not only the xUnit and BDD testing capabilities, but also an assertions and expectations library, several different runners, reporters and MockBox, for mocking and stubbing capabilities.
TestBox does not depend on any other framework for operation and it has a clean and descriptive syntax that will make you giggle when writing your tests. It ships with not only the xUnit and BDD testing capabilities, but also an assertions and expectations library, several different runners, reporters and MockBox, for mocking and stubbing capabilities.
A test suite begins with a call to our TestBox describe()
function with at least two arguments: a title
and a body
function/closure. The title
is the name of the suite to register and the body
function/closure is the block of code that implements the suite. When applying BDD to your tests, this function is used to describe your story scenarios that you will implement. There are more arguments which you can see below:
TestBox relies on the fact of creating testing bundles which are basically CFCs. A bundle CFC will hold all the suites and specs a TestBox runner will execute and produce reports on.
This bundle CFC can contain 2 life-cycle functions and a single run()
function where you will be writing your test suites and specs. The beforeAll()
and afterAll()
methods are called life-cycle methods. They will execute once before the run()
function and once after the run()
function. This is a great way to do any kind of global setup or teardown in your tests.
The run()
function receives the TestBox testResults
object as a reference and testbox
as a reference as well. This way you can have metadata and access to what will be reported to users in a reporter. You can also use it to decorate the results or store much more information that can be picked up later by reports. You also have access to the testbox
class so you can see how the test is supposed to execute, what labels was it passed, directories, options, etc.
We have prepared an awesome PDF ref card for working with TestBox BDD:
With TestBox's BDD syntax, it is possible to create suites dynamically; however, there are a few things to be aware of.
Setup for dynamic suites must be done in the pseudo-constructor (versus in beforeAll()
). This is because variables
-scoped variables set in beforeAll()
are not available in the describe
closure (even though they are available in it
closures). This behavior can be explained by the execution sequence of a BDD bundle: When the bundle's run()
method is called, it collects preliminary test data via describe
s. After preliminary test data are collected, the beforeAll()
runs, followed by the describe
closures.
Additionally, care must be taken to pass data into the it
closures, otherwise strange behavior will result (the values from the last loop iteration will be repeated in the body of each looped it
).
The following bundle creates suites dynamically, by looping over test metadata.
is a style of writing tests where you describe the state of the code you want to test (Given
), the behavior you want to test (When
) and the expected outcome (Then
). (See )
Testbox supports the use of function names given()
and when()
in-place of describe()
function calls. The then()
function call is an alternative for it()
function calls. The advantage of this style of is that you can gather your requirements and write your tests in a common language that can be understood by developers and stake-holders alike. This common language format is often referred to as the language; using it we can gather and document the requirements as:
TestBox provides you with feature()
, scenario()
and story()
wrappers for describe()
blocks. As such we can write our requirements in test form like so:
The output from running the test will read as the original requirements, providing you with not only automated tests but also a living document of the requirements in a business-readable format.
As feature()
, scenario()
and story()
are wrappers for describe()
you can intermix them so that your can create tests which read as the business requirements. As with describe()
, they can be nested to build up blocks.
If you prefer to gather requirements as then you may prefer to take advantage of the story()
wrapper for describe()
instead.
Argument
Required
Default
Type
Description
title
true
---
string
The title of the suite to register
body
true
---
closure/udf
The closure that represents the test suite
labels
false
---
string/array
The list or array of labels this suite group belongs to
asyncAll
false
false
Boolean
If you want to parallelize the execution of the defined specs in this suite group.
skip
false
false
Boolean
A flag or a closure that tells TestBox to skip this suite group from testing if true. If this is a closure it must return boolean.
Specs and suites can be tagged with TestBox labels. Labels allows you to further categorize different specs or suites so that when a runner executes with labels attached, only those specs and suites will be executed, the rest will be skipped.
Expectations are self-concatenated strings that evaluate an actual value to an expected value or condition. These are initiated by the global TestBox method called expect()
which takes in a value called the actual value or expectAll()
which takes in an array or struct which will be the actual value. It is concatenated in our expectations DSL with a matcher function that will most likely take in an expected value or condition to test. You can also concatenate the matchers and do multiple evaluations on a single actual value.
Each matcher implements a comparison or evaluation of the actual value and an expected value or condition. It is responsible for either passing or failing this evaluation and reporting it to TestBox. Each matcher also has a negative counterpart assertion by just prefixing the call to the matcher with a not
expression.
TestBox has a plethora (That's Right! I said Plethora) of matchers that are included in TestBox. The best way to see all the latest matchers is to visit our API and digest the testbox.system.Expectation
class. There is also the ability to register and write custom matchers in TestBox via our addMatchers()
function at runtime.
You can also build and register custom matchers. Please visit the Custom Matchers chapter to read more about custom matchers.
As we have seen before, the describe()
function describes a test suite of related specs in a test bundle CFC. The title of the suite is concatenated with the title of a spec to create a full spec's name which is very descriptive. If you name them well, they will read out as full sentences as defined by BDD style.
Calls to our describe()
function can be nested with specs at any level or point of execution. This allows you to create your tests as a related tree of nested functions. Please note that before a spec is executed, TestBox walks down the tree executing each beforeEach()
and afterEach()
function in the declared order. This is a great way to logically group specs in any level as you see fit.
Please refer to our MockBox section to take advantage of all the mocking and stubbing you can do. However, every BDD TestBundle has the following functions available to you for mocking and stubbing purposes:
makePublic( target, method, newName )
- Exposes private methods from objects as public methods
querySim( queryData )
- Simulate a query
getMockBox( [generationPath] )
- Get a reference to MockBox
createEmptyMock( [className], [object], [callLogging=true])
- Create an empty mock from a class or object
createMock( [className], [object], [clearMethods=false], [callLogging=true])
- Create a spy from an instance or class with call logging
prepareMock( object, [callLogging=true])
- Prepare an instance of an object for method spies with call logging
createStub( [callLogging=true], [extends], [implements])
- Create stub objects with call logging and optional inheritance trees and implementation methods
getProperty( target, name, [scope=variables], [defaultValue] )
- Get a property from an object in any scope
As you can see from our arguments for a test suite, you can pass an asyncAll
argument to the describe()
blocks that will allow TestBox to execute all specs in separate threads for you concurrently.
Caution Once you delve into the asynchronous world you will have to make sure your tests are also thread safe (var-scoped) and provide any necessary locking.
If you are familiar with xUnit style frameworks, the majority of them provide a way to execute functions before and after every single test case or spec in BDD. This is a great way to keep your tests DRY (Do not repeat yourself)! TestBox provides the beforeEach()
and afterEach()
methods that each take in a closure as their argument that receive the name of the spec that's about to be executed or just executed. As their names indicate, they execute before a spec and after a spec in a related describe
block.
The aroundEach()
life-cycle method will completely wrap your spec in another closure. This is an elegant way for you to provide a complete around AOP advice to a specification. You can use it to surround the execution in transaction blocks, ORM rollbacks, logging, and so much more. This life-cycle method will decorate ALL specs within a single suite and any children suites. The method signature is below:
The method receives a structure of data representing the spec. This contains the following elements:
body : The actual closure for the spec that you will use to execute within it.
labels : The labels used in the spec
name : The name of the spec
order : The order of execution of the spec
skip : The skip flag or closure that determines if the spec runs
The method also receives a structure of metadata about the suite this spec is contained in. It has information about life-cycle closures, async information, names, parents, etc. Here is a very practical example of creating and around each closure to provide rollbacks for specs:
This simple around each life-cycle closure will rollback ALL my spec's executions even if they throw exceptions.
TestBox since v2.3.0 also allows you to declare life-cycle methods via annotations. Please see our Annotations section for more information.
When you use beforeEach()
, afterEach()
, and aroundEach()
at the same time, there is a specific order they fire in. For a given describe block, they will fire in this order. Remember, aroundEach()
is split into two parts-- the half of the method before you call spec.body()
and the second half of the method.
beforeEach
aroundEach (first half)
it() (the spec.body()
call)
aroundEach (second half)
afterEach()
Here's an example:
If there are more than one it()
blocks, the process repeats for each one. Steps 1, 2, 4, 5 will wrap every single it()
.
When you nest more than one describe block inside the other, the before/around/after order is the same but drills down to the innermost describe and then bubbles back up. That means the outermost beforeEach()
starts and we end on the outermost afterEach()
.
Here's what an example flow would look like that had before/after/around specified in two levels of describes with a single it()
in the inner most describe.
Outermost beforeEach()
call
Innermost beforeEach()
call
Outermost aroundEach()
call (first half)
Innermost aroundEach()
call (first half)
The it()
block
Innermost aroundEach()
calls (second half)
Outermost aroundEach()
call (second half)
Innermost afterEach()
call
Outermost afterEach()
call
This works regardless of the number of levels and can obviously have many permutations, but the basic order is still the same. Before/around/after and starting at the outside working in, and back out again. This process happens for every single spec or it()
block. This is as opposed to the beforeAll()
and afterAll()
method which only run once for the entire CFC regardless of how many specs there are.
TestBox ships with several test runners internally but we have tried to simplify and abstract it with our testbox
object which can be found in the testbox.system
package. The TestBox object allows you to execute tests from a variety of entry points and languages such as CFC, CFM, HTTP, NodeJS, SOAP or REST. You can also make your CFC's extend from our BaseSpec
class so you can execute it directly via the URL. The main execution methods are:
Info We encourage you to read the API docs included in the distribution for the complete parameters for each method.
run()
ArgumentsHere are the arguments you can use for initializing TestBox or executing the run()
method
runRemote()
argumentsHere are the arguments you can use for executing the runRemote()
method:
The bundles argument which can be a single CFC path or an array of CFC paths or a directory argument so it can go and discover the test bundles from that directory. The reporter argument can be a core reporter name like: json,xml,junit,raw,simple,dots,tap,min,etc or it can be an instance of a reporter CFC. You can execute the runners from any cfm template or any CFC or any URL, that is up to you.
TestBox ships with a global runner that can be used to run pretty much anything. You can customize it or place it wherever you need it:
TestBox ships with a test browser that is highly configurable to whatever URL accessible path you want. It will then show you a test browser where you can navigate and execute not only individual tests, but also directory suites as well.
In our test samples and templates we include an ANT runner that will be able to execute your tests via ANT. It can also leverage our ANTJunit reporter to use the junitreport task to produce JUnit compliant reports as well. You can find this runner in the test samples and runner template directory.
If you make your test bundle CFC inherit from our testbox.system.BaseSpec class, you will be able to execute the CFC directly via the URL: ```javascript http://localhost/test/MyTest.cfc?method=runRemote ``` You can also pass the following arguments to the method via the URL: * **testSuites** : A list or array of suite names that are the ones that will be executed ONLY! * **testSpecs** : A list or array of test names that are the ones that will be executed ONLY! * **reporter** : The type of reporter to run the test with ```javascript http://localhost/test/MyTest.cfc?method=runRemote&reporter=json ```
You can run tests via SOAP by leveraging the runRemote() method. The WSDL URL will be
HTTP/REST Runner You can run tests via HTTP/REST by leveraging the runRemote() endpoint. The URL will be
There is a user-contributed NodeJS Runner that looks fantastic and can be downloaded here:
https://www.npmjs.com/package/testbox-runner
Just use node to install:
Specifications (Specs) are defined by calling the TestBox it()
global function, which takes in a title
and a body
function/closure. The title
is the title of this spec you will write and the body
function/closure is a block of code that represents the spec. A spec will contain most likely one or more expectations that will test the state of the SUT (software under test) or sometimes referred to as code under test. In BDD style, your specifications are what is used to validate your requirements of a scenario which is your describe()
block of your story.
An expectation is a nice assertion DSL that TestBox exposes so you can pretty much read what should happen in the testing scenario. A spec will pass if all expectations pass. A spec with one or more expectations that fail will fail the entire spec.
Since the implementations of the describe()
and it()
functions are closures, they can contain executable code that is necessary to implement the test. All CFML rules of scoping apply to closures, so please remember them. We recommend always using the variables
scope for easy access and distinction.
The data
argument can be used to pass in a structure of data into the spec so it can be used later within the body closure. This is great when doing looping and creating dynamic closure calls:
Specs and suites can be skipped from execution by prefixing certain functions with the letter x
or by using the skip argument in each of them. The reporters will show that these suites or specs where skipped from execution. The functions you can prefix are:
it()
describe()
story()
given()
when()
then()
feature()
The skip
argument can be a boolean value or a closure. If the value is true then the suite or spec is skipped. If the return value of the closure is true then the suite or spec is skipped. Using the closure approach allows you to dynamically at runtime figure out if the desired spec or suite is skipped. This is such a great way to prepare tests for different CFML engines.
TestBox comes also with a nice plethora of reporters:
ANTJunit : A specific variant of JUnit XML that works with the ANT junitreport task
Codexwiki : Produces MediaWiki syntax for usage in Codex Wiki
Console : Sends report to console
Doc : Builds semantic HTML to produce nice documentation
Dot : Builds an awesome dot report
JSON : Builds a report into JSON
JUnit : Builds a JUnit compliant report
Raw : Returns the raw structure representation of the testing results
Simple : A basic HTML reporter
Text : Back to the 80's with an awesome text report
XML : Builds yet another XML testing report
Tap : A test anything protocol reporter
Min : A minimalistic view of your test reports
MinText : A minimalistic view of your test reports in consoles
NodeJS : User-contributed:
Argument
Required
Default
Type
Description
title
true
---
string
the title of the spec
body
true
---
closure/udf
The closure that represents the spec
labels
false
---
string/array
The list or array of labels this suite group belongs to
skip
false
false
Boolean
A flag or a closure that tells TestBox to skip this suite group from testing if true. If this is a closure it must return boolean.
data
false
{}
struct
A struct of data you can bind the spec with so you can use within the body
closure
Argument
Required
Default
Type
Description
bundles
true
---
string/array
The path, list of paths or array of paths of the spec bundle CFCs to run and test
directory
false
---
struct
The directory mapping path or a struct: [ mapping = the path to the directory using dot notation (myapp.testing.specs), recurse = boolean, filter = closure that receives the path of the CFC found, it must return true to process or false to continue process ]
reporter
false
simple
string/struct/instance
The type of reporter to use for the results, by default is uses our 'simple' report. You can pass in a core reporter string type or an instance of a coldbox.system.reports.IReporter. You can also pass a struct with [type="string or classpath", options={}] if a reporter expects options.
labels
false
false
string/array
The string or array of labels the runner will use to execute suites and specs with.
excludes
false
---
string/array
The string or array of labels to exclude from running
options
false
{}
struct
A structure of property name-value pairs that each runner can implement and use at its discretion.
testBundles
false
---
string/array
A list or array of bundle names that are the ones that will be executed ONLY!
testSuites
false
---
string/array
A list or array of suite names that are the ones that will be executed ONLY!
testSpecs
false
---
string/array
A list or array of test names that are the ones that will be executed ONLY
callbacks
false
{}
struct of closures or CFC
A struct of listener callbacks or a CFC with callbacks for listening to progress of the testing: onBundleStart,onBundleEnd,onSuiteStart,onSuiteEnd,onSpecStart,onSpecEnd
eagerFailure
false
false
boolean
If true, then after testing a bundle if there are any failures or errors no more testing will be performed.
Argument
Required
Default
Type
Description
bundles
true
---
string
The path, list of paths or array of paths of the spec bundle CFCs to run and test
directory
false
---
string
The directory mapping to test: directory = the path to the directory using dot notation (myapp.testing.specs)
recurse
false
true
boolean
Recurse the directory mapping or not, by default it does
reporter
false
simple
string/path
The type of reporter to use for the results, by default is uses our 'simple' report. You can pass in a core reporter string type or a class path to the reporter to use.
reporterOptions
false
{}
JSON
A JSON struct literal of options to pass into the reporter
labels
false
false
string
The string array of labels the runner will use to execute suites and specs with.
excludes
false
---
string/array
The string or array of labels to exclude from running
options
false
{}
JSON
A JSON struct literal of configuration options that are optionally used to configure a runner.
testBundles
false
---
string/array
A list or array of bundle names that are the ones that will be executed ONLY!
testSuites
false
---
string
A list of suite names that are the ones that will be executed ONLY!
testSpecs
false
---
string
A list of test names that are the ones that will be executed ONLY
eagerFailure
false
false
boolean
If true, then after testing a bundle if there are any failures or errors no more testing will be performed.