A test suite in TestBox is a collection of specifications that model what you want to test. As we will investigate, the way the suite is expressed can be of many different types.

Test suite is a container that has a set of tests which helps testers in executing and reporting the test execution status.

A test suite begins with a call to our TestBox describe() function with at least two arguments: a title and a body function/closure. The title is the name of the suite to register and the body function/closure is the block of code that implements the suite.

When applying BDD to your tests, this function is used to describe your story scenarios that you will implement.

function run( testResults, testBox ){

     describe("A suite", function(){
          it("contains spec with an awesome expectation", function(){
               expect( true ).toBeTrue();
          });
          it("contains spec with a failure expectation", function(){
               expect( true ).toBeFalse();
          });
     });

}

Arguments

There are more arguments, which you can see below:

BDD stands for Behavioral Driven Development. It is a software development process that aims to improve collaboration between developers, testers, and business stakeholders. BDD involves creating automated tests that are based on the expected behavior of the software, rather than just testing individual code components. This approach helps ensure that the software meets the desired functionality and is easier to maintain and update in the future.

In traditional xUnit, you focused on every component's method individually. In BDD, we will focus on a feature or story to complete, which could include testing many different components to satisfy the criteria. TestBox allows us to create these types of texts with human-readable functions matching our features/stories and expectations.

describe( "Tests of TestBox behaviour", () => {
	it( "rejects 5 as being between 1 and 10", () => {
		expect( () => {
			expect( 5 ).notToBeBetween( 1, 10 );
		} ).toThrow();
	} );
	
	it( "rejects 10 as being between 1 and 10", () => {
		expect( () => {
			expect( 10 ).notToBeBetween( 1, 10 );
		} ).toThrow();
	} );
} );


feature( "Given-When-Then test language support", () => {
	scenario( "I want to be able to write tests using Given-When-Then language", () => {
		given( "I am using TestBox", () => {
			when( "I run this test suite", () => {
				then( "it should be supported", () => {
					expect( true ).toBe( true );
				} );
			} );
		} );
	} );
} );

story( "I want to list all authors", () => {
    given( "no options", () => {
        then( "it can display all active system authors", () => {
            var event = this.get( "/cbapi/v1/authors" );
            expect( event.getResponse() ).toHaveStatus( 200 );
            expect( event.getResponse().getData() ).toBeArray().notToBeEmpty();
            event
                .getResponse()
                .getData()
                .each( function( thisItem ){
                    expect( thisItem.isActive ).toBeTrue( thisItem.toString() );
                } );
        } );
    } );
    given( "isActive = false", () => {
        then( "it should display inactive users", () => {
            var event = this.get( "/cbapi/v1/authors?isActive=false" );
            expect( event.getResponse() ).toHaveStatus( 200 );
            expect( event.getResponse().getData() ).toBeArray().notToBeEmpty();
            event
                .getResponse()
                .getData()
                .each( function( thisItem ){
                    expect( thisItem.isActive ).toBeFalse( thisItem.toString() );
                } );
        } );
    } );
    given( "a search criteria", () => {
        then( "it should display searched users", () => {
            var event = this.get( "/cbapi/v1/authors?search=tester" );
            expect( event.getResponse() ).toHaveStatus( 200 );
            expect( event.getResponse().getData() ).toBeArray().notToBeEmpty();
        } );
    } );
} );

Expectations are self-concatenated strings that evaluate an actual value to an expected value or condition. These are initiated by the global TestBox method called expect() which takes in a value called the actual value or expectAll() which takes in an array or struct which will be the actual value. It is concatenated in our expectations DSL with a matcher function that will most likely take in an expected value or condition to test. You can also concatenate the matchers and do multiple evaluations on a single actual value.

Matchers

Each matcher implements a comparison or evaluation of the actual value and an expected value or condition. It is responsible for either passing or failing this evaluation and reporting it to TestBox. Each matcher also has a negative counterpart assertion by just prefixing the call to the matcher with a not expression.

function run(){

     describe("The 'toBe' matcher evaluates equality", function(){
          it("and has a positive case", function(){
               expect( true ).toBe( true );
          });

          it("and has a negative case", function(){
               expect( false ).notToBe( true );
          });
     });

     describe("Collection expectations", function(){
        it( "can be done easily with TestBox", function(){
            expectAll( {a:2,b:4,c:6} ).toSatisfy( function(x){ return 0 == x%2; });
        });
    });

}

Included Matchers

TestBox has a plethora (That's Right! I said Plethora) of matchers that are included in TestBox. The best way to see all the latest matchers is to visit our API and digest the testbox.system.Expectation class. There is also the ability to register and write custom matchers in TestBox via our addMatchers() function at runtime.

describe("Some Included TestBox Matchers:", function() {

     describe("The 'toBe' matcher", function() {
          it("works for simple values", function() {
               coldbox = 1;
               expect( coldbox )
                    .toBe( 1 )
                    .notToBe( 5 );
          });

          it("works for complex values too (arrays,structs,queries,objects)", function() {
               expect( [1,2] ).toBe( [1,2] );
               expect( { name="luis", awesome=true} ).toBe( { awesome=true, name="luis" } );
               expect( this ).toBe( this );
               expect( queryNew("") ).toBe( queryNew("") );
          });
     });

     it("The 'toBeWithCase' matches strings with case equality", function() {
          expect( 'hello' )
               .toBeWithCase( 'hello' )
               .notToBeWithCase( 'HELLO' );
     });

     it("The 'toBeTrue' and 'toBeFalse' matchers are used for boolean operations", function() {
          coldbox_rocks = true;
          expect( coldbox_rocks ).toBeTrue();
          expect( !coldbox_rocks ).toBeFalse();
     });

     it("The 'toBeNull' expects null values", function() {
          foo = "bar";
          expect( javaCast("null", "") ).toBeNull();
          expect( foo ).notToBeNull();
     });

     it("The 'toBeInstanceOf' expects the object to be of the same class or inheritance or implementation", function() {
          expect( new coldbox.system.Assertions ).toBeInstanceOf( 'coldbox.system.Assertions' );
          expect( this ).notToBeInstanceOf( 'coldbox.system.MockBox' );
     });

     it("The 'toMatch' matcher is for regular expressions with case sensitivity", function() {
          message = 'foo man choo';

          expect( message )
               .toMatch( '^foo' )
               .toMatch( '(man)' )
               .notToMatch( 'superman' );
     });

     it("The 'toMatchNoCase' matcher is for regular expressions with no case sensitivity", function() {
          message = 'foo man choo';

          expect( message )
               .toMatchNoCase( '^FOO' )
               .toMatchNoCase( '(MAN)' )
               .notToMatchNoCase( 'SuperMan' );
     });

     describe("The 'toBeTypeOf' matcher evaluates using the CF isValid() function", function() {
          it("works with direct calls", function() {
               expect( [1,2] ).toBeTypeOf( 'Array' );
               expect( { name="luis", awesome=true} ).toBeTypeOf( 'struct' );
               expect( this ).toBeTypeOf( 'component' );
               expect( '03/01/1990' ).toBeTypeOf( 'usdate' );
          });

          it("works with dynamic calls as well", function() {
               expect( [1,2] ).toBeArray();
               expect( { name="luis", awesome=true} ).toBeStruct();
               expect( this ).toBeComponent();
               expect( '03/01/1990' ).toBeUsDate();
          });
     });

     it("The 'toBeEmpty' checks emptyness of simple or complex values", function() {
          expect( [] ).toBeEmpty();
          expect( { name="luis", awesome=true} ).notToBeEmpty();
          expect( '' ).toBeEmpty();
     });

     it("The 'toHaveLength' checks size of simple or complex values", function() {
          expect( [] ).toHaveLength( 0 );
          expect( { name="luis", awesome=true} ).toHaveLength( 2 );
          expect( 'Hello' ).toHaveLength( 5 );
     });

     it("The 'toHaveKey' checks for existence of keys in structs", function() {
          expect( { name="luis", awesome=true} ).toHaveKey( 'name' );
     });

     it("The 'toHaveDeepKey' checks for existence of keys anywhere in structs", function() {
          expect( { name="luis", { age=35, awesome=true } } ).toHaveDeepKey( 'age' );
     });

     it("The 'toThrow' checks if the actual call fails", function() {
          expect( function(){
               new calculator().divide( 40, 0 );
          }).toThrow();

          expect( function(){
               new calculator().divide( 40, 0 );
          }).toThrow( regex="zero" );
     });

});

Custom Matchers

You can also build and register custom matchers. Please visit the Custom Matchers chapter to read more about custom matchers.

As we have seen before, the describe() function describes a test suite of related specs in a test bundle CFC. The title of the suite is concatenated with the title of a spec to create a full spec's name which is very descriptive. If you name them well, they will read out as full sentences as defined by BDD style.

describe("A spec", function() {
     it("is just a closure, so it can contain any code", function() {
          coldbox = 0;
          coldbox++;

          expect( coldbox ).toBe( 1 );
     });

     it("can have more than one expectation", function() {
          coldbox = 0;
          coldbox++;

          expect( coldbox ).toBe( 1 );
          expect( coldbox ).toBeTrue();
     });
});

Nesting describe Blocks

Calls to our describe() function can be nested with specs at any level or point of execution. This allows you to create your tests as a related tree of nested functions. Please note that before a spec is executed, TestBox walks down the tree executing each beforeEach() and afterEach() function in the declared order. This is a great way to logically group specs in any level as you see fit.

describe("A spec", function() {

     beforeEach(function( currentSpec ) {
          coldbox = 22;
          application.wirebox = new coldbox.system.ioc.Injector();
     });

     afterEach(function( currentSpec ) {
          coldbox = 0;
          structDelete( application, "wirebox" );
     });

     it("is just a function, so it can contain any code", function() {
          expect( coldbox ).toBe( 22 );
     });

     it("can have more than one expectation and talk to scopes", function() {
          expect( coldbox ).toBe( 22 );
          expect( application.wirebox.getInstance( 'MyService' ) ).toBeComponent();
     });

     describe("nested inside a second describe", function() {

          beforeEach(function( currentSpec ) {
               awesome = 22;
          });

          afterEach(function( currentSpec ) {
               awesome = 22 + 8;
          });

          it("can reference both scopes as needed ", function() {
            expect( coldbox ).toBe( awesome );
          });
     });

     it("can be declared after nested suites and have access to nested variables", function() {
          expect( awesome ).toBe( 30 );
     });

});

With TestBox's BDD syntax, it is possible to create suites dynamically; however, there are a few things to be aware of.

Setup for dynamic suites must be done in the pseudo-constructor (versus in beforeAll()). This is because variables-scoped variables set in beforeAll() are not available in the describe closure (even though they are available in it closures). This behavior can be explained by the execution sequence of a BDD bundle: When the bundle's run() method is called, it collects preliminary test data via describes. After preliminary test data are collected, the beforeAll() runs, followed by the describe closures.

Additionally, care must be taken to pass data into the it closures, otherwise strange behavior will result (the values from the last loop iteration will be repeated in the body of each looped it).

Example

The following bundle creates suites dynamically, by looping over test metadata.

component
    extends="testbox.system.BaseSpec"
    hint="This is an example of a TestBox BDD test bundle containing dynamically-defined suites."
{

    /*
    * Need to do config for *dynamic* test suites here, in the
    * pseudo-constructor, versus in `beforeAll()`.
    */
    doDynamicSuiteConfig();

    /*
    * @hint This method is arbitrarily named, but it sets up 
    * metadata needed by the dynamic suites example. The setup
    * could have been done straight in the pseudo-constructor,
    * but it might be nice to organize it into such a method
    * as this.
    */
    function doDynamicSuiteConfig(){
        variables.dynamicSuiteConfig = ["foo","bar","baz"];
    }

    function run( testResults, testBox ){

        /*
        * @hint Dynamic Test Suites Example
        */
        // loop over test metadata
        for ( var thing in dynamicSuiteConfig ) {
            describe("Dynamic Suite #thing#", function(){
                // notice how data is passed into the it() closure:
                //  * data={ keyA=valueA, keyB=ValueB }
                //  * function( data )
                it( title=thing & "test", 
                    data={ thing=thing }, 
                    body=function( data ) {
                      var thing = data.thing;
                      expect( thing ).toBe( thing );
                });
            });
        }

    }

}

TestBox relies on the fact of creating testing bundles which are basically CFCs. A bundle CFC will hold all the suites and specs a TestBox runner will execute and produce reports on. Don't worry, we will cover what's a suite and a spec as well. Usually they will have a name that ends with *Spec or *Test.

component extends="testbox.system.BaseSpec"{

     // executes before all suites
     function beforeAll(){}

     // executes after all suites
     function afterAll(){}

     // All suites go in here
     function run( testResults, testBox ){

     }

}

This bundle CFC can contain 2 life-cycle functions and a single run() function where you will write your test suites and specs.

Life-Cycle Methods

The beforeAll() and afterAll() methods are called life-cycle methods. They will execute once before the run() function and once after the run() function. This is a great way to do any global setup or tear down in your tests.

// executes before all suites
function beforeAll(){}

// executes after all suites
function afterAll(){}

Execution

The run() function receives the TestBox testResults object as a reference and testbox as a reference as well. This way you can have metadata and access to what will be reported to users in a reporter. You can also use it to decorate the results or store much more information that reports can pick up later. You also have access to the testbox class so you can see how the test is supposed to execute, what labels was it passed, directories, options, etc.

function run( testResults, testBox ){

}

A spec is a declaration that will usually test your system with a requirement. They are defined by calling the TestBox it() global function, which takes in a title and a body function/closure. The title is the title of this spec you will write and the body function/closure is a block of code that represents the spec.

A spec will contain most likely one or more expectations that will test the state of the SUT (software under test) or sometimes referred to as code under test. In BDD style, your specifications are what is used to validate your requirements of a scenario which is your describe() block of your story.

function run(){

     describe("A suite", function(){
          it("contains spec with an awesome expectation", function(){
               expect( true ).toBeTrue();
          });

          it("contains a spec with more than 1 expectation", function(){
               expect( [1,2,3] ).toBeArray();
               expect( [1,2,3] ).toHaveLength( 3 );
          });
     });

}

An expectation is a nice assertion DSL that TestBox exposes so you can pretty much read what should happen in the testing scenario. A spec will pass if all expectations pass. A spec with one or more expectations that fail will fail the entire spec.

Arguments

They are closures Ma!

Since the implementations of the describe() and it() functions are closures, they can contain executable code that is necessary to implement the test. All CFML rules of scoping apply to closures, so please remember them. We recommend always using the variables scope for easy access and distinction.

function run(){

     describe("A suite is a closure", function(){
          c = new Calculator();

          it("and so is a spec", function(){
               expect( c ).toBeTypeOf( 'component' );
          });
     });

}

Spec Data Binding

The data argument can be used to pass in a structure of data into the spec so it can be used later within the body closure. This is great when doing looping and creating dynamic closure calls:

// Simple Example
it( title="can handle binding", body=function( data ){
    expect(    data.keep ).toBeTrue();
}, data={ keep = true } );

// Complex Example. Let's iterate over a bunch of files and create dynamic specs
for( var filePath in files ){

  it( 
    title="#getFileFromPath( filePath )# should be valid JSON", 
    // pass in a struct of data to the spec for later evaluation
    data={ filePath = filePath },
    // the spec closure accepts the data for later evaluation
    body=function( data ) {
      var json = fileRead( data.filePath );
      var isItJson = isJSON( json );

      expect( json ).notToBeEmpty();
      expect( isItJson ).toBeTrue();

      if( isItJson ){
          var jsonData = deserializeJSON(json);
          if( getFileFromPath( filePath ) != "index.json"){
              expect( jsonData ).toHaveKey( "name" );
              expect( jsonData ).toHaveKey( "type" );
          }
      }

  });
}

is a style of writing tests where you describe the state of the code you want to test (Given), the behavior you want to test (When) and the expected outcome (Then). (See )

Testbox supports the use of function names given() and when() in-place of describe() function calls. The then() function call is an alternative for it() function calls. The advantage of this style of is that you can gather your requirements and write your tests in a common language that can be understood by developers and stake-holders alike. This common language format is often referred to as the language; using it we can gather and document the requirements as:

TestBox provides you with feature(), scenario() and story() wrappers for describe() blocks. As such we can write our requirements in test form like so:

The output from running the test will read as the original requirements, providing you with not only automated tests but also a living document of the requirements in a business-readable format.

As feature(), scenario() and story() are wrappers for describe() you can intermix them so that your can create tests which read as the business requirements. As with describe(), they can be nested to build up blocks.

Specs and suites can be tagged with TestBox labels. Labels allows you to further categorize different specs or suites so that when a runner executes with labels attached, only those specs and suites will be executed, the rest will be skipped. You can alternatively choose to skip specific labels when a runner executes with excludes attached.

Specs and suites can be focused so ONLY those suites and specs execute. You will do this by prefixing certain functions with the letter f or by using the focused argument in each of them. The reporters will show that these suites or specs where execute ONLY The functions you can prefix are:

• it()

• describe()

• story()

• given()

• when()

• then()

• feature()

You can pass in an argument called data , which is a struct of dynamic data, to all life-cycle methods. This is useful when creating dynamic suites and specifications. This data will then be passed into the executing body for each life-cycle method for you.

Here is a typical example:

Specs and suites can be skipped from execution by prefixing certain functions with the letter x or by using the skip argument in each of them or by using the skip( message, detail ) function. The reporters will show that these suites or specs were skipped from execution. The functions you can prefix are:

• it()

• describe()

• story()

• given()

• when()

• then()

• feature()

Here are some examples:

Skip Argument

The skip argument can be a boolean value or a closure. If the value is true then the suite or spec is skipped. If the return value of the closure is true then the suite or spec is skipped. Using the closure approach allows you to dynamically at runtime figure out if the desired spec or suite is skipped. This is such a great way to prepare tests for different CFML engines.

Skip Method

You can now use the skip( message, dteail ) method to skip any spec or suite a-la-carte instead of as an argument to the function definitions. This lets you programmatically skip certain specs and suites and pass a nice message.

Global Callbacks

Global callbacks affect the execution of the entire test bundle CFC and all of its suites and specs.

beforeAll()

Executes once before all specs for the entire test bundle CFC. A great place to initialize the environment the bundle needs for testing.

afterAll()

Executes once after all specs for the entire test bundle CFC. A great place to teardown the environment the bundle needed for testing.

Copy

run( testResults, testBox )

Executes once so it can capture all your describe and it blocks so they can be executed by a TestBox runner.

Suite CallBacks

The following callbacks influence the execution of specification methods: it(), then(). The great flexibility of the BDD approach is that it allows you to nest describe, feature, story, given, scenario, when suite blocks to create very human readable and organized documentation for your tests. Each suite block can have its own life-cycle methods as well. Not only that, if they are nested, TestBox will walk the tree and call each beforeEach() and afterEach() in the order you declare them.

beforeEach( body, data )

The body closure will receive have the following signature:

afterEach( body, data )

The body closure will receive have the following signature:

Here are some examples:

aroundEach( body, data )

The body closure will receive have the following signature:

The spec is the currently executing specification, the suite is the suite this life-cycle is embedded in and data is the data binding, if any.

Here is an example:

Lifecycle Nesting Order

When you use beforeEach(), afterEach(), and aroundEach() at the same time, there is a specific order they fire in. For a given describe block, they will fire in this order. Remember, aroundEach() is split into two parts-- the half of the method before you call spec.body() and the second half of the method.

1. beforeEach

2. aroundEach (first half)

3. it() (the spec.body() call)

4. aroundEach (second half)

5. afterEach()

Here's an example:

If there are more than one it() blocks, the process repeats for each one. Steps 1, 2, 4, 5 will wrap every single it().

When you nest more than one describe block inside the other, the before/around/after order is the same but drills down to the innermost describe and then bubbles back up. That means the outermost beforeEach() starts and we end on the outermost afterEach().

Here's what an example flow would look like that had before/after/around specified in two levels of describes with a single it() in the inner most describe.

1. Outermost beforeEach() call

2. Innermost beforeEach() call

3. Outermost aroundEach() call (first half)

4. Innermost aroundEach() call (first half)

5. The it() block

6. Innermost aroundEach() calls (second half)

7. Outermost aroundEach() call (second half)

8. Innermost afterEach() call

9. Outermost afterEach() call

This works regardless of the number of levels and can obviously have many permutations, but the basic order is still the same. Before/around/after and starting at the outside working in, and back out again. This process happens for every single spec or it() block. This is as opposed to the beforeAll() and afterAll() method which only run once for the entire CFC regardless of how many specs there are.

Please refer to our section to take advantage of all the mocking and stubbing you can do. However, every BDD TestBundle has the following functions available to you for mocking and stubbing purposes:

• makePublic( target, method, newName ) - Exposes private methods from objects as public methods

• querySim( queryData ) - Simulate a query

• getMockBox( [generationPath] ) - Get a reference to MockBox

• createEmptyMock( [className], [object], [callLogging=true]) - Create an empty mock from a class or object

• createMock( [className], [object], [clearMethods=false], [callLogging=true]) - Create a spy from an instance or class with call logging

• prepareMock( object, [callLogging=true]) - Prepare an instance of an object for method spies with call logging

• createStub( [callLogging=true], [extends], [implements]) - Create stub objects with call logging and optional inheritance trees and implementation methods

• getProperty( target, name, [scope=variables], [defaultValue] ) - Get a property from an object in any scope

TestBox comes also with a nice plethora of reporters:

• ANTJunit : A specific variant of JUnit XML that works with the ANT junitreport task

• Codexwiki : Produces MediaWiki syntax for usage in Codex Wiki

• Console : Sends report to console

• Doc : Builds semantic HTML to produce nice documentation

• Dot : Builds an awesome dot report

• JSON : Builds a report into JSON

• JUnit : Builds a JUnit compliant report

• Raw : Returns the raw structure representation of the testing results

• Simple : A basic HTML reporter

• Text : Back to the 80's with an awesome text report

• XML : Builds yet another XML testing report

• Tap : A test anything protocol reporter

• Min : A minimalistic view of your test reports

• MinText : A minimalistic view of your test reports in consoles

• NodeJS : User-contributed: https://www.npmjs.com/package/testbox-runner

As you can see from our arguments for a test suite, you can pass an asyncAll argument to the describe() blocks that will allow TestBox to execute all specs in separate threads for you concurrently.

describe(title="A spec (with setup and tear-down)", asyncAll=true, body=function() {

     beforeEach(function() {
          coldbox = 22;
          application.wirebox = new coldbox.system.ioc.Injector();
     });

     afterEach(function() {
          coldbox = 0;
          structDelete( application, "wirebox" );
     });

     it("is just a function, so it can contain any code", function() {
          expect( coldbox ).toBe( 22 );
     });

     it("can have more than one expectation and talk to scopes", function() {
          expect( coldbox ).toBe( 22 );
          expect( application.wirebox.getInstance( 'MyService' ) ).toBeComponent();
     });
});

Caution Once you delve into the asynchronous world you will have to make sure your tests are also thread safe (var-scoped) and provide any necessary locking.

Running tests is essential of course. There are many ways to run your tests, we will see the basics here, and you can check out our Running Tests section in our in-depth guide.

TestBox CLI

The easiest way to run your tests is to use the TestBox CLI via the testbox run command. Ensure you are in the web root of your project or have configured the box.json to include the TestBox runner in it as shown below. If not CommandBox will try to run by convention your site + test/runner.cfm for you.

Here is a simple box.json config that has a runner and some watcher config.

"testbox":{
    "runner":"http://localhost:49616/tests/runner.cfm",
    "watchers":[
        "system/**.cfc",
        "tests/**.cfc"
    ],
    "watchDelay":"250"
}

URL Runner

Every test harness also has an HTML runner you can execute. By convention the URL is

http://localhost{port}/tests/runner.cfm

This will execute ALL tests in the tests/specs directory for you.

URL Spec Runner

You can also target a specific spec to execute via the URL

http://localhost{port}/tests/specs/MySpec.cfc

Global Runner

TestBox ships with a global runner that can run pretty much anything. You can customize it or place it wherever you need it:

Test Browser

TestBox ships with a test browser that is highly configurable to whatever URL-accessible path you want. It will then show you a test browser where you can navigate and execute not only individual tests but also directory suites.