In this section you will find the release notes for each version we release under this major version. If you are looking for the release notes of previous major versions use the version switcher at the top left of this documentation book. Here is a breakdown of our major version releases.

Version 4.0 - April 2020

In this release we have dropped support for legacy CFML engines and introduced the ability to mock data, relationships and build JSON documents.

Version 3.0

In this release we focused on dropping engine supports for legacy CFML engines. We had a major breakthrough in introducing Code Coverage thanks to the FusionReactor folks as well. This major release came also with a new UI for all reporters and streamlining the result view ports.

Version 2.0

This version spawned off with over 8 minor releases. We focused on taking TestBox 1 to yet a high level. Much more attention to detail and introducing modern paradigms like given-when-then. Multiple interception points, async executions and ability to chain methods.

Version 1.0

This was our first major version for TestBox. We had completely migrated from MXUnit and it introduced BDD to the ColdFusion (CFML) world.

TestBox BDD v4.x

TestBox is a next generation testing framework for ColdFusion (CFML) that is based on BDD (Behavior Driven Development) for providing a clean obvious syntax for writing tests. It contains not only a testing framework, runner, assertions and expectations library but also ships with MockBox, A mocking and stubbing companion. It also supports xUnit style of testing and MXUnit compatibilities.

Features At A Glance

Here are a simple listing of features TestBox brings to the table:

• BDD style testing

• xUnit style testing

• Code Coverage via FusionReactor

• Testing life-cycle methods

• MockBox integration for mocking and stubbing

• MockDataCFC for mocking json/complex data and relationships

• Ability to extend and create custom test runners

• Ability to extend and create custom test reporters

• Extensible reporters, bundled with tons of them:

  • JSON

  • XML

  • JUnit 4 XML

  • Text (80's style)

  • Console

  • TAP (Test Anything Protocol)

  • Simple HTML

  • Min - Minimalistic Heaven

  • Raw

  • CommandBox

• Asynchronous testing

• Multi-suite capabilities

• Test skipping

• Suite skipping

• Dynamic skipping support via runtime executions

• Test one or more suites exclusively

• Test one or more tests/specs exclusively

• Test labels and tagging

• Testing debug output stream

• Clickable suite titles to filter test execution

• Much more!

Versioning

TestBox is maintained under the Semantic Versioning guidelines as much as possible.Releases will be numbered with the following format:

<major>.<minor>.<patch>

And constructed with the following guidelines:

• Breaking backward compatibility bumps the major (and resets the minor and patch)

• New additions without breaking backward compatibility bumps the minor (and resets the patch)

• Bug fixes and misc changes bumps the patch

License

TestBox and MockBox are open source and licensed under the Apache 2 License. If you use them please try to make mention of it in your code or web site.

• Copyright by Ortus Solutions, Corp

• TestBox is a registered trademark by Ortus Solutions, Corp

Discussion & Help

• Help Group : https://groups.google.com/a/ortussolutions.com/forum/#!forum/testbox

• BoxTeam Slack : https://boxteam.herokuapp.com

• CFML Slack: Look for the box-products channel: http://cfml.slack.com

Reporting a Bug

We all make mistakes from time to time :) So why not let us know about it and help us out. We also love pull requests, so please star us and fork us: https://github.com/Ortus-Solutions/TestBox

• By Jira: https://ortussolutions.atlassian.net/browse/TESTBOX

Professional Open Source

TestBox is a professional open source software backed by Ortus Solutions, Corp offering services like:

• Custom Development

• Professional Support & Mentoring

• Training

• Server Tuning

• Security Hardening

• Code Reviews

• Much More

Resources

• Official Site: https://www.ortussolutions.com/products/testbox

• Current API Docs: https://apidocs.ortussolutions.com/testbox/current

• Help Group: https://community.ortussolutions.com/c/communities/testbox/11

• Source Code: https://github.com/Ortus-Solutions/TestBox

• Bug Tracker : https://ortussolutions.atlassian.net/browse/TESTBOX

• Twitter: @ortussolutions

• Facebook: https://www.facebook.com/ortussolutions

HONOR GOES TO GOD ABOVE ALL

Because of His grace, this project exists. If you don't like this, then don't read it, its not for you.

Therefore being justified by faith, we have peace with God through our Lord Jesus Christ: By whom also we have access by faith into this grace wherein we stand, and rejoice in hope of the glory of God. - Romans 5:5

TestBox 4.0.0 is a major release. It has compatibility changes that you should be aware and lots of good features!

Compatibility

The major compatibility issues are the engine support removals:

• Lucee 4.5 Support Dropped

• Adobe ColdFusion 11 Dropped

Updating

It is easy to update, just type update testbox and you are done!

Major Features

TestBox Output Utilities

Sometimes you need to dump something that is in the CFC you are testing or maybe an asynchronous test. The debug() method is only accessible from your test bundle, so getting to the TestBox output utilities is not easy. We have now implemented the testing utilities into the request scope as request.testbox

This utility struct provides you with the following methods:

This way in your code you can add them for better debugging, especially when testing async code:

request.testbox.console( "I am here" )
request.testbox.debug( "why is this not running" )

Mocking Data

We have included MockDataCFC as a dependency to TestBox 4. This will allow you to mock not only objects but data as well. You can access the mocking method via the new mockData() method in all your specs. This feature is a life-saver when mocking APIs or data within your applications.

# Array of objects
var data = mockData(
    $num = 3,
    "author" = "name",
    "id" = "uuid"
);

# An object
var data = mockData(
    $returnType = "struct",
    "author" = "name",
    "id" = "uuid"
);

Let's imagine the following object graph:

Author
    Has Many Books
        Has Many Categories
    Has Keywords
    Has A Publisher

I can then use this mocking DSL to define it:

mockData(
    fullName    = "name",
    description = "sentence",
    age         = "age",
    id          = "uuid",
    createdDate = "datetime",
    isActive	= "oneof:true: false",

    // one to many complex object definitions
    books = [
        {
            $num = "rand:1:      3",
            "id" = "uuid",
            "title" = "words:1:      5",
            "categories" = {
                "$num"      = "2",
                "id"        = "uuid",
                "category"  = "words"
            }
        }
    ],

    // object definition
    publisher = {
        "id" 	= "uuid",
        "name" 	= "sentence"
    },

    // array of values
    keywords = [
        {
            "$num" 	= "rand:1:      10",
            "$type" = "words"
        }
    ]
);

Release Notes

Bugs

• [TESTBOX-275] - Exception in beforeTests/afterTests not reported in a meaningful way on the ANT Junit Reporter

• [TESTBOX-278] - Fix the coverage % in HTML visualizer

New Features

• [TESTBOX-274] - New testbox output utilities struct: request.testbox

• [TESTBOX-276] - MockdataCFC is now a first class module in TestBox

• [TESTBOX-277] - New mockData() method in your base specs so you can mock any type of data

• [TESTBOX-280] - Add cfconfig.json for controlling output and consistency between testing in diff engines

Tasks

• [TESTBOX-271] - Drop ACF11 Support

• [TESTBOX-273] - Drop old mxunit decorator for ORM Transaction Decorator

TestBox v4.3.0 includes several new features and bug fixes. Here are the release notes and major areas of improvements.

New Text Reporters

This release has revamped the text and mintext reporters. Check them out by adding reporter=text or reporter=mintext when running your tests.

Editor URL Integrations

The simple reporter has been updated to add editor opening integration. Now all the tag contexts and traces are links and will open in your favorite IDE of choice. You can control the editor by using the url.editor parameter in the URL or via your runner.

Release Notes

Bugs

Improvements

New Features

The source code for this book is hosted in GitHub: . You can freely contribute to it and submit pull requests. The contents of this book is copyright by and cannot be altered or reproduced without author's consent. All content is provided "As-Is" and can be freely distributed.

• The majority of code examples in this book are done in cfscript.

• The majority of code generation and running of examples are done via CommandBox: The ColdFusion (CFML) CLI, Package Manager, REPL -

• All ColdFusion examples designed to run on the open soure Railo Platform or Adobe ColdFusion 9.0.2+

External Trademarks & Copyrights

Flash, Flex, ColdFusion, and Adobe are registered trademarks and copyrights of Adobe Systems, Inc.

Notice of Liability

The information in this book is distributed “as is”, without warranty. The author and Ortus Solutions, Corp shall not have any liability to any person or entity with respect to loss or damage caused or alleged to be caused directly or indirectly by the content of this training book, software and resources described in it.

Contributing

We highly encourage contribution to this book and our open source software. The source code for this book can be found in our where you can submit pull requests.

Charitable Proceeds

15% of the proceeds of this book will go to charity to support orphaned kids in El Salvador - . So please donate and purchase the printed version of this book, every book sold can help a child for almost 2 months.

Shalom Children's Home

Shalom now cares for over 80 children in El Salvador, from newborns to 18 years old. They receive shelter, clothing, food, medical care, education and life skills training in a Christian environment. The home is supported by a child sponsorship program.

We have personally supported Shalom for over 6 years now; it is a place of blessing for many children in El Salvador that either have no families or have been abandoned. This is good earth to seed and plant.

TestBox v4.1.0 is a minor update. Quickly update it via CommandBox

Bugs

• [] - request.testbox: Component ... has no accessible Member with name [$TESTID]

• [] - Fix type on test results for bundlestats

• [] - DebugBuffer was being removed instead of resetting to empty for getMemento

New Features

• [] - Added cfml engine and version as part of the test results as properties

• [] - Update all reporters so they can just build and return the report with no content type or context repsonse resets

• [] - make buildReporter public in the testbox core

TestBox is a xUnit style and behavior-driven development (BDD) testing framework for CFML (ColdFusion). It does not depend on any other framework for operation and it has a clean and descriptive syntax that will make you giggle when writing your tests. It ships with not only the xUnit and BDD testing capabilities, but also an assertions and expectations library, several different runners, reporters and MockBox, for mocking and stubbing capabilities.

In this section we will go over the major features of our behavior driven development syntax. Let's begin.

TestBox relies on the fact of creating testing bundles which are basically CFCs. A bundle CFC will hold all the suites and specs a TestBox runner will execute and produce reports on. Don't worry, we will cover what's a suite and a spec as well.

component extends="testbox.system.BaseSpec"{

     // executes before all suites
     function beforeAll(){}

     // executes after all suites
     function afterAll(){}

     // All suites go in here
     function run( testResults, testBox ){

     }

}

This bundle CFC can contain 2 life-cycle functions and a single run() function where you will be writing your test suites and specs.

Life-Cycle Methods

The beforeAll() and afterAll() methods are called life-cycle methods. They will execute once before the run() function and once after the run() function. This is a great way to do any kind of global setup or tear down in your tests.

Execution

The run() function receives the TestBox testResults object as a reference and testbox as a reference as well. This way you can have metadata and access to what will be reported to users in a reporter. You can also use it to decorate the results or store much more information that can be picked up later by reports. You also have access to the testbox class so you can see how the test is supposed to execute, what labels was it passed, directories, options, etc.

Fixed

• TESTBOX-336 When using testSpecs or testSuites via non browser executions decoding is not working

• TESTBOX-330 Catch spec errors with no type

• TESTBOX-328 Lucee 5.3.8 reMatchNoCase() change breaks expected execptions

• TESTBOX-327 Code Coverage in CI only reports on a single file rather than all files

• TESTBOX-323 Testbox refuses to update to current release

• TESTBOX-322 Junit reports issues with describe containing "/"

Added

• Migration to github actions

• TESTBOX-332 toBe{Type} is incomplete

• TESTBOX-329 Full Null support

Before and After Specs

If you are familiar with xUnit style frameworks, the majority of them provide a way to execute functions before and after every single test case or spec in BDD. This is a great way to keep your tests DRY (Do not repeat yourself)! TestBox provides the beforeEach() and afterEach() methods that each take in a closure as their argument that receive the name of the spec that's about to be executed or just executed. As their names indicate, they execute before a spec and after a spec in a related describe block.

describe("A spec (with setup and tear-down)", function() {

     beforeEach(function( currentSpec ) {
          coldbox = 22;
          application.wirebox = new coldbox.system.ioc.Injector();
     });

     afterEach(function( currentSpec ) {
          coldbox = 0;
          structDelete( application, "wirebox" );
     });

     it("is just a function, so it can contain any code", function() {
          expect( coldbox ).toBe( 22 );
     });

     it("can have more than one expectation and talk to scopes", function() {
          expect( coldbox ).toBe( 22 );
          expect( application.wirebox.getInstance( 'MyService' ) ).toBeComponent();
     });
});

Around Specs

The aroundEach() life-cycle method will completely wrap your spec in another closure. This is an elegant way for you to provide a complete around AOP advice to a specification. You can use it to surround the execution in transaction blocks, ORM rollbacks, logging, and so much more. This life-cycle method will decorate ALL specs within a single suite and any children suites. The method signature is below:

aroundEach( function( spec, suite ){
     // execute the spec
     arguments.spec.body();
} );

The method receives a structure of data representing the spec. This contains the following elements:

• body : The actual closure for the spec that you will use to execute within it.

• labels : The labels used in the spec

• name : The name of the spec

• order : The order of execution of the spec

• skip : The skip flag or closure that determines if the spec runs

The method also receives a structure of metadata about the suite this spec is contained in. It has information about life-cycle closures, async information, names, parents, etc. Here is a very practical example of creating and around each closure to provide rollbacks for specs:

aroundEach( function( spec, suite ){

     transaction action="begin"{
          try{/
               // execute the spec
               arguments.spec.body();
          } catch( Any e ){
               rethrow;
          } finally{
               transaction action="rollback";
          }
     }
} );

This simple around each life-cycle closure will rollback ALL my spec's executions even if they throw exceptions.

Annotations

TestBox since v2.3.0 also allows you to declare life-cycle methods via annotations. Please see our Annotations section for more information.

Lifecycle Nesting Order

When you use beforeEach(), afterEach(), and aroundEach() at the same time, there is a specific order they fire in. For a given describe block, they will fire in this order. Remember, aroundEach() is split into two parts-- the half of the method before you call spec.body() and the second half of the method.

1. beforeEach

2. aroundEach (first half)

3. it() (the spec.body() call)

4. aroundEach (second half)

5. afterEach()

Here's an example:

describe( 'my describe', function(){
	
    beforeEach( function( currentSpec ){
        // I run first
    } );
    	 
    aroundEach( function( spec, suite ){
        // I run second
        arguments.spec.body();
        // I run fourth
    });
    
    afterEach( function( currentSpec ){
        // I run fifth
    } );
    
    it( 'my it', function(){
        // I run third
    } );
    
} );

If there are more than one it() blocks, the process repeats for each one. Steps 1, 2, 4, 5 will wrap every single it().

When you nest more than one describe block inside the other, the before/around/after order is the same but drills down to the innermost describe and then bubbles back up. That means the outermost beforeEach() starts and we end on the outermost afterEach().

Here's what an example flow would look like that had before/after/around specified in two levels of describes with a single it() in the inner most describe.

1. Outermost beforeEach() call

2. Innermost beforeEach() call

3. Outermost aroundEach() call (first half)

4. Innermost aroundEach() call (first half)

5. The it() block

6. Innermost aroundEach() calls (second half)

7. Outermost aroundEach() call (second half)

8. Innermost afterEach() call

9. Outermost afterEach() call

This works regardless of the number of levels and can obviously have many permutations, but the basic order is still the same. Before/around/after and starting at the outside working in, and back out again. This process happens for every single spec or it() block. This is as opposed to the beforeAll() and afterAll() method which only run once for the entire CFC regardless of how many specs there are.

With TestBox's BDD syntax, it is possible to create suites dynamically; however, there are a few things to be aware of.

Setup for dynamic suites must be done in the pseudo-constructor (versus in beforeAll()). This is because variables-scoped variables set in beforeAll() are not available in the describe closure (even though they are available in it closures). This behavior can be explained by the execution sequence of a BDD bundle: When the bundle's run() method is called, it collects preliminary test data via describes. After preliminary test data are collected, the beforeAll() runs, followed by the describe closures.

Additionally, care must be taken to pass data into the it closures, otherwise strange behavior will result (the values from the last loop iteration will be repeated in the body of each looped it).

Example

The following bundle creates suites dynamically, by looping over test metadata.

component
    extends="testbox.system.BaseSpec"
    hint="This is an example of a TestBox BDD test bundle containing dynamically-defined suites."
{

    /*
    * Need to do config for *dynamic* test suites here, in the
    * pseudo-constructor, versus in `beforeAll()`.
    */
    doDynamicSuiteConfig();

    /*
    * @hint This method is arbitrarily named, but it sets up 
    * metadata needed by the dynamic suites example. The setup
    * could have been done straight in the pseudo-constructor,
    * but it might be nice to organize it into such a method
    * as this.
    */
    function doDynamicSuiteConfig(){
        variables.dynamicSuiteConfig = ["foo","bar","baz"];
    }

    function run( testResults, testBox ){

        /*
        * @hint Dynamic Test Suites Example
        */
        // loop over test metadata
        for ( var thing in dynamicSuiteConfig ) {
            describe("Dynamic Suite #thing#", function(){
                // notice how data is passed into the it() closure:
                //  * data={ keyA=valueA, keyB=ValueB }
                //  * function( data )
                it( title=thing & "test", 
                    data={ thing=thing }, 
                    body=function( data ) {
                      var thing = data.thing;
                      expect( thing ).toBe( thing );
                });
            });
        }

    }

}

Specs and suites can be tagged with TestBox labels. Labels allows you to further categorize different specs or suites so that when a runner executes with labels attached, only those specs and suites will be executed, the rest will be skipped. You can alternatively choose to skip specific labels when a runner executes with excludes attached.

describe(title="A spec", labels="stg,railo", body=function() {
     it("executes if its in staging or in railo", function() {
          coldbox = 0;
          coldbox++;

          expect( coldbox ).toBe( 1 );
     });
});

describe("A spec", function() {
     it("is just a closure, so it can contain any code", function() {
          coldbox = 0;
          coldbox++;

          expect( coldbox ).toBe( 1 );
     });

     it(title="can have more than one expectation and labels", labels="dev,stg,qa,shopping", body=function() {
          coldbox = 0;
          coldbox++;

          expect( coldbox ).toBe( 1 );
          expect( coldbox ).toBeTrue();
     });
});

Expectations are self-concatenated strings that evaluate an actual value to an expected value or condition. These are initiated by the global TestBox method called expect() which takes in a value called the actual value or expectAll() which takes in an array or struct which will be the actual value. It is concatenated in our expectations DSL with a matcher function that will most likely take in an expected value or condition to test. You can also concatenate the matchers and do multiple evaluations on a single actual value.

Matchers

Each matcher implements a comparison or evaluation of the actual value and an expected value or condition. It is responsible for either passing or failing this evaluation and reporting it to TestBox. Each matcher also has a negative counterpart assertion by just prefixing the call to the matcher with a not expression.

function run(){

     describe("The 'toBe' matcher evaluates equality", function(){
          it("and has a positive case", function(){
               expect( true ).toBe( true );
          });

          it("and has a negative case", function(){
               expect( false ).notToBe( true );
          });
     });

     describe("Collection expectations", function(){
        it( "can be done easily with TestBox", function(){
            expectAll( {a:2,b:4,c:6} ).toSatisfy( function(x){ return 0 == x%2; });
        });
    });

}

Included Matchers

TestBox has a plethora (That's Right! I said Plethora) of matchers that are included in TestBox. The best way to see all the latest matchers is to visit our API and digest the testbox.system.Expectation class. There is also the ability to register and write custom matchers in TestBox via our addMatchers() function at runtime.

describe("Some Included TestBox Matchers:", function() {

     describe("The 'toBe' matcher", function() {
          it("works for simple values", function() {
               coldbox = 1;
               expect( coldbox )
                    .toBe( 1 )
                    .notToBe( 5 );
          });

          it("works for complex values too (arrays,structs,queries,objects)", function() {
               expect( [1,2] ).toBe( [1,2] );
               expect( { name="luis", awesome=true} ).toBe( { awesome=true, name="luis" } );
               expect( this ).toBe( this );
               expect( queryNew("") ).toBe( queryNew("") );
          });
     });

     it("The 'toBeWithCase' matches strings with case equality", function() {
          expect( 'hello' )
               .toBeWithCase( 'hello' )
               .notToBeWithCase( 'HELLO' );
     });

     it("The 'toBeTrue' and 'toBeFalse' matchers are used for boolean operations", function() {
          coldbox_rocks = true;
          expect( coldbox_rocks ).toBeTrue();
          expect( !coldbox_rocks ).toBeFalse();
     });

     it("The 'toBeNull' expects null values", function() {
          foo = "bar";
          expect( javaCast("null", "") ).toBeNull();
          expect( foo ).notToBeNull();
     });

     it("The 'toBeInstanceOf' expects the object to be of the same class or inheritance or implementation", function() {
          expect( new coldbox.system.Assertions ).toBeInstanceOf( 'coldbox.system.Assertions' );
          expect( this ).notToBeInstanceOf( 'coldbox.system.MockBox' );
     });

     it("The 'toMatch' matcher is for regular expressions with case sensitivity", function() {
          message = 'foo man choo';

          expect( message )
               .toMatch( '^foo' )
               .toMatch( '(man)' )
               .notToMatch( 'superman' );
     });

     it("The 'toMatchNoCase' matcher is for regular expressions with no case sensitivity", function() {
          message = 'foo man choo';

          expect( message )
               .toMatchNoCase( '^FOO' )
               .toMatchNoCase( '(MAN)' )
               .notToMatchNoCase( 'SuperMan' );
     });

     describe("The 'toBeTypeOf' matcher evaluates using the CF isValid() function", function() {
          it("works with direct calls", function() {
               expect( [1,2] ).toBeTypeOf( 'Array' );
               expect( { name="luis", awesome=true} ).toBeTypeOf( 'struct' );
               expect( this ).toBeTypeOf( 'component' );
               expect( '03/01/1990' ).toBeTypeOf( 'usdate' );
          });

          it("works with dynamic calls as well", function() {
               expect( [1,2] ).toBeArray();
               expect( { name="luis", awesome=true} ).toBeStruct();
               expect( this ).toBeComponent();
               expect( '03/01/1990' ).toBeUsDate();
          });
     });

     it("The 'toBeEmpty' checks emptyness of simple or complex values", function() {
          expect( [] ).toBeEmpty();
          expect( { name="luis", awesome=true} ).notToBeEmpty();
          expect( '' ).toBeEmpty();
     });

     it("The 'toHaveLength' checks size of simple or complex values", function() {
          expect( [] ).toHaveLength( 0 );
          expect( { name="luis", awesome=true} ).toHaveLength( 2 );
          expect( 'Hello' ).toHaveLength( 5 );
     });

     it("The 'toHaveKey' checks for existence of keys in structs", function() {
          expect( { name="luis", awesome=true} ).toHaveKey( 'name' );
     });

     it("The 'toHaveDeepKey' checks for existence of keys anywhere in structs", function() {
          expect( { name="luis", { age=35, awesome=true } } ).toHaveDeepKey( 'age' );
     });

     it("The 'toThrow' checks if the actual call fails", function() {
          expect( function(){
               new calculator().divide( 40, 0 );
          }).toThrow();

          expect( function(){
               new calculator().divide( 40, 0 );
          }).toThrow( regex="zero" );
     });

});

Custom Matchers

You can also build and register custom matchers. Please visit the Custom Matchers chapter to read more about custom matchers.

A spec is a declaration that will usually test your system with a requirement. They are defined by calling the TestBox it() global function, which takes in a title and a body function/closure. The title is the title of this spec you will write and the body function/closure is a block of code that represents the spec.

A spec will contain most likely one or more expectations that will test the state of the SUT (software under test) or sometimes referred to as code under test. In BDD style, your specifications are what is used to validate your requirements of a scenario which is your describe() block of your story.

function run(){

     describe("A suite", function(){
          it("contains spec with an awesome expectation", function(){
               expect( true ).toBeTrue();
          });

          it("contains a spec with more than 1 expectation", function(){
               expect( [1,2,3] ).toBeArray();
               expect( [1,2,3] ).toHaveLength( 3 );
          });
     });

}

An expectation is a nice assertion DSL that TestBox exposes so you can pretty much read what should happen in the testing scenario. A spec will pass if all expectations pass. A spec with one or more expectations that fail will fail the entire spec.

Arguments

They are closures Ma!

Since the implementations of the describe() and it() functions are closures, they can contain executable code that is necessary to implement the test. All CFML rules of scoping apply to closures, so please remember them. We recommend always using the variables scope for easy access and distinction.

function run(){

     describe("A suite is a closure", function(){
          c = new Calculator();

          it("and so is a spec", function(){
               expect( c ).toBeTypeOf( 'component' );
          });
     });

}

Spec Data Binding

The data argument can be used to pass in a structure of data into the spec so it can be used later within the body closure. This is great when doing looping and creating dynamic closure calls:

// Simple Example
it( title="can handle binding", body=function( data ){
    expect(    data.keep ).toBeTrue();
}, data={ keep = true } );

// Complex Example. Let's iterate over a bunch of files and create dynamic specs
for( var filePath in files ){

  it( 
    title="#getFileFromPath( filePath )# should be valid JSON", 
    // pass in a struct of data to the spec for later evaluation
    data={ filePath = filePath },
    // the spec closure accepts the data for later evaluation
    body=function( data ) {
      var json = fileRead( data.filePath );
      var isItJson = isJSON( json );

      expect( json ).notToBeEmpty();
      expect( isItJson ).toBeTrue();

      if( isItJson ){
          var jsonData = deserializeJSON(json);
          if( getFileFromPath( filePath ) != "index.json"){
              expect( jsonData ).toHaveKey( "name" );
              expect( jsonData ).toHaveKey( "type" );
          }
      }

  });
}

TestBox is a next generation testing framework for ColdFusion (CFML) that is based on BDD (Behavior Driven Development) for providing a clean obvious syntax for writing tests. It contains not only a testing framework, runner, assertions, mocking/stubbing and expectations library. It also supports xUnit style of testing with MXUnit compatibilities.

Useful Resources

• Approaches to Mocking

• Wikipedia Mock Objects

• Using mock objects for complex unit tests IBM developerWorks

• Unit testing with mock objects IBM developerWorks

• Emergent Design by Scott Bain

• Mocks Aren't Stubs by Martin Fowler

Ref Cards

Our RefCards will get you up and running in no time.

BDD RefCard

xUnit RefCard

Luis Fernando Majano Lainez

Luis Majano is a Computer Engineer with over 16 years of software development and systems architecture experience. He was born in San Salvador, El Salvador in the late 70’s, during a period of economical instability and civil war. He lived in El Salvador until 1995 and then moved to Miami, Florida where he completed his Bachelors of Science in Computer Engineering at Florida International University. Luis resides in The Woodlands, Texas with his beautiful wife Veronica, baby girl Alexia and baby boy Lucas!

He is the CEO of Ortus Solutions, a consulting firm specializing in web development, ColdFusion (CFML), Java development and all open source professional services under the ColdBox and ContentBox stack. He is the creator of ColdBox, ContentBox, WireBox, MockBox, LogBox and anything “BOX”, and contributes to many open source ColdFusion projects. He is also the Adobe ColdFusion user group manager for the Inland Empire. You can read his blog at www.luismajano.com

Luis has a passion for Jesus, tennis, golf, volleyball and anything electronic. Random Author Facts:

• He played volleyball in the Salvadorean National Team at the tender age of 17

• The Lord of the Rings and The Hobbit is something he reads every 5 years. (Geek!)

• His first ever computer was a Texas Instrument TI-86 that his parents gave him in 1986. After some time digesting his very first BASIC book, he had written his own tic-tac-toe game at the age of 9. (Extra geek!)

• He has a geek love for circuits, microcontrollers and overall embedded systems.

• He has of late (during old age) become a fan of running and bike riding with his family.

Keep Jesus number one in your life and in your heart. I did and it changed my life from desolation, defeat and failure to an abundant life full of love, thankfulness, joy and overwhelming peace. As this world breathes failure and fear upon any life, Jesus brings power, love and a sound mind to everybody!

“Trust in the LORD with all your heart, and do not lean on your own understanding.” Proverbs 3:5

Contributors

Jorge Emilio Reyes Bendeck

Jorge is an Industrial and Systems Engineer born in El Salvador. After finishing his Bachelor studies at the Monterrey Institute of Technology and Higher Education ITESM, Mexico, he went back to his home country where he worked as the COO of Industrias Bendek S.A.. In 2012 he left El Salvador and moved to Switzerland in persuit of the love of his life. He married her and today he resides in Basel with his lovely wife Marta and their daughter Sofía.

Jorge started working as project manager and business developer at Ortus Solutions, Corp. in 2013, . At Ortus he fell in love with software development and now enjoys taking part on software development projects and software documentation! He is a fellow Cristian who loves to play the guitar, worship and rejoice in the Lord!

Therefore, if anyone is in Christ, the new creation has come: The old has gone, the new is here! 2 Corinthians 5:17

Specs and suites can be skipped from execution by prefixing certain functions with the letter x or by using the skip argument in each of them. The reporters will show that these suites or specs where skipped from execution. The functions you can prefix are:

• it()

• describe()

• story()

• given()

• when()

• then()

• feature()

Skip Argument

The skip argument can be a boolean value or a closure. If the value is true then the suite or spec is skipped. If the return value of the closure is true then the suite or spec is skipped. Using the closure approach allows you to dynamically at runtime figure out if the desired spec or suite is skipped. This is such a great way to prepare tests for different CFML engines.

TestBox v4.4.0 includes several new features and bug fixes. Here are the release notes and major areas of improvements.

Release Notes

Bugs

Runner tries to instantiate abstract classes

Fix HTTP Status Headers Being Removed By Reporters when resetting html head

Chaining "not" matchers before regular matchers doesn't work correctly

Coverage output doesn't escape ending script tag

ConsoleReporter fails with missing functions in assets/text

No matching function [SPACE] found

CF error variable [THISBUNDLE] doesn't exist when running tests

Improvements

Full Null Support Some items of array can be NULL

text and min text white space management

notToBeBetween seems to be the same as toBeBetween

As we have seen before, the describe() function describes a test suite of related specs in a test bundle CFC. The title of the suite is concatenated with the title of a spec to create a full spec's name which is very descriptive. If you name them well, they will read out as full sentences as defined by style.

Nesting describe Blocks

Calls to our describe() function can be nested with specs at any level or point of execution. This allows you to create your tests as a related tree of nested functions. Please note that before a spec is executed, TestBox walks down the tree executing each beforeEach() and afterEach() function in the declared order. This is a great way to logically group specs in any level as you see fit.

As you can see from our arguments for a test suite, you can pass an asyncAll argument to the describe() blocks that will allow TestBox to execute all specs in separate threads for you concurrently.

Caution Once you delve into the asynchronous world you will have to make sure your tests are also thread safe (var-scoped) and provide any necessary locking.

TestBox ships with several test runners internally but we have tried to simplify and abstract it with our testbox object which can be found in the testbox.system package. The TestBox object allows you to execute tests from a variety of entry points and languages such as CFC, CFM, HTTP, NodeJS, SOAP or REST. You can also make your CFC's extend from our BaseSpec class so you can execute it directly via the URL. The main execution methods are:

Info We encourage you to read the API docs included in the distribution for the complete parameters for each method.

run() Arguments

Here are the arguments you can use for initializing TestBox or executing the run() method

runRemote() arguments

Here are the arguments you can use for executing the runRemote() method:

The bundles argument which can be a single CFC path or an array of CFC paths or a directory argument so it can go and discover the test bundles from that directory. The reporter argument can be a core reporter name like: json,xml,junit,raw,simple,dots,tap,min,etc or it can be an instance of a reporter CFC. You can execute the runners from any cfm template or any CFC or any URL, that is up to you.

Global Runner

TestBox ships with a global runner that can be used to run pretty much anything. You can customize it or place it wherever you need it:

Test Browser

TestBox ships with a test browser that is highly configurable to whatever URL accessible path you want. It will then show you a test browser where you can navigate and execute not only individual tests, but also directory suites as well.

ANT Runner

In our test samples and templates we include an ANT runner that will be able to execute your tests via ANT. It can also leverage our ANTJunit reporter to use the junitreport task to produce JUnit compliant reports as well. You can find this runner in the test samples and runner template directory.

Bundle(s) Runner

Test Runner

You can also pass the following arguments to the method via the URL:

• testSuites: A list or array of suite names that are the ones that will be executed ONLY!

• testSpecs: A list or array of test names that are the ones that will be executed ONLY!

Directory Runner

SOAP Runner

You can run tests via SOAP by leveraging the runRemote() method. The WSDL URL will be

HTTP/REST Runner You can run tests via HTTP/REST by leveraging the runRemote() endpoint. The URL will be

NodeJS Runner

There is a user-contributed NodeJS Runner that looks fantastic and can be downloaded here:

Just use node to install:

Please refer to our section to take advantage of all the mocking and stubbing you can do. However, every BDD TestBundle has the following functions available to you for mocking and stubbing purposes:

• makePublic( target, method, newName ) - Exposes private methods from objects as public methods

• querySim( queryData ) - Simulate a query

• getMockBox( [generationPath] ) - Get a reference to MockBox

• createEmptyMock( [className], [object], [callLogging=true]) - Create an empty mock from a class or object

• createMock( [className], [object], [clearMethods=false], [callLogging=true]) - Create a spy from an instance or class with call logging

• prepareMock( object, [callLogging=true]) - Prepare an instance of an object for method spies with call logging

• createStub( [callLogging=true], [extends], [implements]) - Create stub objects with call logging and optional inheritance trees and implementation methods

• getProperty( target, name, [scope=variables], [defaultValue] ) - Get a property from an object in any scope

A test suite in TestBox is a collection of specifications that will model what you want to test. The way the suite is expressed can be of many different types as we will investigate.

Test suite is a container that has a set of tests which helps testers in executing and reporting the test execution status.

A test suite begins with a call to our TestBox describe() function with at least two arguments: a title and a body function/closure. The title is the name of the suite to register and the body function/closure is the block of code that implements the suite.

When applying BDD to your tests, this function is used to describe your story scenarios that you will implement.

Arguments

There are more arguments which you can see below:

TestBox comes also with a nice plethora of reporters:

• ANTJunit : A specific variant of JUnit XML that works with the ANT junitreport task

• Codexwiki : Produces MediaWiki syntax for usage in Codex Wiki

• Console : Sends report to console

• Doc : Builds semantic HTML to produce nice documentation

• Dot : Builds an awesome dot report

• JSON : Builds a report into JSON

• JUnit : Builds a JUnit compliant report

• Raw : Returns the raw structure representation of the testing results

• Simple : A basic HTML reporter

• Text : Back to the 80's with an awesome text report

• XML : Builds yet another XML testing report

• Tap : A test anything protocol reporter

• Min : A minimalistic view of your test reports

• MinText : A minimalistic view of your test reports in consoles

• NodeJS : User-contributed:

Specs and suites can be focused so ONLY those suites and specs execute. You will do this by prefixing certain functions with the letter f or by using the focused argument in each of them. The reporters will show that these suites or specs where execute ONLY The functions you can prefix are:

• it()

• describe()

• story()

• given()

• when()

• then()

• feature()

is a style of writing tests where you describe the state of the code you want to test (Given), the behavior you want to test (When) and the expected outcome (Then). (See )

Testbox supports the use of function names given() and when() in-place of describe() function calls. The then() function call is an alternative for it() function calls. The advantage of this style of is that you can gather your requirements and write your tests in a common language that can be understood by developers and stake-holders alike. This common language format is often referred to as the language; using it we can gather and document the requirements as:

TestBox provides you with feature(), scenario() and story() wrappers for describe() blocks. As such we can write our requirements in test form like so:

The output from running the test will read as the original requirements, providing you with not only automated tests but also a living document of the requirements in a business-readable format.

As feature(), scenario() and story() are wrappers for describe() you can intermix them so that your can create tests which read as the business requirements. As with describe(), they can be nested to build up blocks.

TestBox can be installed easily via or by your manually implementing it.

Commandbox installation

CommandBox is our preferred approach for all package installations, updates and more.

This will install TestBox in a /testbox folder from where you called the command.

Manual Installation

You can also download TestBox from . Extract the downloaded zip and place it anywhere you like. You can either create a mapping called _/testbox_that points to testbox in the distribution folder, which is the most secure approach, or you can just place it in the webroot if you like. You can also create an application-level mapping:

You could also clone Testbox from and help us out:

Generating a testing harness

Once you have testbox installed, you'll need a quick way to set up a testing harness. The generate harness command will add a new /tests folder to your application with a few example tests to get you started.

System Requirements

• Lucee 5.x+

• ColdFusion 2016+

What's Included

The download structure includes:

• apidocs : The API docs for TestBox

• system : The main system framework folder

• test-browser : This is a little utility to facilitate navigating big testing suites. This helps navigate to the suites you want and execute them instead of typing all the time.

• test-harness : A simplified version of the TestBox runner that can be placed anywhere in your application. It includes an ANT build that allows you to execute your tests and produce results via ANT and also JUnit compliant reports via the junitreport task.

• test-runner : The pre-built TestBox Global Runner. You can pass in a directory mapping or bundles and select labels and boom run the tests.

• test-visualizer: A static visualizer of json reports. Just drop in a test-results.json and run it!

• tests : Several sample tests and runners which actually are used to build TestBox

The only required folder is system and it does not need to be web-accessible. The other folders are all optional tools to help you that expect to be web-accessible. You can safely remove everything except the system folder if you want to build your own test browsers. If you are placing TestBox outside of your web root, the /testbox mapping needs to point to the parent folder containing the system directory as all TestBox models start with testbox.system in their package path.

Assertions are self-concatenated strings that evaluate an actual value to an expected value or condition. These are initiated by the global TestBox variable called $assert which contains tons of included assertion methods so you can evaluate your tests.

Evaluators

Each assertion evaluator will compare the actual value and an expected value or condition. It is responsible for either passing or failing this evaluation and reporting it to TestBox. Each evaluator also has a negative counterpart assertion by just prefixing the call to the method with a not expression.

      function testIncludes(){
          $assert.includes( "hello", "HE" );
          $assert.includes( [ "Monday", "Tuesday" ] , "monday" );
     }

     function testNotIncludes(){
          $assert.notIncludes( "hello", "what" );
          $assert.notIncludes( [ "Monday", "Tuesday" ] , "Friday" );
     }

Included Evaluators

TestBox has a plethora (That's Right! I said Plethora) of evaluators that are included in the release. The best way to see all the latest evaluator methods is to visit our API and digest the coldbox.system.Assertion class. There is also the ability to register and write custom assertion evaluators in TestBox via our addAssertions() function.

component displayName="TestBox xUnit suite for CF9" labels="railo,cf"{

/*********************************** LIFE CYCLE Methods ***********************************/

     function beforeTests(){
          application.salvador = 1;
     }

     function afterTests(){
          structClear( application );
     }

     function setup(){
          request.foo = 1;
     }

     function teardown(){
          structClear( request );
     }

/*********************************** Test Methods ***********************************/

     function testIncludes(){
          $assert.includes( "hello", "HE" );
          $assert.includes( [ "Monday", "Tuesday" ] , "monday" );
     }

     function testIncludesWithCase(){
          $assert.includesWithCase( "hello", "he" );
          $assert.includesWithCase( [ "Monday", "Tuesday" ] , "Monday" );
     }

     function testnotIncludesWithCase(){
          $assert.notincludesWithCase( "hello", "aa" );
          $assert.notincludesWithCase( [ "Monday", "Tuesday" ] , "monday" );
     }

     function testNotIncludes(){
          $assert.notIncludes( "hello", "what" );
          $assert.notIncludes( [ "Monday", "Tuesday" ] , "Friday" );
     }

     function testIsEmpty(){
          $assert.isEmpty( [] );
          $assert.isEmpty( {} );
          $assert.isEmpty( "" );
          $assert.isEmpty( queryNew("") );
     }

     function testIsNotEmpty(){
          $assert.isNotEmpty( [1,2] );
          $assert.isNotEmpty( {name="luis"} );
          $assert.isNotEmpty( "HelloLuis" );
          $assert.isNotEmpty( querySim( "id, name
               1 | luis") );
     }

     function testSkipped() skip{
          $assert.fail( "This Test should fail" );
     }

     boolean function isRailo(){
          return structKeyExists( server, "railo" );
     }

     function testSkippedWithConstraint() skip="isRailo"{
          $assert.fail( "This Test should fail" );
     }

     function testFails(){
          //$assert.fail( "This Test should fail" );
     }

     function testFailsShortcut() labels="railo"{
          //fail( "This Test should fail" );
     }

     function testAssert() {
          $assert.assert( application.salvador == 1 );
     }

     function testAssertShortcut() {
          assert( application.salvador == 1 );
     }

     function testisTrue() {
          $assert.isTrue( 1 );
     }

     function testisFalse() {
          $assert.isFalse( 0 );
     }

     function testisEqual() {
          $assert.isEqual( 0, 0 );
          $assert.isEqual( "hello", "HEllO" );
          $assert.isEqual( [], [] );
          $assert.isEqual( [1,2,3, {name="hello", test="this"} ], [1,2,3, {test="this", name="hello"} ] );
     }

     function testisNotEqual() {
          $assert.isNotEqual( this, new coldbox.system.MockBox() );
          $assert.isNotEqual( "hello", "test" );
          $assert.isNotEqual( 1, 2 );
          $assert.isNotEqual( [], [1,3] );
     }

     function testisEqualWithCase() {
          $assert.isEqualWithCase( "hello", "hello" );
     }

     function testnullValue() {
          $assert.null( javaCast("null", "") );
     }

     function testNotNullValue() {
          $assert.notNull( 44 );
     }

     function testTypeOf() {
          $assert.typeOf( "array", [ 1,2 ] );
          $assert.typeOf( "boolean", false );
          $assert.typeOf( "component", this );
          $assert.typeOf( "date", now() );
          $assert.typeOf( "time", timeformat( now() ) );
          $assert.typeOf( "float", 1.1 );
          $assert.typeOf( "numeric", 1 );
          $assert.typeOf( "query", querySim( "id, name
               1 | luis") );
          $assert.typeOf( "string", "hello string" );
          $assert.typeOf( "struct", { name="luis", awesome=true } );
          $assert.typeOf( "uuid", createUUID() );
          $assert.typeOf( "url", "http://www.coldbox.org" );
     }

     function testNotTypeOf() {
          $assert.notTypeOf( "array", 1 );
          $assert.notTypeOf( "boolean", "hello" );
          $assert.notTypeOf( "component", {} );
          $assert.notTypeOf( "date", "monday" );
          $assert.notTypeOf( "time", "1");
          $assert.notTypeOf( "float", "Hello" );
          $assert.notTypeOf( "numeric", "eeww2" );
          $assert.notTypeOf( "query", [] );
          $assert.notTypeOf( "string", this );
          $assert.notTypeOf( "struct", [] );
          $assert.notTypeOf( "uuid", "123" );
          $assert.notTypeOf( "url", "coldbox" );
     }

     function testInstanceOf() {
          $assert.instanceOf( new coldbox.system.MockBox(), "coldbox.system.MockBox" );
     }

     function testNotInstanceOf() {
          $assert.notInstanceOf( this, "coldbox.system.MockBox" );
     }

     function testMatch(){
          $assert.match( "This testing is my test", "(TEST)$" );
     }

     function testMatchWithCase(){
          $assert.match( "This testing is my test", "(test)$" );
     }

     function testNotMatch(){
          $assert.notMatch( "This testing is my test", "(hello)$" );
     }

     function testKey(){
          $assert.key( {name="luis", awesome=true}, "awesome" );
     }

     function testNotKey(){
          $assert.notKey( {name="luis", awesome=true}, "test" );
     }

     function testDeepKey(){
          $assert.deepKey( {name="luis", awesome=true, parent = { age=70 } }, "age" );
     }

     function testNotDeepKey(){
          $assert.notDeepKey( {name="luis", awesome=true, parent = { age=70 } }, "luis" );
     }

     function testLengthOf(){
          $assert.lengthOf( "heelo", 5 );
          $assert.lengthOf( [1,2], 2 );
          $assert.lengthOf( {name="luis"}, 1 );
          $assert.lengthOf( querySim( "id, name
               1 | luis"), 1 );

     }

     function testNotLengthOf(){
          $assert.notLengthOf( "heelo", 3 );
          $assert.notLengthOf( [1,2], 5 );
          $assert.notLengthOf( {name="luis"}, 5 );
          $assert.notLengthOf( querySim( "id, name
               1 | luis"), 0 );

     }

     // railo 4.1+ or CF10+
      function testThrows(){
          $assert.throws(function(){
               var hello = invalidFunction();
          });
     }

      // railo 4.1+ or CF10+
     function testNotThrows(){
          $assert.notThrows(function(){
               var hello = 1;
          });
     }

/*********************************** NON-RUNNABLE Methods ***********************************/

     function nonStandardNamesWillNotRun() {
          fail( "Non-test methods should not run" );
     }

     private function privateMethodsDontRun() {
          fail( "Private method don't run" );
     }

}