Improvement

TESTBOX-370 `toHaveKey` works on queries in Lucee but not ColdFusion

TESTBOX-373 Update to `cbstreams` 2.x series for compat purposes.

New Feature

TESTBOX-375 Updated mixerUtil for faster performance and new approaches to dynamic mixins

TESTBOX-376 Add `bundlesPattern` to testbox.system.TestBox `init` method

TESTBOX-377 TestBox Modules

Bug

TESTBOX-346 expect(sut).toBeInstanceOf("something")) breaks if sut is a query

TESTBOX-374 cbstreams doesn't entirely work outside of ColdBox

Improvement

TESTBOX-20 toBeInstanceOf() Expectation handle Java classes

In this section, you will find the release notes for each version we release under this major version. If you are looking for the release notes of previous major versions, use the version switcher at the top left of this documentation book. Here is a breakdown of our major version releases.

Version 5.x - May 2023

In this release, we have dropped legacy engines and added support for the BoxLang JVM language, Adobe 2023 and Lucee 6. We have also added major updates to spying and expectations. We continue in this series to focus on productivity and fluency in the Testing language in preparation for more ways to test.

Version 4.x - April 2020

In this release, we have dropped support for legacy CFML engines and introduced the ability to mock data and relationships and build JSON documents.

Version 3.x

In this release, we focused on dropping engine supports for legacy CFML engines. We had a major breakthrough in introducing Code Coverage thanks to the FusionReactor folks as well. This major release also came with a new UI for all reporters and streamlined the result viewports.

Version 2.x

This version spawned off with over 8 minor releases. We focused on taking TestBox 1 to yet a high level. Much more attention to detail and introducing modern paradigms like given-when-then. Multiple interception points, async executions, and ability to chain methods.

Version 1.x

This was our first major version of TestBox. We had completely migrated from MXUnit, and it introduced BDD to the ColdFusion (CFML) world.

TestBox BDD v5.x

TestBox is a next-generation testing framework for the BoxLang JVM language and ColdFusion (CFML) based on BDD (Behavior Driven Development) for providing a clean, obvious syntax for writing tests. It contains not only a testing framework, console/web runner, assertions, and expectations library but also ships with MockBox, A mocking and stubbing companion.

class{

  function run(){
  	describe( "My calculator features", () => {
	
		beforeEach( () => {
			variables.calc = new Calculator()
		} )
			
		// Using expectations library
		it( "can add", () => {
			expect( calc.add(1,1) ).toBe( 2 )
		} )
		
		// Using assert library
		test( "it can multiply", () => {
			assertIsEqual( calc.multiply(2,2), 4 )
		} )
	} )
  }

}

/**
 * My calculator features
 */
class{

	property calc;
	
	function setup(){
	    calc = new Calculator()
	}
	
	// Function name includes the word 'test'
	// Using expectations library
	function testAdd(){
	    expect( calc.add(1,1) ).toBe( 2 )
	}
		
	// Any name, but with a test annotation
	// Using assertions library
	@test
	function itCanMultiply(){
	    $assert.isEqual( calc.multiply(2,2), 4 )
	}
}

component{

  function run(){
  	describe( "My calculator features", () => {
	
		beforeEach( () => {
			variables.calc = new Calculator()
		} );
			
		// Using expectations library
		it( "can add", () => {
			expect( calc.add(1,1) ).toBe( 2 )
		} );
		
		// Using assert library
		test( "it can multiply", () => {
			$assert.isEqual( calc.multiply(2,2), 4 )
		} );
	} );
  }

}

/**
 * My calculator features
 */
component{
	
	property calc;
	
	function setup(){
	    calc = new Calculator()
	}
	
	// Function name includes the word 'test'
	// Using expectations library
	function testAdd(){
	    expect( calc.add(1,1) ).toBe( 2 )
	}
		
	// Any name, but with a test annotation
	// Using assertions library
	function itCanMultiply() test{
	    $assert.isEqual( calc.multiply(2,2), 4 )
	}
}

Features At A Glance

Here is a simple listing of features TestBox brings to the table:

• BDD style or xUnit style testing

• Testing life-cycle methods

• MockBox integration for mocking and stubbing

• Mocking data library for mocking JSON/complex data and relationships

• Ability to extend and create custom test runners and reporters

• Extensible reporters, bundled with tons of them:

  • JSON

  • XML

  • JUnit XML

  • Text

  • Console

  • TAP (Test Anything Protocol)

  • Simple HTML

  • Min - Minimalistic Heaven

  • Raw

  • CommandBox

• Asynchronous testing

• Multi-suite capabilities

• Test skipping

• Test labels and tagging

• Testing debug output stream

• Code Coverage via FusionReactor

• Much more!

Versioning

TestBox is maintained under the Semantic Versioning guidelines as much as possible. Releases will be numbered in the following format:

<major>.<minor>.<patch>

And constructed with the following guidelines:

• bumpBreaking backward compatibility bumps the major (and resets the minor and patch)

• New additions without breaking backward compatibility bump the minor (and resets the patch)

• Bug fixes and misc changes bump the patch

License

TestBox is open source and licensed under the Apache 2 License. If you use it, please try to mention it in your code or website.

• Copyright by Ortus Solutions, Corp

• TestBox is a registered trademark by Ortus Solutions, Corp

Discussion & Help

• Help Group: https://community.ortussolutions.com/c/communities/testbox/11

• BoxTeam Slack : https://boxteam.ortussolutions.com

Reporting a Bug

We all make mistakes from time to time :) So why not let us know about it and help us out? We also love pull requests, so please star us and fork us: https://github.com/Ortus-Solutions/TestBox

• By Jira: https://ortussolutions.atlassian.net/browse/TESTBOX

Professional Open Source

TestBox is a professional open source software backed by Ortus Solutions, Corp offering services like:

• Custom Development

• Professional Support & Mentoring

• Training

• Server Tuning

• Security Hardening

• Code Reviews

• Much More

Resources

• Official Site: https://www.ortussolutions.com/products/testbox

• Current API Docs: https://apidocs.ortussolutions.com/testbox/current

• Help Group: https://community.ortussolutions.com/c/communities/testbox/11

• Source Code: https://github.com/Ortus-Solutions/TestBox

• Bug Tracker: https://ortussolutions.atlassian.net/browse/TESTBOX

• Twitter: @ortussolutions

• Facebook: https://www.facebook.com/ortussolutions

HONOR GOES TO GOD ABOVE ALL

Because of His grace, this project exists. If you don't like this, don't read it, it's not for you.

Therefore being justified by faith, we have peace with God through our Lord Jesus Christ: By whom also we have access by faith into this grace wherein we stand, and rejoice in hope of the glory of God. - Romans 5:5

Luis Fernando Majano Lainez

Luis Majano is a Computer Engineer with over 16 years of software development and systems architecture experience. He was born in San Salvador, El Salvador in the late 70’s, during a period of economical instability and civil war. He lived in El Salvador until 1995 and then moved to Miami, Florida where he completed his Bachelors of Science in Computer Engineering at Florida International University. Luis resides in The Woodlands, Texas with his beautiful wife Veronica, baby girl Alexia and baby boy Lucas!

He is the CEO of Ortus Solutions, a consulting firm specializing in web development, ColdFusion (CFML), Java development and all open source professional services under the ColdBox and ContentBox stack. He is the creator of ColdBox, ContentBox, WireBox, MockBox, LogBox and anything “BOX”, and contributes to many open source ColdFusion projects. He is also the Adobe ColdFusion user group manager for the Inland Empire. You can read his blog at www.luismajano.com

Luis has a passion for Jesus, tennis, golf, volleyball and anything electronic. Random Author Facts:

• He played volleyball in the Salvadorean National Team at the tender age of 17

• The Lord of the Rings and The Hobbit is something he reads every 5 years. (Geek!)

• His first ever computer was a Texas Instrument TI-86 that his parents gave him in 1986. After some time digesting his very first BASIC book, he had written his own tic-tac-toe game at the age of 9. (Extra geek!)

• He has a geek love for circuits, microcontrollers and overall embedded systems.

• He has of late (during old age) become a fan of running and bike riding with his family.

Keep Jesus number one in your life and in your heart. I did and it changed my life from desolation, defeat and failure to an abundant life full of love, thankfulness, joy and overwhelming peace. As this world breathes failure and fear upon any life, Jesus brings power, love and a sound mind to everybody!

“Trust in the LORD with all your heart, and do not lean on your own understanding.” Proverbs 3:5

Contributors

Jorge Emilio Reyes Bendeck

Jorge is an Industrial and Systems Engineer born in El Salvador. After finishing his Bachelor studies at the Monterrey Institute of Technology and Higher Education ITESM, Mexico, he went back to his home country where he worked as the COO of Industrias Bendek S.A.. In 2012 he left El Salvador and moved to Switzerland in persuit of the love of his life. He married her and today he resides in Basel with his lovely wife Marta and their daughter Sofía.

Jorge started working as project manager and business developer at Ortus Solutions, Corp. in 2013, . At Ortus he fell in love with software development and now enjoys taking part on software development projects and software documentation! He is a fellow Cristian who loves to play the guitar, worship and rejoice in the Lord!

Therefore, if anyone is in Christ, the new creation has come: The old has gone, the new is here! 2 Corinthians 5:17

TestBox 5.x series is a major bump in our library. Here are the major areas of improvement and the full release notes.

Engine Support

We have dropped Adobe 2016 support and added support for Adobe 2023 and Lucee 6+

Batch Test Coverage Reporting

Due to memory limitations in CI environments, larger codebases cannot run all tests as a single testbox run command. Instead, specs are run in a methodical folder-by-folder sequence, separating the testbox run out over many requests and thus working around the Out-Of-Memory exceptions.

While this works, it prevents accurate code coverage reporting since only a small portion of the tests are executed during any request. The generated code coverage report only shows a tiny fraction of the coverage - say, 2% - and not the whole picture

TestBox 5 introduces a CoverageReporter component which

1. Runs on every TestBox code coverage execution

2. Loads any previous coverage data from a JSON file

3. Combines the previous coverage data with the current execution's coverage data (file by file and line by line)

4. Persists the COMBINED coverage data to a JSON file.

5. Returns the COMBINED coverage data for the CoverageBrowser.cfc to build as an HTML report

When setting url.isBatched=true and executing the batched test runner, the code coverage report will grow with each sequential testbox run command.

Method Spies!

MockBox now supports a $spy( method ) method that allows you to spy on methods with all the call log goodness but without removing all the methods. Every other method remains intact, and the actual spied method remains active. We decorate it to track its calls and return data via the $callLog() method.

Example of CUT:

void function doSomething(foo){
  // some code here then...
  local.foo = variables.collaborator.callMe(local.foo);
  variables.collaborator.whatever(local.foo);
}

Example Test:

function test_it(){
  local.mocked = createMock( "com.foo. collaborator" )
    .$spy( "callMe" )
    .$spy( "whatever" );
  variables.CUT.$property( "collaborator", "variables", local.mocked );
  assertEquals( 1, local.mocked.$count( "callMe" ) );
  assertEquals( 1, local.mocked.$count( "whatever" ) );
}

Performance Improvements

We have focused on this release to lazy load everything as much as possible to allow for much better testing performance. Check it out!

Skip it! Skip it -> Good!

You can now use the skip( message ) method to skip any spec or suite a-la-carte instead of as an argument to the function definitions. This lets you programmatically skip certain specs and suites and pass a nice message.

it( "can do something", () => {
    ...
    if( condition ){
        skip( "Condition is true, skipping spec" )
    }
    ...
} )

Release Notes

Fixed

• TESTBOX-341 toHaveLength param should be numeric

• TESTBOX-354 Element $DEBUGBUFFER is undefined in THIS

• TESTBOX-356 Don't assume TagContext has length on simple reporter

• TESTBOX-357 notToThrow() incorrectly passes when no regex is specified

• TESTBOX-360 full null support not working on Application env test

• TESTBOX-361 MockBox Suite: Key [aNull] doesn't exist

• TESTBOX-362 Cannot create subfolders within testing spec directories.

Improvements

• TESTBOX-333 Add contributing.md to the repo

• TESTBOX-339 full null support automated testing

• TESTBOX-353 allows globbing path patterns in test bundles argument

• TESTBOX-355 Add debugBuffer to JSONReporter

• TESTBOX-366 ANTJunit Reporter better visualization of the failed origin and details

• TESTBOX-368 Support list of Directories for HTMLRunner to allow a more modular tests structure

• TESTBOX-370 `toHaveKey` works on queries in Lucee but not ColdFusion

Added

• TESTBOX-371 Add CoverageReporter for batching code coverage reports

• TESTBOX-137 Ability to spy on existing methods: $spy()

• TESTBOX-342 Add development dependencies to box.json

• TESTBOX-344 Performance optimizations for BaseSpec creations by lazy loading external objects

• TESTBOX-345 add a skip([message]) like fail() for skipping from inside a spec

• TESTBOX-365 New build process using CommandBox

• TESTBOX-372 Adobe 2023 and Lucee 6 Support

5.3.1 - September 13, 2023

Fixed

• The variable thisSuite isn't defined if the for loop in the try/catch is never reached before the error. (#150)

5.3.0 - August 1, 2023

New Features

• TESTBOX-379 New expectations: toBeIn(), toBeInWithCase() so you can verify a needle in string or array targets

• TESTBOX-380 New matchers and assertions: toStartWith(), toStartWithCase(), startsWith(), startsWthCase() and their appropriate negations

• TESTBOX-381 New matchers and assertions: toEndWith(), toEndWithCase(), endsWith(), endsWithCase() and their appropriate negations

Bugs

• TESTBOX-378 onSpecError suiteSpecs is invalid, it's suiteStats

TestBox is fully compliant with MXUnit xUnit test cases. In order to leverage it you will need to create or override the /mxunit mapping and make it point to the /testbox/system/compat folder. That's it, everything should continue to work as expected.

Note you will still need TestBox to be in the web root, or have a /testbox mapping created even when using the MXUnit compat runner.

this.mappings[ "/mxunit" ] = expandPath( "/testbox/system/compat" );

After this, all your test code remains the same but it will execute through TestBox's xUnit runners. You can even execute them via the normal URL you are used to. If there is something that is not compatible, please let us know and we will fix it.

Expected Exceptions

We also support in the compatibility mode the expected exception MXUnit annotation: mxunit:expectedException and the expectException() methods. The expectException() method is not part of the assertion library, but instead is inherited from our BaseSpec.cfc.

Please refer to MXUnit's documentation on the annotation and method for expected exceptions, but it is supported with one caveat. The expectException() method can produce unwanted results if you are running your test specs in TestBox asynchronous mode since it stores state at the component level. Only synchronous mode is supported if you are using the expectException() method. The annotation can be used in both modes.

TestBox is a next-generation testing framework for the BoxLang JVM language and ColdFusion (CFML) language based on BDD (Behavior Driven Development) for providing a clean, obvious syntax for writing tests. It contains not only a testing framework, console/web runner, assertions, and expectations library but also ships with several mocking utilities.

Styles

In TestBox you can write your tests in two different styles or approaches.

BDD (Behavior Driven Development)

BDD stands for behavior-driven development and is highly based on creating specifications and expectations of results in a readable DSL (Domain Specific Language). You are not focusing on a specific unit and method to test, but on functionality, features and more. This can encompass not only unit but also integration testing. You have several methods that you can use in order to denote functionality and specifications:

• describe()

• feature()

• story()

• given(), when(), then()

• it() or test()

describe( "My calculator features", () => {
	
	beforeEach( () => {
		variables.calc = new Calculator()
	} )
		
	// Using expectations library
	it( "can add", () => {
		expect( calc.add(1,1) ).toBe( 2 )
	} )
	
	// Using assert library
	test( "it can multiply", () => {
		$assert.isEqual( calc.multiply(2,2), 4 )
	} )
} )

xUnit (Test Driven Development)

xUnit style of testing is the more traditional TDD or test-driven development approach where you create a test case bundle class that matches the software under test, and for each method in the SUT, you create a test method in the test bundle class.

@DisplayName "My calculator features"
class{

	property calc;
	
	function setup(){
		calc = new Calculator()
	}
	
	// Function name includes the word 'test'
	// Using expectations library
	function testAdd(){
		expect( calc.add(1,1) ).toBe( 2 )
	}
		
	// Any name, but with a test annotation
	// Using assertions library
	@DisplayName "It can multiply two operands"
	@test
	function itCanMultiply(){
		$assert.isEqual( calc.multiply(2,2), 4 )
	}
}

Assertions & Expectations

We also give you two ways to do assertions:

1. Assertions library, which is a traditional approach to assertions

2. Expectations library, which is a more fluent approach to assertions.

// Assertions
assert( expression, "custom message" )

// Expectations
expect( structKeyExists( handler, "mixinTest" ) ).toBeTrue();
expect( structKeyExists( handler, "repeatThis" ) ).toBeTrue();
expect( structKeyExists( handler, "add" ) ).toBeTrue();
expect( target.$callLog().relocate[ 1 ].url ).toInclude( "dashboard" );

Life-Cycles

Both approaches also offer different life-cycle callbacks so you can execute code during different times in the test execution.

beforeEach( function( currentSpec ){
    setup();
    target = prepareMock( getInstance( "coldbox.system.FrameworkSupertype" ) );
} );

afterEach( function( currentSpec ){
    getCache( "template" ).clearAll();
    variables.scheduler.shutdown();
} );


function beforeAll() {
	super.beforeAll();
	
	// Login User
	variables.loggedInData = loginUser();
	
	variables.loggedInUserId = variables.loggedInData.user.getUserId();
	
	variables.testTimeOffEmployeeId = qb
		.select( "FK_userId" )
		.from( "timeOff" )
		.where( "requestType", "vacation" )
		.first()
		.FK_userId;
}

function afterAll() {
	variables.userService.clearCaches()
}

function setup() {
	super.setup();

	if ( !isNull( request.testUserData ) ) {
		getRequestContext().setValue( "x-auth-token", request.testUserData.token.access_token );
	}

	return;
}

function tearDown(){
	request.clear()
}

function beforeTests() {
	
	// Login User
	variables.loggedInData = loginUser();
	
	variables.loggedInUserId = variables.loggedInData.user.getUserId();
	
	variables.testTimeOffEmployeeId = qb
		.select( "FK_userId" )
		.from( "timeOff" )
		.where( "requestType", "vacation" )
		.first()
		.FK_userId;
}

function afterTests() {
	variables.userService.clearCaches()
}

Utilities

TestBox will also offer different utilities:

• Debug output to a debug buffer

• Mock classes, methods and properties

• Extension data mocking and JSON mocking

• Logging facilities

Runners

You can also execute your tests via the CLI, IDE or the web server.

Reports

TestBox can produce many different types of reports for your test executions:

• CLI / Console Output

• VSCode Output

• JSON

• XML

• JUNIT

• TAP

• HTML

• DOC

• Your own

Here is a few samples:

class{

  function run(){
  	describe( "My calculator features", () => {
	
		beforeEach( () => {
			variables.calc = new Calculator()
		} )
			
		// Using expectations library
		it( "can add", () => {
			expect( calc.add(1,1) ).toBe( 2 )
		} )
		
		// Using assert library
		test( "it can multiply", () => {
			$assert.isEqual( calc.multiply(2,2), 4 )
		} )
	} )
  }

}

/**
 * My calculator features
 */
class{

	property calc;
	
	function setup(){
		calc = new Calculator()
	}
	
	// Function name includes the word 'test'
	// Using expectations library
	function testAdd(){
		expect( calc.add(1,1) ).toBe( 2 )
	}
		
	// Any name, but with a test annotation
	// Using assertions library
	@test
	function itCanMultiply(){
		$assert.isEqual( calc.multiply(2,2), 4 )
	}
}

component{

  function run(){
  	describe( "My calculator features", () => {
	
		beforeEach( () => {
			variables.calc = new Calculator()
		} );
			
		// Using expectations library
		it( "can add", () => {
			expect( calc.add(1,1) ).toBe( 2 )
		} );
		
		// Using assert library
		test( "it can multiply", () => {
			$assert.isEqual( calc.multiply(2,2), 4 )
		} );
	} );
  }

}

/**
 * My calculator features
 */
component{
	
	property calc;
	
	function setup(){
		calc = new Calculator()
	}
	
	// Function name includes the word 'test'
	// Using expectations library
	function testAdd(){
		expect( calc.add(1,1) ).toBe( 2 )
	}
		
	// Any name, but with a test annotation
	// Using assertions library
	function itCanMultiply() test{
		$assert.isEqual( calc.multiply(2,2), 4 )
	}
}

Useful Resources

• Approaches to Mocking

• Wikipedia Mock Objects

• Using mock objects for complex unit tests IBM developerWorks

• Unit testing with mock objects IBM developerWorks

• Emergent Design by Scott Bain

• Mocks Aren't Stubs by Martin Fowler

No matter what style you decide to use, you will still end up building a Testing Bundle Class. This class will either contain BDD style suites and specs or xUnit style method tests. You can create as many as you need and group them as necessary in different folders (packages) according to their features. Our test harness can be generated via the CLI or you can grab it from the installation folder /bx or cfml/test-harness. Here is the typical layout of the harness:

• /tests - The test harness

  • /resources - Where you can place any kind of testing helpers, data, etc

  • /specs - Where your test bundles can go

  • Application.bx|cfc - Your test harness applilcation file. Controls life-cycle and application concerns.

  • runner.bx|cfm - A web runner template that executes ALL your tests with tons of different options from a running web server.

  • test.xml - An ANT task to do JUNIT testing.

You will be creating test classes inside the /tests/specs folders. The class can extend our base class: testbox.system.BaseSpec or not. If you do, then the tests will be faster, executable directly from a web server and you will get IDE introspection. However, TestBox doesn't enforce the inheritance.

My First Test

Typically test bundles are suffixed with the word Test or Spec, such as MyServiceSpec.bx or UserServiceTest.cfc

class extends="testbox.system.BaseSpec"{

    function run(){
	describe( "My First Test", ()=>{
	  test( "it can add", ()=>{
		expect( sum( 1, 2 ) ).toBe( 3 )
	  } )
	} )
    }

    private function sum( a, b ){
        return a + b
    }

}

component extends="testbox.system.BaseSpec"{

    function run(){
	describe( "My First Test", ()=>{
	  test( "it can add", ()=>{
		expect( sum( 1, 2 ) ).toBe( 3 );
	  } );
	} );
    }

    private function sum( a, b ){
        return a + b;
    }

}

The CLI can assist you when creating new bundles:

# Create a BDD Bundle
testbox create bdd --help
# Create an xUnit Bundle
testbox create unit --help

# Examples
# Remember that by convention it will create bundles at /tests/specs
testbox create bdd CalculatorTest --open
testbox create unit name=SecurityTest directory="tests/specs/unit/"

Now you can run your tests via the browser (http://localhost:port/tests/runner.cfm)

or via the CLI testbox run

Optional Inheritance

At runtime we provide the inheritance via mixins so you don't have to worry about it. However, if you want to declare the inheritance you can do so and this will give you the following benefits:

• Some IDEs will be able to give you introspection for methods and properties

• You will be able to use the HTML runner by executing directly the runRemote method on the CFC Bundle

• Your tests will run faster

Injected Variables

At runtime, TestBox will inject several public variables into your testing bundle to help you with your testing.

• $mockbox : A reference to MockBox

• $assert : A reference to our Assertions library

• $utility : A utility CFC

• $customMatchers : A collection of custom matchers registered

• $exceptionAnnotation : The annotation used to discover expected exceptions, defaults to expectedException

• $testID : A unique identifier for the test bundle

• $debugBuffer : The debugging buffer stream

Injected/Inherited Methods

Wether you inherit or not, your bundles will have a plethora of methods that will help you in your testing adeventure. Here is a link to the API Docs for the BaseSpec class:

Quick Assertion Methods

The BaseSpec has a few shortcut methods for quick assertions. Please look at the Assertions and Expectations library for more in-depth usage.

// Assert that the expression is truthy
assert( expression, [message=""] )
// Start an expectation with an actual value, returns the Expectation object
expect( actual ) : Expectation
// Fail now!
fail( message )

// In BoxLang you can use the dynamic assertions feature
// Meaning you can execute any assertion method by just prefixing it with 
// the word `asssert{method}`
assertIsEqual()
assertIsTrue()
assertIsFalse()
... etc.
// This leverages the dynamic on missing method in BoxLang but not available in CFML

Extension Methods

These methods allow you to extend TestBox with custom assertions and expectation matchers.

// extensions
addMatchers( struct|path matchers )
addAssertions( struct|path assertions )

Environment Methods

These methods assist you with identifying environment conditions.

// Which language/engine are you running on
isAdobe()
isLucee
isBoxLang()

// What OS are we on
isLinux()
isMac()
isWindows()

// Get the Environment Class
getEnv()

Java Environment

You can use the getEnv() to get access to our Environment utility object. From there you can use the following methods:

// Get a java property or environment setting or a default value
getSystemSetting( required key, [defaultValue] )

// Get a java system property ONLY
getSystemProperty( required key, [defaultValue] )

// Get a java environment setting ONLY
getEnv( required key, [defaultValue] )

// Get the java.lang.System
getJavaSystem()

Utility Methods

These methods are here to help you during the testing process. Every bundle also get's a debug buffer attached to its running results. This is where you as the developer can send pretty much any variable (simple or complex) and attach it to the debug buffer. This will then be presented acordingly in the test reporters or facilities.

// Send variables to the output console
console(any var, [numeric top='9999'], [boolean showUDFs='false'], [string label=''])
// Send data to the TestBox debug buffer
debug([any var], [string label=''], [boolean deepCopy='false'], [numeric top='999'], [boolean showUDFs='false'])
// Clear the buffer
clearDebugBuffer()
// Get the debug buffer
getDebugBuffer()

// Writes to the Output Buffer (CLI, Browser)
// Remember that if your test runs in the CLI, the buffer is the console
// If you are in a webserver, the buffer is the browser output
print( message )
printLn( message )

Mocking Methods

TestBox is bundles with two amazing mocking facilities:

• MockBox - Mocks objects and stubs

• MockDataCFC - Allows you to mock data and JSON data - https://www.forgebox.io/view/mockdatacfc

createMock([string className], [any object], [boolean clearMethods='false'])
createEmptyMock([string className], [any object], [boolean callLogging='true'])
prepareMock([any object], [boolean callLogging='true'])
createStub([boolean callLogging='true'], [string extends=''], [string implements=''])

// Create a mock query of data
querySim( queryData )
// Call the `mockData()` method on the MockDataCFC
mockData( arguments )

// Make a private/package method on a class public
makePublic(any target, string method, [string newName=''])
// Get the value of a private property on a class
getProperty( target, name, scope, defaultValue )

// Get the MockBox class
getMockBox( [string generationPath=''] )

BDD Methods

These methods are used to create the BDD style of testing. You will discover them more in the BDD Tests section.

// Life-cycle methods
afterEach(any body, [struct data='[runtime expression]'])
aroundEach(any body, [struct data='[runtime expression]'])
beforeEach(any body, [struct data='[runtime expression]'])

// Suite/Grouping Methods
describe(string title, any body, [any labels='[runtime expression]'], [boolean asyncAll='false'], [any skip='false'], [boolean focused='false'])
feature(string title, any body, [any labels='[runtime expression]'], [boolean asyncAll='false'], [any skip='false'], [boolean focused='false'])
scenario(string title, any body, [any labels='[runtime expression]'], [boolean asyncAll='false'], [any skip='false'], [boolean focused='false'])
story(string title, any body, [any labels='[runtime expression]'], [boolean asyncAll='false'], [any skip='false'], [boolean focused='false'])
given(string title, any body, [any labels='[runtime expression]'], [boolean asyncAll='false'], [any skip='false'], [boolean focused='false'])
when(string title, any body, [any labels='[runtime expression]'], [boolean asyncAll='false'], [any skip='false'], [boolean focused='false'])

// skip the suite
xdescribe()
xfeature()
xscenario()
xgiven()
xstory()
xwhen()

// focus the suite
fdescribe()
ffeature()
fscenario()
fgiven()
fstory()
fwhen()


// Specs/Testing Methods
it(string title, any body, [any labels='[runtime expression]'], [any skip='false'], [struct data='[runtime expression]'], [boolean focused='false'])
then(string title, any body, [any labels='[runtime expression]'], [any skip='false'], [struct data='[runtime expression]'], [boolean focused='false'])
test(string title, any body, [any labels='[runtime expression]'], [any skip='false'], [struct data='[runtime expression]'], [boolean focused='false'])

// Skip the tests
xit()
xthen()
xtest()

// Focus the tests
fit()
fthen()
ftest()

// Start an expectation expression
expect( actual )

A modern editor can enhance your testing experience. We recommend VSCode due to the extensive modules library. Here are the plugins we maintain for each platform.

VSCode Plugin

The VSCode plugin is the best way for you to interact with TestBox alongside the BoxLang plugin. It allows you to run tests, generate tests, navigate tests and much more.

Sublime Plugin

Framework

TestBox can be installed via CommandBox CLI as a development dependency in the root of your projects:

// Create a new project
mkdir myProject --cd

// latest stable version
box install testbox --saveDev

// latest bleeding edge
box install testbox@be --saveDev

The only requirement is that it be in either in the webroot or in a location where you create a /testbox mapping to it's folder.

System Requirements

• BoxLang 1+ Language (Our preference)

• CFML Engines: Lucee 5.x+ or ColdFusion 2018+

What's Included

Here is a table of what's included in the installation package which goes in /{project}/testbox

In the bx folder you will find specific BoxLang tools:

CFML Tools

In the cfml folder you will find specific BoxLang tools:

IDE Tooling

Now that you are installed, please set up your favorite IDE with our tooling extensions so it can make your testing experience more enjoyable.

TestBox CLI

TestBox comes with its own CLI for CommandBox. You can use it to generate tests, harnesses, and suites and also run executions from the CLI.

install testbox-cli

You will now have the testbox namesapce available to you, try it out

testbox help

Generating a Testing Harness

Once you install TestBox, you'll need a quick way to set up a testing harness in your project. The generate harness command will add a new /tests folder to your application with a few example tests to get you started.

testbox generate harness

/tests - The test harness generated

• /resources - Where you can place any kind of testing helpers, data, etc

• /specs - Where your test bundles can go

  • /unit

  • /integration

• Application.bx|cfc - Your test harness applilcation file. Controls life-cycle and application concerns.

• runner.bx|cfm - A web runner template that executes ALL your tests with tons of different options from a running web server.

• test.xml - An ANT task to do JUNIT testing.

You can then run your tests by executing the testbox run command in CommandBox or by visiting the web runner in the generated harness: http://localhost/tests/runner.cfm

testbox run --help

Generating Tests

You can also use the CLI to generate tests according to your style and also it detects if you are using BoxLang or a CFML engine.

testbox create bdd --help
testbox create unit --help

Language Generation

If you want to be explicit you can use the language id to set it to your preferred language for generation and usage. We recommend this so all CLI tooling can detect what language your project is in.

# For BoxLang Generation
package set language="boxlang"

# For CFML
package set language="cfml"

The source code for this book is hosted on GitHub: https://github.com/Ortus-Solutions/testbox-docs. You can freely contribute to it and submit pull requests. The contents of this book is copyrighted by Ortus Solutions, Corp and cannot be altered or reproduced without the author's consent. All content is provided "As-Is" and can be freely distributed.

External Trademarks & Copyrights

Flash, Flex, ColdFusion, and Adobe are registered trademarks and copyrights of Adobe Systems, Inc.

BoxLang, ColdBox, CommandBox, FORGEBOX, TestBox, ContentBox, and Ortus Solutions are all trademarks and copyrights of Ortus Solutions, Corp.

Notice of Liability

The information in this book is distributed “as is” without warranty. The author and Ortus Solutions, Corp shall not have any liability to any person or entity concerning loss or damage caused or alleged to be caused directly or indirectly by the content of this training book, software, and resources described in it.

Contributing

We highly encourage contributions to this book and our open-source software. The source code for this book can be found in our GitHub repository where you can submit pull requests.

Charitable Proceeds

15% of the proceeds of this book will go to charity to support orphaned kids in El Salvador - http://www.harvesting.org/. So please donate and purchase the printed version of this book; every book sold can help a child for almost 2 months.

Shalom Children's Home

Shalom Children’s Home (https://www.harvesting.org/) is one of the ministries that are dear to our hearts located in El Salvador. During the 12-year civil war that ended in 1990, many children were left orphaned or abandoned by parents who fled El Salvador. The Benners saw the need to help these children and received 13 children in 1982. Little by little, more children came on their own, churches and the government brought children to them for care, and the Shalom Children’s Home was founded.

Shalom now cares for over 80 children in El Salvador, from newborns to 18 years old. They receive shelter, clothing, food, medical care, education, and life skills training in a Christian environment. The home is supported by a child sponsorship program.

We have personally supported Shalom for over 6 years now; it is a place of blessing for many children in El Salvador who either has no families or have been abandoned. This is a good earth to seed and plant.

This is more of an approach than an actual specifc runner. This approach shows you that you can create a script file in BoxLang (bxs) or in CFML (cfs|cfm) that can in turn execute any test bundle(s) with many many runnable configurations.

BoxLang

The BoxLang language allows you to run your scripts via the CLI or the browser if you have a web server attached to your project.

If you want to run it in the CLI, then just use:

If you want to run it via the web server, place it in your /tests/ folder and run it

CFML

CFML engines only allow you to run tests via the browser. So create your script, place it in your web accessible /tests folder and run it.

This is more of an approach than an actual specifc runner. This approach shows you that you can create a script file in BoxLang (bxs) or in CFML (cfs|cfm) that can in turn execute any test directory with many many runnable configurations. It is very similar to the approach.

BoxLang

The BoxLang language allows you to run your scripts via the CLI or the browser if you have a web server attached to your project.

If you want to run it in the CLI, then just use:

If you want to run it via the web server, place it in your /tests/ folder and run it

CFML

CFML engines only allow you to run tests via the browser. So create your script, place it in your web accessible /tests folder and run it.

By installing the CommandBox you can get access to our CommandBox runner. The CommandBox runner leverages the HTTP(s) protocol to test against any server. By default it will inspect your box.json for a default runner or try to connect to /tests/runner.cfm by default.

To see all the running options run the following in your CLI shell:

It can also produce reports for you in JSON, HTML, and JUNIT.

Runner Options

If you type testbox run --help you can see all the arguments you can set for running your tests. However, please note that you can also pre-set them in your box.json under the testbox entry:

Runner URL

You can also set up the default runner URL in your box.json and it will be used for you. Setting the URL is a one-time operation.

You can also use a relative path and CommandBox will look up the host and port from your server settings.

The default runner URL of the testbox run command is /tests/runner.cfm so there's actually no need to even configure it if you're using the default convention location for your runner.

Multiple Runner URLs

You can define multiple URLs for your runners by using a JSON array of objects. Each key will be a nice identifier you can use via the runner=key argument in the command.

Then you can just pass in the name:

More Commands:

Watcher

The CLI also comes with a code watcher and runner. It will watch any paths for you, and if it detects any changes, it will run the tests you want.

In order for this command to work, you need to have started your server and configured the URL to the test runner in your box.json.

You can also control what files to watch.

If you need more control over what tests run and their output, you can set additional options in your box.json which will be picked up automatically by testbox run when it fires.

This command will run in the foreground until you stop it. When you are ready to shut down the watcher, press Ctrl+C.

TestBox ships with a new BoxLang CLI runner for Linux/Mac and Windows. This will allow you to execute your tests from the CLI and, in the future, via VSCode easily and extremely fast. It can also execute and stream the executions so you can see the testing progress when running in verbose mode. The runner can also execute specs/tests that are written in CFML or BoxLang in the BoxLang runtime.

BoxLang allows you to not only build web applications, but CLI, serverless, Android, etc. You can use this runner to test each environment. However, please note that if you will be doing web server testing from the CLI only, you will need to install the web support module into the Operating System runtime.

Web Server Testing

If you want to test from the CLI your web application with no web server, then you will need to install the bx-web-support module into the CLI. Remember that BoxLang is multi-runtime, and you not only can build web applications but CLI or OS based applications.

This will add web support to the CLI (BIFS, components, etc.) and a mock HTTP server so you can do full life-cycle testing from the CLI like if running your app in a web server. This runner does not require a web server to function, thus if you are building a web app, you will need this module if you still want to continue to execute your tests in the CLI Runtime.

Script Locations

The scripts are located in the following directory: /testbox/bin from the TestBox installation package.

• run - For Linux/Mac

• run.bat - For Windows

This is the entry point for executing tests at the CLI level. Please note that the test execution does NOT involve a web server. This is for pure CLI testing.

Examples:

The runner must be run from the root of your BoxLang project:

Mac/Linux

Windows Examples:

Execution Options

• --bundles A list of test bundles to run, defaults to *, ex: path.to.bundle1,path.to.bundle2, . Mutually exclusive with --directory

• --bundles-pattern A pattern to match test bundles defaults to "*Spec*.cfc|*Test*.cfc|*Spec*.bx|*Test*.bx"

• --directory A list of directories to look for tests to execute. Please use dot-notation, not absolute notation. Mutually exclusive with --bundles. Ex: tests.specs Defaults to tests.specs

• --recurse : Recurse into subdirectories, defaults to true

• --eager-failure : Fail fast, defaults to false

• --verbose : Verbose output defaults to false. This will stream the output of the status of the tests as they run.

• --runner-options: A JSON struct literal of options to pass into the test runner. Ex: {"verbose"=true}

Reporting Options

• --reporter The reporter to use.

• --reportpath : The path to write the report file, defaults to the /tests/results folder by convention

• --properties-summary : Generate a properties file with the summary of the test results, defaults to true.

• --properties-filename : The name of the properties file to generate defaults to TEST.properties

• --write-report : Write the report to a file in the report path folder, defaults to true

• --write-json-report : Write the report as JSON alongside the requested report, defaults to false

• --write-visualizer : Write the visualizer to a file in the report path folder, defaults to false

Filtering Options

• --labels : A list of labels to run, defaults to *

• --excludes : A list of labels to exclude, defaults to empty

• --filter-bundles : A list of bundles to filter by, defaults to *

• --filter-suites : A list of suites to filter by, defaults to *

• --filter-specs : A list of test names or spec names to filter by, defaults to *

TestBox tests can be run from different sources from what we call Runners. These can be from different sources:

• CLI

  • TestBox CLI (Powered by CommandBox)

  • BoxLang Scripts

  • NodeJS

• Web Server

  • Runner

  • TestBundle Directly

• Custom

Your test harness already includes the web runner: runner.bx or runner.cfm. You can execute that directly in your browser to get the results or run it via the CLI: testbox run. We invite you to explore the different runners available to you.

Custom Runners

However, you can create your own custom runners as long as you instantiate the TestBox class and execute one of it's runnable methods. The main execution methods are:

run()

Here are the arguments you can use for initializing TestBox or executing the run() method

• The bundles argument which can be a single CFC path or an array of CFC paths or a directory argument so it can go and discover the test bundles from that directory.

• The reporter argument can be a core reporter name like: json,xml,junit,raw,simple,dots,tap,min,etc or it can be an instance of a reporter CFC.

• You can execute the runners from any cfm template or any CFC or any URL, that is up to you.

Every test harness comes with a runner.bx or runner.cfm in the root of the tests folder. This is called the web runner and is executable via the web server you are running your application on. This will execute all the tests by convention found in the tests/specs folder.

You can open that file and customize it as you see fit. Here is an example of such a file:

Test Bundle Execution

If you make your test bundle class inherit from our testbox.system.BaseSpec class, you will be able to execute the class directly via the URL:

Arguments

All the arguments found in the runner are available as well in a direct bundle execution:

• labels: The labels to apply to the execution

• testMethod : A list or array of xunit test names that will be executed ONLY!

• testSuites : A list or array of suite names that are the ones that will be executed ONLY!

• testSpecs : A list or array of test names that are the ones that will be executed ONLY!

• reporter : The type of reporter to run the test with

TestBox relies on the fact of creating testing bundles which are basically CFCs. A bundle CFC will hold all the suites and specs a TestBox runner will execute and produce reports on. Don't worry, we will cover what's a suite and a spec as well. Usually they will have a name that ends with *Spec or *Test.

component extends="testbox.system.BaseSpec"{

     // executes before all suites
     function beforeAll(){}

     // executes after all suites
     function afterAll(){}

     // All suites go in here
     function run( testResults, testBox ){

     }

}

This bundle CFC can contain 2 life-cycle functions and a single run() function where you will write your test suites and specs.

Life-Cycle Methods

The beforeAll() and afterAll() methods are called life-cycle methods. They will execute once before the run() function and once after the run() function. This is a great way to do any global setup or tear down in your tests.

// executes before all suites
function beforeAll(){}

// executes after all suites
function afterAll(){}

Execution

The run() function receives the TestBox testResults object as a reference and testbox as a reference as well. This way you can have metadata and access to what will be reported to users in a reporter. You can also use it to decorate the results or store much more information that reports can pick up later. You also have access to the testbox class so you can see how the test is supposed to execute, what labels was it passed, directories, options, etc.

function run( testResults, testBox ){

}

BDD stands for Behavioral Driven Development. It is a software development process that aims to improve collaboration between developers, testers, and business stakeholders. BDD involves creating automated tests that are based on the expected behavior of the software, rather than just testing individual code components. This approach helps ensure that the software meets the desired functionality and is easier to maintain and update in the future.

In traditional xUnit, you focused on every component's method individually. In BDD, we will focus on a feature or story to complete, which could include testing many different components to satisfy the criteria. TestBox allows us to create these types of texts with human-readable functions matching our features/stories and expectations.

describe( "Tests of TestBox behaviour", () => {
	it( "rejects 5 as being between 1 and 10", () => {
		expect( () => {
			expect( 5 ).notToBeBetween( 1, 10 );
		} ).toThrow();
	} );
	
	it( "rejects 10 as being between 1 and 10", () => {
		expect( () => {
			expect( 10 ).notToBeBetween( 1, 10 );
		} ).toThrow();
	} );
} );


feature( "Given-When-Then test language support", () => {
	scenario( "I want to be able to write tests using Given-When-Then language", () => {
		given( "I am using TestBox", () => {
			when( "I run this test suite", () => {
				then( "it should be supported", () => {
					expect( true ).toBe( true );
				} );
			} );
		} );
	} );
} );

story( "I want to list all authors", () => {
    given( "no options", () => {
        then( "it can display all active system authors", () => {
            var event = this.get( "/cbapi/v1/authors" );
            expect( event.getResponse() ).toHaveStatus( 200 );
            expect( event.getResponse().getData() ).toBeArray().notToBeEmpty();
            event
                .getResponse()
                .getData()
                .each( function( thisItem ){
                    expect( thisItem.isActive ).toBeTrue( thisItem.toString() );
                } );
        } );
    } );
    given( "isActive = false", () => {
        then( "it should display inactive users", () => {
            var event = this.get( "/cbapi/v1/authors?isActive=false" );
            expect( event.getResponse() ).toHaveStatus( 200 );
            expect( event.getResponse().getData() ).toBeArray().notToBeEmpty();
            event
                .getResponse()
                .getData()
                .each( function( thisItem ){
                    expect( thisItem.isActive ).toBeFalse( thisItem.toString() );
                } );
        } );
    } );
    given( "a search criteria", () => {
        then( "it should display searched users", () => {
            var event = this.get( "/cbapi/v1/authors?search=tester" );
            expect( event.getResponse() ).toHaveStatus( 200 );
            expect( event.getResponse().getData() ).toBeArray().notToBeEmpty();
        } );
    } );
} );

In our test harness we include an ANT runner (test.xml) that will be able to execute your tests via ANT.

It can also leverage our ANTJunit reporter to use the junitreport task to produce JUnit compliant reports as well.

<?xml version="1.0"?>
<--<
This ANT build can be used to execute your tests with automation using our included runner.cfm.
You can executes directories, bundles and so much more.  It can also produce JUnit reports using the
ANT junitreport tag.  This is meant to be a template for you to spice up.

There are two targets you can use: run and run-junit

Execute the default 'run' target
ant -f test.xml
OR
ant -f test.xml run

Execute the 'run-junit' target
ant -f test.xml run-junit

PLEASE NOTE THAT YOU MUST ALTER THE RUNNER'S URL ACCORDING TO YOUR ENVIRONMENT.
-->
<project name="testbox-ant-runner" default="run" basedir=".">

     <-- <THE URL TO THE RUNNER, PLEASE CHANGE ACCORDINGLY-->
     <property name="url.runner"             value="http://localhost/tests/runner.cfm?"/>
     <-- <FILL OUT THE BUNDLES TO TEST, CAN BE A LIST OF CFC PATHS -->
     <property name="test.bundles"      value="" />
     <-- <FILL OUT THE DIRECTORY MAPPING TO TEST -->
     <property name="test.directory"         value="test.specs" />
     <-- <FILL OUT IF YOU WANT THE DIRECTORY RUNNER TO RECURSE OR NOT -->
     <property name="test.recurse"      value="true" />
     <-- <FILL OUT THE LABELS YOU WANT TO APPLY TO THE TESTS -->
     <property name="test.labels"       value="" />
     <-- <FILL OUT THE TEST REPORTER YOU WANT, AVAILABLE REPORTERS ARE: ANTJunit, Codexwiki, console, dot, doc, json, junit, min, raw, simple, tap, text, xml -->
     <property name="test.reporter"          value="simple" />
     <-- <FILL OUT WHERE REPORTING RESULTS ARE STORED -->
     <property name="report.dir"        value="${basedir}/results" />
     <property name="junitreport.dir"   value="${report.dir}/junitreport" />

     <target name="init" description="Init the tests">
          <mkdir dir="${junitreport.dir}" />
          <tstamp prefix="start">
               <format property="TODAY" pattern="MM-dd-YYYY hh:mm:ss aa"/>
          </tstamp>
          <concat destfile="${report.dir}/latestrun.log">Tests ran at ${start.TODAY}</concat>
     </target>

     <target name="run" depends="init" description="Run our tests and produce awesome results">

          <-- <Directory Runner
               Executes recursively all tests in the passed directory and stores the results in the
               'dest' param.  So if you want to rename the file, do so here.

                Available params for directory runner:
                - Reporter
                - Directory
                - Recurse
                - Labels
          -->
          <get dest="${report.dir}/results.html"
                src="${url.runner}&directory=${test.directory}&recurse=${test.recurse}&reporter=${test.reporter}&labels=${test.labels}"
                verbose="true"/>

          <-- <Bundles Runner
               You can also run tests for specific bundles by using the runner with the bundles params

               Available params for runner:
                - Reporter
                - Bundles
                - Labels

          <get dest="${report.dir}/results.html"
                src="${url.runner}&bundles=${test.bundles}&reporter=${test.reporter}&labels=${test.labels}"
                verbose="true"/>
           -->

     </target>

     <target name="run-junit" depends="init" description="Run our tests and produce ANT JUnit reports">

          <-- <Directory Runner
               Executes recursively all tests in the passed directory and stores the results in the
               'dest' param.  So if you want to rename the file, do so here.

                Available params for directory runner:
                - Reporter = ANTJunit fixed
                - Directory
                - Recurse
                - Labels
                - ReportPath : The path where reports will be stored, by default it is the ${report.dir} directory.
          -->
          <get dest="${report.dir}/results.xml"
                src="${url.runner}&directory=${test.directory}&recurse=${test.recurse}&reporter=ANTJunit&labels=${test.labels}&reportPath=${report.dir}"
                verbose="true"/>

          <-- <Bundles Runner
               You can also run tests for specific bundles by using the runner with the bundles params

               Available params for runner:
                - Reporter
                - Bundles
                - Labels

          <get dest="${report.dir}/results.html"
                src="${url.runner}&bundles=${test.bundles}&reporter=${test.reporter}&labels=${test.labels}"
                verbose="true"/>
           -->

           <-- <Create fancy junit reports -->
          <junitreport todir="${junitreport.dir}">
               <fileset dir="${report.dir}">
                    <include name="TEST-*.xml"/>
               </fileset>
               <report format="frames" todir="${junitreport.dir}">
                    <param name="TITLE" expression="My Awesome TestBox Results"/>
               </report>
          </junitreport>

     </target>

</project>

Now you can run the commands

# Execute the default 'run' target
ant -f test.xml
# Execute a specific target
ant -f test.xml run-junit

There is a user-contributed NodeJS Runner that looks fantastic and can be downloaded here: https://www.npmjs.com/package/testbox-runner

You can use node to install as well into your projects.

npm install -g testbox-runner

Configuration

Create a config file called .testbox-runnerrc in the root of your web project.

{
	"runner": "http://localhost/testbox/system/runners/HTMLRunner.cfm",
	"directory": "/tests/specs",
	"recurse": true
}

Then use the CLI command to run whatever you configured.

testbox-runner

You can also specify a specific configuration file:

testbox-runner --config /path/to/config/file.json

Command Line Arguments

Simply run the utility and pass the above configuration options prefixed with --.

Example

testbox-runner 
    --runner http://localhost/testbox/system/runners/HTMLRunner.cfm 
    --directory /tests 
    --recurse true

TestBox ships with a test browser that is highly configurable to whatever URL accessible path you want. It will then show you a test browser where you can navigate and execute not only individual tests, but also directory suites as well.

• BoxLang: /testbox/bx/test-browser

• CFML: /testbox/cfml/test-browser

It is also a mini web application that can be configured to whatever root folder you desire. It will read the runners and tests from that folder and present a GUI that you can use to navigate the test folders and execute them easily.

TestBox ships with a global runner that can be used to run pretty much anything. You can customize it or place it wherever you need it. You can find it in your distribution under:

• BoxLang: /testbox/bx/test-browser

• CFML: /testbox/cfml/test-browser

This is a mini web application to help you run bundles, directory, specs and more.

A test suite in TestBox is a collection of specifications that model what you want to test. As we will investigate, the way the suite is expressed can be of many different types.

Test suite is a container that has a set of tests which helps testers in executing and reporting the test execution status.

A test suite begins with a call to our TestBox describe() function with at least two arguments: a title and a body function/closure. The title is the name of the suite to register and the body function/closure is the block of code that implements the suite.

When applying BDD to your tests, this function is used to describe your story scenarios that you will implement.

Arguments

There are more arguments, which you can see below:

As we have seen before, the describe() function describes a test suite of related specs in a test bundle CFC. The title of the suite is concatenated with the title of a spec to create a full spec's name which is very descriptive. If you name them well, they will read out as full sentences as defined by style.

Nesting describe Blocks

Calls to our describe() function can be nested with specs at any level or point of execution. This allows you to create your tests as a related tree of nested functions. Please note that before a spec is executed, TestBox walks down the tree executing each beforeEach() and afterEach() function in the declared order. This is a great way to logically group specs in any level as you see fit.

is a style of writing tests where you describe the state of the code you want to test (Given), the behavior you want to test (When) and the expected outcome (Then). (See )

Testbox supports the use of function names given() and when() in-place of describe() function calls. The then() function call is an alternative for it() function calls. The advantage of this style of is that you can gather your requirements and write your tests in a common language that can be understood by developers and stake-holders alike. This common language format is often referred to as the language; using it we can gather and document the requirements as:

TestBox provides you with feature(), scenario() and story() wrappers for describe() blocks. As such we can write our requirements in test form like so:

The output from running the test will read as the original requirements, providing you with not only automated tests but also a living document of the requirements in a business-readable format.

As feature(), scenario() and story() are wrappers for describe() you can intermix them so that your can create tests which read as the business requirements. As with describe(), they can be nested to build up blocks.

With TestBox's BDD syntax, it is possible to create suites dynamically; however, there are a few things to be aware of.

Setup for dynamic suites must be done in the pseudo-constructor (versus in beforeAll()). This is because variables-scoped variables set in beforeAll() are not available in the describe closure (even though they are available in it closures). This behavior can be explained by the execution sequence of a BDD bundle: When the bundle's run() method is called, it collects preliminary test data via describes. After preliminary test data are collected, the beforeAll() runs, followed by the describe closures.

Additionally, care must be taken to pass data into the it closures, otherwise strange behavior will result (the values from the last loop iteration will be repeated in the body of each looped it).

Example

The following bundle creates suites dynamically, by looping over test metadata.

Expectations are self-concatenated strings that evaluate an actual value to an expected value or condition. These are initiated by the global TestBox method called expect() which takes in a value called the actual value or expectAll() which takes in an array or struct which will be the actual value. It is concatenated in our expectations DSL with a matcher function that will most likely take in an expected value or condition to test. You can also concatenate the matchers and do multiple evaluations on a single actual value.

Matchers

Each matcher implements a comparison or evaluation of the actual value and an expected value or condition. It is responsible for either passing or failing this evaluation and reporting it to TestBox. Each matcher also has a negative counterpart assertion by just prefixing the call to the matcher with a not expression.

Included Matchers

Custom Matchers

Specs and suites can be skipped from execution by prefixing certain functions with the letter x or by using the skip argument in each of them or by using the skip( message, detail ) function. The reporters will show that these suites or specs were skipped from execution. The functions you can prefix are:

• it()

• describe()

• story()

• given()

• when()

• then()

• feature()

Here are some examples:

Skip Argument

The skip argument can be a boolean value or a closure. If the value is true then the suite or spec is skipped. If the return value of the closure is true then the suite or spec is skipped. Using the closure approach allows you to dynamically at runtime figure out if the desired spec or suite is skipped. This is such a great way to prepare tests for different CFML engines.

Skip Method

You can now use the skip( message, dteail ) method to skip any spec or suite a-la-carte instead of as an argument to the function definitions. This lets you programmatically skip certain specs and suites and pass a nice message.

Specs and suites can be tagged with TestBox labels. Labels allows you to further categorize different specs or suites so that when a runner executes with labels attached, only those specs and suites will be executed, the rest will be skipped. You can alternatively choose to skip specific labels when a runner executes with excludes attached.

Specs and suites can be focused so ONLY those suites and specs execute. You will do this by prefixing certain functions with the letter f or by using the focused argument in each of them. The reporters will show that these suites or specs where execute ONLY The functions you can prefix are:

• it()

• describe()

• story()

• given()

• when()

• then()

• feature()

A spec is a declaration that will usually test your system with a requirement. They are defined by calling the TestBox it() global function, which takes in a title and a body function/closure. The title is the title of this spec you will write and the body function/closure is a block of code that represents the spec.

A spec will contain most likely one or more expectations that will test the state of the (software under test) or sometimes referred to as code under test. In BDD style, your specifications are what is used to validate your requirements of a scenario which is your describe() block of your .

An expectation is a nice assertion DSL that TestBox exposes so you can pretty much read what should happen in the testing scenario. A spec will pass if all expectations pass. A spec with one or more expectations that fail will fail the entire spec.

Arguments

They are closures Ma!

Spec Data Binding

The data argument can be used to pass in a structure of data into the spec so it can be used later within the body closure. This is great when doing looping and creating dynamic closure calls: