Improvement

TESTBOX-370 `toHaveKey` works on queries in Lucee but not ColdFusion

TESTBOX-373 Update to `cbstreams` 2.x series for compat purposes.

New Feature

TESTBOX-375 Updated mixerUtil for faster performance and new approaches to dynamic mixins

TESTBOX-376 Add `bundlesPattern` to testbox.system.TestBox `init` method

TESTBOX-377 TestBox Modules

Bug

TESTBOX-346 expect(sut).toBeInstanceOf("something")) breaks if sut is a query

TESTBOX-374 cbstreams doesn't entirely work outside of ColdBox

Improvement

TESTBOX-20 toBeInstanceOf() Expectation handle Java classes

TestBox 5.x series is a major bump in our library. Here are the major areas of improvement and the full release notes.

Engine Support

We have dropped Adobe 2016 support and added support for Adobe 2023 and Lucee 6+

Batch Test Coverage Reporting

Due to memory limitations in CI environments, larger codebases cannot run all tests as a single testbox run command. Instead, specs are run in a methodical folder-by-folder sequence, separating the testbox run out over many requests and thus working around the Out-Of-Memory exceptions.

While this works, it prevents accurate code coverage reporting since only a small portion of the tests are executed during any request. The generated code coverage report only shows a tiny fraction of the coverage - say, 2% - and not the whole picture

TestBox 5 introduces a CoverageReporter component which

1. Runs on every TestBox code coverage execution

2. Loads any previous coverage data from a JSON file

3. Combines the previous coverage data with the current execution's coverage data (file by file and line by line)

4. Persists the COMBINED coverage data to a JSON file.

5. Returns the COMBINED coverage data for the CoverageBrowser.cfc to build as an HTML report

When setting url.isBatched=true and executing the batched test runner, the code coverage report will grow with each sequential testbox run command.

Method Spies!

MockBox now supports a $spy( method ) method that allows you to spy on methods with all the call log goodness but without removing all the methods. Every other method remains intact, and the actual spied method remains active. We decorate it to track its calls and return data via the $callLog() method.

Example of CUT:

void function doSomething(foo){
  // some code here then...
  local.foo = variables.collaborator.callMe(local.foo);
  variables.collaborator.whatever(local.foo);
}

Example Test:

function test_it(){
  local.mocked = createMock( "com.foo. collaborator" )
    .$spy( "callMe" )
    .$spy( "whatever" );
  variables.CUT.$property( "collaborator", "variables", local.mocked );
  assertEquals( 1, local.mocked.$count( "callMe" ) );
  assertEquals( 1, local.mocked.$count( "whatever" ) );
}

Performance Improvements

We have focused on this release to lazy load everything as much as possible to allow for much better testing performance. Check it out!

Skip it! Skip it -> Good!

You can now use the skip( message ) method to skip any spec or suite a-la-carte instead of as an argument to the function definitions. This lets you programmatically skip certain specs and suites and pass a nice message.

it( "can do something", () => {
    ...
    if( condition ){
        skip( "Condition is true, skipping spec" )
    }
    ...
} )

Release Notes

Fixed

• TESTBOX-341 toHaveLength param should be numeric

• TESTBOX-354 Element $DEBUGBUFFER is undefined in THIS

• TESTBOX-356 Don't assume TagContext has length on simple reporter

• TESTBOX-357 notToThrow() incorrectly passes when no regex is specified

• TESTBOX-360 full null support not working on Application env test

• TESTBOX-361 MockBox Suite: Key [aNull] doesn't exist

• TESTBOX-362 Cannot create subfolders within testing spec directories.

Improvements

• TESTBOX-333 Add contributing.md to the repo

• TESTBOX-339 full null support automated testing

• TESTBOX-353 allows globbing path patterns in test bundles argument

• TESTBOX-355 Add debugBuffer to JSONReporter

• TESTBOX-366 ANTJunit Reporter better visualization of the failed origin and details

• TESTBOX-368 Support list of Directories for HTMLRunner to allow a more modular tests structure

• TESTBOX-370 `toHaveKey` works on queries in Lucee but not ColdFusion

Added

• TESTBOX-371 Add CoverageReporter for batching code coverage reports

• TESTBOX-137 Ability to spy on existing methods: $spy()

• TESTBOX-342 Add development dependencies to box.json

• TESTBOX-344 Performance optimizations for BaseSpec creations by lazy loading external objects

• TESTBOX-345 add a skip([message]) like fail() for skipping from inside a spec

• TESTBOX-365 New build process using CommandBox

• TESTBOX-372 Adobe 2023 and Lucee 6 Support

TestBox BDD v5.x

TestBox is a next-generation testing framework for the BoxLang JVM language and ColdFusion (CFML) based on BDD (Behavior Driven Development) for providing a clean, obvious syntax for writing tests. It contains not only a testing framework, console/web runner, assertions, and expectations library but also ships with MockBox, A mocking and stubbing companion.

class{

  function run(){
  	describe( "My calculator features", () => {
	
		beforeEach( () => {
			variables.calc = new Calculator()
		} )
			
		// Using expectations library
		it( "can add", () => {
			expect( calc.add(1,1) ).toBe( 2 )
		} )
		
		// Using assert library
		test( "it can multiply", () => {
			assertIsEqual( calc.multiply(2,2), 4 )
		} )
	} )
  }

}

/**
 * My calculator features
 */
class{

	property calc;
	
	function setup(){
	    calc = new Calculator()
	}
	
	// Function name includes the word 'test'
	// Using expectations library
	function testAdd(){
	    expect( calc.add(1,1) ).toBe( 2 )
	}
		
	// Any name, but with a test annotation
	// Using assertions library
	@test
	function itCanMultiply(){
	    $assert.isEqual( calc.multiply(2,2), 4 )
	}
}

component{

  function run(){
  	describe( "My calculator features", () => {
	
		beforeEach( () => {
			variables.calc = new Calculator()
		} );
			
		// Using expectations library
		it( "can add", () => {
			expect( calc.add(1,1) ).toBe( 2 )
		} );
		
		// Using assert library
		test( "it can multiply", () => {
			$assert.isEqual( calc.multiply(2,2), 4 )
		} );
	} );
  }

}

/**
 * My calculator features
 */
component{
	
	property calc;
	
	function setup(){
	    calc = new Calculator()
	}
	
	// Function name includes the word 'test'
	// Using expectations library
	function testAdd(){
	    expect( calc.add(1,1) ).toBe( 2 )
	}
		
	// Any name, but with a test annotation
	// Using assertions library
	function itCanMultiply() test{
	    $assert.isEqual( calc.multiply(2,2), 4 )
	}
}

Features At A Glance

Here is a simple listing of features TestBox brings to the table:

• BDD style or xUnit style testing

• Testing life-cycle methods

• MockBox integration for mocking and stubbing

• Mocking data library for mocking JSON/complex data and relationships

• Ability to extend and create custom test runners and reporters

• Extensible reporters, bundled with tons of them:

  • JSON

  • XML

  • JUnit XML

  • Text

  • Console

  • TAP (Test Anything Protocol)

  • Simple HTML

  • Min - Minimalistic Heaven

  • Raw

  • CommandBox

• Asynchronous testing

• Multi-suite capabilities

• Test skipping

• Test labels and tagging

• Testing debug output stream

• Code Coverage via FusionReactor

• Much more!

Versioning

TestBox is maintained under the Semantic Versioning guidelines as much as possible. Releases will be numbered in the following format:

<major>.<minor>.<patch>

And constructed with the following guidelines:

• bumpBreaking backward compatibility bumps the major (and resets the minor and patch)

• New additions without breaking backward compatibility bump the minor (and resets the patch)

• Bug fixes and misc changes bump the patch

License

TestBox is open source and licensed under the Apache 2 License. If you use it, please try to mention it in your code or website.

• Copyright by Ortus Solutions, Corp

• TestBox is a registered trademark by Ortus Solutions, Corp

Discussion & Help

• Help Group: https://community.ortussolutions.com/c/communities/testbox/11

• BoxTeam Slack : https://boxteam.ortussolutions.com

Reporting a Bug

We all make mistakes from time to time :) So why not let us know about it and help us out? We also love pull requests, so please star us and fork us: https://github.com/Ortus-Solutions/TestBox

• By Jira: https://ortussolutions.atlassian.net/browse/TESTBOX

Professional Open Source

TestBox is a professional open source software backed by Ortus Solutions, Corp offering services like:

• Custom Development

• Professional Support & Mentoring

• Training

• Server Tuning

• Security Hardening

• Code Reviews

• Much More

Resources

• Official Site: https://www.ortussolutions.com/products/testbox

• Current API Docs: https://apidocs.ortussolutions.com/testbox/current

• Help Group: https://community.ortussolutions.com/c/communities/testbox/11

• Source Code: https://github.com/Ortus-Solutions/TestBox

• Bug Tracker: https://ortussolutions.atlassian.net/browse/TESTBOX

• Twitter: @ortussolutions

• Facebook: https://www.facebook.com/ortussolutions

HONOR GOES TO GOD ABOVE ALL

Because of His grace, this project exists. If you don't like this, don't read it, it's not for you.

Therefore being justified by faith, we have peace with God through our Lord Jesus Christ: By whom also we have access by faith into this grace wherein we stand, and rejoice in hope of the glory of God. - Romans 5:5

In this section, you will find the release notes for each version we release under this major version. If you are looking for the release notes of previous major versions, use the version switcher at the top left of this documentation book. Here is a breakdown of our major version releases.

Version 5.x - May 2023

In this release, we have dropped legacy engines and added support for the JVM language, Adobe 2023 and Lucee 6. We have also added major updates to spying and expectations. We continue in this series to focus on productivity and fluency in the Testing language in preparation for more ways to test.

Version 4.x - April 2020

In this release, we have dropped support for legacy CFML engines and introduced the ability to mock data and relationships and build JSON documents.

Version 3.x

In this release, we focused on dropping engine supports for legacy CFML engines. We had a major breakthrough in introducing Code Coverage thanks to the folks as well. This major release also came with a new UI for all reporters and streamlined the result viewports.

Version 2.x

This version spawned off with over 8 minor releases. We focused on taking TestBox 1 to yet a high level. Much more attention to detail and introducing modern paradigms like given-when-then. Multiple interception points, async executions, and ability to chain methods.

Version 1.x

This was our first major version of TestBox. We had completely migrated from MXUnit, and it introduced BDD to the ColdFusion (CFML) world.

Luis Fernando Majano Lainez

Luis Majano is a Computer Engineer with over 16 years of software development and systems architecture experience. He was born in in the late 70’s, during a period of economical instability and civil war. He lived in El Salvador until 1995 and then moved to Miami, Florida where he completed his Bachelors of Science in Computer Engineering at . Luis resides in The Woodlands, Texas with his beautiful wife Veronica, baby girl Alexia and baby boy Lucas!

Luis has a passion for Jesus, tennis, golf, volleyball and anything electronic. Random Author Facts:

• He played volleyball in the Salvadorean National Team at the tender age of 17

• The Lord of the Rings and The Hobbit is something he reads every 5 years. (Geek!)

• His first ever computer was a Texas Instrument TI-86 that his parents gave him in 1986. After some time digesting his very first BASIC book, he had written his own tic-tac-toe game at the age of 9. (Extra geek!)

• He has a geek love for circuits, microcontrollers and overall embedded systems.

• He has of late (during old age) become a fan of running and bike riding with his family.

Keep Jesus number one in your life and in your heart. I did and it changed my life from desolation, defeat and failure to an abundant life full of love, thankfulness, joy and overwhelming peace. As this world breathes failure and fear upon any life, Jesus brings power, love and a sound mind to everybody!

“Trust in the LORD with all your heart, and do not lean on your own understanding.” Proverbs 3:5

Contributors

Jorge Emilio Reyes Bendeck

Jorge started working as project manager and business developer at Ortus Solutions, Corp. in 2013, . At Ortus he fell in love with software development and now enjoys taking part on software development projects and software documentation! He is a fellow Cristian who loves to play the guitar, worship and rejoice in the Lord!

Therefore, if anyone is in Christ, the new creation has come: The old has gone, the new is here! 2 Corinthians 5:17

5.3.1 - September 13, 2023

Fixed

• The variable thisSuite isn't defined if the for loop in the try/catch is never reached before the error. ()

5.3.0 - August 1, 2023

New Features

• New expectations: toBeIn(), toBeInWithCase() so you can verify a needle in string or array targets

• New matchers and assertions: toStartWith(), toStartWithCase(), startsWith(), startsWthCase() and their appropriate negations

• New matchers and assertions: toEndWith(), toEndWithCase(), endsWith(), endsWithCase() and their appropriate negations

Bugs

• onSpecError suiteSpecs is invalid, it's suiteStats

The source code for this book is hosted on GitHub: . You can freely contribute to it and submit pull requests. The contents of this book is copyrighted by and cannot be altered or reproduced without the author's consent. All content is provided "As-Is" and can be freely distributed.

External Trademarks & Copyrights

Flash, Flex, ColdFusion, and Adobe are registered trademarks and copyrights of Adobe Systems, Inc.

BoxLang, ColdBox, CommandBox, FORGEBOX, TestBox, ContentBox, and Ortus Solutions are all trademarks and copyrights of Ortus Solutions, Corp.

Notice of Liability

The information in this book is distributed “as is” without warranty. The author and Ortus Solutions, Corp shall not have any liability to any person or entity concerning loss or damage caused or alleged to be caused directly or indirectly by the content of this training book, software, and resources described in it.

Contributing

We highly encourage contributions to this book and our open-source software. The source code for this book can be found in our where you can submit pull requests.

Charitable Proceeds

15% of the proceeds of this book will go to charity to support orphaned kids in El Salvador - . So please donate and purchase the printed version of this book; every book sold can help a child for almost 2 months.

Shalom Children's Home

Shalom now cares for over 80 children in El Salvador, from newborns to 18 years old. They receive shelter, clothing, food, medical care, education, and life skills training in a Christian environment. The home is supported by a child sponsorship program.

We have personally supported Shalom for over 6 years now; it is a place of blessing for many children in El Salvador who either has no families or have been abandoned. This is a good earth to seed and plant.