TestBox ships with a new BoxLang CLI runner for Linux/Mac and Windows. This will allow you to execute your tests from the CLI and, in the future, via VSCode easily and extremely fast. It can also execute and stream the executions so you can see the testing progress when running in verbose mode. The runner can also execute specs/tests that are written in CFML or BoxLang in the BoxLang runtime.

BoxLang allows you to not only build web applications, but CLI, serverless, Android, etc. You can use this runner to test each environment. However, please note that if you will be doing web server testing from the CLI only, you will need to install the web support module into the Operating System runtime.

Web Server Testing

If you want to test from the CLI your web application with no web server, then you will need to install the bx-web-support module into the CLI. Remember that BoxLang is multi-runtime, and you not only can build web applications but CLI or OS based applications.

// CommandBox
install bx-web-support

// BoxLang OS Binary
install-bx-module bx-web-support

This will add web support to the CLI (BIFS, components, etc.) and a mock HTTP server so you can do full life-cycle testing from the CLI like if running your app in a web server. This runner does not require a web server to function, thus if you are building a web app, you will need this module if you still want to continue to execute your tests in the CLI Runtime.

Script Locations

The scripts are located in the following directory: /testbox/bin from the TestBox installation package.

• run - For Linux/Mac

• run.bat - For Windows

This is the entry point for executing tests at the CLI level. Please note that the test execution does NOT involve a web server. This is for pure CLI testing.

Examples:

The runner must be run from the root of your BoxLang project:

Mac/Linux

./testbox/bin/run
./testbox/bin/run my.bundle
./testbox/bin/run --directory=tests.specs
./testbox/bin/run --bundles=my.bundle

Windows Examples:

./testbox/bin/run.bat
./testbox/bin/run.bat my.bundle
./testbox/bin/run.bat --directory=tests.specs
./testbox/bin/run.bat --bundles=my.bundle

Execution Options

• --bundles A list of test bundles to run, defaults to *, ex: path.to.bundle1,path.to.bundle2, . Mutually exclusive with --directory

• --bundles-pattern A pattern to match test bundles defaults to "*Spec*.cfc|*Test*.cfc|*Spec*.bx|*Test*.bx"

• --directory A list of directories to look for tests to execute. Please use dot-notation, not absolute notation. Mutually exclusive with --bundles. Ex: tests.specs Defaults to tests.specs

• --recurse : Recurse into subdirectories, defaults to true

• --eager-failure : Fail fast, defaults to false

• --verbose : Verbose output defaults to false. This will stream the output of the status of the tests as they run.

• --runner-options: A JSON struct literal of options to pass into the test runner. Ex: {"verbose"=true}

Reporting Options

• --reporter The reporter to use.

• --reportpath : The path to write the report file, defaults to the /tests/results folder by convention

• --properties-summary : Generate a properties file with the summary of the test results, defaults to true.

• --properties-filename : The name of the properties file to generate defaults to TEST.properties

• --write-report : Write the report to a file in the report path folder, defaults to true

• --write-json-report : Write the report as JSON alongside the requested report, defaults to false

• --write-visualizer : Write the visualizer to a file in the report path folder, defaults to false

Filtering Options

• --labels : A list of labels to run, defaults to *

• --excludes : A list of labels to exclude, defaults to empty

• --filter-bundles : A list of bundles to filter by, defaults to *

• --filter-suites : A list of suites to filter by, defaults to *

• --filter-specs : A list of test names or spec names to filter by, defaults to *

This is more of an approach than an actual specifc runner. This approach shows you that you can create a script file in BoxLang (bxs) or in CFML (cfs|cfm) that can in turn execute any test bundle(s) with many many runnable configurations.

BoxLang

The BoxLang language allows you to run your scripts via the CLI or the browser if you have a web server attached to your project.

// Test the BDD Bundle
r = new testbox.system.TestBox( "tests.specs.BDDTest" )
println( r.run() );

// Test the bundle with ONLY the passed specs
r = new testbox.system.TestBox( "tests.specs.BDDTest" )
println( r.run( testSpecs="OnlyThis,AndThis,AndThis" ) )

// Test the bundle with ONLY the passed suites
r = new testbox.system.TestBox( "tests.specs.BDDTest" )
println( r.run( testSuites="Custom Matchers,A Spec" ) )

// Test the passed array of bundles
r = new testbox.system.TestBox( [ "tests.specs.BDDTest", "tests.specs.BDD2Test" ] )
println( r.run() )

// Test with labels and the minimal reporter
r = new testbox.system.TestBox( bundles: "tests.specs.BDDTest", labels="linux" )
println( r.run( reporter: "mintext" ) )

If you want to run it in the CLI, then just use:

boxlang run.bxs

If you want to run it via the web server, place it in your /tests/ folder and run it

http://localhost/tests/run.bxs

CFML

CFML engines only allow you to run tests via the browser. So create your script, place it in your web accessible /tests folder and run it.

<cfscript>
	// Test the BDD Bundle
	r = new testbox.system.TestBox( "tests.specs.BDDTest" )
	writeOutput( r.run() );
	
	// Test the bundle with ONLY the passed specs
	r = new testbox.system.TestBox( "tests.specs.BDDTest" )
	writeOutput( r.run( testSpecs="OnlyThis,AndThis,AndThis" ) )
	
	// Test the bundle with ONLY the passed suites
	r = new testbox.system.TestBox( "tests.specs.BDDTest" )
	writeOutput( r.run( testSuites="Custom Matchers,A Spec" ) )
	
	// Test the passed array of bundles
	r = new testbox.system.TestBox( [ "tests.specs.BDDTest", "tests.specs.BDD2Test" ] )
	writeOutput( r.run() )
	
	// Test with labels and the minimal reporter
	r = new testbox.system.TestBox( bundles: "tests.specs.BDDTest", labels="linux" )
	writeOutput( r.run( reporter: "mintext" ) )
</cfscript>

This is more of an approach than an actual specifc runner. This approach shows you that you can create a script file in BoxLang (bxs) or in CFML (cfs|cfm) that can in turn execute any test directory with many many runnable configurations. It is very similar to the Bundle Runner approach.

BoxLang

The BoxLang language allows you to run your scripts via the CLI or the browser if you have a web server attached to your project.

// Run all the specs in the tests.specs directory and subdirectories
r = new testbox.system.TestBox( directory="tests.specs" )
println( r.run() )

// Run all the specs in the tests.specs directory ONLY
r = new testbox.system.TestBox(
		directory={
            mapping="tets.specs",
            recurse=false
      } )
println( r.run() )

// Run all the specs in the tests.specs directory and subdirectories using
// a custom lambda filter
r = new testbox.system.TestBox(
      	directory={
            mapping : "tets.specs",
            filter : path -> findNoCase( "test", arguments.path ) ? true : false
      }) >
println( r.run() )

// Run all the specs in the tests.specs directory and subdirectories using
// a custom lambda filter and create a JSON report
r = new testbox.system.TestBox(
      	directory={
            mapping="tets.specs",
            filter : path -> findNoCase( "test", arguments.path ) ? true : false
      }) >
fileWrite( 'testreports.json', r.run() )
println( "JSON report created" )

If you want to run it in the CLI, then just use:

boxlang run.bxs

If you want to run it via the web server, place it in your /tests/ folder and run it

http://localhost/tests/run.bxs

CFML

CFML engines only allow you to run tests via the browser. So create your script, place it in your web accessible /tests folder and run it.

<cfset r = new testbox.system.TestBox( directory="tests.specs" ) >
<cfoutput>#r.run()#</cfoutput>

<cfset r = new testbox.system.TestBox(
      directory={
            mapping="tets.specs",
            recurse=false
      }) >
<cfoutput>#r.run()#</cfoutput>

<cfset r = new testbox.system.TestBox(
      directory={
            mapping="tets.specs",
            recurse=true,
            filter=function(path){
                  return ( findNoCase( "test", arguments.path ) ? true : false );
            }
      }) >
<cfoutput>#r.run()#</cfoutput>

<cfset r = new testbox.system.TestBox(
      directory={
            mapping="tets.specs",
            recurse=true,
            filter=function(path){
                  return ( findNoCase( "test", arguments.path ) ? true : false );
            }
      }) >
<cfset fileWrite( 'testreports.json', r.run() )>

By installing the CommandBox TestBox CLI you can get access to our CommandBox runner. The CommandBox runner leverages the HTTP(s) protocol to test against any server. By default it will inspect your box.json for a default runner or try to connect to /tests/runner.cfm by default.

To see all the running options run the following in your CLI shell:

testbox run help

testbox run directory="tests.specs" outputFormats="json,junit,html"

testbox run runner="http://myremoteapp.com/tests/runner.cfm"

It can also produce reports for you in JSON, HTML, and JUNIT.

Runner Options

If you type testbox run --help you can see all the arguments you can set for running your tests. However, please note that you can also pre-set them in your box.json under the testbox entry:

"testbox":{
    "bundles":"",
    "directory":"tests.specs",
    "excludes":"",
    "labels":"",
    "options":{},
    "recurse":true,
    "reporter":"",
    "runner":[
        {
            "default":""
        }
    ],
    "testBundles":"",
    "testSpecs":"",
    "testSuites":"",
    "verbose":true,
    "watchDelay":500,
    "watchPaths":"**.cfc"
},

Runner URL

You can also set up the default runner URL in your box.json and it will be used for you. Setting the URL is a one-time operation.

package set testbox.runner="http://localhost:8080/tests/runner.cfm"
testbox run

You can also use a relative path and CommandBox will look up the host and port from your server settings.

package set testbox.runner="/tests/runner.cfm"
testbox run

The default runner URL of the testbox run command is /tests/runner.cfm so there's actually no need to even configure it if you're using the default convention location for your runner.

Multiple Runner URLs

You can define multiple URLs for your runners by using a JSON array of objects. Each key will be a nice identifier you can use via the runner=key argument in the command.

"testbox" : {
    "runner" : [
        { "core"   : "http://localhost/tests/runner.cfm" },
        { "api" : "http://localhost/api/tests/runner.cfm" }
    ]
}

Then you can just pass in the name:

testbox run runner="core"

More Commands:

package set testbox.runner="[ { default : 'http://localhost/tests/runner.cfm' } ]" --append
package show testbox.runner
testbox run default

Watcher

The CLI also comes with a code watcher and runner. It will watch any paths for you, and if it detects any changes, it will run the tests you want.

testbox watch help

In order for this command to work, you need to have started your server and configured the URL to the test runner in your box.json.

package set testbox.runner=http://localhost:8080/tests/runner.cfm
server start
testbox watch

You can also control what files to watch.

testbox watch **.cfc

If you need more control over what tests run and their output, you can set additional options in your box.json which will be picked up automatically by testbox run when it fires.

package set testbox.verbose=false
package set testbox.labels=foo
package set testbox.testSuites=bar
package set testbox.watchDelay=1000
package set testbox.watchPaths=/models/**.cfc

This command will run in the foreground until you stop it. When you are ready to shut down the watcher, press Ctrl+C.

TestBox tests can be run from different sources from what we call Runners. These can be from different sources:

• CLI

  • TestBox CLI (Powered by CommandBox)

  • BoxLang Scripts

  • NodeJS

• Web Server

  • Runner

  • TestBundle Directly

• Custom

Your test harness already includes the web runner: runner.bx or runner.cfm. You can execute that directly in your browser to get the results or run it via the CLI: testbox run. We invite you to explore the different runners available to you.

Custom Runners

However, you can create your own custom runners as long as you instantiate the TestBox class and execute one of it's runnable methods. The main execution methods are:

// Run tests and produce reporter results
testbox.run()

// Run tests and get raw testbox.system.TestResults object
testbox.runRaw()

// Run tests and produce reporter results from SOAP, REST, HTTP
testbox.runRemote()

// Run via Spec URL
http://localhost/tests/spec.cfc?method=runRemote

// Via CommandBox
testbox run

run()

Here are the arguments you can use for initializing TestBox or executing the run() method

/**
 * Run me some testing goodness, this can use the constructed object variables or the ones
 * you can send right here.
 *
 * @bundles      The path, list of paths or array of paths of the spec bundle classes to run and test
 * @directory    The directory to test which can be a simple mapping path or a struct with the following options: [ mapping = the path to the directory using dot notation (myapp.testing.specs), recurse = boolean, filter = closure that receives the path of the class found, it must return true to process or false to continue process ]
 * @reporter     The type of reporter to use for the results, by default is uses our 'simple' report. You can pass in a core reporter string type or an instance of a testbox.system.reports.IReporter. You can also pass a struct if the reporter requires options: {type="", options={}}
 * @labels       The list or array of labels that a suite or spec must have in order to execute.
 * @excludes     The list or array of labels that a suite or spec must not have in order to execute.
 * @options      A structure of configuration options that are optionally used to configure a runner.
 * @testBundles  A list or array of bundle names that are the ones that will be executed ONLY!
 * @testSuites   A list or array of suite names that are the ones that will be executed ONLY!
 * @testSpecs    A list or array of test names that are the ones that will be executed ONLY!
 * @callbacks    A struct of listener callbacks or a class with callbacks for listening to progress of the testing: onBundleStart,onBundleEnd,onSuiteStart,onSuiteEnd,onSpecStart,onSpecEnd
 * @eagerFailure If this boolean is set to true, then execution of more bundle tests will stop once the first failure/error is detected. By default this is false.
 */
any function run(
	any bundles,
	any directory,
	any reporter,
	any labels,
	any excludes,
	struct options,
	any testBundles      = [],
	any testSuites       = [],
	any testSpecs        = [],
	any callbacks        = {},
	boolean eagerFailure = false
)

• The bundles argument which can be a single CFC path or an array of CFC paths or a directory argument so it can go and discover the test bundles from that directory.

• The reporter argument can be a core reporter name like: json,xml,junit,raw,simple,dots,tap,min,etc or it can be an instance of a reporter CFC.

• You can execute the runners from any cfm template or any CFC or any URL, that is up to you.

There is a user-contributed NodeJS Runner that looks fantastic and can be downloaded here: https://www.npmjs.com/package/testbox-runner

You can use node to install as well into your projects.

npm install -g testbox-runner

Configuration

Create a config file called .testbox-runnerrc in the root of your web project.

{
	"runner": "http://localhost/testbox/system/runners/HTMLRunner.cfm",
	"directory": "/tests/specs",
	"recurse": true
}

Then use the CLI command to run whatever you configured.

testbox-runner

You can also specify a specific configuration file:

testbox-runner --config /path/to/config/file.json

Command Line Arguments

Simply run the utility and pass the above configuration options prefixed with --.

Example

testbox-runner 
    --runner http://localhost/testbox/system/runners/HTMLRunner.cfm 
    --directory /tests 
    --recurse true

TestBox ships with a test browser that is highly configurable to whatever URL accessible path you want. It will then show you a test browser where you can navigate and execute not only individual tests, but also directory suites as well.

• BoxLang: /testbox/bx/test-browser

• CFML: /testbox/cfml/test-browser

It is also a mini web application that can be configured to whatever root folder you desire. It will read the runners and tests from that folder and present a GUI that you can use to navigate the test folders and execute them easily.

TestBox ships with a global runner that can be used to run pretty much anything. You can customize it or place it wherever you need it. You can find it in your distribution under:

• BoxLang: /testbox/bx/test-browser

• CFML: /testbox/cfml/test-browser

This is a mini web application to help you run bundles, directory, specs and more.

Every test harness comes with a runner.bx or runner.cfm in the root of the tests folder. This is called the web runner and is executable via the web server you are running your application on. This will execute all the tests by convention found in the tests/specs folder.

You can open that file and customize it as you see fit. Here is an example of such a file:

Test Bundle Execution

If you make your test bundle class inherit from our testbox.system.BaseSpec class, you will be able to execute the class directly via the URL:

Arguments

All the arguments found in the runner are available as well in a direct bundle execution:

• labels: The labels to apply to the execution

• testMethod : A list or array of xunit test names that will be executed ONLY!

• testSuites : A list or array of suite names that are the ones that will be executed ONLY!

• testSpecs : A list or array of test names that are the ones that will be executed ONLY!

• reporter : The type of reporter to run the test with

In our test harness we include an ANT runner (test.xml) that will be able to execute your tests via ANT.

It can also leverage our ANTJunit reporter to use the junitreport task to produce JUnit compliant reports as well.

Now you can run the commands