1 of 11

Running Tests

TestBox ships with several test runners internally but we have tried to simplify and abstract it with our TestBox object which can be found in the testbox.system package. The testbox object allows you to execute tests from a CFC, CFM, HTTP, SOAP, NodeJS or REST. You can also make your CFC's extend from our BaseSpec class so you can execute it directly via the URL. The main execution methods are:

// Run tests and produce reporter results
testbox.run()

// Run tests and get raw testbox.system.TestResults object
testbox.runRaw()

// Run tests and produce reporter results from SOAP, REST, HTTP
testbox.runRemote()

// Run via Spec URL
http://localhost/tests/spec.cfc?method=runRemote

We encourage you to read the API docs included in the distribution for the complete parameters for each method.

`run()` Arguments

Here are the arguments you can use for initializing TestBox or executing the run() method

`runRemote()` Arguments

Here are the arguments you can use for executing the runRemote() method of the TestBox object:

Notes

The bundles argument which can be a single CFC path or an array of CFC paths or a directory argument so it can go and discover the test bundles from that directory.
The reporter argument can be a core reporter name like: json,xml,junit,raw,simple,dots,tap,min,etc or it can be an instance of a reporter CFC.
You can execute the runners from any cfm template or any CFC or any URL, that is up to you.

Run Listeners

Every run and runRaw methods now accept a callbacks argument, which can be a CFC with the right listener methods or a struct with the right closure methods. This will allow you to listen to the testing progress and get information about it. This way you can build informative reports or progress bars.

The available callbacks are:

Global Runner

TestBox ships with a global runner that can be used to run pretty much anything. You can customize it or place it wherever you need it:

Test Browser

TestBox ships with a test browser that is highly configurable to whatever URL accessible path you want. It will then show you a test browser where you can navigate and execute not only individual tests, but also directory suites as well.

Bundle(s) Runner

<cfset r = new coldbox.system.TestBox( "coldbox.testing.cases.testing.specs.BDDTest" ) >
<cfoutput>#r.run()#</cfoutput>

<cfset r = new coldbox.system.TestBox( bundles="coldbox.testing.cases.testing.specs.BDDTest" ) >
<cfoutput>#r.run( testSpecs="OnlyThis,AndThis,AndThis" )#</cfoutput>

<cfset r = new coldbox.system.TestBox( bundles="coldbox.testing.cases.testing.specs.BDDTest" ) >
<cfoutput>#r.run( testSuites="Custom Matchers,A Spec" )#</cfoutput>

<cfset r = new coldbox.system.TestBox( [ "coldbox.testing.cases.testing.specs.BDDTest", "coldbox.testing.cases.testing.specs.BDD2Test" ] ) >
<cfoutput>#r.run()#</cfoutput>

<cfset r = new coldbox.system.TestBox( bundles="coldbox.testing.cases.testing.specs.BDDTest", labels="railo" ) >
<cfoutput>#r.run(reporter="json")#</cfoutput>

Test Runner

If you make your test bundle CFC inherit from our testbox.system.BaseSpec class, you will be able to execute the CFC directly via the URL:

http://localhost/test/MyTest.cfc?method=runRemote

You can also pass the following arguments to the method via the URL:

testSuites : A list or array of suite names that are the ones that will be executed ONLY!
testSpecs : A list or array of test names that are the ones that will be executed ONLY!
reporter : The type of reporter to run the test with

http://localhost/test/MyTest.cfc?method=runRemote&reporter=json

Directory Runner

<cfset r = new coldbox.system.TestBox( directory="coldbox.testing.cases.testing.specs" ) >
<cfoutput>#r.run()#</cfoutput>

<cfset r = new coldbox.system.TestBox( 
      directory={ 
            mapping="coldbox.testing.cases.testing.specs", 
            recurse=false 
      }) >
<cfoutput>#r.run()#</cfoutput>

<cfset r = new coldbox.system.TestBox(
      directory={ 
            mapping="coldbox.testing.cases.testing.specs",
            recurse=true,
            filter=function(path){
                  return ( findNoCase( "test", arguments.path ) ? true : false );
            }
      }) >
<cfoutput>#r.run()#</cfoutput>

<cfset r = new coldbox.system.TestBox(
      directory={ 
            mapping="coldbox.testing.cases.testing.specs",
            recurse=true,
            filter=function(path){
                  return ( findNoCase( "test", arguments.path ) ? true : false );
            }
      }) >
<cfset fileWrite( 'testreports.json', r.run() )>

SOAP Runner

You can run tests via SOAP by leveraging the runRemote() method. The WSDL URL will be

http://localhost/testbox/system/TestBox.cfc?wsdl

HTTP REST Runner

You can run tests via HTTP/REST by leveraging the runRemote() endpoint. The URL will be

http://localhost/testbox/system/TestBox.cfc?method=runRemote

ANT Runner

In our test samples and templates we include an ANT runner that will be able to execute your tests via ANT. It can also leverage our ANTJunit reporter to use the junitreport task to produce JUnit compliant reports as well. You can find this runner in the test samples and runner template directory.

<cfsetting showDebugOutput="false">
<cfparam name="url.reporter"       default="simple">
<cfparam name="url.directory"      default="test.specs">
<cfparam name="url.recurse"        default="true" type="boolean">
<cfparam name="url.bundles"        default="">
<cfparam name="url.labels"         default="">
<cfparam name="url.reportpath"     default="#expandPath( "/tests/results" )#">
<cfscript>
// prepare for tests for bundles or directories
if( len( url.bundles ) ){
     testbox = new testbox.system.TestBox( bundles=url.bundles, labels=url.labels );
}
else{
     testbox = new testbox.system.TestBox( directory={ mapping=url.directory, recurse=url.recurse}, labels=url.labels );
}
// Run Tests using correct reporter
results = testbox.run( reporter=url.reporter );
// do stupid JUnitReport task processing, if the report is ANTJunit
if( url.reporter eq "ANTJunit" ){
     xmlReport = xmlParse( results );
     for( thisSuite in xmlReport.testsuites.XMLChildren ){
          fileWrite( url.reportpath & "/TEST-" & thisSuite.XMLAttributes.name & ".xml", toString( thisSuite ) );
     }
}
// Writeout Results
writeoutput( results );
</cfscript>

Ant

This build is our global build for running TestBox via ANT.

<?xml version="1.0"?>
<--<
This ANT build can be used to execute your tests with automation using our included runner.cfm.
You can executes directories, bundles and so much more.  It can also produce JUnit reports using the
ANT junitreport tag.  This is meant to be a template for you to spice up.

There are two targets you can use: run and run-junit

Execute the default 'run' target
ant -f test.xml
OR
ant -f test.xml run

Execute the 'run-junit' target
ant -f test.xml run-junit

PLEASE NOTE THAT YOU MUST ALTER THE RUNNER'S URL ACCORDING TO YOUR ENVIRONMENT.
-->
<project name="testbox-ant-runner" default="run" basedir=".">

     <-- <THE URL TO THE RUNNER, PLEASE CHANGE ACCORDINGLY-->
     <property name="url.runner"             value="http://cf10cboxdev.jfetmac/coldbox/ApplicationTemplates/Advanced/tests/runner.cfm?"/>
     <-- <FILL OUT THE BUNDLES TO TEST, CAN BE A LIST OF CFC PATHS -->
     <property name="test.bundles"      value="" />
     <-- <FILL OUT THE DIRECTORY MAPPING TO TEST -->
     <property name="test.directory"         value="test.specs" />
     <-- <FILL OUT IF YOU WANT THE DIRECTORY RUNNER TO RECURSE OR NOT -->
     <property name="test.recurse"      value="true" />
     <-- <FILL OUT THE LABELS YOU WANT TO APPLY TO THE TESTS -->
     <property name="test.labels"       value="" />
     <-- <FILL OUT THE TEST REPORTER YOU WANT, AVAILABLE REPORTERS ARE: ANTJunit, Codexwiki, console, dot, doc, json, junit, min, raw, simple, tap, text, xml -->
     <property name="test.reporter"          value="simple" />
     <-- <FILL OUT WHERE REPORTING RESULTS ARE STORED -->
     <property name="report.dir"        value="${basedir}/results" />
     <property name="junitreport.dir"   value="${report.dir}/junitreport" />

     <target name="init" description="Init the tests">
          <mkdir dir="${junitreport.dir}" />
          <tstamp prefix="start">
               <format property="TODAY" pattern="MM-dd-YYYY hh:mm:ss aa"/>
          </tstamp>
          <concat destfile="${report.dir}/latestrun.log">Tests ran at ${start.TODAY}</concat>
     </target>

     <target name="run" depends="init" description="Run our tests and produce awesome results">

          <-- <Directory Runner
               Executes recursively all tests in the passed directory and stores the results in the
               'dest' param.  So if you want to rename the file, do so here.

                Available params for directory runner:
                - Reporter
                - Directory
                - Recurse
                - Labels
          -->
          <get dest="${report.dir}/results.html"
                src="${url.runner}&directory=${test.directory}&recurse=${test.recurse}&reporter=${test.reporter}&labels=${test.labels}"
                verbose="true"/>

          <-- <Bundles Runner
               You can also run tests for specific bundles by using the runner with the bundles params

               Available params for runner:
                - Reporter
                - Bundles
                - Labels

          <get dest="${report.dir}/results.html"
                src="${url.runner}&bundles=${test.bundles}&reporter=${test.reporter}&labels=${test.labels}"
                verbose="true"/>
           -->

     </target>

     <target name="run-junit" depends="init" description="Run our tests and produce ANT JUnit reports">

          <-- <Directory Runner
               Executes recursively all tests in the passed directory and stores the results in the
               'dest' param.  So if you want to rename the file, do so here.

                Available params for directory runner:
                - Reporter = ANTJunit fixed
                - Directory
                - Recurse
                - Labels
                - ReportPath : The path where reports will be stored, by default it is the ${report.dir} directory.
          -->
          <get dest="${report.dir}/results.xml"
                src="${url.runner}&directory=${test.directory}&recurse=${test.recurse}&reporter=ANTJunit&labels=${test.labels}&reportPath=${report.dir}"
                verbose="true"/>

          <-- <Bundles Runner
               You can also run tests for specific bundles by using the runner with the bundles params

               Available params for runner:
                - Reporter
                - Bundles
                - Labels

          <get dest="${report.dir}/results.html"
                src="${url.runner}&bundles=${test.bundles}&reporter=${test.reporter}&labels=${test.labels}"
                verbose="true"/>
           -->

           <-- <Create fancy junit reports -->
          <junitreport todir="${junitreport.dir}">
               <fileset dir="${report.dir}">
                    <include name="TEST-*.xml"/>
               </fileset>
               <report format="frames" todir="${junitreport.dir}">
                    <param name="TITLE" expression="My Awesome TestBox Results"/>
               </report>
          </junitreport>

     </target>

</project>

NodeJS Runner

There is a user-contributed NodeJS Runner that looks fantastic and can be downloaded here:

You can use node to install as well:

ANT Runner

<cfsetting showDebugOutput="false">
<cfparam name="url.reporter"       default="simple">
<cfparam name="url.directory"      default="test.specs">
<cfparam name="url.recurse"        default="true" type="boolean">
<cfparam name="url.bundles"        default="">
<cfparam name="url.labels"         default="">
<cfparam name="url.reportpath"     default="#expandPath( "/tests/results" )#">
<cfscript>
// prepare for tests for bundles or directories
if( len( url.bundles ) ){
     testbox = new testbox.system.TestBox( bundles=url.bundles, labels=url.labels );
}
else{
     testbox = new testbox.system.TestBox( directory={ mapping=url.directory, recurse=url.recurse}, labels=url.labels );
}
// Run Tests using correct reporter
results = testbox.run( reporter=url.reporter );
// do stupid JUnitReport task processing, if the report is ANTJunit
if( url.reporter eq "ANTJunit" ){
     xmlReport = xmlParse( results );
     for( thisSuite in xmlReport.testsuites.XMLChildren ){
          fileWrite( url.reportpath & "/TEST-" & thisSuite.XMLAttributes.name & ".xml", toString( thisSuite ) );
     }
}
// Writeout Results
writeoutput( results );
</cfscript>

Ant

This build is our global build for running TestBox via ANT.

<?xml version="1.0"?>
<--<
This ANT build can be used to execute your tests with automation using our included runner.cfm.
You can executes directories, bundles and so much more.  It can also produce JUnit reports using the
ANT junitreport tag.  This is meant to be a template for you to spice up.

There are two targets you can use: run and run-junit

Execute the default 'run' target
ant -f test.xml
OR
ant -f test.xml run

Execute the 'run-junit' target
ant -f test.xml run-junit

PLEASE NOTE THAT YOU MUST ALTER THE RUNNER'S URL ACCORDING TO YOUR ENVIRONMENT.
-->
<project name="testbox-ant-runner" default="run" basedir=".">

     <-- <THE URL TO THE RUNNER, PLEASE CHANGE ACCORDINGLY-->
     <property name="url.runner"             value="http://cf10cboxdev.jfetmac/coldbox/ApplicationTemplates/Advanced/tests/runner.cfm?"/>
     <-- <FILL OUT THE BUNDLES TO TEST, CAN BE A LIST OF CFC PATHS -->
     <property name="test.bundles"      value="" />
     <-- <FILL OUT THE DIRECTORY MAPPING TO TEST -->
     <property name="test.directory"         value="test.specs" />
     <-- <FILL OUT IF YOU WANT THE DIRECTORY RUNNER TO RECURSE OR NOT -->
     <property name="test.recurse"      value="true" />
     <-- <FILL OUT THE LABELS YOU WANT TO APPLY TO THE TESTS -->
     <property name="test.labels"       value="" />
     <-- <FILL OUT THE TEST REPORTER YOU WANT, AVAILABLE REPORTERS ARE: ANTJunit, Codexwiki, console, dot, doc, json, junit, min, raw, simple, tap, text, xml -->
     <property name="test.reporter"          value="simple" />
     <-- <FILL OUT WHERE REPORTING RESULTS ARE STORED -->
     <property name="report.dir"        value="${basedir}/results" />
     <property name="junitreport.dir"   value="${report.dir}/junitreport" />

     <target name="init" description="Init the tests">
          <mkdir dir="${junitreport.dir}" />
          <tstamp prefix="start">
               <format property="TODAY" pattern="MM-dd-YYYY hh:mm:ss aa"/>
          </tstamp>
          <concat destfile="${report.dir}/latestrun.log">Tests ran at ${start.TODAY}</concat>
     </target>

     <target name="run" depends="init" description="Run our tests and produce awesome results">

          <-- <Directory Runner
               Executes recursively all tests in the passed directory and stores the results in the
               'dest' param.  So if you want to rename the file, do so here.

                Available params for directory runner:
                - Reporter
                - Directory
                - Recurse
                - Labels
          -->
          <get dest="${report.dir}/results.html"
                src="${url.runner}&directory=${test.directory}&recurse=${test.recurse}&reporter=${test.reporter}&labels=${test.labels}"
                verbose="true"/>

          <-- <Bundles Runner
               You can also run tests for specific bundles by using the runner with the bundles params

               Available params for runner:
                - Reporter
                - Bundles
                - Labels

          <get dest="${report.dir}/results.html"
                src="${url.runner}&bundles=${test.bundles}&reporter=${test.reporter}&labels=${test.labels}"
                verbose="true"/>
           -->

     </target>

     <target name="run-junit" depends="init" description="Run our tests and produce ANT JUnit reports">

          <-- <Directory Runner
               Executes recursively all tests in the passed directory and stores the results in the
               'dest' param.  So if you want to rename the file, do so here.

                Available params for directory runner:
                - Reporter = ANTJunit fixed
                - Directory
                - Recurse
                - Labels
                - ReportPath : The path where reports will be stored, by default it is the ${report.dir} directory.
          -->
          <get dest="${report.dir}/results.xml"
                src="${url.runner}&directory=${test.directory}&recurse=${test.recurse}&reporter=ANTJunit&labels=${test.labels}&reportPath=${report.dir}"
                verbose="true"/>

          <-- <Bundles Runner
               You can also run tests for specific bundles by using the runner with the bundles params

               Available params for runner:
                - Reporter
                - Bundles
                - Labels

          <get dest="${report.dir}/results.html"
                src="${url.runner}&bundles=${test.bundles}&reporter=${test.reporter}&labels=${test.labels}"
                verbose="true"/>
           -->

           <-- <Create fancy junit reports -->
          <junitreport todir="${junitreport.dir}">
               <fileset dir="${report.dir}">
                    <include name="TEST-*.xml"/>
               </fileset>
               <report format="frames" todir="${junitreport.dir}">
                    <param name="TITLE" expression="My Awesome TestBox Results"/>
               </report>
          </junitreport>

     </target>

</project>

Running Tests

run() Arguments

runRemote() Arguments

Notes

Run Listeners

Global Runner

Test Browser

Bundle(s) Runner

Test Runner

Directory Runner

SOAP Runner

HTTP REST Runner

ANT Runner

Ant

NodeJS Runner

Test Browser

Run Listeners

Test Runner

Directory Runner

HTTP REST Runner

ANT Runner

Ant

SOAP Runner

Bundle(s) Runner

Global Runner

Running Tests

run() Arguments

runRemote() Arguments

Notes

NodeJS Runner

`run()` Arguments

`runRemote()` Arguments

`run()` Arguments

`runRemote()` Arguments