There are a few known issues with code coverage that are currently out of our control. We apoligize for any inconvenience they may give you.
On some sites we've experienced that after a fresh restart of Lucee Server, when the first page you hit is the test suite and code coverage is enabled, Lucee throws a low level compilation error (which you can see in the logs) and a white page is returned to the browser. We haven't figured out the scenario in which occurs, but refreshing the page always seems to work the second time.
If you run into this on an automated build environment, consider configuring your build to hit a page first, or run the tests again if no results are captured the first run.
This has been reported in Lucee for very large files of CF code. Lucee automatically breaks bytecode into additional methods when compiling to avoid the JVM limits of maximum method sizes in a class file. However, when FusionReactor instruments the byte code (adds additional lines of code in), it can push some of the internal methods over the JVM limit. There will be an error logged to your console and TestBox will have no coverage for the file, showing every line as not executed.
The only workaround at this time is to reduce the size of your CF files so the bytecode isn't as close to the JVM limits. Try moving chunks of code out to includes.
Only executable code should be tracked, meaning a comment, whitespace, or HTML that does not run will not count against your total coverage percentage. When using Adobe ColdFusion, if there are any CF files which did not run at all (and therefore were not compiled) they will count every line in the file as executable since FusionReactor is only capable of tracking files which are compiled. Lucee has a workaround to manually force compilation on such files, but Adobe does not.
The best work around is to improve your tests so that these files are being executed! Alternatively, you can add those files to the blacklist until you are ready to write tests for them, but that will make your coverage look better than it really is if you do eventually want to write tests for those files. Minimally, a test that does nothing but create an instance of a CFC would be enough to trigger its compilation so correct coverage can kick in.
Occasionally you may run across some lines of code in the Code Coverage Browser report that doesn't seem correctly tracked. Common offenders are extra curly braces or ending tags floating on a line of their own. The fact is, mapping each line of CF code to the actual lines of compiled bytecode is tricky business and done entirely by the CF engines (Adobe, Lucee) under the hood during compilation. Sometimes bits of code might not seem tracked correctly, but we've never seen it have any significant effect on your overall coverage data. The behavior is specific to each engine/version but typically lines like that just get associated with the next executable line, so once your coverage for a file hits all the possible lines, the issue goes away :) Feel free to report any issues you see. We have some workarounds in place and can see about making it better.
TestBox's code coverage feature relies on byte code instrumentation from Fusion Reactor. It seems that in some instances this process can fall over due to FR's internal behaviour, byte code class caching, internal compilation changes going from one version of your CFML engine to another one and other reasons. While we report these kind of issues upstream, unfortunately they are nearly impossible for us to properly investigate and debug.
One common symptom seems to be that code coverage statistics for individual files or your overall codebase vary extremely between two TestBox executions without having changed code at all or in a meaningful way. Another symptom observed by users is that code coverage drops to 0% for certain files and you know for sure that these files would be executed during your tests.
A restart of your CFML engine and a subsequent run of your tests with code coverage usually fixes this problem.
When writing tests for an app or library, it's generally regarded that more tests is better since you're covering more functionality and more likely to catch regressions as they happen. This is true, but more specifically, it's important that your tests run as much code in your project as possible. Tests obviously can't check code that they doesn't run!
With BDD, there is not a one-to-one correlation between a test and a unit of code. You may have a test for your login page, but how do you know if all the else blocks in your if statements or case blocks in your switch statements were run? Was your error routine tested? What about optional features, or code that only kicks in on the 3rd Tuesday of months in the springtime? These can be difficult questions to answer just by staring at the results of your tests. The answer to this is Code Coverage.
Code coverage does not replace your tests nor does it change how you write your tests. It's additional metrics gathered by the testing framework while your tests are running that actually tracks what lines of code were executed and what lines of code didn't get run. Now you can finally see how much code in your app is "covered" by your tests and what code is currently being untested.
TestBox supports code coverage statistics out-of-the box with no changes to your test suite and you can capture the data in a handful of ways, including a Coverage Browser report which visualizes every CF file in your code and shows you what lines executed and what lines didn't.
TestBox 3.x+
FusionReactor 7+ (separate license required)
Please note that FusionReactor is a separate product not made by Ortus, but by Intergral GmbH. FusionReactor is a performance monitoring tool for Java-based servers and you will need to purchase a license to use it. We understand you may wish to use code coverage for free, but this feature would not have been possible without the line performance tracking feature of FusionReactor that allows us to match your Java bytecode to the actual code lines of your CFML. For personal use, there is a reasonably-priced Developer Edition. Please reach out to FusionReactor's sales team if you have any questions.
By default, code coverage will track all .cfc and .cfm files in your web root. However, for the most correct numbers, you want to only track the code in your app. This means you'll want to ignore things like
3rd party frameworks such as ColdBox or TestBox itself
3rd party modules installed by CommandBox (i.e., your /modules folder)
Backups or build directories that aren't actually part of your app
Parts of the app that you aren't testing such as the /tests folder itself
Most of the coverage settings are devoted to helping TestBox know what files to track, but there are some other ones too. Let's review them.
Code coverage is enabled by default and set with a default configuration. You can control how it behaves with a series of <CFParam> tags in your /tests/runner.cfm file. If you created a fresh new ColdBox app from our app templates using coldbox create app, you'll see there are already configuration options ready for you to change. If you are working with an existing test suite runner, place the following lines PRIOR to the <CFInclude> in your runner.cfm.
Let's go over the options above and what they do. Feel free to comment/uncomment and modify them as you need for your code. Note, any of these settings can be overridden by a URL variable as well.
Set this to true or false to enable the code coverage feature of TestBox. This setting will default to true if TestBox detects that you have FusionReactor installed, false otherwise. Setting this to true without FusionReactor installed will be ignored.
The following setting would turn off code coverage:
Use this to point to the root folder that contains code you wish to gather coverage data from. This must be an absolute path and feel free to use any CF Mappings defined in your /tests/Application.cfc to make the path dynamic. This is especially useful if the app being tested is in a subfolder of the actual web root. There is nominal overhead in gathering the coverage data from files, so set this to the correct folder and instead of using the whitelist to filter down from your web root if possible.
This setting defaults to the web root. Also note, code coverage only looks at files ending in .cfc or .cfm so there's no need to filter additionally on that unless you want to only include, say, .cfc files.
This is a comma-delimited list of file globbing patterns relative to the coveragePathToCapture setting that further filters what files and folders to track. By default this setting is empty, which means ALL CFML code in the coverage path will be tracked. As soon as you specify at least one file globbing path in the whitelist, ONLY the files matching the whitelist will be tracked.
For example, if you only cared about handlers and models in an MVC app, you could configure a whitelist of:
Note, all the basic rules of file globbing apply here. To give you a quick review
Use ? to match a single char like /Application.cf?
Use * to match multiple characters within a folder or file like /models/*Service.cfc.
Use ** to match multiple characters in any sub folder (recursively) like /models/**.cfc.
A pattern like foo will match any file or folder recursively but a leading slash like /foo locks that pattern to the root directory so it's not a recursive match.
A trailing slash forces a directory match like /tests/, but no trailing slash like /tests would also match a file such as /tests-readme.md.
This is a comma-delimited list of file globbing patterns relative to the coveragePathToCapture setting that is applied on top of any whitelist patterns to prune back folders or files you don't want to track. There's no reason to include a path in the blacklist if you have a whitelist specified and that whitelist already doesn't include the path in question. However, a blacklist can be very handy when you want to include everything but a few small exceptions and it's easier to list those exceptions rather than create a large number of paths in the whitelist.
One of the most visually exciting features of Code Coverage is the ability to generate a stand-alone HTML report that allows you inspect every tracked file in your project and see exactly what lines of code are "covered" by your tests and what is not. The Code Coverage Browser is a mini-site of static HTML files in a directory which you can open in a browser on any computer without the need for a CF engine or TextBox being present. (read, you can zip them up and send them to your boss or store as a build artifact!)
To enable the Code Coverage Browser, uncomment the param for it and specify an absolute file path to where you would like the static mini-site created. You will want a dedicated directory such as /tests/results/coverageReport but just remember to expand it so it's absolute. The directory will be created if it doesn't exist.
Also note, Windows is pesky about placing file and folder locks on your report output directory if you have it open in Windows Explorer. If you get an error about not being able to delete the report directory, close all Explorer windows and try again. Sadly, there's no workaround for this as Windows is the one placing the locks!
SonarQube is a code quality tool that gathers information about your code base for further reporting. If you don't use SonarQube, you may ignore this section. You can have TestBox spit out a coverage XML file in the format that SonarQube requires by uncommenting this param and specifying an absolute file path to where you'd like the file written. Please include the file name in the path.
In order to use TestBox Code Coverage, you will need TestBox 3.x or higher installed, a licensed installation of FusionReactor and a working test suite. You may have some or all of these already so skip the sections that don't apply to you.
If you don't have FusionReactor installed, you can do so very easily in CommandBox like so:
That's it! All servers you start now will have FusionReactor configured. You can open FusionReactor's web console via the menu item in your server's tray icon. Note, the FusionReactor web admin is not required to get TestBox code coverage.
If you are not using CommandBox for your server, follow the installation instructions on FusionReactor's website. If you need a license key, please contact FusionReactor to acquire one. Note they have a 2 week trial you can use.
To get the latest version of TestBox into a new project, you can install it via CommandBox like so:
The --saveDev flag will store TestBox as a development dependency.
If you don't have test suite yet, let's install a ColdBox sample app to play with. TestBox does not require ColdBox to work, but the mechanics of the test runner itself are identical so this is the easiest way to get one running. Run these CommandBox commands in an empty directory.
Inside your directory will be a folder called /tests which has our test runner /tests/runner.cfm. You will need to open your runner.cfm and default code coverage enabled to true.
All you need to do now is run your test suite. You can do so by hitting /tests/runner.cfm in the URL of your browser, or use the testbox run command in CommandBox.
You don't need to configure anything for code coverage to work. TestBox will auto-detect if FusionReactor is installed on your server and will generate the code coverage results for you. In the output of your test reporter, you will see a percentage to represents the number of lines of code (LOC) which were executed divided by the total number of lines of code. Note, code coverage only counts executable lines of code against you, so comments, whitespace, or HTML do not count as an executable LOC.
Keep reading in the next section to find out how to configure the details of code coverage to only look at the files you want and also how to generate the Code Coverage Browser.