Travis CI is one of the most popular CI servers for open source software. At Ortus Solutions, we use it for all of our open source software due to its strength of pull request runners and multi-matrix runners. They have both free and commercial versions, so you can leverage it for private projects as well.

Features

• FREE for Open Source Projects

• Runs distributed VM’s and Container Support

• Triggers Build Script via git repository commits (.travis.yml)

• Multiple language support

• Many integrations and extensions

• Many notification types

• No ability to schedule/manual builds

• Great for open source projects!

TestBox Integration

In order to work with Travis you must create a .travis.yml file in the root of your project. Once there are commits in your repository, Travis will process this file as your build file. Please refer to the Travis Documentation for further study.

language: java
sudo: required
dist: trusty

before_install:
  # CommandBox Keys
  - curl -fsSl https://downloads.ortussolutions.com/debs/gpg | sudo apt-key add -
  - sudo echo "deb https://downloads.ortussolutions.com/debs/noarch /" | sudo tee -a
  /etc/apt/sources.list.d/commandbox.list

install:
  - sudo apt-get update && sudo apt-get --assume-yes install commandbox
  - box install
  - box server start

script:
  - box testbox run

This build file is based on the java language and an Ubuntu Trusty image. We start off by executing the before_install step which installs all the OS dependencies we might need. In our case we add the CommandBox repository server keys and install CommandBox as our dependency. We then move to our install step which makes sure we have all the required software dependencies to execute our tests, again this looks at our box.json for TestBox and required project dependencies. After issuing the box install we move to starting up the CFML engine using box server start and we are ready to test.

install:
  - sudo apt-get update && sudo apt-get --assume-yes install commandbox
  - box install
  - box server start

The testing occurs in the script block:

script:
  - box testbox run

In our script we basically install our dependencies for our project using CommandBox and startup a CFML server. We then go ahead and execute our tests via box testbox run.

Box.json

In order for the box testbox run to execute correctly, our box.json in our project must be able to connect to our server and know which tests to execute. Below are all the possiblities for the testbox integration object in CommandBox's box.json. (See the CommandBox docs for box.json for more details.)

{
    "name" : "Package Name",
    // ForgeBox unique slug
    "slug" : "",
    // semantic version of your package
    "version" : "1.0.0+buildID",
    // author of this package
    "author" : "Luis Majano <lmajano@ortussolutions.com>",
    // location of where to download the package, overrides ForgeBox location
    "location" : "URL,Git/svn endpoint,etc",

    // testbox integration
    testbox :{
        // The location of the runner
        runner : [
            { "default": "http://localhost:8080/tests/runner.cfm" }
        ],
        // Which labels to run, empty means all
        "labels" : "",
        // Which reporter to use, default is json
        "reporter" : "",
        // Which CFC bundles to execute, default is all
        "bundles" : "",
        // Which directories to execute
        "directory" : "tests.specs",
        // Recurse the directories for CFCs
        "recurse" : true,
        // Which bundles to filter on
        "testBundles" : "",
        // Which suites to filter on
        "testSuites" : "",
        // Which specs to filter on
        "testSpecs" : "",
        // Display extra details inlcuding passing and skipped tests.
        "verbose" : true,
        // How may milliseconds to wait before polling for changes, defaults to 500 ms
        "watchDelay" : 500,
        // Command delimited list of file globbing paths to watch relative to the working directory
        "watchPaths" : "**.cfc"
    }
}

Online Example: cbVue

You can look at our cbVue sample application online: https://travis-ci.org/coldbox-samples/cbvue which contains all CI server integrations.

About GitHub Actions

GitHub Actions is an automation platform built in to GitHub.com that makes it easy to automate code quality on your github repos. There are a number of integrations that make using TestBox inside GitHub Actions simple, speedy and powerful so you can get back to writing code.

Features

• FREE for public repositories

• 2,000 minutes for private repos

• Can reuse workflows, i.e. for a standard test.yml workflow in both builds and PR testing

• Can schedule workflow runs

• [Can configure a build "matrix"], i.e. for testing against all recent CFML engines

• Thousands of configurable, reusable "Actions"

• Can notify Slack on build failure

TestBox Integration

Testing your application with TestBox in GitHub Actions (GHA) begins a workflow.yml file at the root of a .github/workflows/ directory. You can name this file anything you like - I'd suggest build.yml or test.yml - but if it is not a valid Yaml file the GHA workflow will fail.

This file should start with some GHA metadata to dictate when and how the workflow should run:

# .github/workflows/tests.yml
name: Test

on:
  push:
    branches:
      - main
      - master
      - development

This will run the workflow on each commit to the master or main branch. Next, specify a workflow job to execute:

jobs:
  tests:
    name: Tests
    runs-on: ubuntu-latest
    steps:
      - # All job steps go here

Under the jobs.tests.steps is where we will place each sequential testing step. First, we need to check out the repository code and install Java and CommandBox:

      - name: Checkout Repository
        uses: actions/checkout@v2
        with:
          fetch-depth: 0

      - name: Setup Java
        uses: actions/setup-java@v2
        with:
          distribution: "adopt"
          java-version: "11"

      - name: Setup CommandBox
        uses: Ortus-Solutions/setup-commandbox@main

If we need to install dependencies, we would do that now:

      - name: Install dependencies
        run: box install

And finally, we can start a CFML server of our choice using CommandBox, before using the testbox run command to run our test suite:

      - name: Start server
        run: box server start cfengine=lucee@5.3 --noSaveSettings

      - name: Run TestBox Tests
        run: box testbox run

If you are using a testing matrix to test against multiple CFML engines, replace lucee@5.3 with ${{ matrix.cfengine }}. See an example here

The full example would look something like this:

# .github/workflows/tests.yml
name: Test

on:
  push:
    branches:
      - main
      - master
      - development
jobs:
  tests:
    name: Tests
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v2
        with:
          fetch-depth: 0

      - name: Setup Java
        uses: actions/setup-java@v2
        with:
          distribution: "adopt"
          java-version: "11"

      - name: Setup CommandBox
        uses: Ortus-Solutions/setup-commandbox@main

      - name: Install dependencies
        run: box install

      - name: Start server
        run: box server start cfengine=lucee@5.3 --noSaveSettings

      - name: Run TestBox Tests
        run: box testbox run

Box.json

In order for the box testbox run to execute correctly, our box.json in our project must be able to connect to our server and know which tests to execute. Here's a basic example showing the most important testbox property: the testbox.runner property:

{
    "name" : "Package Name",
    // ForgeBox unique slug
    "slug" : "",
    // semantic version of your package
    "version" : "1.0.0+buildID",
    // author of this package
    "author" : "Luis Majano <lmajano@ortussolutions.com>",
    // location of where to download the package, overrides ForgeBox location
    "location" : "URL,Git/svn endpoint,etc",

    // testbox integration
    testbox :{
        // The location of the runner
        runner : [
            { "default": "http://localhost:8080/tests/runner.cfm" }
        ]
    }
}

We can also skip setting the testbox.runner property and use the box testbox run "http://localhost:8080/tests/runner.cfm" command format instead. Just be aware that the TestBox integration offers a ton of configuration in case you need to skip certain tests, etc. from your GitHub Actions test run.

See the CommandBox docs for box.json for more details.

Online Example

CBORM is an ORM utility wrapper for ColdBox that takes the pain out of using ORM in CFML. CBORM uses GitHub Actions to test all new commits, to package up new module versions, and even to format all CFML code for every new PR.

See All CBORM Workflows, or see recent GitHub Actions workflow runs here.

is one of the most popular collaboration software suites around. It is not only a CI server, but a source code server, docker registry, issue manager, and much much more. They are our personal favorite for private projects due to their power and flexibility.

Features

• All in one tool: CI, repository, docker registry, issues/milestones, etc.

• Same as Travis in concept - CI Runner

• Docker based

• CI Runners can be decoupled to a docker swarm

• Idea of CI pipelines

  • Pipelines composed of jobs executed in stages

  • Jobs can have dependencies, artifacts, services, etc

  • Schedule Pipelines (Cron)

  • Jobs can track environments

• Great stats + charting

TestBox Integration

The build file above leverages the ortussolutions/commandbox:alpine image, which is a compact and secure image for CommandBox. We then have a few stages (build,test,deploy), but let's focus on the run_tests job.

We define which branches it should listen to: development, and then we have a script block that defines what to do for this job. Please note that the when action is commented, because we want to execute this job every time there is a commit. In Gitlab we can determine if a job is manual, scheduled, automatic or dependent on other jobs, which is one of the most flexible execution runners around.

In our script we basically install our dependencies for our project using CommandBox and startup a CFML server. We then go ahead and execute our tests via box testbox run.

Box.json

Online Example: cbVue

(CI) is a development process that requires developers to integrate their code into a shared source code repository (git,svn,mercurial,etc) several times a day, while a monitoring process detects code commits and acts upon those commits. Those actions can be the actual checkout of branches, execution of build processes, quality control, and of course our favorite; automated testing.

TestBox can integrate with all major CI servers as all you need to do is be able to execute your test suites and produce reports. You can see that in our section and .

Benefits of Continuous Integration

• Decrease the feedback loop

• Discover defects faster before production releases

• Developer Accountability

• Increase code visibility and promote communication

• Increase quality control

• Reduce integration issues with other features

• Develop in an Agile/Scrum fashion with continuous improvement

• Much More...

CI Servers

Here is a listing of some of the major CI servers: