Tag Archives: testing

Build Agent Infrastructure Testing in GoCD

In this post I would like to describe a simple technique for reducing the waiting time and stress related to build agent environment volatility when using Continuous Integration / Continuous Delivery tools like GoCD, via infrastructure testing.

The Problem

Given a modern CI server, such as GoCD, and a set of dedicated build machines (agents), it is possible to improve software development agility. Automated build/test/deploy pipelines, built to reflect the value stream, bring transparency and focus into the software delivery activities.

CI automation is software itself, and is thus susceptible to errors. Configuration management can optimize the set up of the environment, in which the build agents run. However, when computational resources are added to a CI infrastructure, i.e. to parallelize the build and thus reduce feedback times, a missing environment dependency can cause stress and pain that CI is trying to eliminate.

Consider a pipeline where a complete cycle (i.e. with slow integration tests followed by a reporting step at the end or long check-outs in the beginning) takes a significant amount of time. If one of the last tasks fails due to a configuration or an environment issue, the whole stage fails. The computational resources have been wasted just to find out that a compiler is missing. This can easily happen when there is variation in the capabilities of the build agents.

Improvement idea: fail fast! Don’t wait for environment or infrastructure mismatches

GoCD Resources as Requirements and Capabilities

If a step in a build pipeline requires a certain compiler or a particular environment, this can be conveniently expressed in the configuration of GoCD as a build agent resource. A resource can be seen as a requirement of a pipeline step that is fulfilled by a corresponding capability of a build agent.

Consider the following set-up with two build agents – one running on Windows, another one on Linux. Some tasks could be completely platform-independent, such as text processing, and thus could be potentially performed on any machine with a required interpreter installed.


Build Agent is the Culprit

We have set up our environment, and have successfully tested our first commit, but the second one fails:


The code is the same, why did the second build fail? The builds ran on different agents, but I expected them to behave similarly…


Oh, that’s embarrassing. While yes, the script is platform-independent, there’s no executable named python2 in my Windows runner environment path.

With a one-file repository and a simple print statement this failure did not cause much damage, but as mentioned earlier, real life builds failing due to a missing executable might be costly.

Unhappy Picture


Infrastructure Test Pipeline

In order to fail fast in situations where new agents are added to a CI infrastructure, or their environment is volatile, I propose to use a single independent pipeline that checks the assumptions that longer builds depend upon.

If a build step requires python in the path, there should be a test for it that gives this feedback in seconds without much additional waiting time. This can be as easy as calling python --version, which will fail with a non-zero return code if the binary is missing. More fine-grained assertions are possible, but should still remain fast.

If a certain binary should not be in the path, this can be asserted as well. The same goes for environment variables and file existence. Dedicated infrastructure testing tools, such as Serverspec could also be used, but having a response time under a minute is crucial in my view.

Run on All Agents

In order to validate the consistency of the CI infrastructure, the validation tasks should run on all agents that advertise a corresponding resource. This is where, in my view, the real power of GoCD comes to light, and the concepts used in it fit in the right places.

GoCD will run the test tasks on all agents that fulfill all resource requirements for the task.


Test fails

Now that we have all the tests, running them gives quick and precise feedback:


Checking out the job run details reveals the offending agent. Note the test duration: under 1 second.


Fixing the Infrastructure


Whatever the resolution of the infrastructure problem, when the infrastructure test has a good coverage of the prerequisites for a pipeline, adding new agents to the CI infrastructure should become as much fun as TDD is: write an infrastructure test, see it fail, fix the infrastructure, feel the good hormones. Add new build agents for speed — still works — great!


Note how the resources that are available only on one machine are only run on one corresponding machine.


Happy Pictures and Developers

happy pipeline

When to Test

It is an open question, when to test the infrastructure. With the system being composed of the CI server and agents, the tests should probably run on any global state change, such as

  • added/removed/reconfigured agents
  • automatic OS updates (controversial)
  • restarts
  • network topology changes

It is also possible to schedule a regular environment check. Having the environment test pipeline be the input for other pipelines unfortunately will not do in the following sequence of events:

  • environment tests pass
  • faulty agents are added
  • downstream pipeline is triggered
  • environment failure causes a pipeline to fail

In any case, there is a REST API available for the GoCD server should automating the automation become a necessity.


I would like to thank all the great minds, authors and developers who have worked and are working to make lives of developers and software users better. Tools and ideas that work and provide value are indispensable.  The articles and the software linked in this blog entry are examples of knowledge that brings the software industry forward. I am also very grateful to my current employer for letting me learn, grow and make a positive impact.



Why wait forever for the tests? Fast tests of slow software.

Time is volatile

Imagine writing a cron-like functionality that should produce some side-effect, such as cleanup. The intervals between such actions might be quite long. How does one test that? One can surely reason about the software, but given a certain complexity, test should be written, proving that certain important scenarios work as intended.

It’s common that software depends on time flow as dictated by the physical time flow, reflected via some clock provider. However, resetting the time to a year ahead won’t make the CPU work faster and make all the computations it should have performed within that year. A clock is also a volatile component that can be manipulated, thus if time is an issue, it’s probably a good idea not to depend on it directly, following the Stable Dependencies Principle and the Dependency Inversion Principle.

Luckily, there is an abstraction for time, at least in Reactive Extensions (Rx), which is the Scheduler.

Slow non-tests

Here’s a slow Groovy non-test, waiting for some output on the console using RxGroovy:

import rx.*
import java.util.concurrent.TimeUnit

def observable = Observable
	.delay(5, TimeUnit.SECONDS)

observable.subscribe { println 'ah, OK, done! Or not?' }

	.subscribe { println 'still waiting...' }

println 'starting to wait for the test to complete ...'


Running it produces the following slow-ticking output:

oldnontest 1

Interpreting such tests without color can be somewhat challenging 2.

Fast tests

Now let’s test something ridiculous, such as waiting for a hundred days using Spock. Luckily, RxJava & RxGroovy also do implement the test scheduler, thus enabling fast tests using virtual time:

import spock.lang.Specification

import rx.Observable
import rx.schedulers.TestScheduler
import java.util.concurrent.TimeUnit

class DontWaitForever extends Specification {
    def "why wait?"() {
            def scheduler = new TestScheduler()

            // system under test: will tick once after a hundred days
            def observable = Observable
                .delay(100, TimeUnit.DAYS, scheduler)
            def done = false

            observable.subscribe {
                done = true

            // still in the initial state
            done == false

            scheduler.advanceTimeBy 100, TimeUnit.DAYS

            done == true

fasttest 3

just checking, advancing the time by 99 days results in a failure:


Delightful, groovy colors!



  1. Caputured with the wonderful pragmatic tool LICEcap by the Reaper developers
  2. Here, the ‘still waiting’ subscription is terminated after the first subscription ends. Try exchanging the order of the subscribe calls.
  3. Building using Gradle

Deterministic Testing of Concurrent Behavior in RxCpp

A Retrospective

After getting inspired by The Reactive Manifesto, it is hard not to get excited about Reactive Extensions. Such excitement has lead to a series of hello-world articles and some code examples. While Reactive Extensions take over the programming world in C#, Java and JavaScript, it seems, the world of C++ is slow to adopt RxCpp.

The new ReactiveX Tutorial link list is a great place to start learning and grokking. This article is an attempt to bring RxCpp closer to C++ developers who might not see yet, how a reactive programming model might help writing better, more robust code.

Testing concurrency with RxCpp

A previous article showed how to test ViewModels in C# by parameterizing the ViewModels with a scheduler. In a UI setting, the scheduler usually involves some kind of synchronization with the GUI thread. Testing keystrokes arriving at certain speed would require some effort to simulate events, probably leading to brittle tests. With the scheduler abstraction, the concurrent behavior of a component is decoupled from physical time, and thus can be tested repeatedly and very fast. This was the C# test:

(new TestScheduler()).With(scheduler =>
    var ticker = new BackgroundTicker(scheduler);

    int count = 0;
    ticker.Ticker.Subscribe(_ => count++);

    // full control of the time without waiting for 1 second

Show Me The Code

Without further ado, the C++ version is not very far from the C# version. In a simple test, we can parameterize a sequence of integer values arriving at specified intervals (a ticker) with a coordination (why coordination and not scheduler, read in the RxCpp developer manual:

auto seq = rxcpp::observable<>::interval(

The deterministic test scheduler API is currently available through a worker created on the test scheduler:

auto sc = rxcpp::schedulers::make_test();
auto worker = sc.create_worker();
auto test = rxcpp::identity_same_worker(worker);

The rest should read like English:

int count = 0;

WHEN("one subscribes to an observable sequence on the scheduler") {
  auto seq = rxcpp::observable<>::interval(
              test // on the test scheduler
             ).filter([](int i) { return i % 2; });


  THEN("the sequence is not run at first") {
    worker.sleep(2 /* ms */);

    CHECK(count == 0);

    AND_WHEN("the test scheduler is advanced manually") {

      THEN("the sequence is run as expected") {
        worker.advance_by(8 /* ms */);
        CHECK(count == 5);

The full test can be seen @github, and is built on Travis CI

RxCpp 2

RxCpp 2 and API

The last article on rxcpp was based on a now obsolete version of RxCpp. The key contributor to the library, Kirk Shoop, has kindly provided a rewrite based on the newer, 2.0 API of the library: see the pull request, upon which this article is based.

Since the first article, the project has been enriched with somewhat more readable GIVEN/WHEN/THEN-style tests using Catch 1.

Still Ticking: Scheduler and Coordination in RxCpp 2

The previous articles give examples of managing periodic events, such as ticker ticks and measurements in c++. The following example creates an event loop that will be used for coordinated output of various events to the console:

auto scheduler = rxcpp::schedulers::make_same_worker(

auto coordination = rxcpp::identity_one_worker(scheduler);

One such sequence of events is some kind of measurement 2

auto measure = rxcpp::observable<>::interval(
        // when to start
        scheduler.now() + std::chrono::milliseconds(250),
        // measurement frequency
    // take Hz values instead of a counter
    .map([&FM](int) { return FM.Hz(); });

auto measure_subscription = measure
    .subscribe([](int val) {
        std::cout << val << std::endl;

Why didn’t it tick?

If this code were the end of the main program, there wouldn’t be any observable ticks, as all the objects would be destroyed before the first scheduled event. To see the code in action, we shall wait for some condition that will change when we’re done. This step is not necessary if there’s a GUI toolkit event loop that keeps objects alive, but it has to be simulated for a console example.

To demonstrate the subscription change and wait for some time, we’ll wait twice for an atomic variable to become zero:

std::atomic<long> pending(2);


// after all subscriptions defined
while (pending) {
    sleep(1000); // wait for ticker and measure to finish

Tick and Stop

The other ticker will have another period, will only tick 10 times, and then decrement the pending counter:

auto ticker = rxcpp::observable<>::interval(
    scheduler.now() + std::chrono::milliseconds(500),

    .subscribe([](int val) {
        std::cout << "tick " << val << std::endl;
        --pending; // take completed the ticker

Now, we can schedule the termination of the measurement (decrement pending) subscription halfway through the 10-tick run. This scheduling is done on the same scheduler that is running all the subscriptions:

scheduler.create_worker().schedule(scheduler.now() + std::chrono::seconds(2), 
    [&](const rxcpp::schedulers::schedulable&) {
        std::cout << "Canceling measurement ..." << std::endl;
        measure_subscription.unsubscribe(); // cancel measurement
        --pending; // signal measurement canceled

The result:

tick 1
tick 2
tick 3
Canceling measurement ...
tick 4
tick 5
tick 6
tick 7
tick 8
tick 9
tick 10

Thanks, Kirk & other library contributors!

Code @ github

Next: deterministic testing of concurrent behavior

  1. i.e. create.cpp
  2. Observe the convergence of the API towards the C# version.

Presenting at TU-Munich: testing on c++ projects, Thursday, March 26, 2015 7:00 PM

Expecting Thank you to all for a superb heated debate! next week

“no excuses for not testing on c++ projects”

Thursday, March 26, 2015
7:00 PM

details: http://www.meetup.com/MUCplusplus/events/220628575/

If only all test were comprehensible…

SCENARIO("acquiring wisdom") {

  GIVEN("an oracle") { 
    oracle gus;
    WHEN("I ask it to speak") {
      auto answer = gus.speak();

      THEN("wisdom is apparent") {
        CHECK( answer != "bla" );


→ The code can be found @github, including the presentation slides.

ReactiveUI 6 and ViewModel Testing

Testability and ReactiveUI

ReactiveUI XAML example

In the previous articles about ReactiveUI I’ve claimed without bringing any evidence that writing ViewModels using ReactiveObjects brings about testability. While the aspects of testing Rx and ReactiveUI have been treated at length in the respective authors’ blogs linked herein, this post is intended as a quick glance for the impatient online surfer at the hello-world testing code, which has been written “post-mortem” 1 as a follow-up to the previous articles.

An update to ReactiveUI 6

Paul Betts and contributors have been busy simplifying and extending the library 2. There are some extension methods now that help creating observables from properties, and transforming observables to properties. In the example ViewModel from previous articles, there’s an observable stream of strings that is simply transformed into a property defined as follows:

ObservableAsPropertyHelper<string> _BackgroundTicker;
public string BackgroundTicker
    get { return _BackgroundTicker.Value; }

In the constructor, the helper is now initialized without using strings:

public WordCounterModel(IObservable<string> someBackgroundTicker)
    .ToProperty(this, x => x.BackgroundTicker, out _BackgroundTicker);

instead of string-based error-prone

_BackgroundTicker = new ObservableAsPropertyHelper<string>(
	someBackgroundTicker, _ => raisePropertyChanged("BackgroundTicker")

For the actual changes in ReactiveUI, consult Paul Betts’ insightful log.

A simple test

Since the tests have been written after writing the example code, I’ve been searching for the “Generate Unit Test” context menu in Visual Studio 2013. The context menu is not there, but luckily some enthusiasts recreated the functionality partly: → Unit Test Generator.

After the initial set-up and


here’s the simple test of the word count property:

public void WordCounterModelTest()
    var mock = new Mock<IObservable<string>>();
    var vm = new WordCounterModel(mock.Object);


    vm.TextInput = "bla!";

    vm.TextInput = "bla, bla!!";

In it, one can observe the use of MOQ for mocking a dummy and FluentAssertions for beautifully readable Spec/BDD-style assertions 3.

So, there’s no UI involved, and the UI is dogmatically bound from XAML with almost no code-behind.

Testing time series

The hello world example program simulated a dependency on some timed series of strings, ticking every second. While this is not specific to ReactiveUI, let’s make use of the test scheduler 4. For that, the time series should optionally depend on an injected IScheduler:

public class BackgroundTicker
    IScheduler scheduler = Scheduler.Default;

    public BackgroundTicker(IScheduler other_scheduler = null)
        if (other_scheduler != null)
            scheduler = other_scheduler;

    public IObservable<string> Ticker
            return Observable
                .Interval(TimeSpan.FromSeconds(1), scheduler)
                .Select(_ => DateTime.Now.ToLongTimeString());

The test instantiates a test scheduler, which is then advanced to make deterministic assertions. The code should speak for itself:

public void BackgroundTickerTest()
    (new TestScheduler()).With(scheduler =>
        var ticker = new BackgroundTicker(scheduler);

        int count = 0;
        ticker.Ticker.Subscribe(_ => count++);





Code: https://github.com/d-led/reactiveexamples

Previous article: The WPF + ReactiveUI Refactored Version of the Responsive UI Hello World.

See also: the c++ version.

  1. as in, not within TDD
  2. which now also targets Xamarin and Windows Phone 8 and Windows Store Apps
  3. I originally intended to use SpecFlow, but the specs refused to flow frictionlessly
  4. see Intro to Rx

Setting travis-ci with github for a c++ project for the first time


Here, I’m trying to use travis-ci, c++, github, CATCH, premake together with my undoredo-cpp library to reduce entropy, try out continuous integration and behavior-style tests.

As a “one-man show” programmer at least at home, I’m trying to keep the discipline of writing tests first. “Growing Object-Oriented Software Guided by Tests” is perhaps a good, although a comparatively dry book for those who are not yet convinced. The blog post by Phil Nash about his latest version of the c++ single-header testing framework CATCH, moved me to finally get my hands on the free continuous integration service travis-ci, along with CATCH with a goal to rewrite the tests for my undo-redo c++ adventure in a more behavior-driven-style.

The undo-redo library is already there, and the tests as well – in gtest (see the master branch). I’d label them “explorative” at the moment since there are just too many assertions per test case, which means I’m repeating myself.

Starting continuous integration at travis-ci for c++

To start my CATCH-“BDD” exploration I’ve setup the branch first: https://github.com/d-led/undoredo-cpp/tree/catchmoci. At my landing page the project is switched on for catching the commit hooks:


The following configuration file .travis.yml is placed for travis-ci to know what to do with my non-conforming repo:

language: cpp

   - catchmoci

before_script: ls

  - make -C Build

As in all TDD practice, the build fails due to the reason that the build doesn’t work yet at all. Adding a status image to my README shines:


Fixing the build

The makefiles are created using premake4, which is a single-file makefile generator based on lua. Unfortunately, I couldn’t force the CI-virtual machine execute my binary premake4, so I had to add the generated makefiles. Now that the make process works, the tests still don’t compile:


Fixing the tests

Once the bulk of the assertions have been rewritten for CATCH, the build still failed due to an ambiguity in serializing std::nullptr_t. Fortunately, Phil has thought of (or rather tested) that, and has a macro which can be defined for the build, fixing it: CATCH_CONFIG_CPP11_NULLPTR.

Voila, travis-ci vm is happy:


Just checking locally if test reporting is fine by adding a spurious test temporarily:


The test-rewrite has been successful and all pass, the badge is green and I can go to bed


But! It’s not the end of the story! BDD! Mock-objects!