Tag Archives: ci

Build Agent Infrastructure Testing in GoCD

In this post I would like to describe a simple technique for reducing the waiting time and stress related to build agent environment volatility when using Continuous Integration / Continuous Delivery tools like GoCD, via infrastructure testing.

The Problem

Given a modern CI server, such as GoCD, and a set of dedicated build machines (agents), it is possible to improve software development agility. Automated build/test/deploy pipelines, built to reflect the value stream, bring transparency and focus into the software delivery activities.

CI automation is software itself, and is thus susceptible to errors. Configuration management can optimize the set up of the environment, in which the build agents run. However, when computational resources are added to a CI infrastructure, i.e. to parallelize the build and thus reduce feedback times, a missing environment dependency can cause stress and pain that CI is trying to eliminate.

Consider a pipeline where a complete cycle (i.e. with slow integration tests followed by a reporting step at the end or long check-outs in the beginning) takes a significant amount of time. If one of the last tasks fails due to a configuration or an environment issue, the whole stage fails. The computational resources have been wasted just to find out that a compiler is missing. This can easily happen when there is variation in the capabilities of the build agents.

Improvement idea: fail fast! Don’t wait for environment or infrastructure mismatches

GoCD Resources as Requirements and Capabilities

If a step in a build pipeline requires a certain compiler or a particular environment, this can be conveniently expressed in the configuration of GoCD as a build agent resource. A resource can be seen as a requirement of a pipeline step that is fulfilled by a corresponding capability of a build agent.

Consider the following set-up with two build agents – one running on Windows, another one on Linux. Some tasks could be completely platform-independent, such as text processing, and thus could be potentially performed on any machine with a required interpreter installed.

agents

Build Agent is the Culprit

We have set up our environment, and have successfully tested our first commit, but the second one fails:

another_agent

The code is the same, why did the second build fail? The builds ran on different agents, but I expected them to behave similarly…

build_fails

Oh, that’s embarrassing. While yes, the script is platform-independent, there’s no executable named python2 in my Windows runner environment path.

With a one-file repository and a simple print statement this failure did not cause much damage, but as mentioned earlier, real life builds failing due to a missing executable might be costly.

Unhappy Picture

unhappy

Infrastructure Test Pipeline

In order to fail fast in situations where new agents are added to a CI infrastructure, or their environment is volatile, I propose to use a single independent pipeline that checks the assumptions that longer builds depend upon.

If a build step requires python in the path, there should be a test for it that gives this feedback in seconds without much additional waiting time. This can be as easy as calling python --version, which will fail with a non-zero return code if the binary is missing. More fine-grained assertions are possible, but should still remain fast.

If a certain binary should not be in the path, this can be asserted as well. The same goes for environment variables and file existence. Dedicated infrastructure testing tools, such as Serverspec could also be used, but having a response time under a minute is crucial in my view.

Run on All Agents

In order to validate the consistency of the CI infrastructure, the validation tasks should run on all agents that advertise a corresponding resource. This is where, in my view, the real power of GoCD comes to light, and the concepts used in it fit in the right places.

GoCD will run the test tasks on all agents that fulfill all resource requirements for the task.

run_on_all_g

Test fails

Now that we have all the tests, running them gives quick and precise feedback:

infrastructure_test_fails

Checking out the job run details reveals the offending agent. Note the test duration: under 1 second.

infrastructure_test_agent

Fixing the Infrastructure

resources_modified

Whatever the resolution of the infrastructure problem, when the infrastructure test has a good coverage of the prerequisites for a pipeline, adding new agents to the CI infrastructure should become as much fun as TDD is: write an infrastructure test, see it fail, fix the infrastructure, feel the good hormones. Add new build agents for speed — still works — great!

infrastructure_test_passes

Note how the resources that are available only on one machine are only run on one corresponding machine.

 

Happy Pictures and Developers

happy pipeline

When to Test

It is an open question, when to test the infrastructure. With the system being composed of the CI server and agents, the tests should probably run on any global state change, such as

  • added/removed/reconfigured agents
  • automatic OS updates (controversial)
  • restarts
  • network topology changes

It is also possible to schedule a regular environment check. Having the environment test pipeline be the input for other pipelines unfortunately will not do in the following sequence of events:

  • environment tests pass
  • faulty agents are added
  • downstream pipeline is triggered
  • environment failure causes a pipeline to fail

In any case, there is a REST API available for the GoCD server should automating the automation become a necessity.

Acknowledgments

I would like to thank all the great minds, authors and developers who have worked and are working to make lives of developers and software users better. Tools and ideas that work and provide value are indispensable.  The articles and the software linked in this blog entry are examples of knowledge that brings the software industry forward. I am also very grateful to my current employer for letting me learn, grow and make a positive impact.

 

 

Deterministic Testing of Concurrent Behavior in RxCpp

A Retrospective

After getting inspired by The Reactive Manifesto, it is hard not to get excited about Reactive Extensions. Such excitement has lead to a series of hello-world articles and some code examples. While Reactive Extensions take over the programming world in C#, Java and JavaScript, it seems, the world of C++ is slow to adopt RxCpp.

The new ReactiveX Tutorial link list is a great place to start learning and grokking. This article is an attempt to bring RxCpp closer to C++ developers who might not see yet, how a reactive programming model might help writing better, more robust code.

Testing concurrency with RxCpp

A previous article showed how to test ViewModels in C# by parameterizing the ViewModels with a scheduler. In a UI setting, the scheduler usually involves some kind of synchronization with the GUI thread. Testing keystrokes arriving at certain speed would require some effort to simulate events, probably leading to brittle tests. With the scheduler abstraction, the concurrent behavior of a component is decoupled from physical time, and thus can be tested repeatedly and very fast. This was the C# test:

(new TestScheduler()).With(scheduler =>
{
    var ticker = new BackgroundTicker(scheduler);

    int count = 0;
    ticker.Ticker.Subscribe(_ => count++);
    count.Should().Be(0);

    // full control of the time without waiting for 1 second
    scheduler.AdvanceByMs(1000);
    count.Should().Be(1);
});

Show Me The Code

Without further ado, the C++ version is not very far from the C# version. In a simple test, we can parameterize a sequence of integer values arriving at specified intervals (a ticker) with a coordination (why coordination and not scheduler, read in the RxCpp developer manual:

auto seq = rxcpp::observable<>::interval(
            std::chrono::milliseconds(1),
            some_scheduler
);

The deterministic test scheduler API is currently available through a worker created on the test scheduler:

auto sc = rxcpp::schedulers::make_test();
auto worker = sc.create_worker();
auto test = rxcpp::identity_same_worker(worker);

The rest should read like English:

int count = 0;

WHEN("one subscribes to an observable sequence on the scheduler") {
  auto seq = rxcpp::observable<>::interval(
              std::chrono::milliseconds(1),
              test // on the test scheduler
             ).filter([](int i) { return i % 2; });

  seq.subscribe([&count](int){
    count++;
  });

  THEN("the sequence is not run at first") {
    worker.sleep(2 /* ms */);

    CHECK(count == 0);

    AND_WHEN("the test scheduler is advanced manually") {

      THEN("the sequence is run as expected") {
        worker.advance_by(8 /* ms */);
        CHECK(count == 5);
      }
    }
  }
}

The full test can be seen @github, and is built on Travis CI

fakeformat: from header-only to dirty to continuous integration

Fakeformat

Some time ago I’ve started a little string-formatting rapid prototyping library called fakeformat (fakeformat@github). The motivation was to have .Net String.Format-like string formatting cheaply without having to use any large library. Fakeformat would allow simple eager formatting of strings like that:

ff::format("Hello {1}!").with("world").now();

(note the index starting from 1, like in Boost.Locale).

The first header-only version allowed the index as well as the format specifier index and the standard library elements used in fakeformat to be configured.

The implementation is a wrapper around the standard string stream std::stringstream.

Extending format specifiers

Boost.Locale’s format specifiers allow key-value format modifiers. These may come handy in many formatting tasks. I decided to implement some format modifiers that do not require complex locale information. The first task was to extend the parser of the format specifiers (or placeholders). The first manual approach to parsing deemed a futile task, hence a dedicated component into light. Parsing could be done via a dedicated generated parser or using a simple generated state machine.

Parser generators

There are wonderful parser generators, such as ANTLR, Bison, Lemon or my favourite, Coco/R. After looking at the generated code I decided against all, still hoping to create something minimal, allowing header-only use.

Minimalistic parsers can be made with Boost libraries Spirit or Xpressive, but if someone is using Boost, there’s no need in fakeformat.

State machine compilers

Having sketched the state machine for parsing the format specifiers, I’ve decided to generate a state machine and incorporate that into fakeformat. Once again, the choices are numerable, but I’ve been having some constraints in mind: while generated code doesn’t have to be “clean”, I’d like it to be. I’d also like it to compile without warnings on modern C++ compilers. Another constraint comes from The Pragmatic Programmer: “Don’t Use Wizard Code You Don’t Understand – Wizards can generate reams of code. Make sure you understand all of it before you incorporate it into your project.”.

A typical parser-related state machine compiler is Ragel. The examples failed to compile without warnings. I had some experience using the SMC but decided to use the old, buggy, but still quite clean-code finite state machine generator by Uncle Bob (pdf). It has a very simple syntax, and getting started is quite bumpy. So, for those trying to figure it out, here’s the command line:

java -cp smc.jar smc.Smc format.sm -g smc.generator.cpp.SMCppGenerator -f

where format.sm is the input file. The ready state machine definition after many cycles:

Context FormatContext
FSMName FormatParser
Initial General

pragma Header format_context.h
{
    General
    {
        ReadLeftBrace   ReadingPlaceholder  StartCollectingPlaceholder
    }

    ReadingPlaceholder
    {
        ReadRightBrace  General             { ParsePlaceholder
                                              FinishCollectingPlaceholder }
        ReadLeftBrace   ReadingPlaceholder  StartCollectingPlaceholder
        ReadComma       ReadingKey          { ParsePlaceholder StartKey }
    }

    ReadingKey
    {
        ReadRightBrace  General             FinishCollectingPlaceholder
        ReadLeftBrace   ReadingPlaceholder  StartCollectingPlaceholder
        ReadComma       ReadingKey          { AddKey ContinueCollectingKeys }
        ReadEqualsSign  ReadingValue        { AddKey StartAddingValue }
    }

    ReadingValue
    {
        ReadRightBrace  General             { AddValue 
                                              FinishCollectingPlaceholder }
        ReadLeftBrace   ReadingPlaceholder  StartCollectingPlaceholder
        ReadComma       ReadingKey          AddValue
    }
}

which can be read like

State
{
    Transition    Next_State   Actions
}

Dirty code

While working on the state machine I’ve committed clean-code-sin. My test has been manual, observing fancy colored console output using the library rlutil.

format_sm

The fancy coloring code is implemented in the state machine context file (s. source).

Extended state

The parsing state machine needs extended state so that the parsed tokens may be collected. The collection is implemented inside the state machine context. The driver of the state machine is however external:

FormatParser f;
f.SetString("bla {1} {2}{}{3,bla,blup}{4,k=akj,nl,jsl=22}{{5}} }}{{");	

while (!f.IsAtEnd()) {
	char c=f.Step();
	switch (c) {
			case '{': f.ReadLeftBrace(); break;
			case '}': f.ReadRightBrace(); break;
			case ',': f.ReadComma(); break;
			case '=': f.ReadEqualsSign(); break;
			default : f.Continue(); break;
	}
	std::cout<<c;
}

Incorporating parser into the formatter

While still hoping for a header-only library, I’ve written a lua script for embedding the generated state machine and the prepared token-collecting context class into fakeformat.hpp. This way I could still work on the token collection and use my fakeformat test to restore the functionality that has been broken since I’ve started working on the extension.

Still header only?

Well, the generated parser is meant to be compiled in one translation unit. I haven’t yet come up with a method to translate the text into the template code of the formatter, so now, a fakeformat.cpp has to be compiled. A simple test, instantiating the formatter from a second compilation unit confirms the usability. But with some amount of manual labor, the generated source file can be transferred into the header without functionality loss.

Formatter structure

Coming to the structure of the formatter:

  • The constructor calls the format string parser. Hence, the constructor

    auto fmt=ff::format("{1}{2}")

    is not trivial and preparses the specifiers.

  • The parameter addition methods with and also_with serialize the parameters eagerly and store them for final formatting. Note that each parameter may be formatted differently a number of times.
  • The final string formatting method now replaces the legal format string placeholders with the serialized parameters

Format modifiers

The following format modifiers are currently supported:

So, here’s a snipped of the Catch test:

REQUIRE(ff::format("{1}{1,width=3}{1}{1,w=0}").with(1).now()=="1  111");

Cleaning up

Before cleaning up, I’ve set up Travis-ci again (as for hiberlite and undoredo-cpp) → https://travis-ci.org/d-led/fakeformat, deleted manually generated Visual Studio project files and cleaned up the embedded parser from the fancy colors.

Build Status

Performance

The parsing of strings into integers is now done via the slowest, but safest version without using Boost or C++11. If performance is needed, changing the string_to_key functions can be helpful. There’s a superb article on options.

Try it out in the browser

http://ideone.com/kYcGJV

To do

Any ideas for further features or improvements?

Header only?