Tag Archives: Lua

Self-contained GoCD Environment Using Docker-Compose

Motivation

Today’s software is often connected, be it for automatic desktop updates or for implementing an internet-scale service. Developers’ tools or toys are no longer solely editors or compilers, but also databases, logging or search services, code sharing platforms, or Continuous Integration servers. Evaluating distributed on-premise software has not been as easy as desktop software, however. Arduous and error-prone installation instructions seem out of place, but are still very common. A number of open source software now comes with a one(or two)-liner installation that is always up to date. GoCD is one of them.

While discovering features of GoCD I sometimes wished for even more simplicity and automation: a one-liner for a whole Continuous Integration environment. This should include a server, several build agents, and several source repositories. With a recent push towards containerized software delivery, the path is quite clear: build, provision and configure the whole infrastructure from code and run it in containers. This way, it is easier to experiment, build and communicate Continuous Delivery prototypes.

GoCD Infrastructure as Code

There are official GoCD Docker images for both the server and base images for the build agents. The containers are configured such that the build agents can register themselves automatically with the server if the auto-registration key of the server is known.

Thus, starting from the official Docker images, to get to a one-liner self-contained infrastructure installation, the following ingredients are missing:

  • starting the server and the agents
  • provisioning the agents to contain desired compilers
  • adding source code repositories to be built

Docker Compose as a Powerful Toy

Docker Compose is a tool to define and run multi-container applications. It is so pragmatic that using it almost feels like playing.

To start the server and an agent, the following Docker Compose configuration would suffice:

Running docker-compose up -d pulls and brings up the minimal infrastructure.

A Custom Build Agent

In the configuration above, the agent is rather empty, and probably does not contain the build infrastructure we need.

Here, we extend the gocd/gocd-agent-ubuntu-16.04:v17.3.0 image, installing some Lua infrastructure:

In the docker-compose configuration, the image tag is replaced with a build one:

Note the resources that can be assigned to the build agents from the docker-compose file and get passed to the agent container as environment variable for correct automatic registration with the server.

After bringing up the infrastructure and the auto-registration key matches that of the server, the agents are registered with the server:

GoCD agents

25.3.2017 Update: in the GoCD 17.3.0 server image, the auto-registration key is a generated one, and is not set to the default above. To enable auto-registration, the provisioning step is used.

Pipeline →⇉→ as Code

The final missing piece of the self-provisioning CD infrastructure is the addition of the repositories to be built. There is a number of GoCD management libraries on Github. Each would probably serve the purpose of setting up the pipelines.

The approach taken here is to centrally manage a list of repositories to be built, while delegating the details of each pipeline to the corresponding repository. This way, the pipeline configuration is part of the repository itself (pipeline as code), and could be portable between different GoCD instances.

The three ingredients are: the pipeline configuration from source control feature #1133, the YAML Config Plugin, and the GoCD REST API, as of the time of the experiment, none of the tools I have seen could add the pipeline-as-code configuration.

Adding the Pipelines (External Provisioning)

for a simpler provisioning via mapping a config file into the container, see the next chapter

To configure the server with the pipelines, and set the auto-registration key, a separate container runs a Python script once. This way, provisioning is decoupled from the generic server configuration and can be replaced by another mechanism without rebuilding the server image. The script provisions the server with the pipelines, and sets the auto-registration key to the one agreed upon.

A common problem is the startup sequence of the containers. Trying to provision the server before it is ready to accept configuration would result in an error. Thus, the provisioning script tries to avoid pushing the configuration too early by waiting for the GoCD web UI to become available. This is accomplished by waiting using urlwait:

if not wait_for_url("http://go-server:8153", 300):
  print("""Go server did not start in a timely fashion.
           Please retry docker-compose up provisioner""")
  sys.exit()

The repositories to configure pipelines from are (here, rather crudely) added directly to the XML config as the yaml config plugin expects them. The XML is first read from the /go/api/admin/config.xml API, then simply extended with the necessary tags, and then posted to the same URL. There is still a chance of a race condition that the configuration is changed between it is read and it is written. As GoCD validates the config upon modification, and the script strives to be idempotent, re-running the container should fix the conflict.

The GoCD XML config needs the following addition for a repository to configure a pipeline:

<config-repos>
 <config-repo plugin="yaml.config.plugin">
  <git url="https://github.com/d-led/gocd-rpi-unicorn-hat-monitor.git" />
 </config-repo>
</config-repos>

In the repository itself, place a ci.gocd.yaml with a corresponding pipeline definition.

After grabbing a coffee, the infrastructure has been started, provisioned and configured, and the UI shows the result:

GoCD pipelines

Yet Simpler Self-Provisioning

With the current official Docker image for the server, it is possible to map configuration-relevant files and folders into the container. In our case this means, the plugin, and the whole server configuration can be directly mapped into the container, thus, provisioning the server without an extra provisioning step.

As GoCD keeps the pipeline configuration in a single file cruise-config.xml, we can simply track it in the same repository as the Docker Compose config. To map the configuration, and the plugin jar, the volumes are added to the container config:

When the server starts, it already has most of its configuration. As the agent auto-registration key is part of the XML config, the agents will automatically register themselves, since they are configured with the same key.

Other CI Tools

A comparable exercise can be performed with almost any tool. Some tools that are natively based on the concept of pipelines are Concourse CI and Drone that both use Docker as build agents (runners). Drone even comes with its own Docker-Compose config. Concourse can be bootstrapped via Vagrant.

A significant difference of GoCD to the more recent tools is its platform-independence. Sometimes, building inside a container is not a choice, e.g. on Windows. GoCD agents can run anywhere, where a JRE8 can run, thus, increasing its reach. A phoenix environment including Windows agents can be achieved with some effort using Chocolatey → Packer → Terraform/Vagrant.

Conclusion

This article has described an experiment to rather quickly arrive at a self-contained and self-provisioned Continuous Delivery infrastructure consisting of “Phoenix Servers” – a phoenix infrastructure.

While the result is rather humble, it demonstrates that continuous delivery techniques can be applied to a continuous delivery infrastructure itself. Using Docker Compose allowed to prototype a distributed development-supporting application and its configuration on a local developer machine with a potential to transfer the prototype into real use.

All this would be impossible without a huge network, or, I’d say, a universe, consisting of online services running and providing open source software that is created by a large number of open source communities, and a yet larger number of individuals collaborating in various ways to envision, create, maintain, and run it. Moreover, times are such that one can experience successful transition of proprietary software into open source (GoCD), and companies, building business around open source software. The OSS ecosystem is a distributed, self-directing system that catalyzes idea creation, mutation and destruction much faster that most smaller systems can do. For the moment, I hope, there is no going back.

Repository

The Github repository to run the self-contained infrastructure can be found here: gocd_docker_compose_example.

The repository can be used to bootstrap demos, further experiments and proofs of concept.

Disclaimer

This post, as any other post on my blog, is not advertisement, and no affiliation or endorsement exists. It is a write-up of my personal experiments, experiences and opinions. The results obtained here can most certainly be achieved using other tools and technologies.

Automatic Lua Properties

Automatic Lua Properties?

Starting with an example using the Lua specification and testing framework Busted:

Here is a little exercise in Lua metaprogramming.

Code

https://github.com/d-led/automatic-lua-property-tables

Spec: autoprop_spec.lua
Implementation: autoprop.lua

P.S. Other implementations: lua-users wiki: Automagic Tables

function argument sequence

In many APIs one is confronted with the question, which parameter stands for what.

Consider a call in a code you’re reading or trying to write:

is_funkier(1,2)

Is 2 funkier than 1 or vice versa? A long time ago I asked a childhood friends’ father if I could try program their game console. The response was that one has to know a programming language, such as basic. On a request to quote a bit from a programming language I heard “begin, end, goto”. ‘Sounds like English to me’, I replied,’I could do that’. And now, years later I want to program close to English: not Shakespeare but rather a DSL.

A minimal example of how I would improve the API above 1. Test:

And an easy way of implementing the API 2:

Depending on the usage, such constrained argument ordering may improve readability and thus perhaps reduce risks associated with writing code.

To further extend the DSL, further mechanisms can be used of the language of choice. There are some languages that can help remove clutter even further by eliminating punctuation. See also: a nice introduction to creating readable DSLs in Groovy.

  1. in Lua here, but not limited to Lua in principle
  2. returning a function would suffice in case of one query

ris – a lightweight cross-platform resource compiler for c++ projects

Why a resource compiler in the 21st century?

Starting a c++ project that will potentially need static string resources (i.e. Lua scripts) makes one search for an easy way to embed large strings in an executable. There are many ways of including static resources, but there seems to be no simple but robust, platform-independent way 1. For fast prototyping and one-shot projects, I’d like to have lightweight, minimum-configuration and install solution to the problem of embedding binary resources without having to use a large framework.

Premake, my favourite light-weight meta-build system contains a number of Lua scripts in its binary. These are embedded using a Lua script into a simple array of C-string constants. This is the simplicity that in my view should be strived for. ris is an attempt to do something similar for general c++ projects with a possibility of embedding binary blobs.

ris – cross-platform resource compiler for c++

The project (ris@github) is in its infancy, but seems already to be usable. Here’s a preview:

Defining and compiling resources

ris <path_to>/<resources>.json

with an input file as this self-explaining one:

{
    "namespace" : "test",
    "header" : "acceptance_test/resource.h",
    "source" : "acceptance_test/resource.cpp",
    "resources" : [
        {
            "name" : "string_test",
            "source_type" : "string",
            "source" : "plain text"
        },
        {
            "name" : "binary_file_test",
            "source_type" : "file",
            "source" : "test.bin"
        }
    ]
}

will generate two c++11 files to include in your project, enabling easy resource retrieval:

std::string res = test::Resource::string_test();

or

std::string res = test::Resource::Get("string_test");

Update 30.07.2015: now resources can be defined more concisely in YAML. A minimal resource definition in YAML looks like the following:

header: "res.h"
source: "res.cpp"

resources:
  -
    compression: "LZ4F"
    name: "some_text"
    source: "some text"
    source_type: "string"

Enumerating the resources

Resource keys in the compiled resource can be enumerated passing a callable to GetKeys:

std::vector<std::string> keys;
test::Resource::GetKeys([&keys](char const* key){
    keys.emplace_back(key);
});

Compression

Using an optional compression library bundle, adding a compression property to a resource description enables transparent (de)compression:

"compression" : "LZ4HC"

Customization

updated 24.11.2014:
ris now uses text resources generated and bootstrapped by its own early version. The goal is to make The code generator is customizable. The default template can be seen in template.json, and the generated header in template.h. The generation sequence can be seen in ris.cpp.

Using the second parameter to ris, it’s possible to override strings in the generator template. See an example below.

C++03

updated 27.11.2014:
One such customization is available in template_cpp03.json, where the C++11 constructs are replaced with generateable C++03 constructs.

To generate the resources using the C++03 template:

ris my_template.json template_cpp03.json

Why C++?

Such code generator as ris could most probably be developed more rapidly using any other programming language with a huge framework and a ton of libraries behind them. My personal preference for certain kinds of small projects lies in the area of self-contained single-binary/single-file executables or libraries, such as Lua. Lua is the primary motivation for this project, as it is itself a compact library for building flexible and extensible prototypes and tools. ris can act as a bootstrapping component to embed resources for building specialized shell-scripting replacements, i.e. for massive scripted file operations.

Source

https://github.com/d-led/ris

There is a number of paths this project can take from here. Features, such as robustness, performance or flexibility could all be addressed, but most probably ris will be shaped by its usage or absence of such.

A simple github + travis-ci project for quick c++ to lua binding tests

It’s sometimes necessary to create a simple example for C++ to Lua bindings on the run. Travis-CI might be of great help in that, while online C++ compilers will not suffice.

Here’s the project and a sample code:

https://github.com/d-led/lua_cpp_tryout

#include <lua.hpp>
#include <LuaBridge.h>
#include <RefCountedPtr.h>
#include <LuaState.h>
#include <iostream>

void luabridge_bind(lua_State* L) {
	class Test {
	public:
		Test() {
			std::cout<<__FUNCTION__<<std::endl;
		}

		~Test() {
			std::cout<<__FUNCTION__<<std::endl;
		}
	};

	luabridge::getGlobalNamespace(L)
		.beginClass<Test>("Test")
			.addConstructor<void(*)(),RefCountedPtr<Test>>()
		.endClass()
	;
}

int main() {
    lua::State state;
    luabridge_bind(state.getState());
    try {
    	state.doString("Test()");
    	state.doString("blabla()");
    } catch (std::exception& e) {
    	std::cerr<<e.what()<<std::endl;
    }
}

Test
[string "blabla()"]:1: attempt to call global 'blabla' (a nil value)
~Test

note the automatic lifetime management

In the process of searching for a quick header-only wrapper for the Lua state I came across LuaState which was the first one that compiled out of the box. Its advantage over other wrappers for my needs was that it did reveal the actual Lua state, with the effect of other binding libraries being usable. It could itself provide an alternative to LuaBridge for writing bindings and could be investigated further.

It’s now easy to fork the project and try out bindings online.

Beginning another quest – scripting business logic in .NET: using dynamiclua

Let’s start with Lua

The current state of things is that the lightweight but mighty Lua has become ubiquitous in the world of computing as a scripting and configuration component. Lua’s strategy of “mechanism, not policies” allows the language to be extremely malleable. It’s worth watching Roberto Ierusalimschy’s talks on the evolution of Lua 1.

The latest adoptions of Lua are for example the Redis’ EVAL Query with Lua scripting, and Wikipedia’s template automation using user-written Lua scripts.

dynamiclua

There have been several attempts to bring Lua into the world of .NET, and one of the most recent efforts by Niklas Rother, dynamiclua, combines the previous achievements with the dynamic type of .NET 4.0, and lays path for an easily reusable business logic scripting component, which is now even available via nuget.

After adding dynamiclua as a reference, here’s your first business logic:

using (dynamic lua = new DynamicLua.DynamicLua()) {
 Console.WriteLine(lua("return 2+2"));
}

note the IDisposable usage for freeing native resources.

Discovering the features

Static configuration

As an author of a configurable component,
In order to reduce my support effort,
I want the users of my component to write their static configuration themselves

While for that user story JSON or XML would suffice, we’re on the Lua track, hence this is how it could work:

// comes from some user input or file
string user_static_config = @"
 config = {
  url = 'https://github.com/nrother/dynamiclua'
 }
";

lua(user_static_config);
string url = lua.config.url;
Console.WriteLine(url);

… it does look like JSON, but it isn’t. Here, one can observe the harmony resulting from the use of the dynamic keyword to interact with a dynamic language from a statically typed one. DynamicLua can leverage reflection capabilities of both languages to create an “automagical” interop, compared sometimes tedious writing of bindings for a language, such as native C++.

The power of Lua at the user’s fingertips

As an author of a customizable business software,
In order to reduce my own support effort,
And create a new business of customizations,
I want the administrators of my software to write their business logic scripts themselves

Many systems grow out of static configurability, and business logic definition and evaluation in some programming language becomes inevitable. Whether this is a good thing or a bad one for one’s business 2 is not in the scope of this article, as it is assumed, we want to make our software customizable via scripting 3.

The static configuration entries can be dynamically evaluated at script run time. Various bindings may be used in the script, as i.e. the math standard library here:

lua("config.answer = nil or math.floor(131.94689145078/math.pi)");
int answer = (int)lua.config.answer;
Console.WriteLine(answer);

Using Lua as an expression evaluator

Lua supports multiple return values, which makes certain scenarios fun:

dynamic result = lua("return 1, 'world'");
Console.WriteLine("{0} {1}", result[0], result[1]);

Binding .NET code

Dynamiclua does not have an explicit class binding syntax 4, but it doesn’t matter much, as functions can be bound, and thus, constructors. Via reflection all public 5 members are bound. The domain code is therefore very easy to bind. Take an example:

class Example
{
    int answer = 42;
    public int Answer { get { return answer; } }

    public int Add(int a, int b)
    {
        return a + b;
    }
}

The binding is quite trivial:

lua.NewExample = new Func<Example>(() => new Example());

// Lua functions may be called via dynamiclua
Example my_example = lua.NewExample();
Console.WriteLine(my_example.Answer);

lua(@"
    local example = NewExample()
    print(example.Answer)
    print(example:Add(2,3))
");

Minimal sandboxing

As an author of a scripting component,
In order to mitigate the risk of damage by malicious user-supplied code,
I want the script executing environment to be sandboxed.

Sandboxing is a very complex issue and perhaps only approximations of an undefined ideal sandbox can be made 6. A sane attitude towards this security-related issue should go along the lines of trusting nobody, even oneself, and even that distrust should not be enough. Depending on the posed security requirements, the logic execution component might even be physically separated from your business software, i.e. running on another machine or in another process. But that is perhaps another concern and should be independent of your script execution component.

Since Lua is malleable, one can configure or manipulate the Lua virtual machine to make certain actions unlikely, such as accessing the file system. This can be achieved by setting the bound names to nil as in 7:

lua("import = nil");
lua("require = nil");

new Action(() => lua("import 'System'"))
    .ShouldThrow<Exception>();

In this little spec you can also observe what should happen (non-specifically) if there’s a Lua syntax or a runtime error. You can also see how to import assemblies into the Lua state.

Here’s another quote from the test:

lua.import("System");

Func<int> func = () => (int)lua("return Console.BufferHeight")[0];
new Action(() => func())
    .ShouldNotThrow();
func().Should().Be(Console.BufferHeight);

Without further ado

Some things should perhaps be further investigated:

  • Is dynamiclua prepared for the Mono runtime on *x platforms?
  • Is it possible to crash the process uncontrollably, even while catching exceptions?
  • Fix crash on finalization

Lua can appear simple enough, such that, given a syntactic constraint, even non-programmers could write business logic scripts and configurations. However, there is a company experimenting with alternative ways to define business logic using different approaches to a logic-definition UI. It’s worth checking out: LogicEditor.

Other language and runtime candidates will hopefully be investigated within the next posts of the quest: IronRuby and Boo. The readers of this blog entry are welcome to criticize, point out omitted or wrong information, and interact freely.

Code

https://github.com/d-led/BusinessLogicScripting

  1. Video 1, more slides, Video 2, more Roberto
  2. User scripts are software, and thus might never terminate
  3. I won’t repeat the meme about great power here
  4. as is the case with many C++ bindings i.e. LuaBridge
  5. In my opinion, access to privates should throw…
  6. see wiki: SandBoxes and Daniel Silverstone’s talk at Lua Workshop 2013 on his approach
  7. More …

lurlutil, a console manipulation library for lua : API documented

The cross-platform Lua wrapper of the console manipulation library rlutil seems to have arrived at a certain milestone:

https://github.com/d-led/lurlutil

The API has been expanded, converging on the original scope of rlutil.

Enjoy:

and much more further console fun for Windows, Linux and MacOS X.

Quickstart for cross-platform c++ projects

A typical dilemma for a c++ developer is creating the initial build configuration. Out of my affection for Lua, I’ve collected my typical premake4 patterns into a separate project to be able to set up c++ projects on any platform within a minute.

Here’s a sample from selfdestructing: