Category Archives: c#

Batching Data by Time or Count Using Reactive Extensions (Rx)

Motivation

The time-series database InfluxDB provides a HTTP API to write data. Data points (measurements) are inserted via a Line protocol, which allows batching of data by writing multiple points in one HTTP request.

While experimenting for a simple InfluxDB C++ client, I wanted to create an asynchronous fire-and-forget API, so that the data points can be sent over HTTP without blocking the instrumented C++ code. Several “readymade” options to implement concurrency in this scenario are available.

A simple PAIR of ZeroMQ sockets would do the job, but I’d have to implement batching separately. Thus, I turned my attention to a higher-level abstraction: Rx

Rx Window Operator

Quickly looking through the cross-language Reactive Extensions site, I found the right operator: Window.

This operator has luckily been implemented in RxCpp, thus I proceeded with the experiment.

Batching Design

Batching using Rx Winodw Operator

Rx Window Operator (CC BY 3.0 reactivex.io)[1. Source: reactivex.io License: (CC BY 3.0)]

The window operator takes an observable sequence of data and splits it into windows (batches) of observables. To batch requests, the observable windows of data are aggregated to a single value upon the last value from the windows (via other aggregating Rx operators).

A Toy Problem

To validate the approach, the following problem is set:

Given a stream of integers, append the integers into a series of strings, either every second, or every N integers

String appends with integer-to-string conversions in C++ will be done via the {fmt} library.

Batching in One Line of Code

A stream of numbers batched either by time or count:

auto values = rxcpp::observable<>::range(1, 1000'000)
    .window_with_time_or_count(std::chrono::seconds(1), 100'000);

Note, there is an almost one-to-one translation into a C# version:

var values = Observable.Range(1, 1000000)
               .Window(TimeSpan.FromSeconds(1), 100000);

This indicates the power of the Rx abstraction across languages. The Rx website provides just the right sorting of the documentation to be able to translate Rx code from one language to another.

Aggregating the Batches

In order to do something useful with the batched data, the Scan operator is used to gather the data in a string buffer, and after the last value has been received, the string buffer is assembled into a string and processed:

values.subscribe(
    [](rxcpp::observable<int> window) {
        // append the number to the buffer
        window.scan(
            std::make_shared<fmt::MemoryWriter>(),
            [](std::shared_ptr<fmt::MemoryWriter> const& w, int v)
        {
            *w << v;
            return w;
        })

        // what if the window is empty? Provide at least one empty value
        .start_with(std::make_shared<fmt::MemoryWriter>())

        // take the last value
        .last()

        // print something fancy
        .subscribe([](std::shared_ptr<fmt::MemoryWriter> const& w) {
            fmt::print(
                "Len: {} ({}...)\n",
                w->size(),
                w->str().substr(0, 42)
            );
        });            
    }
);

The Tale of Two Bugs

In the initial (non-TDD) spike, the batching seemed to work, however, something caught my attention (the code bites back):

[window 0] Create window
Count in window: 170306
Len: 910731 (123456789101112131415161718192021222324252...)

the window wasn’t capped at 100’000. This could have been either a misunderstanding or a bug, thus I formulated a hesitant issue #277. As it turned out, it indeed was a bug, which was then fixed in no time. However, the first bug has hidden another one: the spike implementation started to crash at the end: when all the windows were capped by count, and not by time, last window was empty, as all values fit exactly into 10 batches.

The Last operator rightly caused an exception due to an empty sequence. Obviously, there’s no last value in an empty sequence. Rubber Ducking and a hint from Kirk Shoop fixed the issue by utilizing the StartWith operator to guarantee, the sequence is never empty. An empty string buffer can be ignored easier downstream.

Active Object

The active object pattern was applied to implement a fire-and-forget asynchronous API. A Rx Subject to bridge between the function call and the “control-inverted” observable:

struct async_api {
    //...
    rxcpp::subjects::subject<line> subj;
    //...

    async_api(...)
    {
        auto incoming_requests = subj
            .get_observable()
            .map([](auto line) {
                return line.to_string();
            });

        incoming_requests
            .window_with_time_or_count(
                window_max_ms,
                window_max_lines,
                // schedule window caps on a new thread
                rxcpp::synchronize_new_thread()
            )
            .subscribe(...)
        ;
    }

    // fire-and-forget
    void insert(line const& line)
    {
        subj
            .get_subscriber()
            .on_next(line);
    }
};

in order not to block the caller (which would be the default behavior), the observable watches the values from each window on a new thread. Here, scheduling on a thread pool (currently missing in RxCpp) would probably be beneficial.

While this implementation might not be an optimal one, the declarative nature of Rx, once the basics are understood, allows to “make it work and make it right” pretty quickly by composing the right operators.

Code

The runnable code of the example can be found at Github: C++ version.

In order to show, how similar the high level code can be between different languages when writing, I’ve “ported” the example to C# [2. The C# version appears to run faster on my windows machine while solving the same toy problem].

No Events: ReactiveUI Windows Forms MVVM-Style

Review

designed using http://viperneo.github.io/winforms-modernui/

This is the next post in the series, looking first at Reactive Extensions (RX) to simplify writing Windows Forms UI logic, then using a viewmodel shared between a WPF gui implementation and a rewritten WinForms version using ReactiveUI, stopping at a short article on testing the viewmodels.

passedtest

ReactiveUI News

ReactiveUI API has been quite volatile for the last year, and this series is in need of an update[0. See ReactiveUI Design Guidelines]. A CodeProject author gardner9032 published a nice teaser article, showing the ReactiveUI mechanism for writing simplified Viewmodel-View bindings [1. see article @CodeProject], which serves as primary trigger for this post.

There’s plenty of news and updated articles on Paul Betts’ log, providing a good resource for updates on the API. Phil Haack’s blog is also a superb resource for explanations and commentaries on the use of ReactiveUI in real-world applications.

The ReactiveUI project is quite active, as the community seems to have grokked the jist of it, while the list of supported platforms has become more than convincing.

Getting rid of events

The enabling feature of ReactiveUI is writing declarative UI glue code, and if the viewmodels are based on Reactive Extensions, then declarative C# style can be used throughout the project. The previous ReactiveUI Windows Forms examples converted an event sequence into an observable sequence of values. In this example, that will be accomplished conveniently by the ReactiveUI WinForms lbrary. The viewmodels also contained some imperative code. For this article, the viewmodels will not be reused from the previous articles, but written from scratch.

ViewModel

s. code

The viewmodel’s task is the same: something is ticking in the background, while words are counted in the input text asynchronously, and the calculation is throttled to 0.5 seconds. The viewmodel boilerplate is simplified using ReactiveUI.ReactiveObject.

Output (read-only) properties

The ReactiveUI-way of creating output properties is through ObservableAsPropertyHelper.

private readonly ObservableAsPropertyHelper<string> backgroundTicker;
public string BackgroundTicker
{
    get
    {
        return backgroundTicker.Value;
    }
    //...
}

The constructor of the viewmodel receives an IScheduler for scheduling on the correct thread, and an IObservable, which will be a stream of input from the view. Observe the ToProperty helper:

public MyViewModel(IScheduler scheduler, IObservable<string> input)
{
    Observable.Interval(TimeSpan.FromSeconds(1))
        .ObserveOn(scheduler)
        .Select(_ => DateTime.Now.ToLongTimeString())
        .ToProperty(this, x => x.BackgroundTicker, out backgroundTicker);
    //...
}

Word counting logic is implemented in a similar fashion by transforming the incoming stream of strings.

View

s. code

To remove yet more boilerplate code, WinForms Form specialization implements the ReactiveUI.IViewFor interface. This allows for largely simplified run-time and compile-time checked bindings, avoiding using strings for property names. The implementation is straightforward, and pays off once the views become more complex than this example:

IViewFor

public MyViewModel VM { get; private set; }

object IViewFor.ViewModel
{
    get { return VM; }
    set { VM = (MyViewModel)value; }
}

MyViewModel IViewFor<MyViewModel>.ViewModel
{
    get { return VM; }
    set { VM = value; }
}

Bindings

None of the controls in the designed WinForm have wired events or bindings set from the designer. The glue code is reduced to instantiating the viewmodel …

VM = new MyViewModel(
    new System.Reactive.Concurrency.ControlScheduler(this),
    this.WhenAnyValue(x => x.inputBox.Text)
);

… and declaring the bindings[2. The ReactiveUI WinForms implementation seems not to support fully read-only fields using default bindings yet, hence an empty setter in the viewmodel] [3. The scheduler is from Windows Forms helpers].

this.Bind(VM, x => x.BackgroundTicker, x => x.tickerBox.Text);
this.Bind(VM, x => x.WordCount, x => x.wordCountBox.Text);

Source

@github

Beginning another quest – scripting business logic in .NET: using dynamiclua

Let’s start with Lua

The current state of things is that the lightweight but mighty Lua has become ubiquitous in the world of computing as a scripting and configuration component. Lua’s strategy of “mechanism, not policies” allows the language to be extremely malleable. It’s worth watching Roberto Ierusalimschy’s talks on the evolution of Lua [1. Video 1, more slides, Video 2, more Roberto].

The latest adoptions of Lua are for example the Redis’ EVAL Query with Lua scripting, and Wikipedia’s template automation using user-written Lua scripts.

dynamiclua

There have been several attempts to bring Lua into the world of .NET, and one of the most recent efforts by Niklas Rother, dynamiclua, combines the previous achievements with the dynamic type of .NET 4.0, and lays path for an easily reusable business logic scripting component, which is now even available via nuget.

After adding dynamiclua as a reference, here’s your first business logic:

using (dynamic lua = new DynamicLua.DynamicLua()) {
 Console.WriteLine(lua("return 2+2"));
}

note the IDisposable usage for freeing native resources.

Discovering the features

Static configuration

As an author of a configurable component,
In order to reduce my support effort,
I want the users of my component to write their static configuration themselves

While for that user story JSON or XML would suffice, we’re on the Lua track, hence this is how it could work:

// comes from some user input or file
string user_static_config = @"
 config = {
  url = 'https://github.com/nrother/dynamiclua'
 }
";

lua(user_static_config);
string url = lua.config.url;
Console.WriteLine(url);

… it does look like JSON, but it isn’t. Here, one can observe the harmony resulting from the use of the dynamic keyword to interact with a dynamic language from a statically typed one. DynamicLua can leverage reflection capabilities of both languages to create an “automagical” interop, compared sometimes tedious writing of bindings for a language, such as native C++.

The power of Lua at the user’s fingertips

As an author of a customizable business software,
In order to reduce my own support effort,
And create a new business of customizations,
I want the administrators of my software to write their business logic scripts themselves

Many systems grow out of static configurability, and business logic definition and evaluation in some programming language becomes inevitable. Whether this is a good thing or a bad one for one’s business [2. User scripts are software, and thus might never terminate] is not in the scope of this article, as it is assumed, we want to make our software customizable via scripting [3. I won’t repeat the meme about great power here].

The static configuration entries can be dynamically evaluated at script run time. Various bindings may be used in the script, as i.e. the math standard library here:

lua("config.answer = nil or math.floor(131.94689145078/math.pi)");
int answer = (int)lua.config.answer;
Console.WriteLine(answer);

Using Lua as an expression evaluator

Lua supports multiple return values, which makes certain scenarios fun:

dynamic result = lua("return 1, 'world'");
Console.WriteLine("{0} {1}", result[0], result[1]);

Binding .NET code

Dynamiclua does not have an explicit class binding syntax [4. as is the case with many C++ bindings i.e. LuaBridge], but it doesn’t matter much, as functions can be bound, and thus, constructors. Via reflection all public [5. In my opinion, access to privates should throw…] members are bound. The domain code is therefore very easy to bind. Take an example:

class Example
{
    int answer = 42;
    public int Answer { get { return answer; } }

    public int Add(int a, int b)
    {
        return a + b;
    }
}

The binding is quite trivial:

lua.NewExample = new Func<Example>(() => new Example());

// Lua functions may be called via dynamiclua
Example my_example = lua.NewExample();
Console.WriteLine(my_example.Answer);

lua(@"
    local example = NewExample()
    print(example.Answer)
    print(example:Add(2,3))
");

Minimal sandboxing

As an author of a scripting component,
In order to mitigate the risk of damage by malicious user-supplied code,
I want the script executing environment to be sandboxed.

Sandboxing is a very complex issue and perhaps only approximations of an undefined ideal sandbox can be made [6. see wiki: SandBoxes and Daniel Silverstone’s talk at Lua Workshop 2013 on his approach]. A sane attitude towards this security-related issue should go along the lines of trusting nobody, even oneself, and even that distrust should not be enough. Depending on the posed security requirements, the logic execution component might even be physically separated from your business software, i.e. running on another machine or in another process. But that is perhaps another concern and should be independent of your script execution component.

Since Lua is malleable, one can configure or manipulate the Lua virtual machine to make certain actions unlikely, such as accessing the file system. This can be achieved by setting the bound names to nil as in [7. More …]:

lua("import = nil");
lua("require = nil");

new Action(() => lua("import 'System'"))
    .ShouldThrow<Exception>();

In this little spec you can also observe what should happen (non-specifically) if there’s a Lua syntax or a runtime error. You can also see how to import assemblies into the Lua state.

Here’s another quote from the test:

lua.import("System");

Func<int> func = () => (int)lua("return Console.BufferHeight")[0];
new Action(() => func())
    .ShouldNotThrow();
func().Should().Be(Console.BufferHeight);

Without further ado

Some things should perhaps be further investigated:

  • Is dynamiclua prepared for the Mono runtime on *x platforms?
  • Is it possible to crash the process uncontrollably, even while catching exceptions?
  • Fix crash on finalization

Lua can appear simple enough, such that, given a syntactic constraint, even non-programmers could write business logic scripts and configurations. However, there is a company experimenting with alternative ways to define business logic using different approaches to a logic-definition UI. It’s worth checking out: LogicEditor.

Other language and runtime candidates will hopefully be investigated within the next posts of the quest: IronRuby and Boo. The readers of this blog entry are welcome to criticize, point out omitted or wrong information, and interact freely.

Code

https://github.com/d-led/BusinessLogicScripting