Go Rust Swift, Vala

In the WWDC keynote speech, Apple announced their new programming language named Swift. It looks like a scripting language, but it actually compiles to binary, giving you lots of performance. For years, I've been griping about the need for a programming language that's as easy as scripting, while delivering decently high performance. Lately, we've seen a number of new languages that do exactly this. So, it's time to discuss a few of these new, modern languages.

Go

Go was developed at Google and was first publicly released at the end of 2009. It's no secret that I'm a bit of a fan of Go, and I'll admit that of the four new languages discussed here, it's the only one I actually wrote a decent amount of code in.
Go is awesome for parallel programming. Goroutines and communication channels are plain brilliant. In fact, it completely changes the way you think about parallel programming. Go's duck typing system is marvelous, an enormous sigh of relief after decades of OOP and I just want to hug the creators of Go.
And yet, it has not yet become my primary language of choice. A big reason for that is mandatory CamelCasing. In Go, lower/upper CamelCasing actually controls whether the symbol is being exported or not; a really awkward design decision (golang's only mishap?).
Go is great for systems programming (ie. for UNIX tools, services), however ... it's as clunky as C when trying to parse a config file, and its flag command-line argument parser module isn't great either. Then I tried making a game in Go and simply couldn't get started because there were no stable bindings for the needed libs at hand at the time.
Go has a very “C feel” to it, every time I code in it I feel like I'm constructing a tiny, delicate machine. Everything about Go is excellently engineered. Some clever men with beards really thought things through.

What will golang's popularity be, five years from now? We shall see if it's truly to replace C/C++ as top systems programming language.

Rust

Mozilla's language named Rust started in 2006, but didn't go public until 2010. Rust is quite impressive, but the tutorial kind of lost me once it started on heap allocation, object ownership of memory, control of mutability, references, and confusing syntax involving glyphs. This leads to some pretty horrific code that shouldn't have been that way, in my opinion. Secretly, Rust is trying hard to be like C++.
The tutorial has some notes on features that are currently missing or buggy. In that respect, Rust is not ready for prime time yet. It does show a lot of potential and it did manage to create a buzz in the open source world, so check back once it hits version 1.0.

Swift

Apple's Swift language was announced only last week and is said to have been in development since 2010. At a glance, it's suspiciously a lot like Rust. This is particularly interesting given the fact that Rust is sponsored by rival company Samsung. What's the true story behind this?

Anyway, Swift looks awesome, and it's already miles ahead of Rust. Swift builds upon a foundation laid by Objective-C, all the way from NextStep to OS X and iOS. Cocoa and Cocoa Touch bindings are in place and ready to use. Under the hood Swift uses reference counting for garbage collection, and here lies a major pitfall; you should take care not to create any cyclic references, and/or know the difference between weak and unowned references. If you get this wrong, memory will leak just as it did in Obj-C. [Note that the same is true for Rust and Vala].

Swift lives in an Apple-only world, which should be no surprise — I suppose this compares to Microsoft's strategy regarding Visual Basic, C#, .NET.
There is no doubt that Swift is going to be huge on iOS, and probably also on OS X. Nobody likes Obj-C (apart from some weird guys like me). The matter of truth is, Obj-C is outdated, it has just been obsoleted by Swift. Two years from now, no one will be writing new applications in Obj-C.

Vala

Last but not least, there is Vala, which has been around since 2006 already. Vala is a Java/C#/Mono-like language for developing GNOME applications. It is said Elementary OS's main apps are built in Vala. The Vala compiler converts Vala code to plain C with GObject, which is in turn compiled by a generic C compiler. In a sense, Vala is just syntactical sugar, removing all the bad things that C has (the tedious task of programming intricate details), while keeping all the good (high performance).
Its syntax will be appealing to mostly Java and C# programmers. Personally, I get lost in the soup of keywords. A Vala source can be like a C++ code in disguise. There are just too many keywords; the designers forgot that less is more. It's probably too late now, but my wish for Vala is to be stripped down, simplified, made easier to work with.
Nevertheless it is a great choice for developing GNOME applications, and there is Elementary OS to prove it. Currently it looks like Vala's success is tied to that of GNOME, and it's a bit of a shame because it deserves more attention from a broader audience.
Vala wins the prize for “Best chosen name for new programming language” simply because googling it lands you on the right pages right away.

Links

• Effective Go
• Rust tutorial
• Swift Basics
• Vala tutorial

Programming is hard

I have always seen programming as something creative, like an art. And like in art, it is hard work to create something beautiful. It's hard to get the details right. If you get it right, then people will like the result, and of course, some may not because tastes differ and you can't make everybody happy. But aside from people's reaction to the picture painted, a painter also gets his satisfaction from mere crafting, using this technique here and that trick there, knowing the internals of his work.
Picture a musician, selecting an instrument to play, a rhythm, writing a melody.

That sure is a romantic view of programming. In reality, it's more of a purely technical process. When you want to accomplish goal G you are going to have to make work A and B together. This might be done with technique C. This is engineering. Apply a trick here, and it is called a hack. Hacks can be dirty hacks, or they can be clever hacks. In a security context, they might even be dangerous hacks.

Some say building applications is like building a house, and there should be blueprints for building software as well. Houses are built from small, simple building blocks. All houses have walls and a roof, a kitchen, a living and a bedroom, yet each house is unique. For each room, you can zoom in and work out the details. Users have the option of changing the wallpaper. And of course, a construction error may take down the entire house.
It's a nice metaphore that works well, but I suspect that whoever thought up that idea is not a programmer.

Programming is a mind game. The programmer builds a model in his mind, writes down the code and tries it. If it doesn't work, she tries to find out where it doesn't work. You may use a debugger or sketch it on paper but mostly the problem is cracked in your mind. Like a game of chess, it takes a lot of concentration trying to think six moves ahead, a single mistake will play out badly sooner or later.

Bugs are rarely ever “happy little accidents” like in Bob Ross paintings. They range from apparently random program crashes to worldwide Heartbleed scale disaster. Slowly the world is waking up, it has become apparent that eventually all software is flawed. Consequently, using the internet is as big a risk as a running across a highway, naked.

Programming is hugely underestimated. Programming is hard. It is really, really hard. The human mind was not made for this job. It is incredible that we have computers at all. Steve Jobs said that everyone should learn to program, because it teaches you to think. I agree a little, but it's like saying “Everyone can paint!” Or, “everyone should learn how to build a house.”

The best way to keep out bugs? Keep It Simple, Stupid. Simplify, simplify. Write code in a clear, consistent style. Add useful comments. Sometimes comment on why you did it this way. Test, test, test. Eat your own dog food.

There is a nice test to see if you've written good code. Open up a source file that you wrote six months ago. If you can't make out how it works within five minutes, then it's just not good enough. Now open up a source code that someone else wrote.

Boom. Oh, the horror.

Note that even this test is flawed, because you might not be seeing what you think you are seeing. Think about it. Programming is hard. And so is building houses.

In exceptional conditions

On the topic of condition variables, wikipedia says: “In concurrent programming, a monitor is a synchronization construct that allows threads to have both mutual exclusion and the ability to wait (block) for a certain condition to become true.”

This is commonly used in producer/consumer-like problems.

The code for programming such a monitor looks like this:

lock(&mutex);
while (!predicate) {
    condition_wait(&mutex);
}
consume(&item);
unlock(&mutex);

This can be implemented, for example, with pthread_mutex_lock() and pthread_cond_wait(). If you would wrap things into a class, it might look like the following:

mutex.lock();
while (!predicate) {
    cond.wait(mutex);
}
consume(item);
mutex.unlock();

However, in a language that uses exceptions, this code would be flawed. Suppose that consume() (or even cond.wait(), however unlikely) throws an exception, then the mutex remains locked. Therefore, in a language like Java, you should write:

mutex.lock();
try {
    while (!predicate) {
        cond.wait(mutex);
    }
    consume(item);
} finally {
    mutex.unlock();
}

This ensures the mutex gets unlocked, even in case an exception occurs.
In Python, the mutex is part of the condition object. This simplifies things somewhat by hiding the mutex under the hood:

cond.acquire()
try:
    while not predicate:
        cond.wait()
    consume(item)
finally:
    cond.release()

C++ does not have a finally keyword — it doesn't need it, because it has RAII (“Resource Acquisition Is Initialisation”). In C++, it should be coded as:

std::mutex mx;
{
    std::unique_lock<std::mutex> lk(mx);
    while (!predicate) {
        cond.wait(lk);
    }
    consume(item);
}

Note that the destructor of std::unique_lock will ensure that the mutex is unlocked. The destructor will run when the stack unwinds, which is also done when an exception occurs.

Although entirely correct, and arguably the most elegant solution, I have to say I find the C++ code unintuitive and difficult to follow. The reason is that there are no explicit statements in this code block for dealing with locking; the mutex is locked by the constructor and unlocked by the destructor, that run automagically. It is almost as if the mutex variable isn't doing anything at all. Annoyingly, this code can not be refactored to resemble the other code fragments, because of RAII. And you are being forced to write it this way now that threads, mutexes, and condition variables are part of the std library; there is no other way in which a condition variable can be used because the other way would be incorrect in C++.
This is one of those cases where C++ shines but somehow it doesn't feel gratifying.

Python multiprocessing: I did it my way

Python has a wonderful multiprocessing module for parallel execution. It's well known that Python's threading module does not do real multithreading due to the Global Interpreter Lock (GIL). Hence multiprocessing, which forks a pool of workers so you can efficiently run tasks in parallel.

In principle:

    p = multiprocessing.Process(target=func)
    p.start()
    p.join()

In practice, I have had weird issues with multiprocessing. Two notable issues:

• program does not start any processes at all, and just exits
• program raises an exception in QueueFeederThread at exit

Apparently the multiprocessing module has its problems, and although there are fixes in the most recent versions, I can't rely on that when the users of my program are running on operating systems that don't always include the latest greatest Python.

So, a decision had to be made. I ditched the multiprocessing module and replaced it with my own, that calls os.fork(). The resulting code is much easier to handle than with  multiprocessing, too.

Note that os.fork() does not port to Microsoft Windows. My target platform is UNIX anyway.

Pseudo-code:

    parallel(func, work_array):
        for i < numproc:
            fork()
            if child:
                work_items = part_of(work_array)
                for item in work_items:
                    func(item)

                # child exits
                exit(0)

        wait_for_child_procs()

So, a pool of numproc workers is spawned. Each of the child processes do their part of the work given in work_array. No communication is needed here because fork() causes the child to be a copy of the parent process, and thus getting a copy of work_array.

This is the simplest kind of parallel programming in UNIX. Surprisingly, this bit of code works better for me than the multiprocessing module — which supposedly does the same thing, under the hood.

Having decent language support for parallel programming is of utmost importance in today's world, where having a quadcore CPU is no exception; practically all modern computers are multi-core machines. A modern programming language should offer some easy mechanisms to empower the programmer, enabling you to take advantage of the hardware at hand.

Proper multithreading is exceptionally hard (if not impossible) to do for interpreted languages. This is a fact of life. Using a forking model, you can still get good parallellism.