Let’s talk about bucket lists. Everybody has something that they want to do in life, some big or insignificant little thing that they want to achieve before they die old and ugly. Someone want to climb a mountain, someone want to dive deep into the sea. And I’m guessing a lot of peoples (me included) has one or more persons they would like to meet once in life. Now, one of the small things that I’ve always wanted (being a programmer and all) was for some of the code I write to be “recognized”. I generally write a lot of code (probably many thousand lines a year), and I publish most of it as open source code, so when someone uses something that I’ve made, or builds on it, it makes me really happy. So, when earlier this week none other than Miguel de Icaza over at xamarin added one of my classes to MonoTouch.Dialog (after making a few small changes to it) I felt really honored. It didn’t take him more than a day to remove it though, but it was only moved to another project, so I feel just as good about it xD. Anyways, the image below still makes me think that I can check off one of the points on my bucket list now. This doesn’t mean that I’ll stop doing open source, I’ll probably do even more, but I still feel like I’ve accomplished something.

Image may be NSFW.
Clik here to view.

Writing your own language

Another one of the things that I want to do is to write my own computer language. Now, why would I want to do something stupid like that you might think, but I don’t think it’s stupid. I think it’s awesome, and I learn a lot by doing so. I have actually written my own programming language. It’s called Totem and is a dynamic language that works a lot like javascript. The current version compiles down to CIL (common intermediate language) used by the CLR (common language runtime) to run, this means it runs on the .NET virtual machine and should also probably run just fine on mono (never tested). I use Irony to generate the AST (abstract syntax tree), and IKVM.Reflection to do the emitting of the actual CIL bytecode. There is support for arrays and hasmaps, loops and variables, and most things you need in a general programming-language, and samples such as the following one runs just fine:

var fib = () => {
    var prevs = [0,1];
    var fib = n => {
        if(n < prevs.length) return prevs[n];
        var ret = n < 2 ? n : fib(n-2) + fib(n-1);
        prevs.push(ret);
        return ret;
    };
    return fib;
}();

print(fib(100));

Now, after having reached pretty far with Totem I’ve started to see that there are certain parts of it that really isn’t made well, like the type-system, and I’ve always wanted to change that, however, that would in general require a rewrite of the entire Totem-compiler. And also, I would like to be able to run Totem-code on the fly hosted inside other processes (like a scripting-language). To solve these two I set my eyes on the DLR and started reading.

DLR

For those of you who don’t know what the DLR is, DLR stands for Dynamic Language Runtime, and it can be found here. So, what is the DLR? And how does it sit with regards to .NET 4.0 which has built-in support for dynamic? At first I looked around like a madman in the CLR-assemblies trying to find the classes I needed to build my own DLR language, but I couldn’t, and after which I came up on the following statement in the Discussion-tab of the dlr codeplex-page found here.

The facts: IronRuby and IronPython both use the DLR, in fact most of the DLR’s features are derived from earlier implementation of IronPython. I’m not familiar with how what version of the DLR IronScheme uses. A portion of the DLR (Microsoft.Scripting.Core.dll) was shipped in .NET 4.0 in System.Core.dll, and that portion is what C# 4.0 uses for it’s “dynamic” keyword support, which is why that assembly is not present in .NET 4.0 builds of the DLR. The rest of the DLR’s codebase (Microsoft.Scripting.dll, Microsoft.Dynamic.dll are the most important) are available for download from this CodePlex project, or the IronLanguages GitHub project. They provide the APIs for consumers and producers of hosting languages, while the APIs in .NET 4.0 are for actually doing dynamic dispatch of methods.

To write a dynamic language compiler for .NET 4.0, I’d use C# as the implementation language and use the DLR as a library for simplifying common compiler tasks. For a simple example language on the DLR, look at Sympl.

~Jimmy

Ok, now that that’s cleared out, let’s go download the stuff we need and find us some tutorials. Hitting up some quick tutorials on the internet I quickly discover that the trend is that the tutorials are written in 2008, and some of them are more then 5 years old. This seems to suggest that the DLR was rather hot for a period of time about 4-5 years ago, however, there are NO new tutorials at all. So I think to myself, “ok, fear not, it doesn’t matter if the content is old as long as it’s good”. Then, after reading some more on this vast internet I realize something. All these tutorials released around 2008 was released pre DLR 1.0, and the API has changed since then! After some more searching I find book that seems to be promising, and google books contains the first 2 (or so) chapters as a preview, so I can see if this book explains what I need to know. The book is called “Pro DLR in .NET 4“.

The book starts of nice by creating to simple class, one that extends LanguageContext and one that extends ScriptCode, both found in some Mircosoft.Scripting namespace that is part of the DLR. And after fallowing the tutorial and writing the code therein I end up with the following console window:
Image may be NSFW.
Clik here to view.

My reaction:
Image may be NSFW.
Clik here to view.

I mean, did you see that? I wrote like… 20-30 lines of code + some stupid XML config, and I have a interactive command-prompt in my face already. This thing is looking so good in no-time, and I can’t wait to start implementing my own language on top of this! This book is so awesome, it’s exactly what I need. The book then goes on to explain how the innards of the DLR work, how it’s based around Expressions that always returns something, and stuff like that. Which is interesting, intuitive and fairly easy to comprehend, but I’ve already read a lot about programming languages, so it didn’t come as a bomb that I recognized the Expression-model of the DLR. After explaining about expressions, it goes on to explain about binders which I knew little to nothing at all about. Binders provide an efficient way to do stuff with dynamic objects. There of-cause was a whole lot more to it (like caching and rules and stuff), however, if you want to learn all that go buy the book like I did. However, after some more reading it became apparent that the book has next to no examples on how to build an actual language on top of the DLR. True enough, I’ve learned a lot of stuff that I probably have to know when I do implement the language, but I still have next to no idea of where to start, and how I should implement my ScriptSource class.
Image may be NSFW.
Clik here to view.

So after a mild case of desk-flipping (and a good nights sleep) I start a new, freshly spirited and ready for new challenges. I recall the Sympl language talked about in the quote I posted above and figure I have to start reading sourcecode. However, there is also a 79 pages pdf following allong that I can read to get an understanding of how things work. Nice. However, after I start reading the PDF I quickly realize that the audience this politically correct and higly technical document is intended for is not one I consider myself to be in. The document feels like some sort of research document and goes about explaining a lot of things that I don’t really care about with as many words and bullet-lists that they could possible fit into 80 pages worth of text. Now, before I shoot myself in the leg, I must say that I haven’t read it all, and I’m probably going to do it by the time I post this blogpost (I’m planning to continue this post until I have something that’s working or give up), but still; when I see images like the one bellow I feel like I’m reading some sort of research-document on the DLR, and that’s not what I want to be doing. I want a hands-down example on how to implement a basic dynamic language on top of the DLR with examples in regards to how I should structure the project and where I should do what.
Image may be NSFW.
Clik here to view.

The tideturner

After another couple of hours going around in circles, I stumbled across a project called ASP Classic Compiler. The ASP Classic Compiler had done just what I needed, it had “bridged” a LanguageContext with the Sympl language (except that they didn’t use Sympl of-cause, but rather VBS). Now this was exciting news. Now I have a starting point, and I could finally start writing some working code. It took me about an hour to get the following working:
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.

I was actually planning to share my entire discoveries and how to get it all working, but I’m kind of tired, and this post has already gotten way longer than it was ever supposed to (and ugly and full of rainbow-colors), so I think I’m ending this post now, and uploading the code to github. The source-code (still only basic arithmetic working) can be found here: https://github.com/Alxandr/Totem-2.0/tree/fac78a860e1d100333ebabc249ab7c0cef62bfb5

Image may be NSFW.
Clik here to view.

Image may be NSFW.
Clik here to view.