Coding Day » .Net

Random number Generation with RNG

Can — Thu, 01 Apr 2010 10:38:56 +0000

I would like to mention a question that posed on stackoverflow a year back, with a slight change.

Given a function which produces a random integer in the range 0 to 4, write a function which produces a random integer in the range 0 to 6.

Simply, we have let ran5() = new Random().Next(5), we would like to write ran7 using ran5, and without any other random number generator
The answers are rather interesting for this question, as it is important to have a uniformly distributed random number generator.
The first attempt, is a naive way to generate the random number by defining the sample set small.

//random number generator with 6 numbers
let ran = new Random()
let random5() = ran.Next(5)
 
//naive
let rantbl = [| 0 .. 6 |]
 
let rec random7() = let i,j = random5(), random5()                   
                    let sum = i + j
                    if  sum < rantbl.Length then rantbl.[sum]
                    else random7()

Now, if you run the sample, it will generate the random numbers to 7 just fine, but it won’t be uniformly distributed, and will converge to a number as we run infinite times.
However, if we define our initial set uniformly to match out number generator, in this case we can generate 5 numbers, so a set of 25, we can hit each element by the multiplication of those two generators

//based on
let randomTable = [|for i in 0 .. 24 do if i / 7<3 then yield i % 7
                                        else yield -1 |] 
let rec random7_2() = let i,j = random5(), random5()
                      let index = i  * 5 + j
                      if  randomTable.[index] <> -1 then randomTable.[index]
                      else random7_2()
 
let sampler sample = seq { for i in 0 .. 100000 do yield sample() }

val ran : Random
val random5 : unit -> int
val rantbl : int array
val it : int array = [|0; 1; 2; 3; 4; 5; 6|]
val randomTable : int array
val it : int array = [|0; 1; 2; 3; 4; 5; 6; 0; 1; 2; 3; 4; 5; 6; 0; 1; 2; 3; 4; 5; 6; -1; -1; -1;
-1|]
val random7 : unit -> int
val random7_2 : unit -> int
val sampler : (unit -> ‘a) -> seq<'a>

We can estimate of the value of a random variable and predict the error, which is proportional to the iterations. Since we have 2 sets, we can analyse and compare their statistical properties.

let variance sample N =  let mean = sample |> Seq.average_by (fun i-> float i)
                         (sample |> Seq.sum_by (fun i->(float i - mean) * (float i - mean) ))
                                                     / N
let standardDeviation varian= sqrt (varian)
 
let standardError sample =  let N = float (Seq.length sample)
                            let vari = variance sample N
                            let stDev = standardDeviation vari
                            stDev / (sqrt N)
 
let sample = sampler random7
standardError(sampler random7 )
standardError (sampler random7_2)
 
// uniform distribution
let errorSet mySet = let N = Seq.length mySet |> float
                     mySet |> Seq.count_by (fun e-> e)  |> Seq.map (fun (i,j) -> i, (float j)/N * 100.0) |> Map.of_seq
 
errorSet (sampler random7)
errorSet (sampler random7_2)

val variance : seq -> float -> float
val standardDeviation : float -> float
val standardError : seq -> float
val sample : seq
val errorSet : seq<'a> -> Map<'a,float>

Random Number Generation

Downloads: 61 File Name: random.fs

Happy Pi Day and Monte Carlo Method

Can — Sun, 14 Mar 2010 22:11:17 +0000

Pi Calculation

Today is the day, 03/14 2010. Happy day to all!

The Monte Carlo method generates multiple trials to determine the expected value of a random variable.

Calculating Pi is generally the hello world of Monte Carlo Method in Stochastic Calculus. So for today, I will try to give a sample calculation of pi as monte carlo in F#. There are very good articles on the Monte carlo and how the pi calculation is formulated.

Generally these are the steps to follow, before running the simulation

Define the input set
Generate randomly the input set
Filter the random set using some computations
Fold the result to the final result

Pi calculation formula :

open System
let calcPi() =
    let ran = new Random()
    let distances = seq { for i in 0 .. 100000 do
                            let x,y  = ran.NextDouble(), ran.NextDouble()
                            yield sqrt (x * x + y * y)
         }
 
    let count = distances |> Seq.filter (fun distance -> distance <= 1.0) 
                              |> Seq.length     
  4.0 * (float count) / (float (Seq.length distances))
 
let errorRate() = (1.0 - (calcPi() / Math.PI)) * 100.0
 
let print5 value = for i in 0 .. 5 do printf "%f\n" (value())
print5 (errorRate)

val calcPi : unit -> float
val errorRate : unit -> float
val print5 : (unit -> float) -> unit
Error rates:
-0.024698
-0.093453
-0.013239
-0.094726
-0.373563
0.308887
>

Pi Calculation

Downloads: 103 File Name: pi.fs

Book Review – Real World Functional Programming

Can — Mon, 08 Mar 2010 00:21:41 +0000

Real-World functional programming has been written by Tomas Petricek and Jon Skeet.
Tomasz’s and Jon’s book is built around how to think functionally when programming and how to make best use of functional paradigm for real world scenerios. The book is written mainly for the C# programmers who would like to switch/learn more of the functional world. It also features snippets with some technical background of the ideas involved.
As the title suggests the book is not focussed on one language, instead a mix of C# and F#, and it does a great job in bringing these worlds together,which is to me best of both worlds. Although, samples are all in functional style, when the problem needs the functional beauty to express, F# takes place. This gives any .NET developer to see when to use the necessary language for a particular problem.

The book starts slowly with the functional structures, and goes to monads, and to reactive libraries. I found the language of the book clear to understand, and easy to follow. The samples in the beginning of a chapter starts with some really simple constructs, but at the end of a chapter they become more complicated, and a nice programs to reflect the idea of the chapter.

Applied functional programming is probably the most sophisticated part of the book. Some ideas mainly inspired from research papers (cited at the end) blended with modern languages and libraries and applied somewhat differently. Especially composable functional libraries and reactive functional programs were insightful and open the mind with new possibilities.

Finally, I would recommend this book whoever wants to switch to functional programming and also learn new techniques in general. Each sample is crafted well, and represent real value with its tutorial writing style.

Pex, Unit Test but Better, Parameterized Unit Testing

Can — Sun, 07 Feb 2010 23:39:24 +0000

Finally I had a chance to play with the Pex (Program EXploration) , I heard it on lang.net for the very first time and I was quite impressed, but it took a while to explore it.

I downloaded the latest commercial evaluation package 0.22.50128.1 from Pex web site.

It is about 12 MB, it comes with different installation options with the support of VS 2008 and 2010. It installed without any problems on my VS2010 beta 2.

Here is what you get from the installation :

Pex command line (Visual Studio not required)
Pex Visual Studio Add-in, for Visual Studio 2008 and Visual Studio 2010 Beta2
Moles (which includes Stubs), lightweight test stubs and detours for .NET
API reference, tutorials and documentation
Samples

The documentation is really detailed, and the samples are worth investigating. I quite liked the Pex Tutorial document, which explains in detail what Pex is all about.

The good thing is it supports many different testing frameworks, including .NET’s own, nunit, xunit etc. By the way, I have to say VS2010 is awesome!

Pex is pretty much like a unit test but better in many ways. I’m quite surprised by the dynamic coverage window of a test method that you execute. This way, it is clear to see if the code is fully covered by the Pex test.

Pex is more like test method that takes inputs. In general, unit tests are not taking any inputs, they are isolated from each other and from external access, other than calling them parameterless.

Input generation in Pex based on the static analysis of the methods. It generates inputs such that, the code will be covered in full, however for some special types, the inputs might need to be explicitely told to test framework. The tools it brings to Visual Studio is extremely simple and easy to use. It does a very good job on incorporating those results.

Here is a very simple function that I generated my first Pex tests from

 public static int Compare(int i, string s)
        {
            if (i > 500)
            {
                i *= 2;
            }
            else
            {
                return Compare2(i);
            }
            return i * i;
        }

When you run Pex, it generates potential inputs to cover all the code. According to the documentation, it uses static analysis to figure the potential values. So it is not a random number generated test, or a brute force test, which is quite interesting. Pex can cover some obscure side of the code, which fully supports many Unicode string inputs or regular expressions.

However, when I converted this very simple code, from System.Int32 to System.Numeric.BigInteger (a new type in .NET 4), the test only generated for a null value. Unless I’m missing something, it doesn’t support all the types as yet, but it looks like a promising and valuable product to be added to your toolset

Poker : Programming Problem

Can — Tue, 24 Feb 2009 01:19:36 +0000

Although it has been a while since Brian posted the poker problem in his blog, I haven’t got the chance to look at it until I came across in ProjectEuler Problem 54. It is not the elegant or best solution at all, but just wanted to join the crew and can confirm it works with the problem’s 1000 games’ dataset.

Hope this helps.

#light
(* Problem 54 *)
type suit = |Spades |Hearts |Clubs |Diamonds 
type card=  |Ace = 14 |Two = 2  |Three =3  |Four = 4 |Five = 5 |Six = 6 
            |Seven =7  |Eight = 8 |Nine = 9 |Ten = 10 |Jack = 11 |Queen =12 |King  =13
type acard = (card * suit)  
 
let carder x:card= enum x
 
let card_value = function
    | 'A' -> card.Ace
    | 'K' -> card.King
    | 'Q' -> card.Queen
    | 'J' -> card.Jack  
    | 'T' -> card.Ten
    | c ->  carder(System.Int32.Parse(c.ToString()))
 
let suit_value = function
    | 'S' -> Spades
    | 'H' -> Hearts    
    | 'C' -> Clubs
    | 'D' -> Diamonds    
    | a -> invalid_arg (a.ToString())
 
let Create (str: string) :acard = (card_value str.[0],suit_value str.[1])
 
let isstraigh (mycards:acard list)  = 
    let mycards = List.sort_by (fun (a,b) -> a,b) mycards
    let rec isstr previouscard mycards   (straightlist : acard list) =
        if straightlist.Length >= 5 then straightlist
        else match mycards with  
                | cur :: rest -> if int (fst cur) = int ((fst previouscard)) + 1 then isstr cur rest  (cur::straightlist)
                                 else isstr cur rest  []                           
                | _ -> []
    let head = List.hd mycards
    isstr head (List.tl mycards) [head]
 
let pairl (mycards : acard list) groupfunction minelementCount =
                 mycards |> Seq.group_by groupfunction
                         |> Seq.filter (fun a -> Seq.length (snd a) >= minelementCount)
                         |> Seq.to_list
                         |> List.unzip
 
 
type Ranks =
    | Highest of card
    | Pair of  card
    | TwoPair of card*card
    | Three of card
    | Straight of card
    | Flush of card 
    | FullHouse of  card*card
    | Four of  card
    | StraightFlush of card
 
type Player = |One |Two  |Noone
 
let rank (mycards :  acard list) =     
    let traverseL  (l :'a list)  = if not l.IsEmpty then l  |> List.hd |> Seq.to_list
                                   else []                            
    let isflush mycards = snd (pairl mycards snd 5)  |> traverseL
 
    let FofF l = traverseL l |> List.hd |> fst
 
    let flush = isflush mycards
    let straight = isstraigh mycards
 
    let ispair count= pairl mycards fst count                                                             
 
    let four,fours = let f,s = ispair 4 
                     f,s|> traverseL
    let three,threes = ispair 3 
    let two,twos = ispair 2
 
    let maxcard c = fst (List.max c)
 
    if not flush.IsEmpty && not straight.IsEmpty then StraightFlush(maxcard flush),flush
    elif  not fours.IsEmpty then Four(four.Head),fours
    elif not threes.IsEmpty && not twos.IsEmpty && FofF twos <> FofF threes then FullHouse(two.Head, three.Head), List.append (threes |> traverseL) (traverseL twos)
    elif not flush.IsEmpty then Flush(maxcard flush),flush
    elif not straight.IsEmpty then Straight(maxcard straight),straight
    elif not three.IsEmpty then Three( three.Head), threes |> traverseL
    elif List.length two = 2 then TwoPair(two.Head,two.Tail.Head),Seq.append (twos.Head) (twos.Tail.Head) |> Seq.to_list
    elif not (twos |> traverseL).IsEmpty then Pair(two.Head), twos |> traverseL
    else Highest(maxcard mycards), [List.max mycards]
 
let play input =
    let convert (line: string) = let l = line.Split([|' '|])                                  
                                 [|[for j in 0 .. 4 do yield Create( l.[j])]; [for j in 5 .. 9 do yield Create( l.[j])]|]                                                                  
    let playercrds = convert input              
    let rec iswinner pcards=               
                let ranks,rankcards = pcards |> Array.map (rank)  |> Array.unzip
 
                let removecards (mainlist) (toberemoved)  =                         
                        mainlist |> Array.map2 (fun rem main-> main |> List.filter (fun c->
                                List.fold_left(fun ac x-> if x = c then ac && false else ac && true) true rem)) toberemoved
 
                if ranks.[0]>ranks.[1] then One
                elif ranks.[0]<ranks.[1] then Two
                else iswinner  (removecards  pcards rankcards) 
 
    iswinner playercrds
 
 
play  "5H 5C 6S 7S KD 2C 3S 8S 8D TD" 
play  "5D 8C 9S JS AC 2C 5C 7D 8S QH"
play  "2D 9C AS AH AC 3D 6D 7D TD QD"
play  "4D 6S 9H QH QC 3D 6D 7H QD QS"  // prob pair queens look at the highes
play  "2H 2D 4C 4D 4S 3C 3D 3S 9S 9D" 
 
play  "2H 2D 4C 4D 4S 2H 2D 4C 4D 4S" 
 
play  "TH 8D 6C 4D 3S TH 8D 6C 4D 4S" 
 
 
 
let rdinput =   use file = System.IO.File.OpenText("poker.txt")
                let p1count = ref 0  
 
                while not file.EndOfStream do
                 if play (file.ReadLine()) = One then p1count := !p1count+1
 
                file.Close()
                !p1count

Multiple Inheritance in C# using Dynamic Features

Can — Fri, 20 Feb 2009 16:03:30 +0000

Multiple Inheritance is a feature that a class that can inherit more than one class. Although CLR does not really care about it, it is not possible in the mainstream .NET languages to have multiple inheritance.

It is arguable that why we would ever need multiple inheritance. I think it is the same paradox as having dynamic types in a statically typed language. I am not going into that debate as it is already decided for almost a decade. This is an attempt on how to achieve the same effect using dynamic types easily.

In C++, multiple inheritance can simply be expressed like this :

class Auto {
public:
  virtual void Go(int a) {}
  virtual ~Auto() {}
};
class Animal {
public:
  virtual void Go() {}
  virtual ~Animal() {}
};
class SampleClass : public Auto, public Animal{};
 
int main() {	
	SampleClass *a = new SampleClass(); 
	//a->Go(5); // Compiler error for ambigouity
	//a->Go("going"); // Compiler error for missing method
	Animal* an = dynamic_cast<Animal*>(a);
	an->Go(); 
 
	Auto* at = dynamic_cast<Auto*>(a);
	at->Go(7);
}

When C# introduced dynamic type system and shared type system with C# 4, I thought it should be possible to express multiple inheritance using the new syntax. More like a forced Javascript prototypal inheritance instead of classical inheritance. However it turned out that it was even easier than that. It is relying on the runtime to operate on the types underlying with dynamic conversions, simply reflection.

Obviously it is not forced by the compiler, and you wouldn’t get intellisense but it will behave like a multiple inherited object. To achieve that the dynamic features of C# 4 has been used.

The same implementation in C# using Minherit object that has been created for reflected multiple inheritance.
So in C#, it would look like this :

    public class Auto
    {
        public string Go(int a) {}            
    }
    public class Animal
    {
        public string Go() {}        
    }
    public class SampleClass : Minherit<Auto, Animal>
    {                        
        static void Main(string[] args)
        {
            dynamic sample = new SampleClass();            
            a.Go(4); // this will succeed at runtime and call the first Go(int)            
            a.Go("going"); // runtime error for missing method
            Animal an = (Animal)sample;
            Auto x = (Auto)sample;
 
            an.Go();
            x.Go(7);            
        }
    }

It uses dynamic sample that has come with .NET Framework 4 CTP. I actually think that sample will go to mscorlib with the final release in the System namespace

As the code got a bit long, I post it as project zip file if you are interested.

C# 4 Dynamic Multiple Inheritance

Downloads: 373 File Name: cs4trials.zip

It is possible to create multiple inheritance abstraction as shown below. Here are the selected four methods to display how the GetMember and SetMember is implemented.

  public override object Call(CallAction action, params object[] args)
        {
            var objMethod = GetMember<MethodInfo>(this, action.Name).Concat(
                               GetMember<MethodInfo>(base1, action.Name)).Concat(
                                   GetMember<MethodInfo>(base2, action.Name));                                                     
 
            object result = null;
            bool executionSuccess = false;
            foreach (var item in objMethod)
            {
                try
                {                     
                    result = item.Second.Invoke(item.First, args);
                    executionSuccess = true;
                    break;
                }
                catch 
                {                    
                }
            }
            if (executionSuccess)
            {
                return result;
            }
            else
            {
                throw new MissingMethodException(action.Name);
            }                    
        }
 private IEnumerable<Pair<object, T>> GetMember<T>(object obj, string name)
        {
            return from member in obj.GetType().GetMember(name).OfType<T>()
                   select new Pair<object, T>(obj, member);
        }
        public override object GetMember(GetMemberAction action)
        {
            var property = GetMember<PropertyInfo>(this, action.Name).Concat(
                                     GetMember<PropertyInfo>(base1, action.Name)).Concat(
                                         GetMember<PropertyInfo>(base2, action.Name)).FirstOrDefault();
            var field = GetMember<FieldInfo>(this, action.Name).Concat(
                            GetMember<FieldInfo>(base1, action.Name)).Concat(
                                GetMember<FieldInfo>(base2, action.Name)).FirstOrDefault();
            if (property != null)
            {
                return property.Second.GetValue(property.First, null);
            }
            else if (field != null)
            {
                return field.Second.GetValue(field.First);
            }
            else
            {
                throw new MissingMemberException(action.Name);
            }
 
        }
        public override void SetMember(SetMemberAction action, object value)
        {
            var properties = GetMember<PropertyInfo>(this, action.Name).Concat(
                                    GetMember<PropertyInfo>(base1, action.Name)).Concat(
                                        GetMember<PropertyInfo>(base2, action.Name));
            var fields = GetMember<FieldInfo>(this, action.Name).Concat(
                            GetMember<FieldInfo>(base1, action.Name)).Concat(
                                GetMember<FieldInfo>(base2, action.Name));
 
            bool issucceess = false;
            foreach (var property in properties)
            {
                try
                {
                    property.Second.SetValue(property.First, value, null);
                    issucceess = true;
                    break;
                }
                catch { }             
            }
 
            if (!issucceess)
            {
                foreach (var field in fields)
                {
                    try
                    {
                        field.Second.SetValue(field.First, value);
                        issucceess = true;
                        break;
                    }
                    catch { }
                }
            }
            if (!issucceess)
            {
                throw new MissingMemberException(action.Name);
            }
        }

Next Generation Cryptography (CNG) for .NET

Can — Sun, 20 Jul 2008 00:23:33 +0000

Last year, Windows Kernel team released a new cryptography libraries called Next Generation Cryptography (CNG). Although it is only available on Windows Vista and Windows Server 2008 kernels so far, I think it is going to replace the CAPICOM with its powerful new implementations.

I quite liked the new CNG libraries as it gets over some limitations of the previous library.

CLR security team released a new project on codeplex called CLR Security for that new cryptography library on .NET. So it is possible to benefit from those libraries in managed worlds as well. For a detailed description be sure to check security blog.

Also, the team is very quick to respond and provide fixes if necessary.

String Stream for .NET

Can — Wed, 09 Apr 2008 01:11:17 +0000

I was working with C++ Standard Template Library (STL) library and when using the StringStream class I felt like we need this for .NET Yes we have a great class StringBuilder but we can’t use it as a Stream (because it is not).

Most of the cryptographic classes works with the stream object. I wanted to save the encrypted information as text. Ideally we could use memoryStream but it is not really efficient, because it is bits and we need to build a model for serializing/deserializing as text.

String Stream class (as I call it) is just an ugly but functional wrapper around StringBuilder class as shown some snippets below. I just wanted it to use with encryption and decryption a string. So only the functions needed is there which is enough to use. But I doubt it leaks a little, so use with care if needed.

public class StringStream : Stream
{
        private StringBuilder strBuilder;
        public StringStream()
        {            strBuilder = new StringBuilder();        }
 
        public StringStream(string str)
        {            strBuilder = new StringBuilder(str);        }
 
        public override int Read(byte[] buffer, int offset, int count)
        {
            int howMuchRead = 0;
            for (int i = offset; i &lt; count; i++)
            {
                var actualIndex = i * 2;                
                howMuchRead = i+1;
                if (actualIndex &gt;= strBuilder.Length) 
                {
                    howMuchRead = count;
                    break;
                }                                   
                string s = strBuilder[actualIndex].ToString() + strBuilder[actualIndex + 1].ToString();
                buffer[i] = Convert.ToByte(s,16);
            }
            return howMuchRead;
        }
      public override void Write(byte[] buffer, int offset, int count)
        {            
            for (int i = offset; i &lt; count; i++)
            {
                strBuilder.Append(buffer[i].ToString("x2"));
            }
        }
}

StringStream

Downloads: 257 File Name: StringStream.cs

The main magic is happening on reading and writing. each byte is represented as 2 characters in the string. It is using the helper function Convert.ToByte to make it happen.
In the next post I will use it with a symmetric encryption helper class.

Visual Studio Dark Color Scheme

Can — Sun, 09 Mar 2008 18:26:49 +0000

After reading interesting posts about the fonts and colors in Visual Studio, I gave a try to some of them. Because I work quite a lot with Visual Studio, I realized that it is actually very important for the eyes and for productivity. Unfortunately none of the themes have pleased my IDE concept, low contrast, dark background, pale colours, clear type fonts, and follow metaphors.

I also put some time to design and experiment with some fonts and colours. The themes that I found were to dark for me with very high contrast, and the fonts were glowing.

This theme is still dark but with pale colours. More importantly it follows the metaphors we used to, so it will be OK when discussing with developers. What I mean? You know greenish parts of your code are always comments, or red parts of SQL code is bad (strings for dynamic SQL), as we all used to from visual studio, SQL or any popular IDEs.

One more thing is that when working with F# some of the color features are not the same as C#. So this theme is prepared with the F# projects in mind as well. Also HTML, CSS, JavaScript, and XAML are formatted following my colour preference.

For fonts I have used DejaVu font with size 10. If Consolas is your favourite font than, it would be better to put on size 9.

Moreover if you use, ReSharper code refactoring tool it has also some colour highlights for it as well. But it is not a problem if you don’t.

Dark Color Scheme for Visual Studio 2010

Downloads: 59 File Name: Dark_IDE_2010.zip

Friendly IDE for Visual Studio 2008

Downloads: 1608 File Name: Friendly_IDE_2008.zip

Friendly IDE for Visual Studio 2005

Downloads: 572 File Name: Friendly_IDE_2005.zip

C# – F# Scheme

ASPX – HTML – XML Scheme

LINQ Expression Trees-Lambdas to CodeDom Conversion

Can — Mon, 25 Feb 2008 00:23:34 +0000

Introduction

Some people are working to make the meta-programming possible. Some says as language oriented programming or domain specific language, but I prefer in general as meta-programming. For years programming languages supported to generate code with the powerful libraries or developers worked just with string concatenations and external linkers.

Nowadays meta-programming is getting more and more important as the domain expertise required. So the languages make meta programming possible at the compiler level with compiler directives.

Indeed there a lot of ideas coming from functional programming world where everything treated as expressions. The code becomes data and data usage happens in the code. It should sound familiar with LINQ to SQL efforts to make this possible.

Libraries

.NET Framework had code generators since the beginning. CodeDom is probably the best known for tree based code generation. Codedom made possible to develop the ASP.NET engine, Windows Form designer, Web form designer, Web services wrapper LINQ entity objects and more. It is used extensively by the framework for the key technologies.

Although there are other APIs in .NET framework such as System.Reflection, System.Reflection.Emit, in this post we will focus on CodeDom and the new comer Expression Trees.

Expression Tree is the key API behind LINQ to SQL or IQueryable interface in general. Every query is expressed as typed trees that is parsed and converted to SQL later by the library.

The syntax of expressing queries is very readable with query comprehension syntax. However sometimes I want to know about the generated tree, like actually which functions are getting involved in the query. I have used Expression Tree Debugger Visualizer to draw the tree. It is pretty handy tool but for big trees it is difficult to see what is going on. This was my main motivation actually, although we had the code, we don’t see what’s the magic going on with query comprehension.

Implementation

So the idea is to have the code regenerated from the tree. In the real world this will involve a parser, interpreter and some more compiler theory which requires a lot of research. And because this is just for fun and since we have a powerful CodeDom library to generate code, I tried to convert the expression tree to CodeDom tree. Than used the CodeDom to generate code in any language. Finally I wrote the extension methods so that the debuggers and my code can use it directly from the type.

The compiler generates automatically the expression trees if we use the proper syntax. So from the beginning we have the tree. In order to convert to CodeDom objects, we need to traverse the tree and generate the necessary CodeDom objects. So I wrote a tree walker that generates a CodeDom object to is parent while going to the last children. I didn’t realise how far it is going but that was it. When the tree walker finished with some more few lines of code the converter was just working.

I would like to put the code here as well but unfortunately it is too long for a blog post, so here are some snippets. Feel free to provide suggestions or bug reports.

LINQ Expression Visitor that generates CodeDom Trees

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
 
using System.Collections.ObjectModel;
using System.Linq.Expressions;
using System.Reflection;
using System.CodeDom;
using System.CodeDom.Compiler;
using System.IO;
 
namespace ExpressionToCodedom
{
    public  class CodeDomExpressionVisitor
    {
 
        Expression m_exp;
        Dictionary<string, CodeTypeMember> m_members;
 
        public CodeDomExpressionVisitor(Expression e)
        {
            m_exp = e;
        }
        internal string GenerateSource(CodeDomProvider codeProvider)
        {
            StringBuilder sb = new StringBuilder();
            TextWriter tWriter = new IndentedTextWriter(new StringWriter(sb));
            CodeCompileUnit ccu = GenerateCode();
            codeProvider.GenerateCodeFromCompileUnit(ccu, tWriter, new CodeGeneratorOptions());
            codeProvider.Dispose();
 
            tWriter.Close();
 
            return sb.ToString();
        }
 
        internal string GenerateSource(string language)
        {
 
 
            CodeDomProvider codeProvider=null;
            if (language == "cs")
                codeProvider = new Microsoft.CSharp.CSharpCodeProvider();
            else if (language == "vb")
                codeProvider = new Microsoft.VisualBasic.VBCodeProvider();
            else
            {                
 
                    throw new Exception("make sure you are trying to load a CodeDomProvider assembly");
 
            }
            return GenerateSource(codeProvider); 
 
        }
 
        public string GenerateSource()
        {
            return GenerateSource("cs"); 
        }
 
 
        private CodeCompileUnit GenerateCode()
        {
            var code = new CodeCompileUnit();
            m_members = new Dictionary<string, CodeTypeMember>();
 
            var LambdaTypeClass = new CodeTypeDeclaration("LambdaExpression");
            var ns = new CodeNamespace("Runtime");
 
            ns.Types.Add(LambdaTypeClass);            
            ns.Imports.Add(new CodeNamespaceImport("System"));
            // add more types in case I want to compile
 
            code.Namespaces.Add(ns);
 
            CodeObject cEvaluationResult = Visit(m_exp);
 
            var constructor = new CodeConstructor();
 
            if (cEvaluationResult is CodeStatement)
                constructor.Statements.Add(cEvaluationResult as CodeStatement);
 
            else if (cEvaluationResult is CodeExpression)
                constructor.Statements.Add(cEvaluationResult as CodeExpression);
 
            LambdaTypeClass.Members.Add(constructor);
 
 
            foreach (var item in m_members)
            {
                LambdaTypeClass.Members.Add(item.Value);
            }
 
            return code;
 
        }
 
        protected virtual CodeObject Visit(Expression exp)
        {
            if (exp == null)
                return null;
            switch (exp.NodeType)
            {
                case ExpressionType.Negate:
                case ExpressionType.NegateChecked:
                case ExpressionType.Not:
                case ExpressionType.Convert:
                case ExpressionType.ConvertChecked:
                case ExpressionType.ArrayLength:
                case ExpressionType.Quote:
                case ExpressionType.TypeAs:
                    return this.VisitUnary((UnaryExpression)exp);
                case ExpressionType.Add:
                case ExpressionType.AddChecked:
                case ExpressionType.Subtract:
                case ExpressionType.SubtractChecked:
                case ExpressionType.Multiply:
                case ExpressionType.MultiplyChecked:
                case ExpressionType.Divide:
                case ExpressionType.Modulo:
                case ExpressionType.And:
                case ExpressionType.AndAlso:
                case ExpressionType.Or:
                case ExpressionType.OrElse:
                case ExpressionType.LessThan:
                case ExpressionType.LessThanOrEqual:
                case ExpressionType.GreaterThan:
                case ExpressionType.GreaterThanOrEqual:
                case ExpressionType.Equal:
                case ExpressionType.NotEqual:
                case ExpressionType.Coalesce:
                case ExpressionType.ArrayIndex:
                case ExpressionType.RightShift:
                case ExpressionType.LeftShift:
                case ExpressionType.ExclusiveOr:
                    return this.VisitBinary((BinaryExpression)exp);
                case ExpressionType.TypeIs:
                    return this.VisitTypeIs((TypeBinaryExpression)exp);
                case ExpressionType.Conditional:
                    return this.VisitConditional((ConditionalExpression)exp);
                case ExpressionType.Constant:
                    return this.VisitConstant((ConstantExpression)exp);
                case ExpressionType.Parameter:
                    return this.VisitParameter((ParameterExpression)exp);
                case ExpressionType.MemberAccess:
                    return this.VisitMemberAccess((MemberExpression)exp);
                case ExpressionType.Call:
                    return this.VisitMethodCall((MethodCallExpression)exp);
                case ExpressionType.Lambda:
                    return this.VisitLambda((LambdaExpression)exp);
                case ExpressionType.New:
                    return this.VisitNew((NewExpression)exp);
                case ExpressionType.NewArrayInit:
                case ExpressionType.NewArrayBounds:
                    return this.VisitNewArray((NewArrayExpression)exp);
                case ExpressionType.Invoke:
                    return this.VisitInvocation((InvocationExpression)exp);
                case ExpressionType.MemberInit:
                    return this.VisitMemberInit((MemberInitExpression)exp);
                case ExpressionType.ListInit:
                    return this.VisitListInit((ListInitExpression)exp);
                default:
                    throw new Exception(string.Format("Unhandled expression type: '{0}'", exp.NodeType));
            }
        }
 
        protected virtual CodeObject VisitBinding(MemberBinding binding)
        {            
            switch (binding.BindingType)
            {
                case MemberBindingType.Assignment:
                    return this.VisitMemberAssignment((MemberAssignment)binding);
                case MemberBindingType.MemberBinding:
                    return this.VisitMemberMemberBinding((MemberMemberBinding)binding);
                case MemberBindingType.ListBinding:
                    return this.VisitMemberListBinding((MemberListBinding)binding);
                default:
                    throw new Exception(string.Format("Unhandled binding type '{0}'", binding.BindingType));
            }
        }
 
        protected virtual CodeExpression VisitElementInitializer(ElementInit initializer)
        {            
            ReadOnlyCollection<CodeExpression> arguments = this.VisitExpressionList(initializer.Arguments);
 
            return new CodeMethodInvokeExpression(new CodeMethodReferenceExpression(new CodeThisReferenceExpression(),initializer.AddMethod.Name), arguments.ToArray());                               
        }
 
        protected virtual CodeObject VisitUnary(UnaryExpression u)
        {
            CodeObject operand = this.Visit(u.Operand);
 
            return operand;
        }
 
        private CodeBinaryOperatorType BindOperant(ExpressionType e)
        {
            switch (e)
            {
                case ExpressionType.Add:
                case ExpressionType.AddChecked:
                    return CodeBinaryOperatorType.Add;
 
                case ExpressionType.And:
                    return CodeBinaryOperatorType.BitwiseAnd;
 
                case ExpressionType.AndAlso:
                    return CodeBinaryOperatorType.BooleanAnd;                  
 
                case ExpressionType.Or:
                    return CodeBinaryOperatorType.BitwiseOr;                    
 
                case ExpressionType.OrElse:
                    return CodeBinaryOperatorType.BooleanOr;                    
 
                case ExpressionType.ExclusiveOr:
                case ExpressionType.ArrayIndex:
                case ExpressionType.Coalesce:
                case ExpressionType.RightShift:
                case ExpressionType.LeftShift:
                    throw new NotSupportedException("no direct equivalent in codedom,so workarounds not implemented");
 
                case ExpressionType.Equal:
                    return CodeBinaryOperatorType.IdentityEquality;
 
                case ExpressionType.NotEqual:
                    return CodeBinaryOperatorType.IdentityInequality;                    
 
                case ExpressionType.GreaterThan:
                    return CodeBinaryOperatorType.GreaterThan;                    
 
                case ExpressionType.GreaterThanOrEqual:
                    return CodeBinaryOperatorType.GreaterThanOrEqual;                    
 
                case ExpressionType.LessThan:
                    return CodeBinaryOperatorType.LessThan;                    
 
                case ExpressionType.LessThanOrEqual:
                    return CodeBinaryOperatorType.LessThanOrEqual;                    
 
                case ExpressionType.Multiply:
                case ExpressionType.MultiplyChecked:
                    return CodeBinaryOperatorType.Multiply;
 
                case ExpressionType.Subtract:
                case ExpressionType.SubtractChecked:
                    return CodeBinaryOperatorType.Subtract;
 
                case ExpressionType.Power:
                case ExpressionType.Divide:
                    return CodeBinaryOperatorType.Divide;
 
                case ExpressionType.Modulo:
                    return CodeBinaryOperatorType.Modulus;
 
                default:
                    throw new Exception("are you sure you are right?");
            }
        }
 
        protected virtual CodeBinaryOperatorExpression VisitBinary(BinaryExpression b)
        {
            var left = this.Visit(b.Left) as CodeExpression;
            var right = this.Visit(b.Right) as CodeExpression;
            CodeObject conversion = this.Visit(b.Conversion);
 
            CodeBinaryOperatorType operant = BindOperant(b.NodeType);           
            var condExpr = new CodeBinaryOperatorExpression(left, operant, right);
            return condExpr;
        }
 
        protected virtual CodeObject VisitTypeIs(TypeBinaryExpression b)
        {            
            CodeObject expr = this.Visit(b.Expression);          
            return expr;
        }
 
        protected virtual CodeExpression VisitConstant(ConstantExpression c)
        {
            if (c.Value == null)
            {
                return new CodePrimitiveExpression(null);
            }
            else if (c.Value.GetType().IsValueType || c.Value.GetType() == typeof(string))
            {
                   return new CodePrimitiveExpression(c.Value);
            }
            else
            {
                return new CodeVariableReferenceExpression(c.Value.ToString());             
            }                        
        }
 
        protected virtual CodeObject VisitConditional(ConditionalExpression c)
        {            
            CodeObject test = this.Visit(c.Test);
            CodeExpression ifTrue = this.Visit(c.IfTrue) as CodeExpression;
            CodeExpression ifFalse = this.Visit(c.IfFalse) as CodeExpression;
 
            var ifStatement = new CodeConditionStatement(test as CodeExpression,
                                                         new CodeStatement[] {new CodeExpressionStatement(ifTrue) }, 
                                                         new CodeStatement[] {new CodeExpressionStatement(ifFalse) });                    
            return ifStatement;
        }
 
        protected virtual CodeObject VisitParameter(ParameterExpression p)
        {
            return new CodeArgumentReferenceExpression(p.Name);            
        }
 
        protected virtual CodeObject VisitMemberAccess(MemberExpression m)
        {
 
            CodeObject exp = this.Visit(m.Expression);
 
            if (exp is CodePrimitiveExpression)
            {
                return exp;
            }
            else
            {
                Type memType;
                if (m.Member.MemberType == MemberTypes.Field)
                    memType = (m.Member as FieldInfo).FieldType;
                else memType = (m.Member as PropertyInfo).PropertyType;
 
 
                m_members[m.Member.Name] = new CodeMemberField(memType, m.Member.Name);
                return new CodeVariableReferenceExpression(m.Member.Name);
            }
        }    
 
        protected virtual CodeObject VisitMethodCall(MethodCallExpression m)
        {           
            CodeObject obj = this.Visit(m.Object);
            IEnumerable<CodeExpression> args = this.VisitExpressionList(m.Arguments);
 
            if (obj == null)
            {  //static method call
                return new CodeMethodInvokeExpression(new CodeTypeReferenceExpression(m.Method.DeclaringType),m.Method.Name,args.ToArray());                
            }
            else
            {
                return new CodeMethodInvokeExpression(obj as CodeExpression, m.Method.Name, args.ToArray());
            }   
        }
 
        protected virtual ReadOnlyCollection<CodeExpression> VisitExpressionList(ReadOnlyCollection<Expression> original)
        {
            List<CodeExpression> list = new List<CodeExpression>();
            for (int i = 0, n = original.Count; i < n; i++)
            {
                CodeExpression p = (CodeExpression)this.Visit(original[i]);                
                    list.Add(p);                
            }            
            return list.AsReadOnly();
        }
 
        protected virtual CodeExpression VisitMemberAssignment(MemberAssignment assignment)
        {// thhose are properties
 
            CodeObject e = this.Visit(assignment.Expression);
            return e as CodeExpression;
 
        }
 
        protected virtual CodeObject VisitMemberMemberBinding(MemberMemberBinding binding)
        {
 
            IEnumerable<CodeExpression> bindings = this.VisitBindingList(binding.Bindings) as IEnumerable<CodeExpression>;
            return new CodeObjectCreateExpression(binding.Member.Name, bindings.ToArray());            
        }
 
        protected virtual CodeObject VisitMemberListBinding(MemberListBinding binding)
        {
 
            IEnumerable<CodeExpression> initializers = this.VisitElementInitializerList(binding.Initializers);
 
            return new CodeObjectCreateExpression(binding.Member.Name, initializers.ToArray());
 
        }
 
        protected virtual IEnumerable<CodeExpression> VisitBindingList(ReadOnlyCollection<MemberBinding> original)
        {
            List<CodeExpression> list = new List<CodeExpression>();
            for (int i = 0, n = original.Count; i < n; i++)
            {
                CodeExpression b = this.VisitBinding(original[i]) as CodeExpression;
 
                    list.Add(b);
 
            }
            return list;
        }
 
        protected virtual IEnumerable<CodeExpression> VisitElementInitializerList(ReadOnlyCollection<ElementInit> original)
        {
            List<CodeExpression> list = new List<CodeExpression>();
            for (int i = 0, n = original.Count; i < n; i++)
            {
                CodeExpression init = this.VisitElementInitializer(original[i]);
 
                list.Add(init);
 
            }
 
            return list;
        }
 
        protected CodeMethodReferenceExpression VisitLambda(LambdaExpression lambda)
        {
            var  body = this.Visit(lambda.Body);
            var lambdaMethod = new CodeMemberMethod();
 
            lambdaMethod.Name = lambda.Type.Name;
            if (lambdaMethod.Name.Contains("Func"))
                lambdaMethod.ReturnType = new CodeTypeReference(lambda.Body.Type);
 
            foreach (var item in lambda.Parameters)
            {
                lambdaMethod.Parameters.Add(new CodeParameterDeclarationExpression(item.Type, item.Name));
            }
 
            if (body is CodeExpression)
            {
                if (lambdaMethod.ReturnType.BaseType.Contains("Void"))
                    lambdaMethod.Statements.Add((body as CodeExpression ));
 
                else
                    lambdaMethod.Statements.Add(new CodeMethodReturnStatement(body as CodeExpression));
            }
            else if (body is CodeStatement)
            {
                    lambdaMethod.Statements.Add((body as CodeStatement));
            }
            else
            {
                throw new Exception("investigate...");
            }
 
            m_members[lambda.Type.FullName] = lambdaMethod;
            return new CodeMethodReferenceExpression(new CodeThisReferenceExpression(), lambdaMethod.Name) ;
        }
 
        protected virtual CodeObject VisitNew(NewExpression nex)
        {            
            IEnumerable<CodeExpression> args = this.VisitExpressionList(nex.Arguments);
 
 
            return new CodeObjectCreateExpression(nex.Type.Name,args.ToArray());
 
        }
 
        protected virtual CodeObject VisitMemberInit(MemberInitExpression init)
        {
            CodeObject n = this.VisitNew(init.NewExpression);
            CodeExpression[] bindings = this.VisitBindingList(init.Bindings).ToArray(); //binding will return property initialisation
 
 
            for (int i = 0; i < init.Bindings.Count; i++)            
            {
                                                                    // need to do something with that////
                var assignProperty = new CodeAssignStatement(new CodePropertyReferenceExpression(
                            n as CodeExpression, init.Bindings[i].Member.Name), bindings[i]);
            }                                   
 
            return n;
        }
 
        protected virtual CodeObject VisitListInit(ListInitExpression init)
        {
 
            CodeObject n = this.VisitNew(init.NewExpression);
            IEnumerable<CodeExpression> initializers = this.VisitElementInitializerList(init.Initializers);
 
            return n;
        }
 
        protected virtual CodeObject VisitNewArray(NewArrayExpression na)
        {
 
            IEnumerable<CodeExpression> exprs = this.VisitExpressionList(na.Expressions);
 
 
            return new CodeArrayCreateExpression(new CodeTypeReference(na.Type), exprs.ToArray());
        }
 
        protected virtual CodeObject VisitInvocation(InvocationExpression iv)
        {            
            IEnumerable<CodeExpression> args = this.VisitExpressionList(iv.Arguments);
 
            var expr = this.Visit(iv.Expression) as CodeExpression;
 
            return new CodeMethodInvokeExpression(new CodeMethodReferenceExpression(expr, "Method"), args.ToArray());            
        }      
    }
}

Expression Tree to CodeDom Source Code

Downloads: 227 File Name: Expression_Source.zip

Expression Tree to CodeDom DLL

Downloads: 126 File Name: ExpressionToCodedom.dll

Example

The extension methods enables to see the source code of any IQueryable and any Expression. Any of them have a GenerateSourceCodeMethod that gives back a string.

GenerateSourceCode(); // default C#

GenerateSourceCode(string language); // either cs or vb as input or Fully qualified name of the CodeDomProvider (like Microsoft.FSharp.Compiler.CodeDom.FSharpCodeProvider) It should be added as a reference to the project if you’re going to use it.

Sample program that manipulates the expression trees and usage of CodeDom Converter with “item.GetCodeDomSource(“vb”)”

int a = 3, c = 2, d = 0;
 
var e1 = Expression.Constant(5);
var e2 = Expression.And(e1, e1);
Expression<Func<string, Func<bool>>> e3 = tbool => () => a < b && 8 > d || c == d;
Expression<Func<bool>> e4 = () => b < 4;
Expression<Func<RecordName, bool>> e5 = rn => rn.LastName == "ALFKI";
Expression<Func<StringBuilder>> e6 = () => new StringBuilder { Capacity = 20 };
Expression<Func<string, string>> e7 = word => word == "hello" ? "yes" : "no";
 
 
foreach (var item in new Expression[] { e1, e2, e3, e4, e5, e6,e7 })
{
    Console.WriteLine(item.GetCodeDomSource("vb"));
}

Visual Basic Output

Namespace Runtime
 
    Public Class LambdaExpression
 
        Private LastName As String
 
        Private Sub New()
            MyBase.New
            Me.Func`2
        End Sub
 
        Private Function Func`2(ByVal rn As Demo.Program.RecordName) As Boolean
            Return (LastName Is "ALFKI")
        End Function
    End Class
End Namespace

C# Output

namespace Runtime {
    using System;
 
 
    public class LambdaExpression {
 
        private string LastName;
 
        private LambdaExpression() {
            this.Func`2;
        }
 
        private bool Func`2(Demo.Program.RecordName rn) {
            return (LastName == "ALFKI");
        }
    }
}

Conclusion

Codedom is too much C# centric, so it’s hard to make it available for every language. The difference between Code Statement and Code Expressions sometimes makes it hard to convert from expression trees.

On the on the other hand Expression trees are too much LINQ oriented. They are less powerful than CodeDom but more easy to express. In expression trees everything is an expression unlike CodeDom. Some constructs are missing from expression trees like the assignment, but we will probably see the improvements in the expression trees in the future. So it might not be a true DSL or language generator, but sure it is enough to get the most of the databases.

There are some other more powerful meta-programming tools and libraries. F# quotation library supports all the available full-set language features expressed as quotations. Dynamic Language Runtime is another expression tree like library focussed more on compiler developers.

Finally this library is not build for runtime code conversion from expression tree to CodeDom, although it is possible. The CodeDom generated code is mainly for debugging to print the source code of the query. It might also be helpful for seeing what is going on under the hood.