Introduction

Some people are working to make the meta-programming possible. Some says as language oriented programming or domain specific language, but I prefer in general as meta-programming. For years programming languages supported to generate code with the powerful libraries or developers worked just with string concatenations and external linkers.

Nowadays meta-programming is getting more and more important as the domain expertise required. So the languages make meta programming possible at the compiler level with compiler directives.

Indeed there a lot of ideas coming from functional programming world where everything treated as expressions.  The code becomes data and data usage happens in the code. It should sound familiar with LINQ to SQL efforts to make this possible.

Libraries

.NET Framework had code generators since the beginning. CodeDom is probably the best known for tree based code generation. Codedom made possible to develop the ASP.NET engine, Windows Form designer, Web form designer, Web services wrapper LINQ entity objects and more. It is used extensively by the framework for the key technologies.

Although there are other APIs in .NET framework such as System.Reflection, System.Reflection.Emit, in this post we will focus on CodeDom and the new comer Expression Trees.

Expression Tree is the key API behind LINQ to SQL or IQueryable interface in general. Every query is expressed as typed trees that is parsed and converted to SQL later by the library.

The syntax of expressing queries is very readable with query comprehension syntax. However sometimes I want to know about the generated tree, like actually which functions are getting involved in the query. I have used Expression Tree Debugger Visualizer to draw the tree. It is pretty handy tool but for big trees it is difficult to see what is going on. This was my main motivation actually, although we had the code, we don’t see what’s the magic going on with query comprehension.

Implementation

So the idea is to have the code regenerated from the tree. In the real world this will involve a parser, interpreter and some more compiler theory which requires a lot of research. And because this is just for fun and since we have a powerful CodeDom library to generate code, I tried to convert the expression tree to CodeDom tree. Than used the CodeDom to generate code in any language. Finally I wrote the extension methods so that the debuggers and my code can use it directly from the type.

The compiler generates automatically the expression trees if we use the proper syntax. So from the beginning we have the tree. In order to convert to CodeDom objects, we need to traverse the tree and generate the necessary CodeDom objects. So I wrote a  tree walker that generates a CodeDom object to is parent while going to the last children. I didn’t realise how far it is going but that was it. When the tree walker finished with some more few lines of code the converter was just working.

I would like to put the code here as well but unfortunately it is too long for a blog post, so here are some snippets. Feel free to provide suggestions or bug reports.

LINQ Expression Visitor that generates CodeDom Trees

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
 
using System.Collections.ObjectModel;
using System.Linq.Expressions;
using System.Reflection;
using System.CodeDom;
using System.CodeDom.Compiler;
using System.IO;
 
namespace ExpressionToCodedom
{
    public  class CodeDomExpressionVisitor
    {
 
        Expression m_exp;
        Dictionary<string, CodeTypeMember> m_members;
 
        public CodeDomExpressionVisitor(Expression e)
        {
            m_exp = e;
        }
        internal string GenerateSource(CodeDomProvider codeProvider)
        {
            StringBuilder sb = new StringBuilder();
            TextWriter tWriter = new IndentedTextWriter(new StringWriter(sb));
            CodeCompileUnit ccu = GenerateCode();
            codeProvider.GenerateCodeFromCompileUnit(ccu, tWriter, new CodeGeneratorOptions());
            codeProvider.Dispose();
 
            tWriter.Close();
 
            return sb.ToString();
        }
 
        internal string GenerateSource(string language)
        {
 
 
            CodeDomProvider codeProvider=null;
            if (language == "cs")
                codeProvider = new Microsoft.CSharp.CSharpCodeProvider();
            else if (language == "vb")
                codeProvider = new Microsoft.VisualBasic.VBCodeProvider();
            else
            {                
 
                    throw new Exception("make sure you are trying to load a CodeDomProvider assembly");
 
            }
            return GenerateSource(codeProvider); 
 
        }
 
        public string GenerateSource()
        {
            return GenerateSource("cs"); 
        }
 
 
        private CodeCompileUnit GenerateCode()
        {
            var code = new CodeCompileUnit();
            m_members = new Dictionary<string, CodeTypeMember>();
 
            var LambdaTypeClass = new CodeTypeDeclaration("LambdaExpression");
            var ns = new CodeNamespace("Runtime");
 
            ns.Types.Add(LambdaTypeClass);            
            ns.Imports.Add(new CodeNamespaceImport("System"));
            // add more types in case I want to compile
 
            code.Namespaces.Add(ns);
 
            CodeObject cEvaluationResult = Visit(m_exp);
 
            var constructor = new CodeConstructor();
 
            if (cEvaluationResult is CodeStatement)
                constructor.Statements.Add(cEvaluationResult as CodeStatement);
 
            else if (cEvaluationResult is CodeExpression)
                constructor.Statements.Add(cEvaluationResult as CodeExpression);
 
            LambdaTypeClass.Members.Add(constructor);
 
 
            foreach (var item in m_members)
            {
                LambdaTypeClass.Members.Add(item.Value);
            }
 
            return code;
 
        }
 
        protected virtual CodeObject Visit(Expression exp)
        {
            if (exp == null)
                return null;
            switch (exp.NodeType)
            {
                case ExpressionType.Negate:
                case ExpressionType.NegateChecked:
                case ExpressionType.Not:
                case ExpressionType.Convert:
                case ExpressionType.ConvertChecked:
                case ExpressionType.ArrayLength:
                case ExpressionType.Quote:
                case ExpressionType.TypeAs:
                    return this.VisitUnary((UnaryExpression)exp);
                case ExpressionType.Add:
                case ExpressionType.AddChecked:
                case ExpressionType.Subtract:
                case ExpressionType.SubtractChecked:
                case ExpressionType.Multiply:
                case ExpressionType.MultiplyChecked:
                case ExpressionType.Divide:
                case ExpressionType.Modulo:
                case ExpressionType.And:
                case ExpressionType.AndAlso:
                case ExpressionType.Or:
                case ExpressionType.OrElse:
                case ExpressionType.LessThan:
                case ExpressionType.LessThanOrEqual:
                case ExpressionType.GreaterThan:
                case ExpressionType.GreaterThanOrEqual:
                case ExpressionType.Equal:
                case ExpressionType.NotEqual:
                case ExpressionType.Coalesce:
                case ExpressionType.ArrayIndex:
                case ExpressionType.RightShift:
                case ExpressionType.LeftShift:
                case ExpressionType.ExclusiveOr:
                    return this.VisitBinary((BinaryExpression)exp);
                case ExpressionType.TypeIs:
                    return this.VisitTypeIs((TypeBinaryExpression)exp);
                case ExpressionType.Conditional:
                    return this.VisitConditional((ConditionalExpression)exp);
                case ExpressionType.Constant:
                    return this.VisitConstant((ConstantExpression)exp);
                case ExpressionType.Parameter:
                    return this.VisitParameter((ParameterExpression)exp);
                case ExpressionType.MemberAccess:
                    return this.VisitMemberAccess((MemberExpression)exp);
                case ExpressionType.Call:
                    return this.VisitMethodCall((MethodCallExpression)exp);
                case ExpressionType.Lambda:
                    return this.VisitLambda((LambdaExpression)exp);
                case ExpressionType.New:
                    return this.VisitNew((NewExpression)exp);
                case ExpressionType.NewArrayInit:
                case ExpressionType.NewArrayBounds:
                    return this.VisitNewArray((NewArrayExpression)exp);
                case ExpressionType.Invoke:
                    return this.VisitInvocation((InvocationExpression)exp);
                case ExpressionType.MemberInit:
                    return this.VisitMemberInit((MemberInitExpression)exp);
                case ExpressionType.ListInit:
                    return this.VisitListInit((ListInitExpression)exp);
                default:
                    throw new Exception(string.Format("Unhandled expression type: '{0}'", exp.NodeType));
            }
        }
 
        protected virtual CodeObject VisitBinding(MemberBinding binding)
        {            
            switch (binding.BindingType)
            {
                case MemberBindingType.Assignment:
                    return this.VisitMemberAssignment((MemberAssignment)binding);
                case MemberBindingType.MemberBinding:
                    return this.VisitMemberMemberBinding((MemberMemberBinding)binding);
                case MemberBindingType.ListBinding:
                    return this.VisitMemberListBinding((MemberListBinding)binding);
                default:
                    throw new Exception(string.Format("Unhandled binding type '{0}'", binding.BindingType));
            }
        }
 
        protected virtual CodeExpression VisitElementInitializer(ElementInit initializer)
        {            
            ReadOnlyCollection<CodeExpression> arguments = this.VisitExpressionList(initializer.Arguments);
 
            return new CodeMethodInvokeExpression(new CodeMethodReferenceExpression(new CodeThisReferenceExpression(),initializer.AddMethod.Name), arguments.ToArray());                               
        }
 
        protected virtual CodeObject VisitUnary(UnaryExpression u)
        {
            CodeObject operand = this.Visit(u.Operand);
 
            return operand;
        }
 
        private CodeBinaryOperatorType BindOperant(ExpressionType e)
        {
            switch (e)
            {
                case ExpressionType.Add:
                case ExpressionType.AddChecked:
                    return CodeBinaryOperatorType.Add;
 
                case ExpressionType.And:
                    return CodeBinaryOperatorType.BitwiseAnd;
 
                case ExpressionType.AndAlso:
                    return CodeBinaryOperatorType.BooleanAnd;                  
 
                case ExpressionType.Or:
                    return CodeBinaryOperatorType.BitwiseOr;                    
 
                case ExpressionType.OrElse:
                    return CodeBinaryOperatorType.BooleanOr;                    
 
                case ExpressionType.ExclusiveOr:
                case ExpressionType.ArrayIndex:
                case ExpressionType.Coalesce:
                case ExpressionType.RightShift:
                case ExpressionType.LeftShift:
                    throw new NotSupportedException("no direct equivalent in codedom,so workarounds not implemented");
 
                case ExpressionType.Equal:
                    return CodeBinaryOperatorType.IdentityEquality;
 
                case ExpressionType.NotEqual:
                    return CodeBinaryOperatorType.IdentityInequality;                    
 
                case ExpressionType.GreaterThan:
                    return CodeBinaryOperatorType.GreaterThan;                    
 
                case ExpressionType.GreaterThanOrEqual:
                    return CodeBinaryOperatorType.GreaterThanOrEqual;                    
 
                case ExpressionType.LessThan:
                    return CodeBinaryOperatorType.LessThan;                    
 
                case ExpressionType.LessThanOrEqual:
                    return CodeBinaryOperatorType.LessThanOrEqual;                    
 
                case ExpressionType.Multiply:
                case ExpressionType.MultiplyChecked:
                    return CodeBinaryOperatorType.Multiply;
 
                case ExpressionType.Subtract:
                case ExpressionType.SubtractChecked:
                    return CodeBinaryOperatorType.Subtract;
 
                case ExpressionType.Power:
                case ExpressionType.Divide:
                    return CodeBinaryOperatorType.Divide;
 
                case ExpressionType.Modulo:
                    return CodeBinaryOperatorType.Modulus;
 
                default:
                    throw new Exception("are you sure you are right?");
            }
        }
 
        protected virtual CodeBinaryOperatorExpression VisitBinary(BinaryExpression b)
        {
            var left = this.Visit(b.Left) as CodeExpression;
            var right = this.Visit(b.Right) as CodeExpression;
            CodeObject conversion = this.Visit(b.Conversion);
 
            CodeBinaryOperatorType operant = BindOperant(b.NodeType);           
            var condExpr = new CodeBinaryOperatorExpression(left, operant, right);
            return condExpr;
        }
 
        protected virtual CodeObject VisitTypeIs(TypeBinaryExpression b)
        {            
            CodeObject expr = this.Visit(b.Expression);          
            return expr;
        }
 
        protected virtual CodeExpression VisitConstant(ConstantExpression c)
        {
            if (c.Value == null)
            {
                return new CodePrimitiveExpression(null);
            }
            else if (c.Value.GetType().IsValueType || c.Value.GetType() == typeof(string))
            {
                   return new CodePrimitiveExpression(c.Value);
            }
            else
            {
                return new CodeVariableReferenceExpression(c.Value.ToString());             
            }                        
        }
 
        protected virtual CodeObject VisitConditional(ConditionalExpression c)
        {            
            CodeObject test = this.Visit(c.Test);
            CodeExpression ifTrue = this.Visit(c.IfTrue) as CodeExpression;
            CodeExpression ifFalse = this.Visit(c.IfFalse) as CodeExpression;
 
            var ifStatement = new CodeConditionStatement(test as CodeExpression,
                                                         new CodeStatement[] {new CodeExpressionStatement(ifTrue) }, 
                                                         new CodeStatement[] {new CodeExpressionStatement(ifFalse) });                    
            return ifStatement;
        }
 
        protected virtual CodeObject VisitParameter(ParameterExpression p)
        {
            return new CodeArgumentReferenceExpression(p.Name);            
        }
 
        protected virtual CodeObject VisitMemberAccess(MemberExpression m)
        {
 
            CodeObject exp = this.Visit(m.Expression);
 
            if (exp is CodePrimitiveExpression)
            {
                return exp;
            }
            else
            {
                Type memType;
                if (m.Member.MemberType == MemberTypes.Field)
                    memType = (m.Member as FieldInfo).FieldType;
                else memType = (m.Member as PropertyInfo).PropertyType;
 
 
                m_members[m.Member.Name] = new CodeMemberField(memType, m.Member.Name);
                return new CodeVariableReferenceExpression(m.Member.Name);
            }
        }    
 
        protected virtual CodeObject VisitMethodCall(MethodCallExpression m)
        {           
            CodeObject obj = this.Visit(m.Object);
            IEnumerable<CodeExpression> args = this.VisitExpressionList(m.Arguments);
 
            if (obj == null)
            {  //static method call
                return new CodeMethodInvokeExpression(new CodeTypeReferenceExpression(m.Method.DeclaringType),m.Method.Name,args.ToArray());                
            }
            else
            {
                return new CodeMethodInvokeExpression(obj as CodeExpression, m.Method.Name, args.ToArray());
            }   
        }
 
        protected virtual ReadOnlyCollection<CodeExpression> VisitExpressionList(ReadOnlyCollection<Expression> original)
        {
            List<CodeExpression> list = new List<CodeExpression>();
            for (int i = 0, n = original.Count; i < n; i++)
            {
                CodeExpression p = (CodeExpression)this.Visit(original[i]);                
                    list.Add(p);                
            }            
            return list.AsReadOnly();
        }
 
        protected virtual CodeExpression VisitMemberAssignment(MemberAssignment assignment)
        {// thhose are properties
 
            CodeObject e = this.Visit(assignment.Expression);
            return e as CodeExpression;
 
        }
 
        protected virtual CodeObject VisitMemberMemberBinding(MemberMemberBinding binding)
        {
 
            IEnumerable<CodeExpression> bindings = this.VisitBindingList(binding.Bindings) as IEnumerable<CodeExpression>;
            return new CodeObjectCreateExpression(binding.Member.Name, bindings.ToArray());            
        }
 
        protected virtual CodeObject VisitMemberListBinding(MemberListBinding binding)
        {
 
            IEnumerable<CodeExpression> initializers = this.VisitElementInitializerList(binding.Initializers);
 
            return new CodeObjectCreateExpression(binding.Member.Name, initializers.ToArray());
 
        }
 
        protected virtual IEnumerable<CodeExpression> VisitBindingList(ReadOnlyCollection<MemberBinding> original)
        {
            List<CodeExpression> list = new List<CodeExpression>();
            for (int i = 0, n = original.Count; i < n; i++)
            {
                CodeExpression b = this.VisitBinding(original[i]) as CodeExpression;
 
                    list.Add(b);
 
            }
            return list;
        }
 
        protected virtual IEnumerable<CodeExpression> VisitElementInitializerList(ReadOnlyCollection<ElementInit> original)
        {
            List<CodeExpression> list = new List<CodeExpression>();
            for (int i = 0, n = original.Count; i < n; i++)
            {
                CodeExpression init = this.VisitElementInitializer(original[i]);
 
                list.Add(init);
 
            }
 
            return list;
        }
 
        protected CodeMethodReferenceExpression VisitLambda(LambdaExpression lambda)
        {
            var  body = this.Visit(lambda.Body);
            var lambdaMethod = new CodeMemberMethod();
 
            lambdaMethod.Name = lambda.Type.Name;
            if (lambdaMethod.Name.Contains("Func"))
                lambdaMethod.ReturnType = new CodeTypeReference(lambda.Body.Type);
 
            foreach (var item in lambda.Parameters)
            {
                lambdaMethod.Parameters.Add(new CodeParameterDeclarationExpression(item.Type, item.Name));
            }
 
            if (body is CodeExpression)
            {
                if (lambdaMethod.ReturnType.BaseType.Contains("Void"))
                    lambdaMethod.Statements.Add((body as CodeExpression ));
 
                else
                    lambdaMethod.Statements.Add(new CodeMethodReturnStatement(body as CodeExpression));
            }
            else if (body is CodeStatement)
            {
                    lambdaMethod.Statements.Add((body as CodeStatement));
            }
            else
            {
                throw new Exception("investigate...");
            }
 
            m_members[lambda.Type.FullName] = lambdaMethod;
            return new CodeMethodReferenceExpression(new CodeThisReferenceExpression(), lambdaMethod.Name) ;
        }
 
        protected virtual CodeObject VisitNew(NewExpression nex)
        {            
            IEnumerable<CodeExpression> args = this.VisitExpressionList(nex.Arguments);
 
 
            return new CodeObjectCreateExpression(nex.Type.Name,args.ToArray());
 
        }
 
        protected virtual CodeObject VisitMemberInit(MemberInitExpression init)
        {
            CodeObject n = this.VisitNew(init.NewExpression);
            CodeExpression[] bindings = this.VisitBindingList(init.Bindings).ToArray(); //binding will return property initialisation
 
 
            for (int i = 0; i < init.Bindings.Count; i++)            
            {
                                                                    // need to do something with that////
                var assignProperty = new CodeAssignStatement(new CodePropertyReferenceExpression(
                            n as CodeExpression, init.Bindings[i].Member.Name), bindings[i]);
            }                                   
 
            return n;
        }
 
        protected virtual CodeObject VisitListInit(ListInitExpression init)
        {
 
            CodeObject n = this.VisitNew(init.NewExpression);
            IEnumerable<CodeExpression> initializers = this.VisitElementInitializerList(init.Initializers);
 
            return n;
        }
 
        protected virtual CodeObject VisitNewArray(NewArrayExpression na)
        {
 
            IEnumerable<CodeExpression> exprs = this.VisitExpressionList(na.Expressions);
 
 
            return new CodeArrayCreateExpression(new CodeTypeReference(na.Type), exprs.ToArray());
        }
 
        protected virtual CodeObject VisitInvocation(InvocationExpression iv)
        {            
            IEnumerable<CodeExpression> args = this.VisitExpressionList(iv.Arguments);
 
            var expr = this.Visit(iv.Expression) as CodeExpression;
 
            return new CodeMethodInvokeExpression(new CodeMethodReferenceExpression(expr, "Method"), args.ToArray());            
        }      
    }
}

Expression Tree to CodeDom Source Code

Downloads: 482 File Name: Expression_Source.zip

Expression Tree to CodeDom DLL

Downloads: 298 File Name: ExpressionToCodedom.dll

Example

The extension methods enables to see the source code of any IQueryable and any Expression. Any of them have a GenerateSourceCodeMethod that gives back a string.

GenerateSourceCode(); // default C#

GenerateSourceCode(string language); // either cs or vb as input or  Fully qualified name of the CodeDomProvider (like Microsoft.FSharp.Compiler.CodeDom.FSharpCodeProvider) It should be added as a reference to the project if you’re going to use it.

Sample program that manipulates the expression trees and usage of CodeDom Converter with “item.GetCodeDomSource(“vb”)”

int a = 3, c = 2, d = 0;
 
var e1 = Expression.Constant(5);
var e2 = Expression.And(e1, e1);
Expression<Func<string, Func<bool>>> e3 = tbool => () => a < b && 8 > d || c == d;
Expression<Func<bool>> e4 = () => b < 4;
Expression<Func<RecordName, bool>> e5 = rn => rn.LastName == "ALFKI";
Expression<Func<StringBuilder>> e6 = () => new StringBuilder { Capacity = 20 };
Expression<Func<string, string>> e7 = word => word == "hello" ? "yes" : "no";
 
 
foreach (var item in new Expression[] { e1, e2, e3, e4, e5, e6,e7 })
{
    Console.WriteLine(item.GetCodeDomSource("vb"));
}

Visual Basic Output

Namespace Runtime
 
    Public Class LambdaExpression
 
        Private LastName As String
 
        Private Sub New()
            MyBase.New
            Me.Func`2
        End Sub
 
        Private Function Func`2(ByVal rn As Demo.Program.RecordName) As Boolean
            Return (LastName Is "ALFKI")
        End Function
    End Class
End Namespace

C# Output

namespace Runtime {
    using System;
 
 
    public class LambdaExpression {
 
        private string LastName;
 
        private LambdaExpression() {
            this.Func`2;
        }
 
        private bool Func`2(Demo.Program.RecordName rn) {
            return (LastName == "ALFKI");
        }
    }
}

Conclusion

Codedom is too much C# centric, so it’s hard to make it available for every language. The difference between Code Statement and Code Expressions sometimes makes it hard to convert from expression trees.

On the on the other hand Expression trees are too much LINQ oriented. They are less powerful than CodeDom but more easy to express. In expression trees everything is an expression unlike CodeDom. Some constructs are missing from expression trees like the assignment, but we will probably see the improvements in the expression trees in the future. So it might not be a true DSL or language generator, but sure it is enough to get the most of the databases.

There are some other more powerful meta-programming tools and libraries. F# quotation library supports all the available full-set language features expressed as quotations. Dynamic Language Runtime is another expression tree like library focussed more on compiler developers.

Finally this library is not build for runtime code conversion from expression tree to CodeDom, although it is possible. The CodeDom generated code is mainly for debugging to print the source code of the query. It might also be helpful for seeing what is going on under the hood.

Introduction

For many years, parallel computing is an important area for research in high performance computing. Super computers dominated the industry all the time. However with the cost of obtaining a fast computer and a fast network, cluster computing considered as a good alternative. High Performance Computing market grew rapidly, mainly because of the clusters intensified. According to a research, clusters represent 50% of the High Performance Computing system revenue at the end of 2005.

The idea of cluster computing is to have many machines on a high-speed network, clusters of computers running the same program. Recently, with the invention and adoption of multi-core CPU systems for desktops, it has become even more important. MPI makes even easier for people to build supercomputers by the usage of powerful computers, high speed networks and powerful libraries.

Message Passing Interface (MPI) is the standard of message passing in a distributed computing environment. Its benefit for researchers is invaluable.

MPICH is an open source, portable implementation of Message Passing Interface (MPI) for developing distributed memory application .

The goal of MPI Tools is to make easy to write programs that runs on a cluster of machines. Also make the transition and the portability easy for existing programs in cluster. Using MPITools, it is possible to create distributed functional applications with F#. Although it is primarily developed for .NET framework, it can run on any CLI implementation.

Implementation

The first step involved to make MPICH available to use for F# platform. A wrapping library is implemented for MPICH. Mainly used MPICH functions made available to F#. Those MPI functions are implemented with the effective usage of types. MPI_Init,, MPI_Comm_size, MPI_Comm_rank, MPI_Finalize, MPI_Send, MPI_Recv, MPI_Abort,  MPI_Barrier, MPI_Bcast , MPI_Gather, MPI_Scatter, MPI_Reduce. In reality, you could write distributed programs with just the first six of those function as I will show on the samples.  When using the library you don’t have to worry about the types and data sizes as you usually do in C programming. The only thing important is the order of communication the same as socket programming.

Because MPICH is unmanaged library, it important to make the data types compatible using the interoperability libraries in .NET framework.  All of the exposed data types and functions are defined in “mpi.h” file in MPICH distribution. If you want to use a different MPI implementation then it is needed to change those functions definitions appropriately based on the documentation.

[]
extern int MPI_Send( void *buf, int count, int MPI_Datatype,
int dest, int tag, int MPI_Comm)

Once the value data types such as int, char, byte, double and float types implemented which are pretty same with C implementation. Next step was to make the reference types of the virtual machine available. Unfortunately not all reference types are possible to send out to wire because of the state or impureness of the type. The types have to be serializable in order to send or receive. To make that possible binary serialization is used and passed as a byte array to the MPI. Implementing the reference types made also possible to pass the functions and lambda functions in to the channel.

For the MPI development, the key factor is the data types. The parties have to agree with the file types. Also the size of the file types should be fixed in order to communicate. However the types are properly handled by the library using the sophisticated type system capabilities. In the programs the order becomes really important. In order to get to the internals of MPI Tools, here is an implementation for standard MPICH type definitions and a type converter for it (shortened for simplicity).

type MPI_Datatype =
| MPI_CHAR           =  0x4c000101
| MPI_SIGNED_CHAR    =  0x4c000118

let private TypeConvert (t) =
let res =
match (box t) with
| : ? byte -> MPI_Datatype.MPI_BYTE
| : ? char -> MPI_Datatype.MPI_CHAR
| _ -> failwith “not implemented data type
Enum.to_int res

The complicated, many parameter function calls in the unmanaged MPICH library becomes powerful function with a few arguments in the .NET library. For instance previously defined 6 argument MPI_Send function becomes a three argument polymorphic function. To make it easy, actually send function becomes in different flavours. Actually most of the communication functions come in different versions for different types. The version below is used for singular types. There are two more flavours one for arrays and other for matrix types.

let send(data,destination, tag)
'a * int * int -> unit

let sendArray(data : 'a array,destination,tag)
a array * int * int -> unit

let sendMatrix (data : matrix,destination,tag)
matrix * int * int -> unit

Similarly, the same pattern goes for the receive function. However, this time it is needed to specify the return type as a generic argument of the function.

let receive<'a>(source,tag)
int*int -> 'a
 
let receiveArray<'a> (source, tag)
int * int -> 'a array
 
let receiveMatrix(source, tag)
int * int -> matrix

The other functions of MPI are implemented in a similar manner. You could also check the project as a tutorial as well. The library uses effectively active patterns, discriminated unions, interoperability and other functional structures

Usage

First of all MPICH needs to be installed prior to usage. The library is used just like another .NET library in your programs. However the execution is relatively different than usual. The programs have to be executed using the MPI daemon called "mpiexec". You could look at more on how to configure a cluster in the MPICH documentation. To run the process in n processor or processes “-n” switch needs to be given as a command line argument followed by the name of the compiled program.

mpiexec -n 2 test.exe

Here is a very simple ping pong application using MPI Tools. You can find more samples on the MPI Tools Source code.

#light
#I @"..\MPITools.Bindings\"
#r @"MPITools.Bindings.dll"
open MPITools
MPI.initialize()
 
let procSize = MPI.size()
let curProcess =  MPI.rank()
 
let pingpong() =
if curProcess = 0 then
let i =  0
MPI.send(i,1,0)
let b = MPI.receive(1,0)
()
elif curProcess = 1 then
let b = MPI.receive(0,0)
MPI.send(b+1,0,0)
pingpong()
MPI.finalize()

Conclusion

You can download MPI Tools from codeplex. Using MPI Tools, the distributed programs will be short, expressive and well typed with the help of the glorified type system of F#.

MPI Tools is built with F# 1.9.3.7 Compiler for the .NET Framework 2.0. However it would possibly work with any CLI implementation. In the future, some more MPI functions will be implemented, including some helper functions that hides the imperative style programming. and the side effects.

I hope it will help to solve your high computation problems effectively. Please feel free to ask questions or to contribute to the project.

Have fun!

Power of Functional Programming

Functional programming is one the oldest of major programming paradigms. Functional languages have been with us for a while. Languages like Lisp, Scheme, ML, OCaml, Haskell, Erlang and F# have well built compilers and tools and large user and development communities. However, the early success of the imperative programming languages made the procedural languages extremely popular for more than three decades. This lead to the rise of the object-oriented paradigm and formed the basis of commercial software development. Object oriented programming is still the most popular paradigm today.

Functional programming is a programming paradigm that uses the functions in their real mathematical sense. This means that functions are only computation objects where there is no mutable data and state information. This way it is more close to mathematical expressions. In contrast to imperative programming that is desperately dependent on the state of the objects, functional programming views all programs as collections of functions that accept arguments and return values.

Modern functional programming languages have a number of different techniques to help program development. That includes but not limited to the

• Powerful typing systems : It is accomplished with the support of polymorphism, and type inference
• Higher order functions: The higher-order functions are based on the concept of the first class values as functions mainly functions are actually like data.
• Implicit recursion: Implicit recursion is supported functional languages through the powerful functional data structures and also with higher-order functions such as fold, map and filter.

In programming languages, static typing means the type of expression is determined at compile by a technique called static program analysis. On the other hand dynamically typed languages, determines the types at the runtime by the aid of a runtime type checker. Personally, I like everything to be typed at the compile time with an extensive support of polymorphic types; because it allows seeing the errors at compile time and also it gives an optimised performance because there is no need for a runtime type checker. Luckily most of the functional languages benefit from the polymorphic statically typed systems.

Pureness is another concept applied by some functional languages like Haskell. On the other hand, it might be a style of programming even if it is not forced by the compiler. In pure functional programming, side effects are not allowed in the program with immutable variable and no loops. Immutability means that when a value is assigned to an identifier (not a variable), it cannot change anymore. Loops can be achieved with recursive functions. This style is a bit more difficult to program and read as well, but the benefits could be massive. For instance, it might be possible to optimise the compiled code for multiple cores, because of the compositionality of the functions that form the program.

The Features of Functional Programming

Functional Programming is not all about functions treated as first class values and immutable state. Functional programming also provides powerful features that every programmer should benefit.

Lambda calculus is a formal system designed to investigate function definition, function application and recursion. Lambda calculus could be used to define a computable function. Lambda calculus is used to develop a formal set theory. Function can be passed as an argument to other functions. There is another concept called curried functions when using lambda functions. The function reduces the term and returns another function with the normal form. This is called curried functions. In the lambda calculus, functions can only be created using another method, with higher-order functions and currying, another function creation is provided. A function in curried form is called partial application.

The calculus has only functions of one argument. In the curried function systems, a function with multiple arguments is expressed using a function whose result is another function. Every argument is reduced by default and returns a function.

Pattern matching is another powerful concept that functional and logic languages sport. It is used for assigning values to variables and for controlling the execution flow of a program. Pattern matching is used to match patterns with terms. If a pattern and term have the same shape then the match will succeed and any variables occurring in the pattern will be bound to the data structures which occur in the corresponding positions in the term.

Recursion is a mechanism for iterating an instruction or simply for a code reuse. By recursion it is possible to write a compact program that can help generalization. It is mainly used in functional programming.

Nowadays collections are the most important data structures when writing programs in any programming languages because of the big datasets. In most of the functional languages there are special functions to make the programmers jobs easy. By the usage of first class functions and those methods ends up in a good way to handle the collections. Some of those higher-order functions are map, fold, filter etc.

• Map : applies the function passed to each element of the collection. Resembles to for each loops for collections in procedural languages.
• Fold : applies the function to each element while sharing a resulting object. One common uses us to apply a sum operation to each member. It is similar to SQL aggregate functions but we pass the applying function.
• Filter : applies the filter function to each member of the collection and returns a list of object satisfying the conditional function.

Some pure languages implement those higher-order functions in a continuation style. But it could be implemented by simple loops in unpure languages as well.

Functional languages help rapid prototyping with the aid of powerful type systems and usually with an interactive window that allows executing expressions one or more at a time. It enables to code complex algorithms without ignoring the mathematical representation. It is much easier to create domain-specific languages (DSLs). It improves the productivity of the developer.

DSLs are specific programming languages made for use in a specific application domain. Programs written with a DSL are more clear and readable than those written with general-purpose languages. DSLs are more declarative than the imperative languages and they focus on the problem rather than the rules of the language. Meta programming is building a program which manipulates the syntactic structures of other programs. DSLs and meta programming are closely related to each other. It is possible to build a domain specific language by the meta programming features of a language.

The Future of Functional Programming

Object oriented paradigm is crucial in programming current industrial applications. A functional language has usually been considered as an academic language but I believe that this is going to change in the real feature, because functional programming has a big potential in this demanding industry. The expressiveness, powerful unique concepts such as laziness, immutability, powerful pattern matching, continuations etc. and the elegant style of programming makes the implementation of some tools easier than other paradigms. Tools such as static analysis tools, high-level modelling, compilation, interpretation and verification tools are one of the target areas of functional programming. Moreover one of the todays hot topics domain specific languages, eventually meta programming and need for a “uniformed single language” for all solutions makes functional programming very important. Those tasks could be implemented with other languages as well, but functional languages fit better than the others. Also the distribution of tasks in multi-cores or multiple machines might be possible with functional programming if the program is written in an immutable style.

Considering today’s facts, running systems, interoperability with the systems, non-functional world; using functional programming shouldn’t mean staying functional all the time. For the difficulty of some problems and previous experience of the developers it is inevitable to use procedural and object oriented paradigms. That’s how the mix functional object oriented programming emerged and affected the programming languages.

Not surprisingly, some companies already work with functional programming in industry. For the moment their main areas of development is design, modelling, specification, or build compiler tools. But this is yet to change and augment.

I really liked the XML expressiveness of Visual Basic, let’s build a very simple MSN History Search Engine using LINQ and XML Literals in Visual Basic.

The best thing is those literals could be used in LINQ expressions. Remember the simple XML file that MSN stores as a history.

• .@AttributeName : Accesses the attribute element in XML
• .<ElementName> : Accesses the element in XML
• …<Descendant name>: Accesses the descendant name in XML

Modifying the XML content is very neat as well either using the LINQ expressions or even in loops.

In .NET Framework 2.0 VB has one more feature called MY namespaces. It is very nice to access some dynamic data available like application or forms information. It also contains some helper functions to do some common tasks. Now I also found it very handy in a Windows Forms application.

Get the history files from the location and operate the XLINQ query:

Private Sub btnSearch_Click(ByVal sender As System.Object, ByVal e As System.EventArgs)
 
For Each file In My.Computer.FileSystem.GetFiles(dirLocation)
ProcessFile(file)
Next
txtOutput.Text = sBuild.ToString()
 
End Sub

The XLINQ query that does search magic for the messages is as follows :

Function ProcessFile(ByVal s As String) As Boolean
If s.EndsWith("xml") Then
Dim msn = XElement.Load(s)
Dim q = From message In msn.<message> _
Where message.<text>.Value.Contains(txtSearch.Text) _
Select From = (message.<from>.@FriendlyName), Too = (message.<to>.@FriendlyNam), _
Message = message.<text>.Value
For Each msgFound In q
sBuild.AppendLine(msgFound.From + " says to " + msgFound.Too + _
" :  " + msgFound.Message)
Next
End If
End Function

What makes this different is the usage of literals. In C# that query would be longer than that.
In the sample message.<From>.@FriendlyName
means that it will get the From element and get the friendlyname attribute from it.

It is basically like having the XML data in your hands but there is no need to parse it or access the elements using the classes provided rather this work is done by the compiler at the compile time.

In a couple lines of code we have a fully featured MSN history searching. Let me know if you still want the source code (although that is all about it) or even the executable in case you are not into programming.

Here is the summary of new language features as mentioned on the “What’s new on ORCAS

• C# 3.0 Language Support: This CTP implements all of the C#3.0 language features from the May LINQ CTP including:
  • Query Expressions
  • Object and Collection Initializers
  • Extension Methods
  • Local Variable Type Inference and Anonymous Types
  • Lambdas bound to Delegates and Expression trees
  • Complete design-time support: Intellisense, Formatting, Colorization

• VB 9.0 Language Support: This CTP contains the following language features:
  • Query Expressions: Basic querying, filtering, and ordering support
  • Object Initializers
  • Extension Methods
  • Local Variable Type Inference
  • Anonymous Types
  • XML literals
  • XML properties
  • New Line and Expression IntelliSense

LINQ is all in both of the languages and indeed this is the main feature for .NET Framework 3.5. Writing any type of queries is the purpose of LINQ at the end. Considering the abilities of LINQ, everything was possible before as well. LINQ makes us to get rid of the strings (the red coloured stuff) from the program in order to minimise the typo errors, easy to read programs by syntax highlighting. But all the best is that gives the ability to write declarative and functional style programs.

Beside the new language features, as a compiler improvement, it is very surprising that C# still doesn’t have background compilation. There is background syntax checking but no compilation. I believe this is a definite need for C# because it is really helpful. For instance the F# projects do background compilation and syntax checks, that way it easy to investigate “silly errors” while coding. Also this was one of the powerful features that I found on eclipse while working on a Java project.

Visual basic has that feature moreover it has also automatic syntax fixing as well. Likewise if you call a method with lower case letters it is automatically converted to the actual method on the next line. Actually in a type inferenced language this is needed, because it is not easy to recognise the type information of all the members.

Anyway I just wrote a quick macro to give the feeling of background compilation for C#. It is not really sophisticated but it works. Just put it into EnvironmentEvent macro in Visual Stuio.

Dim lastbuilt As DateTime
    Private Sub TextDocumentKeyPressEvents_AfterKeyPress(ByVal Keypress As String, ByVal Selection As EnvDTE.TextSelection, ByVal InStatementCompletion As Boolean) Handles TextDocumentKeyPressEvents.AfterKeyPress
        Dim doc = DTE.ActiveDocument
        Dim diff = DateTime.Now.Subtract(lastbuilt)
 
        If Not Char.IsLetterOrDigit(Keypress(0)) And diff.Seconds > 5 Then
            DTE.ExecuteCommand("Build.BuildSelection")
            lastbuilt = DateTime.Now
            doc.Activate()
        End If
    End Sub

XML in Language

All of the best is that now XML is a first class citizen in VB. I wouldn’t expect this as a serious feature but after trials it makes extremely relevant to use XML in Visual Basic. You get syntax highlighting and even intellisense for XML if the namespaces are specified and even more.

Having XML literals in the language, it makes really sense to use XLinq with VB.

Like consider the xml stored by messenger. You could just assign to a variable just like that.

        Dim msn = <?xml version="1.0"?>
                  <?xml-stylesheet type='text/xsl' href='MessageLog.xsl'?>
                  <Log FirstSessionID="1" LastSessionID="1">
                      <Message Date="25/03/2007" Time="22:35:47" DateTime="2007-03-25T21:35:47.173Z" SessionID="1">
                          <From><User FriendlyName="koko"/></From>
                          <To><User FriendlyName="opopop"/></To>
                          <Text Style="font-family:Comic Sans MS; font-weight:bold; color:#0000a0; ">151515</Text>
                      </Message>
                      <Message Date="25/03/2007" Time="22:35:55" DateTime="2007-03-25T21:35:55.344Z" SessionID="1">
                          <From><User FriendlyName="koko"/></From>
                          <To><User FriendlyName="opopop"/></To>
                          <Text Style="font-family:Comic Sans MS; font-weight:bold; color:#0000a0; ">5959959</Text></Message>
                  </Log>

It will have the type of System.Xml.Linq.XDocument.

Let’s define the XML Stylesheet :

        Dim xslt = <?xml version="1.0"?>
                   <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
                       <xsl:template match="Log">
                           <html>
                               <head>
                                   <title> Message Log for </title>
                               </head>
                               <body style='margin:0'>
                                   <table border='1'>
                                       <tr>
                                           <td> From </td>
                                           <td> To </td>
                                           <td> Message </td>
                                       </tr>
                                       <xsl:for-each select="/Log/Message">
                                           <tr>
                                               <td><xsl:value-of select="From/User/@FriendlyName"/></td>
                                               <td><xsl:value-of select="To/User/@FriendlyName"/></td>
                                               <td><xsl:value-of select="Text"/></td>
                                           </tr>
                                       </xsl:for-each>
                                   </table>
                               </body>
                           </html>
                       </xsl:template>
                   </xsl:stylesheet>

If we want to do an XSLT transformation to that snippet, it is even easier than it used to be.

Dim xTransform = New System.Xml.Xsl.XslCompiledTransform()
xTransform.Load(xslt.CreateReader())
xTransform.Transform(msn.CreateReader(), New System.Xml.XmlTextWriter("test.html", New System.Text.UnicodeEncoding()))

I think working with xml data using Visual Basic should be considered as a manipulation language. Since we are all becoming multilingual this shouldn’t be a problem.

Introduction

Software agent is a program that handles the behaviour of a user or software with the aid of an agency. The software agents are not called by a user or by a process, they are acting themselves, they decide the next task to do. This drives the system to the concept of intelligent agents.

Agent system is a system composed of several software agents. It provides the collective environment for making the goals possible for individual agent. It helps the agents to communicate and delegate the task. Although software agents are considered as autonomous, all agents operate on a human supervision.

For agent systems, there are no currently implemented commercial applications; however there are a lot of ongoing research projects in the software agents and agent systems area. I have read a couple of research papers in software agent systems and software agents. One of the research papers is focussing on the different architectures behind the software agent systems. The other research paper is about different implications of the software agents for different scenarios.

Software agents and agent systems will be the key elements in information technology. They will solve many problems of information overload and this will change the human computer interaction. It will also change the way we develop software for large systems.

It is possible to use modern agent platforms to implement large scale agent systems. They are some working applications of software agents in an e-commerce system and software agents as a resource broker in the grid. I will try to investigate on different types of software agents and their implications.

There are different types of agent systems. Intelligent agents are capable of modifying their behaviour based on their learning and reasoning. Distributed agents are executed physically in distinct machines. Multi-agent systems are basically distributed agents that are not capable of doing alone the objective. They need to talk and communicate. Mobile agents are the agents that can execute on different processors.

Software Agents and Agent Systems

In real life, agents are expert people on some areas. For example a travel agent has the information needed for you to travel. They have the information and they have the connections to make the travel for us. Another example is the insurance agent, they have a deep knowledge in their fields and they provide this information for us.

These are the characteristics of software agents as well. They are specialised and provide only the needed knowledge. It is possible to find, filter or customise the interaction with software agents. Software agents will affect relatively the software evolution. We have some technologies used in software development like object orientation, and multi-threaded objects. The main concept is the execution in threads and communication between threads. Next thing will be the software agents. The agent based approach will be the revolution for building complex software systems. They are some implications of software agents. For instance Bio-agents work on a cellular automaton. They have a simple behaviour and they don’t have any intelligence.

Software agents are designed with dynamic transportations. In dynamic transportation the parts are changing, the environment does not build thousand parts. It works with real-time scheduling and it chooses the best scenario based on the real-life conditions.

A tool called JADE can be used as a software agent system. In JADE, agents communicate with messages. They send and receive ACL messages. JADE also provides queue for the agents.

Some real world examples exist for software agents. Agents can operate in an e-shop application. They can go to e-shop; buy some things and can kill themselves can be a simple usage of the software agents. There exist also agent systems for large distributed area like GRID systems. Agents can travel in the grid for resource broking.

There are some assumptions of the agents in the grid. The agents work in teams, and each agent has a team leader. Incoming agents can join any team based on the criteria they have been created. Teams can accept or reject the incoming agents based on their criteria of acceptance. There exists different type of architectures. One of the approaches is the usage of thread pool. Task-per-thread paradigm is being used in that architecture. Another approach is the use of databases. Database agents are located on remote machines contributing additional computational power.

Developing high-quality industry ready is difficult to achieve. It is one of the complex construction tasks to build software in telecommunication or other big industrial areas because of the size of the data and because of the rules to implement.

I think complex software systems can be built and designed by the use of autonomous agents. It can give some advantages to the software engineers especially for some areas. GRID computing is the most complex system that can be handled with software agents.

Computers are becoming more important for everyday activities of every business and every people. The speed of access to the information is less than seconds and the information is increasing exponentionally every single hour. Current software implementations are enough to handle that huge request for the moment; however without an autonomous system this task is getting harder to accomplish with the technological developments. The human driven mechanism for handling the information will collapse soon or later. This metaphor has to change from the software systems perspective. Software agents can help in organising the data without human interaction and make the data ready for people’s use.

Conclusion

Agent-oriented techniques are used in many big areas. There are some experiments on the GRID computing which makes this work very important. GRID computing is very important especially in the research area. However it will be more relevant for the industry as well.

JADE is one of the great multi-agent development tools that support different architectures. None of the architectures can be good for a particular solution. Each problem has different situations and each might have a different architecture. As the research and development continues in that area, every problem will be investigated and will have a particular solution.

Software agents can be applied to many different areas. Software engineers will accept the agent-based approach soon or later to make their system autonomous and intelligent. We might see a combination of paradigms in the future. Agent-oriented concepts and techniques are well suited to developing complex and distributed systems as an extension to the current paradigms.

Software engineering started to be like a car manufacturing company. Many things are getting automated and many software paradigms are becoming de facto standard for a particular problem. It is sometimes make the process easier; on the other hand it makes difficult to implement for some scenarios.