Archive for the ‘Tools’ Category.

Page 1 of 212

Eclipse Reference Project Management with Maven

20 July, 2008, 00:50

I was using Eclipse for one for the previous projects as a development environment. It has some powerful features. But I have to say not as powerful as Visual Studio. Anyway since it is possibly the best IDE for Java development it was my choice on the day of implementation.

The project that I was working was done using Borland’s Jbuilder. Unfortunately that IDE doesn’t exist anymore as it is discontinued by a new company. Thanks to Eclipse import project wizard, the project migration wasn’t difficult at all.

I just opened the project XML and the project was ready to be working. However, when using with different big open source libraries, Eclipse has a problem in managing them. Actually I have a problem in managing them, because I need to find the required version of each dependency and so on. It is the difficulty of package and build management of eclipse.

It is often difficult to reference a library and get all the dependencies on Eclipse. Somehow it doesn’t warn at compile time as well and you get exception at runtime which is not nice enough.

Than I found a project called Maven for doing proper build systems. What it does is has an online repository that have all the dependencies filed and it downloads the required assemblies for your project. So you don’t have download each package separately and link to project. All the best it is very well integrated with eclipse with its marvelous plug-in.

You load the plug-in and tell maven to add a reference to your project. I know Java programmers call this as something else but since I’m mainly a .NET programmer I hope somebody will find it useful.

Anyway, it stores the configuration in XML file that you can tweak it later if needed. All the updates to the references can happen automatically as well which is quite nice.

Tags: , ,
Category: Tools, java  |  3 Comments

LINQ Expression Trees-Lambdas to CodeDom Conversion

25 February, 2008, 00:23

Introduction

Some people are working to make the meta-programming possible. Some says as language oriented programming or domain specific language, but I prefer in general as meta-programming. For years programming languages supported to generate code with the powerful libraries or developers worked just with string concatenations and external linkers.

Nowadays meta-programming is getting more and more important as the domain expertise required. So the languages make meta programming possible at the compiler level with compiler directives.

Indeed there a lot of ideas coming from functional programming world where everything treated as expressions.  The code becomes data and data usage happens in the code. It should sound familiar with LINQ to SQL efforts to make this possible.

Libraries

.NET Framework had code generators since the beginning. CodeDom is probably the best known for tree based code generation. Codedom made possible to develop the ASP.NET engine, Windows Form designer, Web form designer, Web services wrapper LINQ entity objects and more. It is used extensively by the framework for the key technologies.

Although there are other APIs in .NET framework such as System.Reflection, System.Reflection.Emit, in this post we will focus on CodeDom and the new comer Expression Trees.

Expression Tree is the key API behind LINQ to SQL or IQueryable interface in general. Every query is expressed as typed trees that is parsed and converted to SQL later by the library.

The syntax of expressing queries is very readable with query comprehension syntax. However sometimes I want to know about the generated tree, like actually which functions are getting involved in the query. I have used Expression Tree Debugger Visualizer to draw the tree. It is pretty handy tool but for big trees it is difficult to see what is going on. This was my main motivation actually, although we had the code, we don’t see what’s the magic going on with query comprehension.

Implementation

So the idea is to have the code regenerated from the tree. In the real world this will involve a parser, interpreter and some more compiler theory which requires a lot of research. And because this is just for fun and since we have a powerful CodeDom library to generate code, I tried to convert the expression tree to CodeDom tree. Than used the CodeDom to generate code in any language. Finally I wrote the extension methods so that the debuggers and my code can use it directly from the type.

The compiler generates automatically the expression trees if we use the proper syntax. So from the beginning we have the tree. In order to convert to CodeDom objects, we need to traverse the tree and generate the necessary CodeDom objects. So I wrote a  tree walker that generates a CodeDom object to is parent while going to the last children. I didn’t realise how far it is going but that was it. When the tree walker finished with some more few lines of code the converter was just working.

I would like to put the code here as well but unfortunately it is too long for a blog post, so here are some snippets. Feel free to provide suggestions or bug reports.

LINQ Expression Visitor that generates CodeDom Trees 

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
 
using System.Collections.ObjectModel;
using System.Linq.Expressions;
using System.Reflection;
using System.CodeDom;
using System.CodeDom.Compiler;
using System.IO;
 
namespace ExpressionToCodedom
{
    public  class CodeDomExpressionVisitor
    {
 
        Expression m_exp;
        Dictionary<string, CodeTypeMember> m_members;
 
        public CodeDomExpressionVisitor(Expression e)
        {
            m_exp = e;
        }
        internal string GenerateSource(CodeDomProvider codeProvider)
        {
            StringBuilder sb = new StringBuilder();
            TextWriter tWriter = new IndentedTextWriter(new StringWriter(sb));
            CodeCompileUnit ccu = GenerateCode();
            codeProvider.GenerateCodeFromCompileUnit(ccu, tWriter, new CodeGeneratorOptions());
            codeProvider.Dispose();
 
            tWriter.Close();
 
            return sb.ToString();
        }
 
        internal string GenerateSource(string language)
        {
 
 
            CodeDomProvider codeProvider=null;
            if (language == "cs")
                codeProvider = new Microsoft.CSharp.CSharpCodeProvider();
            else if (language == "vb")
                codeProvider = new Microsoft.VisualBasic.VBCodeProvider();
            else
            {                
 
                    throw new Exception("make sure you are trying to load a CodeDomProvider assembly");
 
            }
            return GenerateSource(codeProvider); 
 
        }
 
        public string GenerateSource()
        {
            return GenerateSource("cs"); 
        }
 
 
        private CodeCompileUnit GenerateCode()
        {
            var code = new CodeCompileUnit();
            m_members = new Dictionary<string, CodeTypeMember>();
 
            var LambdaTypeClass = new CodeTypeDeclaration("LambdaExpression");
            var ns = new CodeNamespace("Runtime");
 
            ns.Types.Add(LambdaTypeClass);            
            ns.Imports.Add(new CodeNamespaceImport("System"));
            // add more types in case I want to compile
 
            code.Namespaces.Add(ns);
 
            CodeObject cEvaluationResult = Visit(m_exp);
 
            var constructor = new CodeConstructor();
 
            if (cEvaluationResult is CodeStatement)
                constructor.Statements.Add(cEvaluationResult as CodeStatement);
 
            else if (cEvaluationResult is CodeExpression)
                constructor.Statements.Add(cEvaluationResult as CodeExpression);
 
            LambdaTypeClass.Members.Add(constructor);
 
 
            foreach (var item in m_members)
            {
                LambdaTypeClass.Members.Add(item.Value);
            }
 
            return code;
 
        }
 
        protected virtual CodeObject Visit(Expression exp)
        {
            if (exp == null)
                return null;
            switch (exp.NodeType)
            {
                case ExpressionType.Negate:
                case ExpressionType.NegateChecked:
                case ExpressionType.Not:
                case ExpressionType.Convert:
                case ExpressionType.ConvertChecked:
                case ExpressionType.ArrayLength:
                case ExpressionType.Quote:
                case ExpressionType.TypeAs:
                    return this.VisitUnary((UnaryExpression)exp);
                case ExpressionType.Add:
                case ExpressionType.AddChecked:
                case ExpressionType.Subtract:
                case ExpressionType.SubtractChecked:
                case ExpressionType.Multiply:
                case ExpressionType.MultiplyChecked:
                case ExpressionType.Divide:
                case ExpressionType.Modulo:
                case ExpressionType.And:
                case ExpressionType.AndAlso:
                case ExpressionType.Or:
                case ExpressionType.OrElse:
                case ExpressionType.LessThan:
                case ExpressionType.LessThanOrEqual:
                case ExpressionType.GreaterThan:
                case ExpressionType.GreaterThanOrEqual:
                case ExpressionType.Equal:
                case ExpressionType.NotEqual:
                case ExpressionType.Coalesce:
                case ExpressionType.ArrayIndex:
                case ExpressionType.RightShift:
                case ExpressionType.LeftShift:
                case ExpressionType.ExclusiveOr:
                    return this.VisitBinary((BinaryExpression)exp);
                case ExpressionType.TypeIs:
                    return this.VisitTypeIs((TypeBinaryExpression)exp);
                case ExpressionType.Conditional:
                    return this.VisitConditional((ConditionalExpression)exp);
                case ExpressionType.Constant:
                    return this.VisitConstant((ConstantExpression)exp);
                case ExpressionType.Parameter:
                    return this.VisitParameter((ParameterExpression)exp);
                case ExpressionType.MemberAccess:
                    return this.VisitMemberAccess((MemberExpression)exp);
                case ExpressionType.Call:
                    return this.VisitMethodCall((MethodCallExpression)exp);
                case ExpressionType.Lambda:
                    return this.VisitLambda((LambdaExpression)exp);
                case ExpressionType.New:
                    return this.VisitNew((NewExpression)exp);
                case ExpressionType.NewArrayInit:
                case ExpressionType.NewArrayBounds:
                    return this.VisitNewArray((NewArrayExpression)exp);
                case ExpressionType.Invoke:
                    return this.VisitInvocation((InvocationExpression)exp);
                case ExpressionType.MemberInit:
                    return this.VisitMemberInit((MemberInitExpression)exp);
                case ExpressionType.ListInit:
                    return this.VisitListInit((ListInitExpression)exp);
                default:
                    throw new Exception(string.Format("Unhandled expression type: '{0}'", exp.NodeType));
            }
        }
 
        protected virtual CodeObject VisitBinding(MemberBinding binding)
        {            
            switch (binding.BindingType)
            {
                case MemberBindingType.Assignment:
                    return this.VisitMemberAssignment((MemberAssignment)binding);
                case MemberBindingType.MemberBinding:
                    return this.VisitMemberMemberBinding((MemberMemberBinding)binding);
                case MemberBindingType.ListBinding:
                    return this.VisitMemberListBinding((MemberListBinding)binding);
                default:
                    throw new Exception(string.Format("Unhandled binding type '{0}'", binding.BindingType));
            }
        }
 
        protected virtual CodeExpression VisitElementInitializer(ElementInit initializer)
        {            
            ReadOnlyCollection<CodeExpression> arguments = this.VisitExpressionList(initializer.Arguments);
 
            return new CodeMethodInvokeExpression(new CodeMethodReferenceExpression(new CodeThisReferenceExpression(),initializer.AddMethod.Name), arguments.ToArray());                               
        }
 
        protected virtual CodeObject VisitUnary(UnaryExpression u)
        {
            CodeObject operand = this.Visit(u.Operand);
 
            return operand;
        }
 
        private CodeBinaryOperatorType BindOperant(ExpressionType e)
        {
            switch (e)
            {
                case ExpressionType.Add:
                case ExpressionType.AddChecked:
                    return CodeBinaryOperatorType.Add;
 
                case ExpressionType.And:
                    return CodeBinaryOperatorType.BitwiseAnd;
 
                case ExpressionType.AndAlso:
                    return CodeBinaryOperatorType.BooleanAnd;                  
 
                case ExpressionType.Or:
                    return CodeBinaryOperatorType.BitwiseOr;                    
 
                case ExpressionType.OrElse:
                    return CodeBinaryOperatorType.BooleanOr;                    
 
                case ExpressionType.ExclusiveOr:
                case ExpressionType.ArrayIndex:
                case ExpressionType.Coalesce:
                case ExpressionType.RightShift:
                case ExpressionType.LeftShift:
                    throw new NotSupportedException("no direct equivalent in codedom,so workarounds not implemented");
 
                case ExpressionType.Equal:
                    return CodeBinaryOperatorType.IdentityEquality;
 
                case ExpressionType.NotEqual:
                    return CodeBinaryOperatorType.IdentityInequality;                    
 
                case ExpressionType.GreaterThan:
                    return CodeBinaryOperatorType.GreaterThan;                    
 
                case ExpressionType.GreaterThanOrEqual:
                    return CodeBinaryOperatorType.GreaterThanOrEqual;                    
 
                case ExpressionType.LessThan:
                    return CodeBinaryOperatorType.LessThan;                    
 
                case ExpressionType.LessThanOrEqual:
                    return CodeBinaryOperatorType.LessThanOrEqual;                    
 
                case ExpressionType.Multiply:
                case ExpressionType.MultiplyChecked:
                    return CodeBinaryOperatorType.Multiply;
 
                case ExpressionType.Subtract:
                case ExpressionType.SubtractChecked:
                    return CodeBinaryOperatorType.Subtract;
 
                case ExpressionType.Power:
                case ExpressionType.Divide:
                    return CodeBinaryOperatorType.Divide;
 
                case ExpressionType.Modulo:
                    return CodeBinaryOperatorType.Modulus;
 
                default:
                    throw new Exception("are you sure you are right?");
            }
        }
 
        protected virtual CodeBinaryOperatorExpression VisitBinary(BinaryExpression b)
        {
            var left = this.Visit(b.Left) as CodeExpression;
            var right = this.Visit(b.Right) as CodeExpression;
            CodeObject conversion = this.Visit(b.Conversion);
 
            CodeBinaryOperatorType operant = BindOperant(b.NodeType);           
            var condExpr = new CodeBinaryOperatorExpression(left, operant, right);
            return condExpr;
        }
 
        protected virtual CodeObject VisitTypeIs(TypeBinaryExpression b)
        {            
            CodeObject expr = this.Visit(b.Expression);          
            return expr;
        }
 
        protected virtual CodeExpression VisitConstant(ConstantExpression c)
        {
            if (c.Value == null)
            {
                return new CodePrimitiveExpression(null);
            }
            else if (c.Value.GetType().IsValueType || c.Value.GetType() == typeof(string))
            {
                   return new CodePrimitiveExpression(c.Value);
            }
            else
            {
                return new CodeVariableReferenceExpression(c.Value.ToString());             
            }                        
        }
 
        protected virtual CodeObject VisitConditional(ConditionalExpression c)
        {            
            CodeObject test = this.Visit(c.Test);
            CodeExpression ifTrue = this.Visit(c.IfTrue) as CodeExpression;
            CodeExpression ifFalse = this.Visit(c.IfFalse) as CodeExpression;
 
            var ifStatement = new CodeConditionStatement(test as CodeExpression,
                                                         new CodeStatement[] {new CodeExpressionStatement(ifTrue) }, 
                                                         new CodeStatement[] {new CodeExpressionStatement(ifFalse) });                    
            return ifStatement;
        }
 
        protected virtual CodeObject VisitParameter(ParameterExpression p)
        {
            return new CodeArgumentReferenceExpression(p.Name);            
        }
 
        protected virtual CodeObject VisitMemberAccess(MemberExpression m)
        {
 
            CodeObject exp = this.Visit(m.Expression);
 
            if (exp is CodePrimitiveExpression)
            {
                return exp;
            }
            else
            {
                Type memType;
                if (m.Member.MemberType == MemberTypes.Field)
                    memType = (m.Member as FieldInfo).FieldType;
                else memType = (m.Member as PropertyInfo).PropertyType;
 
 
                m_members[m.Member.Name] = new CodeMemberField(memType, m.Member.Name);
                return new CodeVariableReferenceExpression(m.Member.Name);
            }
        }    
 
        protected virtual CodeObject VisitMethodCall(MethodCallExpression m)
        {           
            CodeObject obj = this.Visit(m.Object);
            IEnumerable<CodeExpression> args = this.VisitExpressionList(m.Arguments);
 
            if (obj == null)
            {  //static method call
                return new CodeMethodInvokeExpression(new CodeTypeReferenceExpression(m.Method.DeclaringType),m.Method.Name,args.ToArray());                
            }
            else
            {
                return new CodeMethodInvokeExpression(obj as CodeExpression, m.Method.Name, args.ToArray());
            }   
        }
 
        protected virtual ReadOnlyCollection<CodeExpression> VisitExpressionList(ReadOnlyCollection<Expression> original)
        {
            List<CodeExpression> list = new List<CodeExpression>();
            for (int i = 0, n = original.Count; i < n; i++)
            {
                CodeExpression p = (CodeExpression)this.Visit(original[i]);                
                    list.Add(p);                
            }            
            return list.AsReadOnly();
        }
 
        protected virtual CodeExpression VisitMemberAssignment(MemberAssignment assignment)
        {// thhose are properties
 
            CodeObject e = this.Visit(assignment.Expression);
            return e as CodeExpression;
 
        }
 
        protected virtual CodeObject VisitMemberMemberBinding(MemberMemberBinding binding)
        {
 
            IEnumerable<CodeExpression> bindings = this.VisitBindingList(binding.Bindings) as IEnumerable<CodeExpression>;
            return new CodeObjectCreateExpression(binding.Member.Name, bindings.ToArray());            
        }
 
        protected virtual CodeObject VisitMemberListBinding(MemberListBinding binding)
        {
 
            IEnumerable<CodeExpression> initializers = this.VisitElementInitializerList(binding.Initializers);
 
            return new CodeObjectCreateExpression(binding.Member.Name, initializers.ToArray());
 
        }
 
        protected virtual IEnumerable<CodeExpression> VisitBindingList(ReadOnlyCollection<MemberBinding> original)
        {
            List<CodeExpression> list = new List<CodeExpression>();
            for (int i = 0, n = original.Count; i < n; i++)
            {
                CodeExpression b = this.VisitBinding(original[i]) as CodeExpression;
 
                    list.Add(b);
 
            }
            return list;
        }
 
        protected virtual IEnumerable<CodeExpression> VisitElementInitializerList(ReadOnlyCollection<ElementInit> original)
        {
            List<CodeExpression> list = new List<CodeExpression>();
            for (int i = 0, n = original.Count; i < n; i++)
            {
                CodeExpression init = this.VisitElementInitializer(original[i]);
 
                list.Add(init);
 
            }
 
            return list;
        }
 
        protected CodeMethodReferenceExpression VisitLambda(LambdaExpression lambda)
        {
            var  body = this.Visit(lambda.Body);
            var lambdaMethod = new CodeMemberMethod();
 
            lambdaMethod.Name = lambda.Type.Name;
            if (lambdaMethod.Name.Contains("Func"))
                lambdaMethod.ReturnType = new CodeTypeReference(lambda.Body.Type);
 
            foreach (var item in lambda.Parameters)
            {
                lambdaMethod.Parameters.Add(new CodeParameterDeclarationExpression(item.Type, item.Name));
            }
 
            if (body is CodeExpression)
            {
                if (lambdaMethod.ReturnType.BaseType.Contains("Void"))
                    lambdaMethod.Statements.Add((body as CodeExpression ));
 
                else
                    lambdaMethod.Statements.Add(new CodeMethodReturnStatement(body as CodeExpression));
            }
            else if (body is CodeStatement)
            {
                    lambdaMethod.Statements.Add((body as CodeStatement));
            }
            else
            {
                throw new Exception("investigate...");
            }
 
            m_members[lambda.Type.FullName] = lambdaMethod;
            return new CodeMethodReferenceExpression(new CodeThisReferenceExpression(), lambdaMethod.Name) ;
        }
 
        protected virtual CodeObject VisitNew(NewExpression nex)
        {            
            IEnumerable<CodeExpression> args = this.VisitExpressionList(nex.Arguments);
 
 
            return new CodeObjectCreateExpression(nex.Type.Name,args.ToArray());
 
        }
 
        protected virtual CodeObject VisitMemberInit(MemberInitExpression init)
        {
            CodeObject n = this.VisitNew(init.NewExpression);
            CodeExpression[] bindings = this.VisitBindingList(init.Bindings).ToArray(); //binding will return property initialisation
 
 
            for (int i = 0; i < init.Bindings.Count; i++)            
            {
                                                                    // need to do something with that////
                var assignProperty = new CodeAssignStatement(new CodePropertyReferenceExpression(
                            n as CodeExpression, init.Bindings[i].Member.Name), bindings[i]);
            }                                   
 
            return n;
        }
 
        protected virtual CodeObject VisitListInit(ListInitExpression init)
        {
 
            CodeObject n = this.VisitNew(init.NewExpression);
            IEnumerable<CodeExpression> initializers = this.VisitElementInitializerList(init.Initializers);
 
            return n;
        }
 
        protected virtual CodeObject VisitNewArray(NewArrayExpression na)
        {
 
            IEnumerable<CodeExpression> exprs = this.VisitExpressionList(na.Expressions);
 
 
            return new CodeArrayCreateExpression(new CodeTypeReference(na.Type), exprs.ToArray());
        }
 
        protected virtual CodeObject VisitInvocation(InvocationExpression iv)
        {            
            IEnumerable<CodeExpression> args = this.VisitExpressionList(iv.Arguments);
 
            var expr = this.Visit(iv.Expression) as CodeExpression;
 
            return new CodeMethodInvokeExpression(new CodeMethodReferenceExpression(expr, "Method"), args.ToArray());            
        }      
    }
}

Download

Expression Tree to CodeDom Source Code

Downloads: 235 File Name: Expression_Source.zip


Download

Expression Tree to CodeDom DLL

Downloads: 129 File Name: ExpressionToCodedom.dll

Example

The extension methods enables to see the source code of any IQueryable and any Expression. Any of them have a GenerateSourceCodeMethod that gives back a string.

Expression Tree to CodeDom Visualizer

GenerateSourceCode(); // default C#

GenerateSourceCode(string language); // either cs or vb as input or  Fully qualified name of the CodeDomProvider (like Microsoft.FSharp.Compiler.CodeDom.FSharpCodeProvider) It should be added as a reference to the project if you’re going to use it.

Sample program that manipulates the expression trees and usage of CodeDom Converter with “item.GetCodeDomSource(“vb”)”

int a = 3, c = 2, d = 0;
 
var e1 = Expression.Constant(5);
var e2 = Expression.And(e1, e1);
Expression<Func<string, Func<bool>>> e3 = tbool => () => a < b && 8 > d || c == d;
Expression<Func<bool>> e4 = () => b < 4;
Expression<Func<RecordName, bool>> e5 = rn => rn.LastName == "ALFKI";
Expression<Func<StringBuilder>> e6 = () => new StringBuilder { Capacity = 20 };
Expression<Func<string, string>> e7 = word => word == "hello" ? "yes" : "no";
 
 
foreach (var item in new Expression[] { e1, e2, e3, e4, e5, e6,e7 })
{
    Console.WriteLine(item.GetCodeDomSource("vb"));
}

Visual Basic Output

Namespace Runtime
 
    Public Class LambdaExpression
 
        Private LastName As String
 
        Private Sub New()
            MyBase.New
            Me.Func`2
        End Sub
 
        Private Function Func`2(ByVal rn As Demo.Program.RecordName) As Boolean
            Return (LastName Is "ALFKI")
        End Function
    End Class
End Namespace

C# Output

namespace Runtime {
    using System;
 
 
    public class LambdaExpression {
 
        private string LastName;
 
        private LambdaExpression() {
            this.Func`2;
        }
 
        private bool Func`2(Demo.Program.RecordName rn) {
            return (LastName == "ALFKI");
        }
    }
}

Conclusion

Codedom is too much C# centric, so it’s hard to make it available for every language. The difference between Code Statement and Code Expressions sometimes makes it hard to convert from expression trees.

On the on the other hand Expression trees are too much LINQ oriented. They are less powerful than CodeDom but more easy to express. In expression trees everything is an expression unlike CodeDom. Some constructs are missing from expression trees like the assignment, but we will probably see the improvements in the expression trees in the future. So it might not be a true DSL or language generator, but sure it is enough to get the most of the databases.

There are some other more powerful meta-programming tools and libraries. F# quotation library supports all the available full-set language features expressed as quotations. Dynamic Language Runtime is another expression tree like library focussed more on compiler developers.

Finally this library is not build for runtime code conversion from expression tree to CodeDom, although it is possible. The CodeDom generated code is mainly for debugging to print the source code of the query. It might also be helpful for seeing what is going on under the hood.

Category: Articles, C#, Functional Programming, Software, Tools, linq  |  3 Comments

Distributed Functional Programming with F# MPI Tools for .NET

20 January, 2008, 03:44

Introduction

For many years, parallel computing is an important area for research in high performance computing. Super computers dominated the industry all the time. However with the cost of obtaining a fast computer and a fast network, cluster computing considered as a good alternative. High Performance Computing market grew rapidly, mainly because of the clusters intensified. According to a research, clusters represent 50% of the High Performance Computing system revenue at the end of 2005.

The idea of cluster computing is to have many machines on a high-speed network, clusters of computers running the same program. Recently, with the invention and adoption of multi-core CPU systems for desktops, it has become even more important. MPI makes even easier for people to build supercomputers by the usage of powerful computers, high speed networks and powerful libraries.

Message Passing Interface (MPI) is the standard of message passing in a distributed computing environment. Its benefit for researchers is invaluable.

MPICH is an open source, portable implementation of Message Passing Interface (MPI) for developing distributed memory application .

The goal of MPI Tools is to make easy to write programs that runs on a cluster of machines. Also make the transition and the portability easy for existing programs in cluster. Using MPITools, it is possible to create distributed functional applications with F#. Although it is primarily developed for .NET framework, it can run on any CLI implementation.

Implementation

The first step involved to make MPICH available to use for F# platform. A wrapping library is implemented for MPICH. Mainly used MPICH functions made available to F#. Those MPI functions are implemented with the effective usage of types. MPI_Init,, MPI_Comm_size, MPI_Comm_rank, MPI_Finalize, MPI_Send, MPI_Recv, MPI_Abort,  MPI_Barrier, MPI_Bcast , MPI_Gather, MPI_Scatter, MPI_Reduce. In reality, you could write distributed programs with just the first six of those function as I will show on the samples.  When using the library you don’t have to worry about the types and data sizes as you usually do in C programming. The only thing important is the order of communication the same as socket programming.

Because MPICH is unmanaged library, it important to make the data types compatible using the interoperability libraries in .NET framework.  All of the exposed data types and functions are defined in “mpi.h” file in MPICH distribution. If you want to use a different MPI implementation then it is needed to change those functions definitions appropriately based on the documentation.

[]
extern int MPI_Send( void *buf, int count, int MPI_Datatype,
int dest, int tag, int MPI_Comm)

Once the value data types such as int, char, byte, double and float types implemented which are pretty same with C implementation. Next step was to make the reference types of the virtual machine available. Unfortunately not all reference types are possible to send out to wire because of the state or impureness of the type. The types have to be serializable in order to send or receive. To make that possible binary serialization is used and passed as a byte array to the MPI. Implementing the reference types made also possible to pass the functions and lambda functions in to the channel.

For the MPI development, the key factor is the data types. The parties have to agree with the file types. Also the size of the file types should be fixed in order to communicate. However the types are properly handled by the library using the sophisticated type system capabilities. In the programs the order becomes really important. In order to get to the internals of MPI Tools, here is an implementation for standard MPICH type definitions and a type converter for it (shortened for simplicity).

type MPI_Datatype =
| MPI_CHAR           =  0x4c000101
| MPI_SIGNED_CHAR    =  0x4c000118

let private TypeConvert (t) =
let res =
match (box t) with
| : ? byte -> MPI_Datatype.MPI_BYTE
| : ? char -> MPI_Datatype.MPI_CHAR
| _ -> failwith “not implemented data type
Enum.to_int res

The complicated, many parameter function calls in the unmanaged MPICH library becomes powerful function with a few arguments in the .NET library. For instance previously defined 6 argument MPI_Send function becomes a three argument polymorphic function. To make it easy, actually send function becomes in different flavours. Actually most of the communication functions come in different versions for different types. The version below is used for singular types. There are two more flavours one for arrays and other for matrix types.

let send(data,destination, tag)
'a * int * int -&gt; unit

let sendArray(data : 'a array,destination,tag)
a array * int * int -> unit

let sendMatrix (data : matrix,destination,tag)
matrix * int * int -> unit

Similarly, the same pattern goes for the receive function. However, this time it is needed to specify the return type as a generic argument of the function.

let receive&lt;'a&gt;(source,tag)
int*int -&gt; 'a
 
let receiveArray&lt;'a&gt; (source, tag)
int * int -&gt; 'a array
 
let receiveMatrix(source, tag)
int * int -&gt; matrix

The other functions of MPI are implemented in a similar manner. You could also check the project as a tutorial as well. The library uses effectively active patterns, discriminated unions, interoperability and other functional structures

Usage

First of all MPICH needs to be installed prior to usage. The library is used just like another .NET library in your programs. However the execution is relatively different than usual. The programs have to be executed using the MPI daemon called "mpiexec". You could look at more on how to configure a cluster in the MPICH documentation. To run the process in n processor or processes “-n” switch needs to be given as a command line argument followed by the name of the compiled program.

mpiexec -n 2 test.exe

Here is a very simple ping pong application using MPI Tools. You can find more samples on the MPI Tools Source code.

#light
#I @"..\MPITools.Bindings\"
#r @"MPITools.Bindings.dll"
open MPITools
MPI.initialize()
 
let procSize = MPI.size()
let curProcess =  MPI.rank()
 
let pingpong() =
if curProcess = 0 then
let i =  0
MPI.send(i,1,0)
let b = MPI.receive(1,0)
()
elif curProcess = 1 then
let b = MPI.receive(0,0)
MPI.send(b+1,0,0)
pingpong()
MPI.finalize()

Conclusion

You can download MPI Tools from codeplex. Using MPI Tools, the distributed programs will be short, expressive and well typed with the help of the glorified type system of F#.

MPI Tools is built with F# 1.9.3.7 Compiler for the .NET Framework 2.0. However it would possibly work with any CLI implementation. In the future, some more MPI functions will be implemented, including some helper functions that hides the imperative style programming. and the side effects.

I hope it will help to solve your high computation problems effectively. Please feel free to ask questions or to contribute to the project.

Have fun!

Tags: , , , , , ,
Category: Articles, Concurrency, Distributed, F#, Functional Programming, Tools  |  2 Comments

MsBee, Write Compile Run .Net 1.1 Applications with Visual Studio 2005

28 December, 2006, 02:59

I was just willing to write .Net 1.1 applications from Visual Studio 2005. Maybe you may already know but I found that great utility MsBee. You need also .Net Framework 1.1 SDK to make it install and work if you don’t have and surprisingly 1.1 SDK has a size of more than 100Mb.

Anyway, it is good to write .Net Framework 1.1 targeted applications from Visual Studio 2005.

Tags: ,
Category: .Net, Tools  |  Comment

XML Notepad 2.0

2 September, 2006, 00:55

Friday surprise from Microsoft, XML Notepad has rewritten in .net framework 2.0 from C++. It seems that the team did a great job. And best part is it will be available on Codeplex as a  community project.

Here are the features that I liked at first view

  • Open source
  • Comes with source code, samples (look at the directory you’ve installed)
  • Very simple user interface
  • Intellisense support based on XSD.
  • Great navigation and adding of XML objects (like element, attribute, comment) simply by providing before and after points.
  • Copy cut paste XML objects
  • Find feature is also nice, it is possible to use regular expressions or Xpath language.
  • XSL Output after the XSLT transformation.
  • Error list similar façon of Visual Studio

You can try and download XML Notepad 2.0

XMLNotepad

Tags: ,
Category: Tools, XML  |  Comment

Another hidden .Net Tool usage basics: ilMerge

22 August, 2006, 00:18

Ilmerge is a great utility for any .net developer. What it does is basically merge the assemblies. Let’s say you have three dll and one exe file, you can just merge those 4 files to 1 file using ilmerge. 
As a matter of fact, it is easy to use, you don’t need any recompilation or any other modification. You don’t need to provide installers since there is only one file.

Ilmerge exists in two versions one for .Net 1.1 and the other is .Net 2.0

Basic Usage

Usage: ilmerge [/lib:directory]* [/log[:filename]] [/keyfile:filename [/delaysig
n]] [/internalize[:filename]] [/t[arget]:(library|exe|winexe)] [/closed] [/ndebu
g] [/ver:version] [/copyattrs [/allowMultiple]] [/xmldocs] [/attr:filename] [/pu
blickeytokens] [/wildcards] [/zeroPeKind] [/allowDup:type]* /out:filename <prima
ry assembly> [<other assemblies>...]

Although the help explains very well, here is the usage.

C:\Program Files\Default Company Name\ILMergeBinaryDistribution7>Ilmerge /t:exe
/out:outputfile.exe file1.exe file2.dll

/t us used for the output of the file, exe is for a console application, winexe is for windows forms application and library is for the dlls.

/out is for the name of the output file.

the last parameters are the assemblies to be compined. Attention here is that the first one should be the exe one instead of the class library, becuase it merges the following files to the first file.

Advanced Usage

ILmerge is also usable as a class library for your project. What you need to do is to add the ilmerge executable as a reference and use the namespace of it to do some tasks. The class library is very similar to the console commands.

Tags: , , ,
Category: .Net, Tools  |  1 Comment