Connecting to Tibco EMS from a C# Client

Introduction

This is a slightly esoteric post, but hopefully it will be of use somewhere.  Below is some reasonably simple code that shows the basic use cases for connecting to a Tibco EMS Server topic from a C# client using Tibco’s TIBCO.EMS.dll.

It actually points at a locally installed instance of EMS.  That is, the server code is running on the workstation.  If you have the Windows EMS installation you can just install it in the default path and the instructions below should work.  Alternatively it’s relatively simple to change the server name from ‘localhost’ and put a username and password in the code to connect to a secure EMS Server.

How to Use the Code

  • Get hold of the Windows Tibco EMS install, and install it on your workstation on the default path.  Note that Tibco do NOT provide this for free: you need to have paid them for a licence.  If you work for a large organization you will probably find you already have a licence.
  • Create a C# console application called ‘TestEMS’, and reference TIBCO.EMS.dll (the C# EMS assembly) from the install.
  • Paste in the code below.
  • Run the code.  You should see a console window showing that a message has been sent by a publisher and received by a subscriber.

If you want to connect to an existing EMS server in your organization all you need is the TIBCO.EMS.dll assembly, not a full client install. You need to change the appropriate parts of the code below to point to the server: change ‘localhost’ to the server name, and add a username and password if they are needed.  Obviously you don’t need to start the server locally if you do this.

What the Sample Does

As you can see, the code starts the local EMS server, then creates a topic publisher and a subscriber to the same topic.  It then sends a ‘Hello World’ message using the publisher.  The subscriber’s message handler prints out the message to the console when it receives it.

The sample code is slightly more sophisticated than the most basic use case.  It also shows how to set a property on a message, and then to how filter the subscription using a message selector based on the property.  The message is sent with an additional property of ‘Owner’ with value ‘Rich Newman’.  The subscriber is only listening for messages with an Owner property that contains the string ‘Rich Newman’.  You can see this if you change the name of the owner in the message that’s sent: the message listener will not get the message.

Code

using System;
using System.Diagnostics;
using System.Threading;
using TIBCO.EMS;

namespace TestEMS
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Test started");
            new Program().Run();
            Console.ReadLine();
        }

        private void Run()
        {
            StartEMSServer();
            CreateEMSServerTopicPublisher();
            CreateClientTopicSubscriber("Owner LIKE '%Rich Newman%'"); // Pass "" for no message selector
            EMSServerPublishThisMessage("Hello World""Owner""Rich Newman");
        }

        #region EMS Server
        private const string tibcoEMSPath = @"C:\tibco\ems\5.0\bin\";
        private readonly string tibcoEMSExecutable = tibcoEMSPath + "tibemsd.exe";
        private Process tibcoEMSProcess;
        public void StartEMSServer()
        {
            tibcoEMSProcess = new Process();
            ProcessStartInfo processStartInfo = new ProcessStartInfo(tibcoEMSExecutable);
            tibcoEMSProcess.StartInfo = processStartInfo;
            processStartInfo.WorkingDirectory = tibcoEMSPath;
            bool started = tibcoEMSProcess.Start();
            Thread.Sleep(500);
        }

        TopicConnection publisherConnection;
        TopicSession publisherSession;
        TopicPublisher emsServerPublisher;
        private void CreateEMSServerTopicPublisher()
        {
            TopicConnectionFactory factory = new TIBCO.EMS.TopicConnectionFactory("localhost");
            publisherConnection = factory.CreateTopicConnection(""""); // Username, password
            publisherSession = publisherConnection.CreateTopicSession(falseSession.AUTO_ACKNOWLEDGE);
            Topic generalTopic = publisherSession.CreateTopic("GeneralTopic");
            emsServerPublisher = publisherSession.CreatePublisher(generalTopic);

            publisherConnection.Start();
        }

        internal void EMSServerPublishThisMessage(string messagestring propertyNamestring propertyValue)
        {
            TextMessage textMessage = publisherSession.CreateTextMessage();
            textMessage.Text = message;
            textMessage.SetStringProperty(propertyNamepropertyValue);
            emsServerPublisher.Publish(textMessage);
            Console.WriteLine("EMS Publisher published message: " + message);
        }

        #endregion

        #region EMS Client
        TopicConnection subscriberConnection;
        TopicSession subscriberSession;
        private void CreateClientTopicSubscriber(string messageSelector)
        {
            TopicConnectionFactory factory = new TIBCO.EMS.TopicConnectionFactory("localhost");
            subscriberConnection = factory.CreateTopicConnection("""");  // Username, password
            subscriberConnection.Start();
            subscriberSession = subscriberConnection.CreateTopicSession(falseSession.AUTO_ACKNOWLEDGE);
            Topic clientTopic = subscriberSession.CreateTopic("GeneralTopic");
            TopicSubscriber clientTopicSubscriber = subscriberSession.CreateSubscriber(clientTopicmessageSelectortrue);
            clientTopicSubscriber.MessageHandler += new EMSMessageHandler(test_MessageHandler);
        }

        void test_MessageHandler(object senderEMSMessageEventArgs args)
        {
            Console.WriteLine("EMS Client received message: " + args.Message.ToString());
        }

        #endregion
    }
}

Asynchronous Programming in .Net: Async and Await for Beginners

Introduction

There are several ways of doing asynchronous programming in .Net.  Visual Studio 2012 introduces a new approach using the ‘await’ and ‘async’ keywords.  These tell the compiler to construct task continuations in quite an unusual way.

I found them quite difficult to understand using the Microsoft documentation, which annoyingly keeps saying how easy they are.

This series of articles is intended to give a quick recap of some previous approaches to asynchronous programming to give us some context, and then to give a quick and hopefully easy introduction to the new keywords

Example

By far the easiest way to get to grips with the new keywords is by seeing an example.  For this initially I am going to use a very basic example: you click a button on a screen, it runs a long-running method, and displays the results of the method on the screen.

Since this article is about asynchronous programming we will want the long-running method to run asynchronously on a background thread.  This means we need to marshal the results back on to the user interface thread to display them.

In the real world the method could be running a report, or calling a web service.  Here we will just use the method below, which sleeps to simulate the long-running process:

        private string LongRunningMethod(string message)
        {
            Thread.Sleep(2000);
            return "Hello " + message;
        }

The method will be called asynchronously from a button click method, with the results assigned to the content of a label.

Coding the Example with Previous Asynchronous C# Approaches

There are at least five standard ways of coding the example above in .Net currently.  This has got so confusing that Microsoft have started giving the various patterns acronyms, such as the ‘EAP‘ and the ‘APM‘.   I’m not going to talk about those as they are effectively deprecated.  However it’s worth having a quick look at how to do our example using some of the other approaches.

Coding the Example by Starting our Own Thread

This simple example is fairly easy to code by just explicitly starting a new thread and then using Invoke or BeginInvoke to get the results back onto the UI thread.  This should be familiar to you:

        private void Button_Click_1(object sender, RoutedEventArgs e)
        {
            new Thread(() => { 
                string result = LongRunningMethod("World");
                Dispatcher.BeginInvoke((Action)(() => Label1.Content = result)); 
            }).Start();
            Label1.Content = "Working...";
        }

We start a new thread and hand it the code we want to run.  This calls the long-running method and then uses Dispatcher.BeginInvoke to call back onto the user interface thread with the result and update our label.

Note that immediately after we start the new thread we set the content of our label to ‘Working…’.  This is to show that the button click method continues immediately on the user interface thread after the new thread is started.

The result is that when we click the button our label says ‘Working…’ almost immediately, and then shows ‘Hello World’ when the long-running method returns.  The user interface will remain responsive whilst the long-running thread is running.

Coding the Example Using the Task Parallel Library (TPL)

More instructive is to revisit how we would do this with tasks using the Task Parallel Library.  We would typically use a task continuation as below.

        private void Button_Click_2(object sender, RoutedEventArgs e)
        {
            Task.Run<string>(() => LongRunningMethod("World"))
                .ContinueWith(ant => Label2.Content = ant.Result, 
                              TaskScheduler.FromCurrentSynchronizationContext());
            Label2.Content = "Working...";
        }

Here we’ve started a task on a background thread using Task.Run.  This is a new construct in .Net 4.5.  However, it is nothing more complicated than Task.Factory.StartNew with preset parameters.  The parameters are the ones you usually want to use.  In particular Task.Run uses the default Task Scheduler and so avoids one of the hidden problems with StartNew.

The task calls the long-running method, and does so on a threadpool thread.  When it is done a continuation runs using ContinueWith.  We want this to run on the user interface thread so it can update our label.  So we specify that it should use the task scheduler in the current synchronization context, which is the user interface thread when the task is set up.

Again we update the label after the task call to show that it returns immediately.  If we run this we’ll see a ‘Working…’ message and then ‘Hello World’ when the long-running method returns.

Coding the Example Using Async and Await

Code

Below is the full code for the async/await implementation of the example above.  We will go through this in detail.

       private void Button_Click_3(object sender, RoutedEventArgs e)
        {
            CallLongRunningMethod();
            Label3.Content = "Working...";        
        }

        private async void CallLongRunningMethod()
        {
            string result = await LongRunningMethodAsync("World");
            Label3.Content = result;
        }

        private Task<string> LongRunningMethodAsync(string message)
        {
            return Task.Run<string>(() => LongRunningMethod(message));
        }

        private string LongRunningMethod(string message)
        {
            Thread.Sleep(2000);
            return "Hello " + message;
        }

Asynchronous Methods

The first thing to realize about the async and await keywords is that by themselves they never start a thread.  They are a way of controlling continuations, not a way of starting asynchronous code.

As a result the usual pattern is to create an asynchronous method that can be used with async/await, or to use an asynchronous method that is already in the framework.  For these purposes a number of new asynchronous methods have been added to the framework.

To be useful to async/await the asynchronous method has to return a task.  The asynchronous method has to start the task it returns as well, something that maybe isn’t so obvious.

So in our example we need to make our synchronous long-running method into an asynchronous method.  The method will start a task to run the long-running method and return it.  The usual approach is to wrap the method in a new method.   It is usual to give the method the same name but append ‘Async’.  Below is the code to do this for the method in our example:

        private Task<string> LongRunningMethodAsync(string message)
        {
            return Task.Run<string>(() => LongRunningMethod(message));
        }

Note that we could use this method directly in our example without async/await.  We could call it and use ‘ContinueWith’ on the return value to effect our continuation in exactly the same way as in the Task Parallel Library code above.  This is true of the new async methods in the framework as well.

Async/Await and Method Scope

Async and await are a smart way of controlling continuations through method scope.  They are used as a pair in a method as shown below:

        private async void CallLongRunningMethod()
        {
            string result = await LongRunningMethodAsync("World");
            Label3.Content = result;
        }

Here async is simply used to tell the compiler that this is an asynchronous method that will have an await in it.  It’s the await itself that’s interesting.

The first line in the method calls LongRunningMethodAsync, clearly.  Remember that LongRunningMethodAsync is returning a long-running task that is running on another thread.  LongRunningMethodAsync starts the task and then returns reasonably quickly.

The await keyword ensures that the remainder of the method does not execute until the long-running task is complete.  It sets up a continuation for the remainder of the method. Once the long-running method is complete the label content will update: note that this happens on the same thread that CallLongRunningMethod is already running on, in this case the user interface thread.

However, the await keyword does not block the thread completely.  Instead control is returned to the calling method on the same thread.  That is, the method that called CallLongRunningMethod will execute at the point after the call was made.

The code that calls LongRunningMethod is below:

        private void Button_Click_3(object sender, RoutedEventArgs e)
        {
            CallLongRunningMethod();
            Label3.Content = "Working...";        
        }

So the end result of this is exactly the same as before.  When the button is clicked the label has content ‘Working…’ almost immediately, and then shows ‘Hello World’ when the long-running task completes.

Return Type

One other thing to note is that LongRunningMethodAsync returns a Task<string>, that is, a Task that returns a string.  However the line below assigns the result of the task to the string variable called ‘result’, not the task itself.

string result = await LongRunningMethodAsync("World");

The await keyword ‘unwraps’ the task.  We could have attempted to access the Result property of the task (string result = LongRunningMethodAsync(“World”).Result.  This would have worked but would have simply blocked the user interface thread until the method completed, which is not what we’re trying to do.

I’ll discuss this further below.

Recap

To recap, the button click calls CallLongRunningMethod, which in turn calls LongRunningMethodAsync, which sets up and runs our long-running task.  When the task is set up (not when it’s completed) control returns to CallLongRunningMethod, where the await keyword passes control back to the button click method.

So almost immediately the label content will be set to “Working…”, and the button click method will exit, leaving the user interface responsive.

When the task is complete the remainder of CallLongRunningMethod executes as a continuation on the user interface thread, and sets the label to “Hello World”.

Async and Await are a Pair

Async and await are always a pair: you can’t use await in a method unless the method is marked async, and if you mark a method async without await in it then you get a compiler warning.  You can of course have multiple awaits in one method as long as it is marked async.

Aside: Using Anonymous Methods with Async/Await

If you compare the code for the Task Parallel Library (TPL) example with the async/await example you’ll see that we’ve had to introduce two new methods for async/await: for this simple example the TPL code is shorter and arguably easier to understand.  However, it is possible to shorten the async/await code using anonymous methods, as below. This shows how we can use anonymous method syntax with async/await, although I think this code is borderline incomprehensible:

        private void Button_Click_4(object sender, RoutedEventArgs e)
        {
            new Action(async () =>
            {
                string result = await Task.Run<string>(() => LongRunningMethod("World"));
                Label4.Content = result;
            }).Invoke();
            Label4.Content = "Working...";
        }

Using the Call Stack to Control Continuations

Overview of Return Values from Methods Marked as Async

There’s one other fundamental aspect of async/await that we have not yet looked at.  In the example above our method marked with the async keyword did not return anything.  However, we can make all our async methods return values wrapped in a task, which means they in turn can be awaited on further up the call stack.  In general this is considered good practice: it means we can control the flow of our continuations more easily.

The compiler makes it easy for us to return a value wrapped in a task from an async method.  In a method marked async the ‘return’ statement works differently from usual.  The compiler doesn’t simply return the value passed with the statement, but instead wraps it in a task and returns that instead.

Example of Return Values from Methods Marked as Async

Again this is easiest to see with our example.  Our method marked as async was CallLongRunningMethod, and this can be altered to return the string result to the calling method as below:

        private async Task<string> CallLongRunningMethodReturn()
        {
            string result = await LongRunningMethodAsync("World");
            return result;
        }

We are returning a string (‘return result’), but the method signature shows the return type as Task<string>.  Personally I think this is a little confusing, but as discussed it means the calling method can await on this method.  Now we can change the calling method as below:

        private async void Button_Click_5(object sender, RoutedEventArgs e)
        {
            Label5.Content = "Working...";
            string result = await CallLongRunningMethodReturn();
            Label5.Content = result;
        }

We can await the method lower down the call stack because it now returns a task we can await on.  What this means in practice is that the code sets up the task and sets it running and then we can await the results from the task when it is complete anywhere in the call stack.  This gives us a lot of flexibility as methods at various points in the stack can carry on executing until they need the results of the call.

As discussed above when we await on a method returning type Task<string> we can just assign the result to a string as shown.  This is clearly related to the ability to just return a string from the method: these are syntactic conveniences to avoid the programmer having to deal directly with the tasks in async/await.

Note that we have to mark our method as ‘async’ in the method signature (‘private async void Button_Click_5′) because it now has an await in it, and they always go together.

What the Code Does

The code above has exactly the same result as the other examples: the label shows ‘Working…’ until the long-running method returns when it shows ‘Hello World’.  When the button is clicked it sets up the task to run the long-running method and then awaits its completion both in CallLongRunningMethodReturn and Button_Click_5.  There is one slight difference in that the click event is awaiting: previously it exited.  However, if you run the examples you’ll see that the user interface remains responsive whilst the task is running.

What’s The Point?

If you’ve followed all the examples so far you may be wondering what the point is of the new keywords.  For this simple example the Task Parallel Library syntax is shorter, cleaner and probably easier to understand than the async/await syntax.  At first sight async/await are a little confusing.

The answer is that for basic examples async/await don’t seem to me to be adding a lot of value, but as soon as you try to do more complex continuations they come into their own.  For example it’s possible to set up multiple tasks in a loop and write very simple code to deal with what happens when they complete, something that is tricky with tasks.  I suggest you look at the examples in the Microsoft documentation which do show the power of the new keywords.

Code

The full code for these examples is available to download.

Conclusion

This article has only covered the basics of the async/await keywords, although I think it’s addressed all the things that were confusing me when trying to learn about them from the Microsoft documentation.  There are some obvious things it hasn’t covered such as cancelling tasks, exception handling, unwrapping tasks (and why you might need to do that) and how to deal with the reentrancy problems that arise.  All of these are covered reasonably well in the documentation.

Personally I think async and await are far from intuitive: the compiler is performing some magic of a kind we don’t usually see in C#.  The result is that we are yielding control in the middle of a method to the calling method until some other task is complete.  Of course we can do similar things with regular task continuations, but the syntax makes regular continuations look slightly less magical.

However, async/await are a powerful way of controlling multithreaded code once you understand what they are doing.  They can make fairly complex threading look simple.

Why Starting a New Task in the Task Parallel Library (TPL) Doesn’t Always Start a New Thread

The Task Parallel Library in .Net is a wonderful resource.  However, one thing that is a little confusing is that it is actually possible to start a task that runs on the same thread as the current code.  That is, we can do Task.StartNew and it still runs on the same thread.  This isn’t the behaviour we expect.

This can happen particularly when we are writing .Net user interface applications where we marshal data on to the UI thread.

Consider the sample code below, which runs in the button click event of a simple WPF application.  This is a not unusual pattern for a user interface application: we start a Task on a new thread and then perform a continuation on the user interface thread using the current synchronization context, which is the user interface thread synchronization context (TaskScheduler.FromCurrentSynchronizationContext()).  We would typically do this so that we could write the results of the Task into the user interface.  There are several articles on the internet that describe this.

private void button1_Click(object sender, RoutedEventArgs e)
{
    Debug.WriteLine("UI thread: " + Thread.CurrentThread.ManagedThreadId);
    Task.Factory.StartNew(() => 
        Debug.WriteLine("Task 1 thread: " + Thread.CurrentThread.ManagedThreadId))
    .ContinueWith(ant =>
    {
        Debug.WriteLine("Continuation on UI thread: " + Thread.CurrentThread.ManagedThreadId);
        Task.Factory.StartNew(() 
            => Debug.WriteLine("Task 2 thread is UI thread: " + Thread.CurrentThread.ManagedThreadId));
    }, TaskScheduler.FromCurrentSynchronizationContext());
}

However, in the code above we then start a new Task from the continuation (task 2).  This runs on the user interface thread by default.  This is almost certainly not what we are trying to do.  We probably want to run something on a background thread, which is what usually happens when we start a Task.  As a result it’s slightly hazardous behaviour; we can easily end up blocking the user interface thread because we think the code is running on a background thread.

The output from a couple of button clicks using this code is as below:

UI thread: 8
Task 1 thread: 9
Continuation on UI thread: 8
Task 2 thread is UI thread: 8
UI thread: 8
Task 1 thread: 10
Continuation on UI thread: 8
Task 2 thread is UI thread: 8

The reason for this behaviour is that the user interface thread synchronization context is still in force when we start our second task.

Fortunately, in this case at least there is a simple solution, even if it isn’t so obvious.  We can tell the new Task to run explicitly using the default synchronization context instead of the user interface synchronization context when we start it.  The simplest syntax for this changes the code as below.  Notice it passes TaskScheduler.Default as an argument to Task.Start.  The terminology here for a user interface application is a little confusing.  The ‘current synchronization context’ is accessed using TaskScheduler.FromCurrentSynchronizationContext, which we typically do from the user interface thread.  Then the ‘current’ context that results in delegates being queued back to the user interface thread.  The ‘default synchronization context’ is the context that results in delegates being queued to the thread pool.

private void button1_Click(object sender, RoutedEventArgs e)
{
    Debug.WriteLine("UI thread: " + Thread.CurrentThread.ManagedThreadId);
    Task.Factory.StartNew(() => 
        Debug.WriteLine("Task 1 thread: " + Thread.CurrentThread.ManagedThreadId))
    .ContinueWith(ant =>
    {
        Debug.WriteLine("Continuation on UI thread: " + Thread.CurrentThread.ManagedThreadId);
        Task task2 = new Task(() 
            => Debug.WriteLine("Task 2 thread: " + Thread.CurrentThread.ManagedThreadId));
        task2.Start(TaskScheduler.Default);
    }, TaskScheduler.FromCurrentSynchronizationContext());
}

The output from this is now as we would expect:

UI thread: 9
Task 1 thread: 10
Continuation on UI thread: 9
Task 2 thread: 11
UI thread: 9
Task 1 thread: 11
Continuation on UI thread: 9
Task 2 thread: 10

For more detailed discussion on this see:

http://stackoverflow.com/questions/6800705/why-is-taskscheduler-current-the-default-taskscheduler

http://stackoverflow.com/questions/12245935/is-task-factory-startnew-guaranteed-to-use-another-thread-than-the-calling-thr

Delegate Syntax in C# for Beginners

Introduction

I have been programming with C# since it came out but I still find the delegate syntax confusing.  This is at least partially because Microsoft have changed the recommended syntax regularly over the years.  This article is a quick recap of the various syntaxes.  It also looks at some of the issues with using them in practice.  It’s worth knowing about all the various syntaxes as you will almost certainly see all of them used.

This article is just a recap: it assumes that you know what a delegate is and why you’d want to use one.

.Net and Visual Studio Versions

The first thing to note is that you can use any of these syntaxes as long as you are using Visual Studio 2008 or later and targeting .Net 2.0 or later.

Named methods were available in .Net 1.0, anonymous methods were introduced in .Net 2.0, and lambda expressions were introduced in .Net 3.0.  However, like much of .Net 3.0, which is based on the .Net 2.0 assemblies, lambda expressions will compile to .Net 2.0 assuming you have the appropriate version of Visual Studio.

Note also that lambda expressions can do (almost) everything anonymous methods can do, and effectively supersede them as the preferred way of writing inline delegate code.

Code

A listing of the code for this article is availableThe complete working program is also available.

The Delegate

For all of these examples we need a delegate definition.  We’ll use the one below initially.

        private delegate void TestDel(string s);

Named Methods

Named methods are perhaps the easiest delegate syntax to understand intuitively.  A delegate is a typesafe method pointer.  So we define a method:

        private void Test(string s)
        {
            Console.WriteLine(s);
        }

Now we create an instance of our method pointer (the delegate above) and point it at our method.  Then we can call our method by invoking the delegate.  The code below prints out ‘Hello World 1’.  This is easy enough, but all a little cumbersome.

            TestDel td = new TestDel(Test);
            td("Hello World 1");

There’s one slight simplification we can use.  Instead of having to explicitly instantiate our delegate with the new keyword we can simply point the delegate directly at the method, as shown below.  This syntax does exactly the same thing as the syntax above, only (maybe) it’s slightly clearer.

            TestDel td2 = Test;
            td2("Hello World 2");

There is an MSDN page on named methods.

Anonymous Methods

The anonymous method syntax was introduced to avoid the need to create a separate method.  We just create the method in the same place we create the delegate.  We use the ‘delegate’ keyword as below.

            TestDel td3 = 
                delegate(string s)
                {
                    Console.WriteLine(s);
                };
            td3("Hello World 3");

Now when we invoke td3 (in the last line) the code between the curly braces executes.

One advantage of this syntax is that we can capture a local variable in the calling method without explicitly passing it into our new method.  We can form a closure.  Since in this example we don’t need to pass our string in as a parameter we use a different delegate:

        private delegate void TestDelNoParams();

We can use this as below.  Note that the message variable is not explicitly passed into our new method, but can nevertheless be used.

            string message = "Hello World 4";
            TestDelNoParams td4 = 
                delegate()
                {
                    Console.WriteLine(message);
                };
            td4();

There is an MSDN page on anonymous methods.

Lambda Expressions

Lambda expressions were primarily introduced to support Linq, but they can be used with delegates in a very similar way to anonymous methods.

There are two basic sorts of lambda expressions.  The first type is an expression lambda.  This can only have one statement (an expression) in its method.  The syntax is below.

            TestDel td5 =  s => Console.WriteLine(s);
            td5("Hello World 5");

The second type is a statement lambda: this can have multiple statements in its method as below.

            string message2 = "Hello World 8";
            TestDel td6 =
                s => 
                { 
                    Console.WriteLine(s); 
                    Console.WriteLine("Hello World 7");
                    Console.WriteLine(message2);
                };
            td6("Hello World 6");

Note that this example also shows a local variable being captured (a closure being created).  We can also capture variables with expression lambdas.

There is an MSDN page on lambda expressions.

Return Values

Nearly all of the examples above can be extended in a simple way to return a value.  The exception is expression lambda which cannot return a value. Doing this is usually an obvious change: we change our delegate signature so that the method it points to returns a value, and then we simply change the method definition to return a value as usual.  For example the statement lambda example above becomes as below.  The invocation of tdr6 now returns “Hello ” + message2, which we write to the console after the invocation returns:

            string message2 = "World 8";
            TestDelReturn tdr6 =
                s =>
                {
                    Console.WriteLine(s);
                    Console.WriteLine("Hello World 7");
                    return "Hello " + message2;
                };
            Console.WriteLine(tdr6("Hello World 6"));

The full list of all the examples above modified to return a value can be seen in the code listing in the method ExamplesWithReturnValues.

Events

All of these syntaxes can be used to set up a method to be called when an event fires.  To add a delegate instance to an event we used the ‘+=’ syntax of course.  Suppose we define an event of type TestDel:

        private event TestDel TestDelEventHandler;

We can add a delegate instance to this event using any of the syntaxes in an obvious way.  For example, to use a statement lambda the syntax is below.  This looks a little odd, but certainly makes it easier to set up and understand event handling code.

            TestDelEventHandler += s => { Console.WriteLine(s); };
            TestDelEventHandler("Hello World 24");

Examples of setting up events using any of the syntaxes above can be found in the code listing.

Passing Delegates into Methods as Parameters: Basic Case

Similarly all of the syntaxes can be used to pass a delegate into a method, which again gives some odd-looking syntax.  Suppose we have a method as below that takes a delegate as a parameter.

        private void CallTestDel(TestDel testDel)
        {
            testDel("Hello World 30");
        }

Then all of the syntaxes below are valid:

            CallTestDel(new TestDel(Test));  // Named method
            CallTestDel(Test);               // Simplified named method
            CallTestDel(delegate(string s) { Console.WriteLine(s); });  // Anonymous method
            CallTestDel(s => Console.WriteLine(s));  // Expression lambda
            CallTestDel(s => { Console.WriteLine(s); Console.WriteLine("Hello World 32"); });  // Statement lambda

Passing Delegates into Methods as Parameters: When You Actually Need a Type of ‘Delegate’

Now suppose we have a method as below that expects a parameter of type Delegate.

        private void CallDelegate(Delegate del)
        {
            del.DynamicInvoke(new object[] { "Hello World 31" });
        }

The Delegate class is the base class for all delegates, so we can pass any delegate into CallDelegate.  However, because the base Delegate class doesn’t know the method signature of the delegate we can’t call Invoke with the correct parameters on the Delegate instance.  Instead we call DynamicInvoke with an object[] array of parameters as shown.

Note that there are some methods that take Delegate as a parameter in the framework (e.g. BeginInvoke on a WPF Dispatcher object).

There’s a slightly unobvious change to the ‘Basic Case’ syntax above if we want to call this method using the anonymous method or lambda expression syntax.  The code below for calling CallDelegate with an expression lambda does NOT work.

            CallDelegate(s => Console.WriteLine(s));  // Expression lambda

The reason is that the compiler needs to create a delegate of an appropriate type, cast it to the base Delegate type, and pass it into the method.  However, it has no idea what type of delegate to create.

To fix this we need to tell the compiler what type of delegate to create (TestDel in this example).  We can do this with the usual casting syntax (and a few more parentheses) as shown below.

            CallDelegate((TestDel)(s => Console.WriteLine(s)));  // Expression lambda

This looks a little strange as we don’t normally need a cast when assigning a derived type to a base type, and in any case we’re apparently casting to a different type to the type the method call needs.  However, this syntax is simply to tell the compiler what type of delegate to create in the first place: the cast to the base type is still implicit.

We need to do this for any of the syntaxes apart from the very basic named method syntax (where we’re explicitly creating the correct delegate):

            CallDelegate(new TestDel(Test));  // Named method
            CallDelegate((TestDel)Test);      // Simplified named method
            CallDelegate((TestDel)delegate(string s) { Console.WriteLine(s); });  // Anonymous method
            CallDelegate((TestDel)(s => Console.WriteLine(s)));  // Expression lambda
            CallDelegate((TestDel)(s => { Console.WriteLine(s); Console.WriteLine("Hello World 32"); }));  // Statement lambda

Actions/Funcs

There is one further simplification that we can use in the examples in this article.  Instead of defining our own delegates (TestDel etc.) we can use the more generic Action and Func delegates provided in the framework.  So, for example, everywhere we use TestDel, which takes a string and returns void, we could use Action<string> instead, since it has the same signature.

Closures in C#

Introduction

There seems to be some confusion about closures in C#: people are mystified as to what they are and there’s even an implication that they don’t work the way you’d expect.

As this short article will explain they are actually quite simple, and do work the way you’d expect if you’re an object oriented programmer.  The article will also briefly look at why there is confusion surrounding them, and discuss whether they are an appropriate tool in an object oriented program.

Closures

Wikipedia defines a closure as ‘a first-class function with free variables that can be bound in the lexical environment’.  What that means is that a ‘closure’ is a function that can access variables from the environment where it is declared without them being explicitly passed in as parameters.  In object-oriented programming a method in a class is a closure of sorts: it can access fields of the class directly without needing them to be passed in.  However, more usually in C# ‘closure’ refers to a function declared as an anonymous delegate that uses variables that are not explicitly passed into it, but are available at the point it is created.

Basic Example

Consider the code below:

        internal void MyFunction()
         {
             int x = 1;
             Action action = () => { x++; Console.WriteLine(x); };
             action();              // Outputs '2'
         }

Here we define an anonymous delegate (a function) ‘action’, and call it.  This increments x and outputs it to the console, even though x isn’t passed into it.  x is simply declared in the ‘lexical environment’ (the calling method).

‘action’ is a closure, and we can say it is ‘closed over’ x, and that x is an ‘upvalue’.  Note that it’s only a closure because of the way it uses x.  Not all anonymous delegates are closures.

By the way, in the Java community anonymous functions frequently are referred to as ‘closures’.  (The Java community has been debating whether ‘closures’ should be added to the language for some time.)

Where’s the Confusion?

The example above is pretty clear and simple.  So how has it caused confusion?

The answer is that the value of x in MyFunction after the ‘action’ call is now 2.  Furthermore, x is completely shared between MyFunction and the action delegate: any code that changes the value in one changes the value in the other.

Consider the code below:

        internal void MyFunction()
         {
             int x = 1;
             Action action = () => { x++; Console.WriteLine(x); };
             action();              // Outputs '2'
             x++;
             action();              // Outputs '4'
         }

Here we call ‘action’, increment x in the calling method (MyFunction), and then call ‘action’ again.  Overall we started with x at 1, and incremented it 3 times, twice in our ‘action’ delegate and once in the calling method.  So it’s no surprise that the shared variable ends up with a value of 4.

This shows that we can change our ‘upvalue’ in the calling method and it is then reflected in our next call to the function: x is genuinely shared in this example.

Whilst this is a little odd (see below) it’s perfectly logical: x is shared between the calling method and any calls to our ‘action’ function.  There’s only one version of x.

This isn’t the way closures work in most functional programming languages (not that you could easily implement the example above, since all variables are immutable in functional languages).  The concept of closures has come from functional languages, so many people are surprised to find them working this way in C#.  There is more on this later.

Closures and Scope

This becomes even odder if we allow the local variable x to go out of scope (which would usually lead to it being destroyed), but retain a reference to the delegate that uses it:

        internal void Run()
        {
             Action action = MyFunction();
             action();             // Outputs '5'
        }

        internal Action MyFunction()
        {
             int x = 1;
             Action action = () => { x++; Console.WriteLine(x); };
             action();              // Outputs '2'
             x++;
             action();              // Outputs '4'
             return action;
        }

Here our Run method retrieves the action delegate from MyFunction and calls it.  When it calls it the value of x is 4 (from the activity in MyFunction), so it increments that and outputs 5.  At this point MyFunction is out of scope so the local variable x would normally have been destroyed.

Again, logically this is what we’d expect, but it looks strange.

This also gives an indication that closures are hard to implement.

A Little Odd?

For an object-oriented programmer this is a little odd.  This is because we’re not used to being able to share local variables with a separate method in this way.

Normally if we want a method to act on a local variable we have to pass it in as a parameter.  If the local variable is a value type as above it gets copied, and changing it in the method will not affect its value in the calling method.  So if we wanted to use it in the calling method we’d have to pass it back explicitly as a return value or an output parameter.

Of course, there are good reasons why we don’t usually allow a method to access any variable in a calling method (quite apart from the practicalities of actually being able to do it with methods other than anonymous functions).  These are to do with encapsulation and ensuring we can maintain state in a way that’s easy to deal with.  We only really allow data to be shared at a class level, or globally if we’re using static variables, although in general we try to keep them to a minimum.

So in some ways closures of this kind break our usual object-oriented encapsulation.  My feeling is that they should be used sparingly in regular object-oriented code as a result.

Other writers have gone further than this, because if you don’t understand that an upvalue is fully shared between the anonymous function and the calling code you can get unexpected behaviour.  See, for example, this article ‘Closures in C# Can Be Evil’.

Conclusion

The concepts behind closures in C# are actually fairly straightforward.  However, if we use them it’s important we understand them and the effects on scope, or we may get behaviour we don’t expect.

Table of Contents: Introduction to using Financial Products Markup Language (FpML) with Microsoft .NET Tools

Part 1 Introduction

An overview of how we can use Visual Studio to examine the FpML XSDs. Shows how to create a Visual Studio project containing the FpML schemas, and how to use that to navigate through them. Also shows how to validate the XML examples that are provided in the FpML download, both using Visual Studio and in C# code.

Part 2 How FpML Validates A Basic Trade Document

Looks at the structure of an FpML trade document in some detail. Explains how the FpML schemas fit together to validate such a document, and examines the use of base and extension types, and of substitution groups, in the XSDs.

Part 3 Generating .Net Classes From The FpML XSD Schema

Shows how to generate .Net classes in C# from the XML schema documents using Xsd.exe.  Explains how to load data into these classes from an FpML document using C# code, and to save the document back out again from the classes.

Part 4 Problems With Using Xsd.exe To Generate .Net Classes From The FpML XSD Schema

Goes through some of the problems with the C# code generated in part 3, and discusses how to fix them.

A Beginner’s Guide to calling a .NET Library from Excel

Introduction

It’s actually very easy to call a .NET library directly from Excel, particularly if you are using Visual Studio 2005. You don’t need Visual Studio Tools for Office. However there doesn’t seem to be an easy guide on the internet anywhere. MSDN help is quite good on the subject, but can be a little confusing. This article is an attempt to redress the situation.

This article was updated 24th August 2007 to cover Excel 2007 and to clarify the issues with intellisense.

A Basic Walk Through

We’ll start by walking through a very basic example. We’ll get Excel to call a .NET method that takes a string as input (for example “ World”) and returns “Hello” concatenated with that input string (so, for example, “Hello World”).

1. Create a C# Windows class library project in Visual Studio 2005 called ‘DotNetLibrary’. It doesn’t matter which folder this is in for the purposes of this example.

2. To call a method in a class in our library from Excel we need the class to have a default public constructor. Obviously the class also needs to contain any methods we want to call. For this walk through just copy and paste the following code into our default class file:

using System;
using System.Collections.Generic;
using System.Text;
 
namespace DotNetLibrary
{
    public class DotNetClass
    {
        public string DotNetMethod(string input)
        {
            return "Hello " + input;
        }
    }
}

That’s it: if you look at existing articles on the web, or read the MSDN help, you might think you need to use interfaces, or to decorate your class with attributes and GUIDs. However, for a basic interop scenario you don’t need to do this.

3. Excel is going to communicate with our library using COM. For Excel to use a COM library there need to be appropriate entries in the registry. Visual Studio can generate those entries for us.

To do this bring up the project properties (double-click ‘Properties’ in Solution Explorer). Then:
i) On the ‘Application’ tab click the ‘Assembly Information…’ button. In the resulting dialog check the ‘Make assembly COM-visible’ checkbox. Click ‘OK’.
ii) On the ‘Build’ tab check the ‘Register for COM interop’ checkbox (towards the bottom: you may need to scroll down).

4. Build the library.

5. Now start Excel and open a new blank workbook. Open the VBA code editor:
i) In Excel 2007 this is a little difficult to find. You have to get the Developer tab visible on the Ribbon if it’s not already set up. To do this click the Microsoft Office Button (top left of the screen), then click Excel Options (at the very bottom). Check the ‘Show Developer tab in the Ribbon’ checkbox in the resulting Options dialog. Click OK. This adds ‘Developer’ to the end of the ribbon menu: click this. Then click the ‘Visual Basic’ icon at the left end of the ribbon.
ii) In earlier versions of Office (2003, XP, 2000) just go to Tools/Macro/Visual Basic Editor on the menu bar.

6. We now need to include a reference to our new library. Select ‘References’ on the Visual Basic Editor’s ‘Tools’ menu. If you scroll down in the resulting dialog you should find that ‘DotNetLibrary’ is in the list. Check the checkbox alongside it and click ‘OK’.

7. Now open the code window for Sheet1 (double click Sheet1 in the Project window). Paste the VBA code below into the code window for Sheet1:

Private Sub TestDotNetCall()
Dim testClass As New DotNetClass
MsgBox testClass.DotNetMethod(“World”)
End Sub

8. Click anywhere in the code you’ve just pasted in and hit ‘F5’ to run the code. You should get a ‘Hello World’ message box.

Getting Intellisense Working in Excel

Whilst the VBA code above compiles and executes, you will discover that intellisense is not working in the code editor. This is because by default our library is built with a late binding (run-time binding) interface only. The code editor therefore doesn’t know about the types in the library at design time.

There are good reasons for only using a late-bound interface by default: with COM versioning libraries can become difficult with early-bound interfaces. In particular, if you change the early-bound interface by adding, for example, a method in between two existing methods you are likely to break existing clients as they are binding based on the order of the methods in the interface.

For similar reasons you are heavily encouraged to code your interface separately as a C# interface and then implement it on your class, rather than using the default public interface of the class as here. You then should not change that interface: you would implement a new one if it needed to change.

For more on this see:

http://msdn2.microsoft.com/en-us/library/system.runtime.interopservices.classinterfaceattribute(vs.80).aspx
http://msdn2.microsoft.com/en-us/library/system.runtime.interopservices.classinterfacetype(VS.80).aspx

However, we can build our library to use early bound interfaces, which means intellisense will be available. To do this we need to add an attribute from the System.Runtime.InteropServices namespace as below:

using System;
using System.Collections.Generic;
using System.Text;
using System.Runtime.InteropServices;
 
namespace DotNetLibrary
{
    [ClassInterface(ClassInterfaceType.AutoDual)]
    public class DotNetClass
    {
        public DotNetClass()
        {
        }
        public string DotNetMethod(string input)
        {
            return "Hello " + input;
        }
    }
}

If you change your code as above it will expose an ‘AutoDual’ interface to COM. This means it is still exposing the late-bound interface as before, but now also exposes an early-bound interface. This means intellisense will work.

To get this working:

1. Save your workbook and close Excel. Excel will lock the DotNetLibrary dll and prevent Visual Studio from rebuilding it unless you close it. Remember you need to save your new code module. If you are using Excel 2007 you will need to save as type Excel Macro-Enabled Workbook (*.xlsm). In earlier versions you can just save as a standard xls.

2. Go back into Visual Studio, change the DotNetClass as shown above, and rebuild the library.

3. Re-open your Excel spreadsheet. Once again if you are using Excel 2007 there is an extra step: you need to explicitly enable macros. A warning bar will appear beneath the ribbon saying the ‘Macros have been disabled’. Click the ‘Options’ button next to this, select ‘Enable this content’, and click OK.

4. Get the VBA code window up again (see item 5 above).

5. Excel can get confused about the interface changes unless you re-reference the library. To do this go to Tools/References. The DotNetLibrary reference should be near the top of the list now. Uncheck it and close the window. Now open the window again, find the library in the list, and re-check it (trust me, you need to do this).

6. Now run the code and it should still work (put a breakpoint in the routine and hit F5).

7. Enter a new line in the routine after the ‘MsgBox’ line, and type ‘testClass.’. When you hit the ‘.’ you should get an intellisense dropdown which shows that DotNetMethod is available. See below.

Intellisense in Excel

Let me re-iterate that this works and is fine for development, but for release code you are better off using the default late binding interfaces unless you understand the full versioning implications. That is, you should remove the ClassInterface attribute from your code when you do a release.

Deployment

In the example here we are using Visual Studio to register our .NET assembly on the workstation so that Excel can find it via COM interop. However, if we try to deploy this application to client machines we’re not going to want to use Visual Studio.

Microsoft have provided a command-line tool, regasm.exe, which can be used to register .NET assemblies for COM interop on client workstations. It can also be used to generate a COM type library (.tlb) separate from the main library (.dll), which is considered good practice in general.

As usual with .NET assemblies you have the choice of strong-naming your assembly and installing it in the GAC, or of not strong-naming it and including it in a local path. If you have strong-named your assembly and installed it in the GAC all you need to do is bring up a Visual Studio 2005 command prompt and run:

regasm DotNetLibrary.dll

If you have not strong-named your assembly you need to tell regasm.exe where it is so that it can find it to register it. To do this you need to run the command below, where c:\ExcelDotNet is the path where DotNetLibrary.dll can be found. This works fine, although it will warn you that you should really strong-name your assembly:

regasm /codebase c:\ExcelDotNet\DotNetLibrary.dll

Note that you can unregister an assembly with the /u option of regasm.

For more detail on this see http://msdn2.microsoft.com/en-us/library/tzat5yw6(vs.80).aspx

Debugging into .NET from Excel

You may want to debug from Excel into your class library. To do this:

1. Using Visual Studio 2005 bring up the Properties window for the class library.

2. Go to the Debug tab and select the ‘Start external program’ option under ‘Start Action’. In the textbox alongside enter the full path including file name to Excel.exe for the version of Excel you are using (usually in Program Files/Microsoft Office/Office).

3. On the same Debug tab under ‘Command line arguments’ enter the full path including file name to your test workbook (the .xls file, or .xlsm if you are using Excel 2007). Once you’re done it should something like below::

Project Properties for Excel

4. Now put a breakpoint in the code (in our example the sensible place is in method DotNetMethod) and hit F5 in the .NET project. The .NET code should compile and Excel should start with your workbook opened. If you now run the VBA code to call the .NET library again, as above, you should find that the code will break at the breakpoint you set in the .NET code.

Possible Problem with these Examples

One problem we have had with these examples is that Excel can get confused about which version of the .NET Framework to load if you have more than one version installed. If this happens you will get an automation error when you try to instantiate .NET objects at runtime from Excel. The .NET types will appear correctly in the Excel object browser.

The workaround for this is to tell Excel explicitly that the version of the .NET Framework that you are using is supported. To do this create a text file called Excel.exe.config and put it in the same directory as Excel.exe itself. The file should contain the text below (with the version number replaced with the .NET Framework version you are using):

<?xml version="1.0"?>
<configuration>
  <startup>
    <supportedRuntime version="v2.0.50727"/>
  </startup>
</configuration>
 

References:

Index page from MSDN
http://msdn2.microsoft.com/en-us/library/zsfww439.aspx

More on COM Interop from COM clients into .NET:
http://www.codeproject.com/vb/net/MusaExposingCOM.asp

A COM Class Wizard for C#
http://www.codeproject.com/csharp/cscomtemplate.asp

Guidelines for COM Interoperability from .NET
http://blogs.gotdotnet.com/heaths/archive/2005/03/09/391358.aspx

In Defense of regasm /codebase
http://weblogs.asp.net/savanness/archive/2003/05/29/7749.aspx

Excel/.NET versioning problems
http://krgreenlee.blogspot.com/2006/01/software-running-excel-with-net-11.html