Rich Newman

January 28, 2008

Problems with Using xsd.exe to Generate .NET Classes from the FpML XSD Schema (Introduction to using FpML with .NET Tools Part 4)

Introduction

Part 3 of this series of articles showed how we can generate .NET classes from the FpML XSD schema using xsd.exe. It showed how we can then use standard .NET serialization syntax to populate the classes from FpML documents, and vice versa.

However, as mentioned in part 3, xsd.exe generates buggy code when used with the FpML XSDs. This article will go through some of the problems with this code and describe how to fix them.

Corrected .NET Classes

Corrected generated classes, which are the end result of the work in this article, are available. This code has been corrected such that it appears to work in most circumstances. However we cannot be certain that it is free of all bugs.

The corrected code is based on FpML 4.2. The main aim of this article is to give you a starting point if you are trying to generate .NET code from other versions of the specification.

Please note also that this article was written using the xsd.exe supplied with Visual Studio 2005 Service Pack 1. Different versions of Visual Studio, or Visual Studio 2005 without the service pack applied, may give different results. See the discussion in the Comments for this article for more details.

Generating the Code

Part 3 of this series of articles described how to use xsd.exe to generate C# code from the FpML schemas. This gave us a file called fpml-main-4-2_xmldsig-core-schema.cs, which contains a Document class which should be the root class for our serialization. Unfortunately if we attempt to create an XmlSerializer object using this code we get exceptions.

Problem 1: Substitution Groups and Extension

The most fundamental problem with our generated code is that xsd.exe has got confused about the substitution groups and associated extension that were discussed in some detail in part 2 of this series of articles.

In particular it has not decorated the Product property of our Trade class correctly to allow it to deal with all the possible products correctly.

In part 2 we saw that the product element in the FpML is replaced with various individual product types using the substitutionGroup syntax. The product element is a sub-element of the trade element in the XML. As a result in our C# code we have a Trade class which contains a data member of type Product and a public property that gets and sets this. Excerpts from our generated code are as below:

[System.CodeDom.Compiler.GeneratedCodeAttribute("xsd", "2.0.50727.42")]
[System.SerializableAttribute()]
[System.Diagnostics.DebuggerStepThroughAttribute()]
[System.ComponentModel.DesignerCategoryAttribute("code")]
[System.Xml.Serialization.XmlTypeAttribute(Namespace="http://www.fpml.org/2005/FpML-4-2")]
public partial class Trade {
 
    private Product productField;
 
    public Product product {
        get {
            return this.productField;
        }
        set {
            this.productField = value;
        }
    }

The problem with this code is that, as we saw in part 2, we need to be able to put a Swap object (for example) in to our Product data member. We have a Swap class in our code, and this inherits Product, as we’d expect. So we can store a Swap object in our Product data member.

However, .NET doesn’t know how to serialize or deserialize a swap element in the XML into our product property without being told. In particular, if the deserializer finds an element called ‘swap’ where it’s expecting ‘product’ it doesn’t know that it should deserialize into a Swap object. As we saw in part 2, in ird_ex01_vanilla_swap.xml there is a ‘swap’ element where you might expect a ‘product’ element.

This is very easily fixed in this case. We simply decorate the Product property with an XmlElementAttribute that tells the serializer what to do:

    [System.Xml.Serialization.XmlElementAttribute("swap", typeof(Swap))]
    public Product product {
        get {
            return this.productField;
        }
        set {
            this.productField = value;
        }
    }

This does solve the problem for swap. However, there are multiple other product types that need the XmlElementAttribute added to Product to get the serialization to work (there are over twenty in fact). Also it turns out that this problem isn’t limited to Product. There are other elements that use the substitution group and extension property in this way and have the same problem

At first glance it appears there is no easy solution to this: we are going to have to go through all the possible places where there might be errors and correct the code by hand. However, there’s a much easier solution.

Problem 1 Solution

For simple examples of this problem xsd.exe will correctly generate the required XmlElementAttribute. It’s not immediately obvious why it fails to do so with the full FpML schemas.

It turns out that the reason xsd.exe gets this wrong is because the FpML schemas are spread across multiple files. If we create one big xsd file containing all of the FpML schema files and then run xsd.exe on this the problem goes away. I’ll leave you to draw your own conclusions about the quality of the xsd.exe code.

So to fix the problem we can cut and paste all of the FpML xsd files into one file, removing the include statements that become redundant. An example of this is available. We then use xsd.exe as described in part 3 to create our C# classes.

Problem 2: RoutingIds

If we fix problem 1 as above, we still get an error when we try to create our XmlSerializer object. The error message says ‘Connot convert type ‘RoutingId[]’ to ‘RoutingId’’.

This exception arises because xsd.exe has got an XmlArrayItemAttribute wrong in class RoutingIdsAndExplicitDetails. The generated code for the routingIds property in this class is as below:

    [System.Xml.Serialization.XmlArrayItemAttribute("routingId", typeof(RoutingId), IsNullable=false)]
    public RoutingId[][] routingIds {
        get {
            return this.routingIdsField;
        }
        set {
            this.routingIdsField = value;
        }
    }

The XmlArrayItemAttribute says that the property relates to an array of type RoutingId. However, the property (correctly) is of type RoutingId[][] which is an array of arrays of type RoutingId. So the XmlArrayItemAttribute should be changed as below:

    [System.Xml.Serialization.XmlArrayItemAttribute("routingId", typeof(RoutingId[]), IsNullable=false)]
    public RoutingId[][] routingIds {
        get {
            return this.routingIdsField;
        }
        set {
            this.routingIdsField = value;
        }
    }

Testing the Generated Classes

The two fixes above appear to make the generated classes work correctly. With these changes we can deserialize our ird_ex01_vanilla_swap.xml FpML document into the classes. We can then serialize it back into XML, and we end up with the same document we started. We saw this in one of our code examples from part 3. It’s not easy to test this code is working in all cases however. One approach is to take all the sample FpML files provided with the FpML download and attempt to deserialize them and reserialize them. The code is working if the final document is the same as the original one.

Testing Program

A testing program that does this is available. This contains the code we have seen before (in part 3) for serialization and deserialization.

It also contains a basic class for comparing the original and final FpML documents and outputting any differences. It does this simply by iterating through the lines in the two files and comparing them. This may not be the best way to do this as it is quite difficult with XML.

It is difficult because there can be valid differences between the files that are hard to deal with. For example we have a ‘difference’ where the original line is:

<hourMinuteTime>09:00:00</hourMinuteTime>

After deserialization and reserialization this becomes:

<hourMinuteTime>09:00:00.0000000+00:00</hourMinuteTime>

These are clearly the same thing but our file comparer has to be able to deal with it. It does so in a very basic way by hard-coding such differences to be ignored in method ‘CompareLines’.

Extent of Testing of Generated Code

Because of the difficulties described above the testing program has only been used to test that the generated code works with the interest rate derivatives and credit derivatives sample files.

Usefulness of Generated Classes

In the last two articles we have demonstrated that we can generate C# classes based on the FpML XSD specification, and with a little work can deserialize FpML documents into this object model, manipulate the objects, and serialize back into FpML.

However, my personal opinion is that we need to think carefully as to whether we want to use these classes. FpML has a very hierarchical structure, and as a result in the generated code we have very many classes interacting to represent even a simple trade. Our object model is not very easy to understand or use as a result.

For example, suppose we want to change the notional on the fixed leg of our interest rate swap (ird_ex01_vanilla_swap.xml) once we have it in the object model. Starting with the top-level Document object that we have deserialized, the syntax is as below:

        private void ChangeNotional(Document document)
        {
            DataDocument dataDocument = (DataDocument)document;
            Trade trade = (Trade)dataDocument.Items[0];
            Swap swap = (Swap)trade.Item;
            InterestRateStream interestRateStream0 = swap.swapStream[0];
            InterestRateStream interestRateStream1 = swap.swapStream[1];
            Calculation calculation1 = (Calculation)interestRateStream1.calculationPeriodAmount.Item;
            Notional notional = (Notional)calculation1.Item;
            notional.notionalStepSchedule.initialValue = 1000000;
        }

In fact, I don’t think this routine is complex enough, since it should really check which of interestRateStream0 and interestRateStream1 is the fixed leg.

A code example incorporating this code is available.

It’s hard to argue that this code is straightforward: for instance we have a number of ‘Item’ properties referenced that can be of various types. We have to know which type we want. In addition we have the issue that we are not entirely sure that the code generated by xsd.exe is free of bugs, even after the work we have done to patch it up.

As a result in a current project I am working on we have decided not to use the classes generated by xsd.exe, but instead to deserialize into flatter structures of our own design.

Conclusion

This article has shown that it is possible to fix the code generated by xsd.exe from the FpML schemas such that we can deserialize/serialize FpML documents into/out of the object model. We have also shown that it is difficult to test that this will work correctly in all cases, and that the resulting object model is not all that easy to use.

Licensing of FpML Specifications

The FpML Specifications of this document are subject to the FpML Public License (the “License”); you may not use the FpML Specifications except in compliance with the License. You may obtain a copy of the License at http://www.FpML.org.
The FpML Specifications distributed under the License are distributed on an “AS IS” basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License.
The Licensor of the FpML Specifications is the International Swaps and Derivatives Association, Inc. All Rights Reserved.

http://www.fpml.org/documents/license.html

January 27, 2008

Generating .NET Classes from the FpML XSD Schema (Introduction to using FpML with .NET Tools Part 3)

Introduction

Part one of this series of articles looked at how we can use Visual Studio to examine the FpML XML schema documents (XSDs), and the associated example XML instance documents.

Part two of the series looked in some detail at the structure of an FpML trade XML document, and showed how to investigate its validation against the schemas using Visual Studio.

This article will show how to generate .NET classes in C# from the XML schema documents. These classes can in theory be used to load FpML documents into objects. These objects can then be manipulated by the code and saved out as FpML. However, there are some problems with the Microsoft tools for doing this, as we shall see.

Motivation

FpML is a powerful way of representing financial products in a standard way. It can be used to pass trade data between financial institutions in an XML format that both parties can understand. It can also be used to pass trade data between computer systems within a financial institution. If all systems know about FpML then we have a standard platform-independent human-readable representation of our data that everyone can use.

Clearly there are some disadvantages to using FpML in this way (it’s very verbose, and we will probably need to modify it for our own needs anyway, in which case it stops being standard). However there’s a prevailing view that FpML is a good place to start when defining what trade data and messages we will pass around in a service-oriented architecture in a bank.

However if we want to use trade data passed to us as FpML we need to get it into a format we can program against. Obviously if we are programming in .NET languages we want to have objects, or even DataSets. Furthermore we’d like to be able to modify those objects or DataSets and then turn them back into valid FpML documents.

The XML Schema Definition Tool (xsd.exe)

Microsoft have provided an ‘XML Schema Definition Tool’ called xsd.exe that purports to allow us to do this. It claims to be able to turn XSDs into classes. These classes can then be used to automatically load associated XML instance documents into objects, and back into XML again after manipulation. xsd.exe is also capable of creating DataSets from XSD files.

Obviously loading XML instance documents into objects is XML deserialization, and turning them back into XML is serialization. Once we have the classes we can use .NET’s standard serialization mechanisms for this.

As we shall see, this doesn’t work particularly well with the FpML schemas.

Generating C# Classes from FpML Schemas

To use xsd.exe to generate C# classes is relatively straightforward:

  1. 1. Start a Visual Studio command prompt (this is under ‘Microsoft Visual Studio/Visual Studio Tools’ on your start menu).
  2. Navigate to a folder where the FpML XSD files are. If you created the console application used in parts 1 and 2 of this series of articles navigate to the project folder (which has the XSDs in it).
  3. Run the command below:
    xsd /c fpml-main-4-2.xsd xmldsig-core-schema.xsd
    This generates a file called fpml-main-4-2_xmldsig-core-schema.cs which contains the classes we need.

Note that the /c parameter asks xsd to generate classes. There is also a /d parameter that asks xsd to generate DataSets. We will discuss this option later.

Note also that we only need reference the root schema file (fpml-main-4-2.xsd) for this to work: the other schema files are referenced from this file (with include statements) and xsd can work this out. However xsd can’t work out what to do with the xmldsig-core-schema.xsd file unless we tell it to process it. This is because only the namespace is referenced in the schema files, not the file itself.

Using the Generated C# Classes

If we look at the fpml-main-4-2_xmldsig-core-schema.cs file we see that we have nearly 37,000 lines of code, including over 650 classes. As you’d expect these classes use .NET’s XML serialization attributes throughout, so we can serialize into and deserialize from XML correctly. The root class is Document.

To use these classes we need to create an XmlSerializer object based on the root Document in code. This is standard .NET serialization code:

        XmlSerializer xmlSerializer = new XmlSerializer(typeof(Document));

Then in theory we should be able to deserialize any FpML document into these classes using the XmlSerializer. The syntax we’d use for this is as below:

        internal Document DeserializeXMLToDocument(FileInfo inputXMLFile)
        {
            using (FileStream fileStream = File.OpenRead(inputXMLFile.FullName))
            {
                return (Document)xmlSerializer.Deserialize(fileStream);
            }
        }

Once we’ve deserialized into objects based on our classes, we should be able to serialize those back into XML. Clearly the final XML should be the same as the initial XML. The syntax for the serialization is as below:

        internal void SerializeDocumentToXML(Document document, FileInfo outputXMLFile)
        {
            using (FileStream outFileStream = new FileStream(outputXMLFile.FullName, FileMode.Create))
            {
                xmlSerializer.Serialize(outFileStream, document);
            }
        }

This is all standard .NET XML serialization code.

First Attempt to Use the Generated Classes

We can write a basic harness that uses our generated classes and the code above to attempt to deserialize and serialize FpML files.

A version of this is available. It tries to deserialize the ird_ex01_vanilla_swap.xml that was examined in part 2 of this series of articles.

Unfortunately the classes generated by xsd.exe have a number of problems, and unless we correct these the basic harness will not work. In fact we can’t even create the XmlSerializer object successfully with the generated code.

Part four of this series of articles will examine the various problems with the code that xsd.exe has generated, and will discuss how to correct them.

Corrected Generated Classes and a Working Harness

Corrected generated classes are available. As will be discussed in part four this code has been corrected such that it appears to work in most circumstances. However we cannot be certain that it is free of all bugs.

A version of the harness that uses the corrected code is also available. As you can see if you run it, this does correctly deserialize and then reserialize ird_ex01_vanilla_swap.xml.

Conclusion

This article has shown that in theory we can generate classes from the FpML XSDs using xsd.exe. We should then be able to deserialize FpML documents into these classes, manipulate the resulting objects, and then reserialize back into valid FpML documents. However, xsd.exe has some problems that prevent this from working correctly.

Part four of this series of articles will look in more detail at the problems in the generated code, and how to fix them.

Licensing of FpML Specifications

The FpML Specifications of this document are subject to the FpML Public License (the “License”); you may not use the FpML Specifications except in compliance with the License. You may obtain a copy of the License at http://www.FpML.org.
The FpML Specifications distributed under the License are distributed on an “AS IS” basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License.
The Licensor of the FpML Specifications is the International Swaps and Derivatives Association, Inc. All Rights Reserved.

http://www.fpml.org/documents/license.html

The Shocking Blue Green Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 83 other followers