Rich Newman

January 28, 2008

Problems with Using xsd.exe to Generate .NET Classes from the FpML XSD Schema (Introduction to using FpML with .NET Tools Part 4)

Introduction

Part 3 of this series of articles showed how we can generate .NET classes from the FpML XSD schema using xsd.exe. It showed how we can then use standard .NET serialization syntax to populate the classes from FpML documents, and vice versa.

However, as mentioned in part 3, xsd.exe generates buggy code when used with the FpML XSDs. This article will go through some of the problems with this code and describe how to fix them.

Corrected .NET Classes

Corrected generated classes, which are the end result of the work in this article, are available. This code has been corrected such that it appears to work in most circumstances. However we cannot be certain that it is free of all bugs.

The corrected code is based on FpML 4.2. The main aim of this article is to give you a starting point if you are trying to generate .NET code from other versions of the specification.

Please note also that this article was written using the xsd.exe supplied with Visual Studio 2005 Service Pack 1. Different versions of Visual Studio, or Visual Studio 2005 without the service pack applied, may give different results. See the discussion in the Comments for this article for more details.

Generating the Code

Part 3 of this series of articles described how to use xsd.exe to generate C# code from the FpML schemas. This gave us a file called fpml-main-4-2_xmldsig-core-schema.cs, which contains a Document class which should be the root class for our serialization. Unfortunately if we attempt to create an XmlSerializer object using this code we get exceptions.

Problem 1: Substitution Groups and Extension

The most fundamental problem with our generated code is that xsd.exe has got confused about the substitution groups and associated extension that were discussed in some detail in part 2 of this series of articles.

In particular it has not decorated the Product property of our Trade class correctly to allow it to deal with all the possible products correctly.

In part 2 we saw that the product element in the FpML is replaced with various individual product types using the substitutionGroup syntax. The product element is a sub-element of the trade element in the XML. As a result in our C# code we have a Trade class which contains a data member of type Product and a public property that gets and sets this. Excerpts from our generated code are as below:

[System.CodeDom.Compiler.GeneratedCodeAttribute("xsd", "2.0.50727.42")]
[System.SerializableAttribute()]
[System.Diagnostics.DebuggerStepThroughAttribute()]
[System.ComponentModel.DesignerCategoryAttribute("code")]
[System.Xml.Serialization.XmlTypeAttribute(Namespace="http://www.fpml.org/2005/FpML-4-2")]
public partial class Trade {
 
    private Product productField;
 
    public Product product {
        get {
            return this.productField;
        }
        set {
            this.productField = value;
        }
    }

The problem with this code is that, as we saw in part 2, we need to be able to put a Swap object (for example) in to our Product data member. We have a Swap class in our code, and this inherits Product, as we’d expect. So we can store a Swap object in our Product data member.

However, .NET doesn’t know how to serialize or deserialize a swap element in the XML into our product property without being told. In particular, if the deserializer finds an element called ‘swap’ where it’s expecting ‘product’ it doesn’t know that it should deserialize into a Swap object. As we saw in part 2, in ird_ex01_vanilla_swap.xml there is a ‘swap’ element where you might expect a ‘product’ element.

This is very easily fixed in this case. We simply decorate the Product property with an XmlElementAttribute that tells the serializer what to do:

    [System.Xml.Serialization.XmlElementAttribute("swap", typeof(Swap))]
    public Product product {
        get {
            return this.productField;
        }
        set {
            this.productField = value;
        }
    }

This does solve the problem for swap. However, there are multiple other product types that need the XmlElementAttribute added to Product to get the serialization to work (there are over twenty in fact). Also it turns out that this problem isn’t limited to Product. There are other elements that use the substitution group and extension property in this way and have the same problem

At first glance it appears there is no easy solution to this: we are going to have to go through all the possible places where there might be errors and correct the code by hand. However, there’s a much easier solution.

Problem 1 Solution

For simple examples of this problem xsd.exe will correctly generate the required XmlElementAttribute. It’s not immediately obvious why it fails to do so with the full FpML schemas.

It turns out that the reason xsd.exe gets this wrong is because the FpML schemas are spread across multiple files. If we create one big xsd file containing all of the FpML schema files and then run xsd.exe on this the problem goes away. I’ll leave you to draw your own conclusions about the quality of the xsd.exe code.

So to fix the problem we can cut and paste all of the FpML xsd files into one file, removing the include statements that become redundant. An example of this is available. We then use xsd.exe as described in part 3 to create our C# classes.

Problem 2: RoutingIds

If we fix problem 1 as above, we still get an error when we try to create our XmlSerializer object. The error message says ‘Connot convert type ‘RoutingId[]’ to ‘RoutingId’’.

This exception arises because xsd.exe has got an XmlArrayItemAttribute wrong in class RoutingIdsAndExplicitDetails. The generated code for the routingIds property in this class is as below:

    [System.Xml.Serialization.XmlArrayItemAttribute("routingId", typeof(RoutingId), IsNullable=false)]
    public RoutingId[][] routingIds {
        get {
            return this.routingIdsField;
        }
        set {
            this.routingIdsField = value;
        }
    }

The XmlArrayItemAttribute says that the property relates to an array of type RoutingId. However, the property (correctly) is of type RoutingId[][] which is an array of arrays of type RoutingId. So the XmlArrayItemAttribute should be changed as below:

    [System.Xml.Serialization.XmlArrayItemAttribute("routingId", typeof(RoutingId[]), IsNullable=false)]
    public RoutingId[][] routingIds {
        get {
            return this.routingIdsField;
        }
        set {
            this.routingIdsField = value;
        }
    }

Testing the Generated Classes

The two fixes above appear to make the generated classes work correctly. With these changes we can deserialize our ird_ex01_vanilla_swap.xml FpML document into the classes. We can then serialize it back into XML, and we end up with the same document we started. We saw this in one of our code examples from part 3. It’s not easy to test this code is working in all cases however. One approach is to take all the sample FpML files provided with the FpML download and attempt to deserialize them and reserialize them. The code is working if the final document is the same as the original one.

Testing Program

A testing program that does this is available. This contains the code we have seen before (in part 3) for serialization and deserialization.

It also contains a basic class for comparing the original and final FpML documents and outputting any differences. It does this simply by iterating through the lines in the two files and comparing them. This may not be the best way to do this as it is quite difficult with XML.

It is difficult because there can be valid differences between the files that are hard to deal with. For example we have a ‘difference’ where the original line is:

<hourMinuteTime>09:00:00</hourMinuteTime>

After deserialization and reserialization this becomes:

<hourMinuteTime>09:00:00.0000000+00:00</hourMinuteTime>

These are clearly the same thing but our file comparer has to be able to deal with it. It does so in a very basic way by hard-coding such differences to be ignored in method ‘CompareLines’.

Extent of Testing of Generated Code

Because of the difficulties described above the testing program has only been used to test that the generated code works with the interest rate derivatives and credit derivatives sample files.

Usefulness of Generated Classes

In the last two articles we have demonstrated that we can generate C# classes based on the FpML XSD specification, and with a little work can deserialize FpML documents into this object model, manipulate the objects, and serialize back into FpML.

However, my personal opinion is that we need to think carefully as to whether we want to use these classes. FpML has a very hierarchical structure, and as a result in the generated code we have very many classes interacting to represent even a simple trade. Our object model is not very easy to understand or use as a result.

For example, suppose we want to change the notional on the fixed leg of our interest rate swap (ird_ex01_vanilla_swap.xml) once we have it in the object model. Starting with the top-level Document object that we have deserialized, the syntax is as below:

        private void ChangeNotional(Document document)
        {
            DataDocument dataDocument = (DataDocument)document;
            Trade trade = (Trade)dataDocument.Items[0];
            Swap swap = (Swap)trade.Item;
            InterestRateStream interestRateStream0 = swap.swapStream[0];
            InterestRateStream interestRateStream1 = swap.swapStream[1];
            Calculation calculation1 = (Calculation)interestRateStream1.calculationPeriodAmount.Item;
            Notional notional = (Notional)calculation1.Item;
            notional.notionalStepSchedule.initialValue = 1000000;
        }

In fact, I don’t think this routine is complex enough, since it should really check which of interestRateStream0 and interestRateStream1 is the fixed leg.

A code example incorporating this code is available.

It’s hard to argue that this code is straightforward: for instance we have a number of ‘Item’ properties referenced that can be of various types. We have to know which type we want. In addition we have the issue that we are not entirely sure that the code generated by xsd.exe is free of bugs, even after the work we have done to patch it up.

As a result in a current project I am working on we have decided not to use the classes generated by xsd.exe, but instead to deserialize into flatter structures of our own design.

Conclusion

This article has shown that it is possible to fix the code generated by xsd.exe from the FpML schemas such that we can deserialize/serialize FpML documents into/out of the object model. We have also shown that it is difficult to test that this will work correctly in all cases, and that the resulting object model is not all that easy to use.

Licensing of FpML Specifications

The FpML Specifications of this document are subject to the FpML Public License (the “License”); you may not use the FpML Specifications except in compliance with the License. You may obtain a copy of the License at http://www.FpML.org.
The FpML Specifications distributed under the License are distributed on an “AS IS” basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License.
The Licensor of the FpML Specifications is the International Swaps and Derivatives Association, Inc. All Rights Reserved.

http://www.fpml.org/documents/license.html

About these ads

21 Comments »

  1. What a great job !!! I’m also working on a project that may use fpml with C# and this article is exactly the one I was searching.
    Thanks a lot. Pleased to read you soon again …

    Comment by takikrd — January 29, 2008 @ 9:14 am

  2. Good article. For problem 1 you may be interested to read

    http://www.codeproject.com/KB/XML/xmlserializerforunknown.aspx

    The article is quite long but in short, you would make a wrapper class for the base class, (say ProductWrapper) in this instance and decorate the Product property like so

    [System.Xml.Serialization.XmlElement(Type=typeof(ProductWrapper))]
    public Product product {
    get {
    return this.productField;
    }
    set {
    this.productField = value;
    }
    }

    ProductWrapper then defines methods to cast between Product and ProductWrapper and also how to XMLSerialize itself. A very elegant hack, if there is such a thing.

    Comment by Wal — February 19, 2008 @ 6:18 am

  3. Richard, did you manually fix any additional problems (by hand?). When executing the following line:
    XmlSerializer xmlSerializer = new XmlSerializer(typeof(Document));

    I get an error: There was an error processing type ‘FxCurve’. Type member ‘quotedCurrencyPair’ declared in ‘FxCurve’ is missing required ‘Order’ property. If one class in the class hierarchy uses explicit sequencing feature (Order), then its base class and all derived classes have to do the same

    When looking at the generated classes, indeed FxCurve (amongst others) is missing the required attribute ([System.Xml.Serialization.XmlElementAttribute(Order=0)])

    I am using your fpml-main-all-4-2.xsd as input and the command
    xsd /c fpml-main-all-4-2.xsd xmldsig-core-schema.xsd

    to generate the class file.

    Comment by Wal — February 20, 2008 @ 9:17 am

  4. Wal

    Actually I did have that problem early on, and then it disappeared once I’d pasted everything into one file, which I thought must have been what fixed it. Subsequently I couldn’t replicate it even with separate files, so I took it out of what I posted. Below are my original notes on the issue:

    “Take all Order parameters out of CreditCurve and PricingStructure – not quite sure why they’re there. We have sequences in YieldCurve as well but it’s not adding order tags there. If you take them out it’ll correctly parse any valid XML, and regenerate valid XML when it serializes (because it outputs in field order, which is correct). It will also parse invalid XML with sequence items in the wrong order, but I’m not sure that’s a problem.”

    Hope that helps

    Rich

    Comment by richnewman — February 20, 2008 @ 10:17 am

  5. Ok Rich, I noticed you are using .NET2.0 SP1 (2.0.50727.1433), I was using .NET 2.0 without the service pack. This makes a *massive* difference to the output. My output is now identical to yours, and installing the SP was the only thing I changed (and rebooting).

    Note that the xsd.exe version remains the same.

    Did you (or auto updates) install the SP at some stage?

    Comment by Wal — February 20, 2008 @ 10:56 am

  6. There is some bugs fixed in SP1 that relate to schemas. (see http://support.microsoft.com/kb/921968/)

    Don’t really want to go through the rigmorale of uninstalling SP1 to confirm.

    Comment by Wal — February 20, 2008 @ 11:09 am

  7. Apologies, that was the wrong link. try http://support.microsoft.com/kb/945757.

    I *swear* your page said there was 4 problems initially – and then I came back to the page and there was only 2? I thought I was going mad…

    Comment by Wal — February 20, 2008 @ 12:43 pm

  8. Wal

    Different machines is the answer here: I must not have SP1’d my laptop.

    Rich

    Comment by richnewman — February 20, 2008 @ 6:07 pm

  9. OK. Could you post problems 3&4 back on? Also, your conclusion is:

    “As a result in a current project I am working on we have decided not to use the classes generated by xsd.exe, but instead to deserialize into flatter structures of our own design”

    This will prove to be time-consuming /error prone ?

    Comment by Wal — February 21, 2008 @ 2:53 am

  10. Rich,

    Good article! A few points from my foray..

    1. The Product abstract partial class compiled up to be any of the following, with the attributes automatically added, which includes ‘Swap’. Not sure why you didn’t get this and needed to manually add [System.Xml.Serialization.XmlElementAttribute("swap", typeof(Swap))].

    ///
    [System.Xml.Serialization.XmlIncludeAttribute(typeof(EquityDerivativeBase))]
    [System.Xml.Serialization.XmlIncludeAttribute(typeof(EquityDerivativeShortFormBase))]
    [System.Xml.Serialization.XmlIncludeAttribute(typeof(EquityOptionTransactionSupplement))]
    [System.Xml.Serialization.XmlIncludeAttribute(typeof(BrokerEquityOption))]
    [System.Xml.Serialization.XmlIncludeAttribute(typeof(EquityDerivativeLongFormBase))]
    [System.Xml.Serialization.XmlIncludeAttribute(typeof(EquityOption))]
    [System.Xml.Serialization.XmlIncludeAttribute(typeof(EquityForward))]
    [System.Xml.Serialization.XmlIncludeAttribute(typeof(Strategy))]
    [System.Xml.Serialization.XmlIncludeAttribute(typeof(ReturnSwapBase))]
    [System.Xml.Serialization.XmlIncludeAttribute(typeof(EquitySwapTransactionSupplement))]
    [System.Xml.Serialization.XmlIncludeAttribute(typeof(ReturnSwap))]
    [System.Xml.Serialization.XmlIncludeAttribute(typeof(Swaption))]
    [System.Xml.Serialization.XmlIncludeAttribute(typeof(Swap))]
    [System.Xml.Serialization.XmlIncludeAttribute(typeof(Fra))]
    [System.Xml.Serialization.XmlIncludeAttribute(typeof(CapFloor))]
    [System.Xml.Serialization.XmlIncludeAttribute(typeof(BulletPayment))]
    [System.Xml.Serialization.XmlIncludeAttribute(typeof(CreditDefaultSwap))]
    [System.Xml.Serialization.XmlIncludeAttribute(typeof(TermDeposit))]
    [System.Xml.Serialization.XmlIncludeAttribute(typeof(FxSwap))]
    [System.Xml.Serialization.XmlIncludeAttribute(typeof(FxOptionLeg))]
    [System.Xml.Serialization.XmlIncludeAttribute(typeof(FxBarrierOption))]
    [System.Xml.Serialization.XmlIncludeAttribute(typeof(FxLeg))]
    [System.Xml.Serialization.XmlIncludeAttribute(typeof(FxDigitalOption))]
    [System.Xml.Serialization.XmlIncludeAttribute(typeof(FxAverageRateOption))]
    [System.CodeDom.Compiler.GeneratedCodeAttribute("xsd", "2.0.50727.42")]
    [System.SerializableAttribute()]
    [System.Diagnostics.DebuggerStepThroughAttribute()]
    [System.ComponentModel.DesignerCategoryAttribute("code")]
    [System.Xml.Serialization.XmlTypeAttribute(Namespace="http://www.fpml.org/2005/FpML-4-2")]
    public abstract partial class Product {

    2. Thought that it should have correctly an array of RoutingId and modified the property and private member.

    svcutil.exe generates an array rather than an array of arrays from a metadata endpoint that is annotated with XmlSerializerFormat

    [XmlSerializerFormat(Style=OperationFormatStyle.Document, Use=OperationFormatUse.Literal)]
    public interface IOrderContract
    {

    (Unfortunately with svcutil 3.0.4506.30 /serializer option only works with a metadata endpoint not an xsd)

    3. Had to refactor the BasketIdentifier.model to slightly simplify it.

    4. Went down the Message side of the tree rather than DataDocument. If you read the Architecture guidance it indicates DataDocument is the ‘old’ style. However I have seen quite a few going down this route today. Message is is good route as it caters identifier history in the header. So one has a ContractCreated and ContractCancelled message for an IRS trade. An update is a new ContractCreated with a reference to the previous version in the header.

    5. Because FpML is big but very extensible it maybe wise to derive your own ‘bank-ml’ schema that includes just the parts you want. Did this for interest rate derivatives and credit derivatives to include IRS and CDS with the corresponding message types (ContractCreated, ContractCancelled, ContractNovation, ContractPartialTermination, ContractFullTermination).

    6. Extensibility is very good and was able to build bond, repo and collateral trades as these are not included by default. Probably rightly as they are not derivatives but it demonstrates the extensibility and reuse one can make with FpML.

    7. BizTalk’s (2006 R2) Mapper in visual mode, will not handle mapping FpML (results in an out of memory error on compile); drop back to XSLT when creating the map. However message validation works fine on any in a pipeline component when sending or receiving.

    8. In the view of pt 5 & 6 consider creating a canonical data model in a shared library and sharing types if you are building WCF clients and services, by using svcutil’s the /r option or VS 2008. I’ve come to think this is perfectly acceptable.

    Comment by benwalker — March 17, 2008 @ 7:04 pm

  11. When I try to run my program the following Excpetion was thrown: The XML element from namespace is already present in the current scope. Use XML attributes to specify another XML name or namespace for the element. I search file and there was few definiction of filed RequestQuoteResponse but it was definated in seperate classes. I use FpML ver. 4.3. Please help me.

    Agnieszka – beginner in FpML developing

    Comment by Agnieszka — June 2, 2008 @ 7:37 am

  12. xsd.exe can take multiple input files.
    It is useful if you have references to other schema definitions, like in this case.
    For example xsd /c epp-1.0.xsd eppcom-1.0.xsd produces a single cs file.

    Comment by Andras Vass — July 10, 2008 @ 10:13 pm

  13. Hey kudos on this article! I was able to infer from your solution that my own XSD.exe generated classes (for a totally unrelated project to your FpML stuff) needed to be tweaked slightly (with the multidimensional array fix) to get the XML serialization to work.

    I kept getting a “cannot implicitly convert type ‘blabla[][]’ to ‘blabla[]'” until I tweaked the generated class definition’s XmlArrayItemAttribute definitions — there is definitely a bug in the XSD.exe utility it seems.

    I’ve used Altova’s MapForce to great ends, but wanted to get into the XML serialization provided with the standard Dot Net framework namespaces.

    Thanks many times! I am totally new to XML so I would have been struggling a lot longer on this.

    Comment by Packdude — July 19, 2008 @ 12:41 am

  14. Any experience with 3rd party class generators – XMLSpy or Liquid XML ?

    Great info. Thanks.

    Comment by FSI — September 19, 2008 @ 7:54 pm

  15. here’s another way to go about this :

    http://www.hosca.com/blog/post/2008/12/26/Generating-C-classes-from-FpML-Schema.aspx

    Comment by Erhan Hosca — March 4, 2009 @ 4:45 pm

  16. iam have class file with element attributes and array attributes i tried ur second solution bt it is getting error in generted xml at both element and array atributes plz give me a suggetion

    Comment by mercury — April 7, 2010 @ 8:44 am

  17. Rich, Great article.

    I was planning to use the xsd.exe generated and corrected classes in my project but noticed that you have not used them in your project so let me know what do you mean by “deserialize into flatter structures of our own design”, how to do that and was that sucessful?

    Thanks,
    KP

    Comment by KP — July 8, 2010 @ 9:46 pm

  18. What a great job!!! Thanks for writing it down. It is really helpful for all other peoples who are working on this tool.

    Comment by Vishal — July 20, 2012 @ 10:23 pm

    • Rich, it’s really helpful article. Thanks.
      Two notes:
      1. In our project we worked around Product abstract class problem, and similar problem with other classes, by overriding XmlTextWriter, in particular WriteStartElement function. The code captures element’s type and replaces existing element name to string representing actual class type, for example replaces ‘product’ to ‘swap’.
      2. We found issue with number of elements defined as “xsd:time” in FpML xsd. The element should contain only time part of DateTime value. Since C# cannot handle time part separately from date part in DateTime class we’ve changed these element types to string in C# file generated by xsd.exe.

      Thanks,
      Leonid

      Comment by Leonid — December 20, 2012 @ 6:11 pm

  19. Sorry, left comment in wrong place.

    Rich, it’s really helpful article. Thanks.
    Two notes:
    1. In our project we worked around Product abstract class problem, and similar problem with other classes, by overriding XmlTextWriter, in particular WriteStartElement function. The code captures element’s type and replaces existing element name to string representing actual class type, for example replaces ‘product’ to ‘swap’.
    2. We found issue with number of elements defined as “xsd:time” in FpML xsd. The element should contain only time part of DateTime value. Since C# cannot handle time part separately from date part in DateTime class we’ve changed these element types to string in C# file generated by xsd.exe.

    Thanks,
    Leonid

    Comment by Leonid — December 20, 2012 @ 6:12 pm

  20. This article helps me very much – thank you :-)

    Comment by Romicus — August 14, 2014 @ 11:08 am


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Shocking Blue Green Theme. Create a free website or blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 82 other followers

%d bloggers like this: