Beginner’s Guide to Techniques for Refreshing Web Pages: Ajax, Comet, HTML5

Introduction

This article briefly discusses the technologies used in modern browsers to display web pages, and goes into a little more detail about the user experience on those web pages, in particular how we can get part of a web page to refresh automatically when data changes on a web server.

Browsers and HTML

I’m sure anyone who’s reading this page is aware that the web is based on a request and response process that returns web pages of data.  If I click on a link my browser makes a request for the web page specified in the link, the request gets routed to the appropriate server, and the server responds with a page of HTML which my browser displays.

HTML (hypertext markup language), of course, is a simple markup language that tells a browser where to put text and images on a web page using tags (e.g. <header>).  The request format is a text URL (uniform resource locator) of the kind you see all the time in your browser’s navigation bar.  Furthermore, the returned text can contain additional links that the browser will show underlined and that I can click on.

Anyone who uses the internet understands this, but the success of the web is at least in part due to the simplicity of that model.  The HTML is just text with a few special tags in angle brackets, and all a browser has to do is know how to send a request, handle the response, and then draw a screen using the HTML that’s returned.  Similarly all a web server has to do is know how to receive a request, get the right HTML text to send back, and send it.  Often web servers will simply store the text in text files on their hard drive, and just load and send the right one in response to a request depending on the text of the request.

At root it’s unbelievably simple; just look what it’s turned into.

Other Technologies Used In Web Browsers

Of course modern browsers aren’t as simple as described above and there are a number of other technologies that they understand and developers can use.

Firstly, developers want to write code, so there’s also a programming language embedded into every modern browser.  This is Javascript.

Javascript allows programmers to write little bits of code that can run when events happen in the browser.  The Javascript can manipulate what’s displayed in the browser programmatically, or can perform other actions.

For the Javascript to change what’s displayed it needs to manipulate the HTML.  Obviously this can be done by simply changing the text.  However, there’s a programmatic representation of a web page that Javascript can use to manipulate elements within it.  This format is the Document Object Model or DOM.

Another baseline technology for what gets displayed to the client is Cascading Style Sheets (CSS).  These allow a common look and feel to be applied to a group of web pages without the need for detailed coding in each page.

Drawbacks of the Basic HTML Request/Response Page-Based Model

HTML + Javascript + CSS allows us to create quite sophisticated web pages.  However, there’s one big drawback with the model as described above: to display new data we have to click on a link to request a whole new page and wait whilst it loads.

For a more sophisticated user experience there are a few things we might like to have:

  1. The ability to refresh part of a web page without reloading the entire page.  Initially this could be initiated by the user clicking a button, but we want just the relevant data to update, not the entire page.
  2. The ability to do this refresh whilst allowing the user to continue to interact with the rest of the page.  That is, the request shouldn’t block the user, it should be asynchronous.
  3. The ability to update the page when data changes on the server without the user having to refresh in any way.

1.  Refreshing Part of a Web Page

The first problem that developers tried to solve was updating part of a web page in place without reloading the entire page.  There are several ways of doing this.

IFrames

However, there are some simple approaches that predate Ajax.  One is to use IFrames.  These are HTML elements within a page that can issue their own request/responses to a web site and render the results independently of the rest of the page.  They have a SRC property that can be set to a URL.  If you set the src to a different or the same URL (say on a button click) the new data will appear without a full page reload.

Many developers don’t like IFrames.  Search engines can get confused by them.  They may show scrollbars if your content doesn’t fit correctly.  Worse if your user has scrolled to the bottom of a page and then you load a new shorter page in the same frame they may be off the bottom of it.  Because of restrictions in HTML IFrames can usually only issue requests to the same site as the main site of the page they are on.  All of this means people have looked for better solutions.

Script Injection

Another approach to refreshing part of a web page is client-side script injection.  This takes advantage of the fact that Javascript code in a web page can be retrieved from a server using a URL via a src tag.

The basic approach is the same as for IFrames: we can set or reset the src of the script code, and the browser will retrieve the script from the URL and execute it.  If we send back valid Javascript that updates part of our web page, or calls a function that does, then we don’t have to refresh the entire page.

One advantage of this approach is that script tags can issue requests to any URL, not just the same site as the page they are on.  One disadvantage of this approach is that it can lead to security vulnerabilities in the code.

JSONP

JSONP is just a way of using client-side script injection across domains to get data from a different website: we request the script from the different server and it returns it as the parameters of a Javascript function which immediately executes and uses the payload.

2.  Refreshing Part of a Web Page Asynchronously

Ajax (Asynchronous Javascript and XML) is probably the primary technology for this.  Ajax is actually a label applied to a way of using many of the technologies described above to allow web pages to be displayed and then be updated asynchronously without reloading the entire page.

The main distinguishing feature of Ajax is that it uses a relatively new request response mechanism.  This is called XMLHttpRequest.  When a browser makes a request using XMLHttpRequest it provides the name of a Javascript function that will be called back by the server.  This function will have access to the data sent back from the server.

The original call to the server will not block and will not reload the page.  The user can carry on interacting with the page as usual, even if the call to the server takes some time.

It is up to the callback function to make whatever changes it needs to make to the web page using the usual Javascript techniques described above.  This will typically involve updating just a part of the screen.

One thing to note here is that the data returned is just text.  It doesn’t have to be XML, in spite of the names (XMLHttpRequest, AjaX).

3.  Updating a Page Automatically when Data Changes on the Server

Ajax as described so far updates a page in place, but only in response to a request from the web page.  This means that the user has to click a button or something similar for the page to update.

Obviously there are situations where data is changing and we would like it to update on our web page without the need for the user to manually refresh.

There are quite a few ways of doing this, some of them direct extensions to the Ajax model described above:

Polling

Javascript allows us to fire events that run code in the browser at set intervals.  So a very simple approach is to automatically request a refresh of the part of the screen we are interested in updating periodically.  We can do this using the Ajax techniques above, so that the rest of the screen remains responsive.

The drawback to this is we may make requests when no data has changed, putting unnecessary load on our servers.  Also our data on the client may well be out of date at any given time if we are between polling requests.

We really want a way for our server to send data only when it’s changed, and at the moment it has changed.

Long Polling

Another approach is long polling.  Here the browser fires off a request with a long timeout and sets up a handler for the results using Ajax as before.  However, the server itself doesn’t respond until it has data that has changed, and then it sends the data in response to the original request.  The browser handles the request and immediately sets up another long timeout request for future updates.

The disadvantage of this approach is that the server has to keep the connection (a network socket) open until it has data.  In general it will have as many connections as it has clients waiting for data.  This obviously puts load on the server and the number of sockets that the server can possibly use becomes a limiting factor.  Also this is clearly a more complex solution to implement than normal (short) polling.

Streaming

In streaming the client makes a request and the server responds with an open response that keeps the communication channel open and allows subsequent responses to be sent to the client.  The server may eventually time out the connection, or may keep it open indefinitely.  If the connection times out the client will have to make another request to refresh the data.  So this approach is like long polling with the client needing to make fewer requests.

One drawback of this approach is that many proxy servers buffer http responses until they are complete: that is, they won’t send on the message until they have the completion.  This means the client won’t get timely updates.  Another obvious drawback is that this is a fairly complex way of keeping data up to date.

With all of these approaches the call backs from the server tend to tie up one http communication channel.  As a result many approaches to solving the problem use (at least) two channels: one for polling or streaming to  update the data in place, and one for regular requests from the client to the server.

A number of commercial frameworks have been built using these techniques.

Comet

Comet is a name that’s been applied to the techniques described above to update a web page in place automatically when data changes on the server using a longlasting HTTP connection.

HTML 5 Web Sockets

HTML 5 web sockets are the new way to do bidirectional communication between a web page and a server.  They don’t use the old HTTP request/response at all, but instead set up one dedicated channel for communication between client and server.  This is fast, and the messages involve very little redundant header information, unlike conventional HTTP requests.

The main drawback of this new technology currently is that many browsers do not support it.  For example, it doesn’t work in the last version of Internet Explorer, IE9, although it works in IE10.

References

http://aspalliance.com/1391_Four_Ways_to_Locally_Update_a_Web_Page.8

http://www.xfront.com/REST-Web-Services.html

http://www.websocket.org/quantum.html

http://www.w3schools.com/jsref/tryit.asp?filename=tryjsref_button_disabled

Credit Default Swap Index Trading: Bid/Offer, Rolls, Roll Conventions

Introduction

I apologize in advance for the rather esoteric subject here, but I have in the past done some work with our CDS index trading desk.  The roll dates for the US indexes are once again approaching.  Immediately after the roll date we trade both the old index, the new index, and the roll between them.  The terminology, and what we are buying and selling, confuses us poor developers no end.  It is immensely confusing because of the price/spread conventions.

At the last roll I wrote some notes on this, which I’m publishing here to help anyone else struggling with the market.

This article focuses on the US CDS index trading market, and uses the Markit CDX indexes as examples.  This isn’t a general introduction to those markets, rather it’s a discussion of the mechanics of the market and what the bid/offer spreads actually mean.

I have written a general introduction to the credit default swap market.  Markit themselves have guides to the index markets (see the Primers section).

CDS Index Pricing

Some indexes trade on spread (e.g. CDX IG), some on price (e.g. CDX HY).  The series of articles on credit default swaps include a description of what these terms mean.

Indexes That Trade On Spread

In many ways spread is easier to think about.  In March 2012 some real prices quoted by a dealer were 90.0/90.5 for the CDX NA IG 18 index, 84.25/85.0 for the CDX NA IG 17, and 5.50/5.69 for roll.  I explain what these numbers actually mean below.

The Index Bid/Offer When Trading on Spread

The 18 price of 90.0/90.5 means that as a customer we pay the equivalent of a running premium of 90.5 to buy protection against defaults in the index, we receive 90.0 to sell protection.  Of course we don’t actually pay that premium, we pay a fixed premium of 100 and an upfront fee that adjusts for the difference to the quoted spread of 90.5.

As a customer we always buy at the offer, and sell at the bid.  Here we are buying protection at the offer (90.5).  So for a customer to ‘buy’ here means to buy protection.

The Roll When Trading On Spread

A roll trade is designed to swap a position in the old index into a position in the new index.  So you will sell protection in the old index and buy it in the new index, or vice versa.

The roll was trading at 5.50/5.69.  When you ‘buy the roll’ you, as usual, buy at the offer (at 5.69).  By convention for securities trading on spread you are buying protection on the new series (the 18), selling it on the old series (the 17).  I’m swapping a ‘long’ (bought protection) position in the 17 for a long position in the 18.

However, for reasons we shall see later it’s easier to think about bid and offer here.  If I buy the roll I’m trading at the offer price (5.69): I’m lifting the offer.  When I do that I’m also trading at the offer price for the 18, but trading at the bid price (hitting the bid) for the 17.

To calculate what trades you enter into at what prices if you trade the roll, you start with the 18 and work out the 17 from the roll spread.  You subtract the roll spread from the 18 price.  Note that this applies to indexes that trade on spread only: see below.

In this case if I buy the roll, I’m buying the 18 at the 18 offer (90.5) and calculate the 17 bid as 90.5-5.69 = 84.81.  Note that this is a better price for me than the actual outright 17 bid (84.25): I effectively receive premium at 84.81 instead of 84.25.

Note also that I haven’t crossed the spread on the 17: I’m trading at an implied 17 bid of 84.81 but this is still less than the offer of 85.00.  For obvious reasons this is an important check for a trader.

If I sell the roll at 5.50, I’m selling the 18 (at 90.0) and the implied 17 offer is 90.0-5.5 = 84.5, again a better price to me than the outright price of 85.0.

Indexes That Trade On Price

The CDX HY index trades on price.  In March 2012 some real prices quoted by a dealer were 97.56/97.75 for the CDX NA HY 18 index, 98.75/98.94  for the CDX NA HY 17 index and 1.19/1.31 for the roll.

The Index Bid/Offer When Trading On Price

Here buy protection/sell protection is reversed from above because of the way the index is quoted.  The 18 price of 97.56/97.75 means that as a customer I still ‘buy’ at the offer (97.75).  However, now this number is used to calculate my upfront fee directly, rather than referring to a notional spread over the life of the trade.  As discussed in my earlier article, the calculation is 100-price = points, and points is the percentage we apply to the notional to calculate the fee.

For the dealer to make money the customer has to pay more to buy protection than they would receive to sell protection.  Here if I ‘buy’ at 97.75 the associated fee is 2.25% of the notional, if I ‘sell’ at 97.56 the associated fee is 2.44% of notional.

What this means is that I’m actually buying protection at the bid (97.56) and selling protection at the offer (97.75), which is the reverse of trading on spread.

So for the HY index we are quoting ‘like a bond’.  If I buy the 18s at the offer (97.75) this is like entering a long bond position (buying a bond), going short protection.

To recap, the percentage fee I actually pay is 100 minus the quoted spread, so for the 18s this is 2.44/2.25.  So I receive 2.25 if I sell protection, pay 2.44 if I buy protection (percent of the notional).

The Roll When Trading On Price

The roll was trading at 1.19/1.31.

If I lift the offer on the roll (1.31) then I’m lifting the offer on the 17 and hitting the bid on the 18 (97.56).  Note that’s the reverse of what I did above for the IG.  This is just the convention: I explain why it’s the convention below.

However this is the SAME trade as lifting the offer on the roll for the IG in terms of buying/selling protection: I’ve sold protection on the 17, bought protection on the 18.

As for the IG we derive the 17 price for the roll from the 18 price and the roll spread.  Here we add.  Again I explain why this is the convention below.

So if I lift the offer on the roll I’ll calculate the implied 17 offer price by adding the roll offer price to the 18 bid price.

That is, the implied 17 = 97.56 + 1.31 = 98.87.  So we’ve sold protection on the 17 at 98.87, which means we receive more upfront fee than if we’d just lifted the offer on the outright 17 (98.94): remember we actually receive 100 minus the value here.  So the customer again gets a better deal by trading the roll than by trading the two outrights.

Note also we haven’t crossed the spread for the 17 (98.75 < 98.87 < 98.94).

Summary of Calculations When You Trade a Roll

Trading on Spread

New index 90.0/90.5

Roll 5.50/5.69

Implies

Customer buys roll (lifts offer), old index bid price = 90.5-5.69 = 84.81 (old bid = new offer – roll offer)

Customer sells roll (hits bid), old index offer price  = 90.0-5.50 = 84.5 (old offer = new bid – roll bid)

Trading on Price

New index 97.56/97.75

Roll 1.19/1.31

Implies

Customer buys roll (lifts offer), old index offer price = 97.56+1.31 = 98.87 (old offer = new bid + roll offer)

Customer sells roll (hits bid), old index bid price = 97.75+1.19 = 98.95 (old bid = new offer + roll bid)

Explanations for the Roll Conventions

The first thing to note is that in general it costs more to buy protection and you receive more if you sell protection in the newer index.  This is because the protection is for six months longer.  You may also have already had defaults in the old index so fewer names are being covered.  However, the index constituents are revised to include more liquid and better-quality credits and this can offset this effect to some extent.

Roll Spread Convention

This means that if you are trading on spread (new index spread- old index spread) is positive, because the new spread is larger, regardless of buy protection/sell protection.   We want to quote a positive spread, so we make that the calculation for our roll spread.

If you are trading on price then things are reversed:  (old index price – new index price) will be positive, so we make that our roll spread.  This is because the upfront fee is larger for the new index (it costs more to buy protection), so the actual price we quote is smaller for the new.

Hence the difference in the roll calculations: if trading on spread to get the old index spread we subtract the roll spread from the new index spread, if trading on price to get the old index price we add the roll spread to the new index price.

Bid/Offer Convention

The bid/offer conventions now follow from the above.  For both price conventions we will always have one trade at the bid and one at the offer in a roll.  We are always buying one index and selling the other.  There are alternative ways of quoting to those described above.  However, if we quote with the conventions described above we always have offer greater than bid, and are always buying protection on the newer index if we buy the roll, so it makes sense to do it this way.

Conclusion

I warned you it was confusing: hopefully this article at least gives some idea of what’s going on.