JavaScript

Setting up an Optimised Deployment Version of your AJAX ASP.Net Application

Eden Ridgway — Mon, 28 Jan 2008 04:02:57 GMT

With rich AJAX web applications one needs to look at optimising resources such as your CSS, JavaScript and image files. Yahoo has collected together a great collection of best practices that have been put into their YSlow Firefox extension. These should be taken as a guideline and not gospel. In general however the core principals are as follows:

Keep server roundtrips to a minimum. The techniques for the various resource types are as follows:
- JavaScript - combine them into a single file if possible. One has to make certain they are combined in the correct order in the file.
- CSS - again merge them as far as possible.
- Images - Use CSS sprites where one combines several images into one and uses the background image offset to display the one you are after. To create the sprites you can always use this online sprite generator.
Reduce the size of each request. This is done in two ways:
- GZip/Deflate compress text resources by configuring IIS and/or using an ASP.Net HTTP Module.
- Reduce file sizes. The tools I use for each type are:
  - JavaScript - Jazmin provides pretty good minification by removing whitespace. If you want better minification then you should take a look at the Dojo ShrinkSafe compressor which will shorten the name of local variables and functions to reduce the file size even more. The downside to this is it makes troubleshooting a production issue more difficult.
  - CSS - CSSTidy not only reduces whitespace but optimises the CSS expressions as well.
  - Images - As PNGs are my image format of choice I use PNGGauntlet to reduce the image file size.
Ensure resources are cached. If you know that the files are going to change infrequently then all you have to do is set IIS to enable content expiration for a folder/file to expire after a set number of days, say 10. This will ensure that the client browser caches the resources and does not re-download them. However depending on the frequency of your deployments and your server architecture you most likely want to ensure that your etags are setup correctly. These are used to indicate whether or not the cached browser resources have changed. When loading a page the browser will send a request to the server, with the etag, for each resource and if it has not changed then the server will only return a 304 status code. Note that this advice is contrary to Yahoo where they tell you to remove etags for a web farm scenario. What you should rather do is ensure that the etag change number is the same for each server in the web farm. This ensures that the etag generated for the same resource on a different server is the same (how to do this varies depending on your version of IIS - read this post for more IIS6 information). If you are using a single web server however the whole this is a non-issue for you.

Okay so we know we want to combine and minify our CSS and JavaScript files when we deploy to our test, staging and production environments, but we don't want to develop and debug our applications with monolithic files with no whitespace and cryptic names. The problem is that our pages contain references to these files and we are merge them them together for deployment our references will now be incorrect. Well this is quite easily solved using compiler directives, like so:

<% #if DEBUG %>
    <script type="text/javascript" src='<%= ResolveUrl("~/JavaScript/File1.js") %>'></script>
    <script type="text/javascript" src='<%= ResolveUrl("~/JavaScript/File2.js") %>'></script>
    <script type="text/javascript" src='<%= ResolveUrl("~/JavaScript/File3.js") %>'></script>
<% #else %>
    <script type="text/javascript" src='<%= ResolveUrl("~/JavaScript/Section.minified.js") %>'></script>
<% #endif %>

For CSS files you can either use the directives around alternative <link> elements or around @import statements, like this:

There is a catch to this however, since the aspx pages are compiled separately, setting your application to compile as a release build will not cause the references to change. What you need to do is change the compilation setting in the application web.config file, which should be set to debug="false" in the production environment:

Now on to the merging and minification. This could be accomplished using MsBuild or a batch file, but a batch file is simpler, so I'll discuss that. On one of the build steps for the web site(s) invoke the batch file which then ensures that the optimised files are always up to date. So you want to add something like this to your pre/post build compilation step in the project property window:

I setup the "Build Scripts" folder to contain the optimisation programs as well as text files with lists of files to merge, for example:

You'll notice that I group the JavaScript files on the site by Section of the site and not by page. The granularity of file grouping really depends on the application, but by section or function is a pretty good starting point. Each line of the file will contain a file name, like this:

SubDir\File1.js
SubDir\File2.js

I don't need to specify the full relative path of the files, in respect to the build folder because when I develop web sites all JavaScript files are placed within a JavaScript folder off the root of the web site.

In my simplified scenario the "Optimise Web Resources.bat" batch file contains the following:

set OLDDIR=%CD%

:: Change to the current directory
cd /d %0\.. 

::Ensure there are no old temporary files lying around 
del *.js
del *.css

::Create a single file for the JavaScript in the admin section
for /F "tokens=*" %%f in (AdminJsFiles.txt) do call type "..\Web Sites\Site\JavaScript\%%f" >> AdminCombined.js
type AdminCombined.js | jazmin > AdminCombined.minified.js
copy AdminCombined.minified.js "..\Web Sites\Site\JavaScript\AdminCombined.minified.js"

::Combine the Stylesheets into a single file
for /F "tokens=*" %%f in (CssFilesToMerge.txt) do call type "..\Web Sites\Site\CSS\%%f" >> CombinedStyles.css
csstidy.exe CombinedStyles.css --template=high CombinedStyles.minified.css
copy CombinedStyles.minified.css "..\Web Sites\Site\CSS\CombinedStyles.minified.css"

::Remove the temporary files
del *.js
del *.css

::Restore the old "current directory" 
chdir /d %OLDDIR%

The one part of the batch file that confuses most people is the FOR /F "token=*" statement. That reads in a line at a time from the file specified in brackets and puts it in the %%f variable. The contents of each file is then appended into a single file using a "type SourceFile >> DestinationFile" statement. In the case of the JavaScript files they are then piped through the Jazmin compressor/minifier, the output of which is redirected in an output file, i.e. AdminCombined.minified.js. With the CSS files, the command line syntax is a little different, but the outcome is the same.

Hopefully the rest of the batch file is pretty self evident and there is enough information here to get you started on implementing the approach yourself. If there are areas that need more explanation please leave a comment on the blog post.

Simple JavaScript Undo Manager for DTOs

Eden Ridgway — Wed, 07 Nov 2007 06:47:10 GMT

When looking at implementing undo functionality, two common patterns that are used are the Memento or Command design patterns. For an example of using the Memento pattern on JavaScript objects take a look at Lawrence Carvalho's ActAsUndoable post. This approach however did not suite the needs of the application I'm currently working on because the changes I'm making are on Data Transfer Objects (DTOs) that have been serialized from the server using ASP.Net web services. As such I don't want to add any functionality to them and I don't really care about breaking encapsulation because all object values are public. I take these client side objects and "databind" (logically speaking) them to various UI elements, such as tree nodes and form fields. What I want to be able to offer however is the ability to cancel out changes without going back to the server to fetch the original data. I also want any undone changes to apply to the original objects and therefore simply cloning the objects by serializing and deserializing them using ASP.Net JSON serialization methods is not an option (it would make the object to UI element association management a pain). If however you don't care about preserving object references then this is a perfect solution for you, and can be done like this:

function cloneObjectGraph(targetObject)
{
var jsonSerializedData = Sys.Serialization.JavaScriptSerializer.serialize(targetObject);
return Sys.Serialization.JavaScriptSerializer.deserialize(jsonSerializedData);
}

So what I have created is a very simple pair of objects, UndoManager and UndoItem, that are ideally suited to taking a snapshot of state on simple JavaScript object graph, where all the values you care about are "public", and then being able to rollback the object graph to the snapshot point while preserving the original object references. If you want functionality such as the ability store state of more complex objects that have getters/setters and "private" variables (as is present with the Module pattern and others) then don't use these objects, rather take a look at the ActAsUndoable memento approach. Also if you want to implement an "action by action" undo/redo interface like word then rather look at using the command pattern.

So getting on to UndoManager and UndoItem objects. As you can see from the diagram below, they are quite straightforward. The UndoManager exists to allow you to simply have a stack of changes that you can undo on a last in last out basis. The UndoItem object handles the storing of a single undo operation on an object graph.

The way UndoItem takes a snapshot of an objects state is to "iterate" over all the object attributes using a for..in loop and copy any simple type to the _undoState variable. Any complex object it encounters, such as an object, date or array, it creates another UndoItem for that and adds it to _undoState in the attributes place. Hence it builds up a hierarchy of Undo Items it can call on when undo is called. When undoing the changes it uses the _objectReference it stored when constructing the UndoItem to copy the values back from the _undoState on to the original object.

This gives one a really simple and unobtrusive way of capturing and restoring state. Say you have a person object that you want to save to save the state for, make changes and then undo them, you use an UndoItem to do it in the following way:

var person =
{
IsActive: true,
Team: { Name: 'Admin' }
};

var undoItem = new UndoItem(person);

person.IsActive = false;
person.Team.Name = 'General User';

undoItem.undo();

If you want to manage a stack of changes then you use the UndoManager like so:

var team = { Name: 'Admin' };
var person = { Name: 'Joe' };

var undoManager = new UndoManager();

undoManager.saveUndoPoint(team);
undoManager.saveUndoPoint(person);

team.Name = 'General User';
person.Name = 'Bob';

//Will undo the person name change
undoManager.undo();

//Will undo the team name change
undoManager.undo();

If you are keen to use the objects then you can download it here (the code is not minified). I have also created a set of Scriptaculous unit tests that you can run here.

Using Synchronous ASP.Net AJAX Web Service Calls and Scriptaculous to Test your JavaScript

Eden Ridgway — Tue, 30 Oct 2007 06:17:07 GMT

As our web applications become more JavaScript heavy it is becomes increasingly important that we have tests for this code. In the past I have managed to get away with it but in retrospect it was a risky approach to take. So in my latest project I started assessing the various JavaScript unit testing frameworks and decided Scriptaculous was the way to go. I excluded JsUnit as an option because it is not object oriented and didn't give me enough information about which tests had passed. The Scriptaculous unit testing framework also has some useful features like the ability to wait before completing the test and benchmarking of calls.

The one big problem you have when testing an AJAX application however is that the out-of-band calls are made asynchronously. This means that if you are testing a method that does a load and want to ensure that certain values are set or calls made once the data has loaded you have to have a bit of a problem. The Scriptaculous framework's solution to this is the ability to wait and execute a function that may then contain your asserts. Here is a basic example of what the test would look like:

new Test.Unit.Runner(
{
    testShouldLoadDataOnInit: function()
    {
        var manager = new TestManager();

        //This makes a web service call
        manager.onInit();

        this.wait(500, function()
                       {
                          this.assertNotEqual(manager._TestData, null);
                       });
    },
});

I found that this led to inconsistent test results because for various reasons the webservice request callbacks would take a varying amount of time. Of course you could try to cover yourself by making the wait time larger but then your tests take too long to run. Both of these problems will discourage others from using and extending your tests.

Having hand coded XMLHTTP calls in the days before the approach was known as AJAX I knew that it was possible to make the calls synchronously, so all I needed to do was get the generated ASP.Net AJAX webservice proxies to use a new synchronous executor. I came across Amit's SJAX post that demonstrated how to created a synchronous executor for a Sys.Net.WebRequest object which was very useful. I discovered that one could change the executor for generated webservice proxy calls by changing the default executor type for the Sys.Net.WebRequestManager, like so:

Sys.Net.WebRequestManager.set_defaultExecutorType("Example.Executor");

So I took Amit's example, changed it somewhat and got it to work work as a synchronous request handler (using the debug version of the ASP.Net AJAX framework JavaScript as a guide). This resulted in a handler that looked like this (note I have removed code and replaced them with comments for brevity) [download full file here]:

Type.registerNamespace('Sjax');

Sjax.XMLHttpSyncExecutor = function()
{
    Sjax.XMLHttpSyncExecutor.initializeBase(this);

    this._started = false;
    this._responseAvailable = false;
    this._onReceiveHandler = null;
    this._xmlHttpRequest = null;

    this.get_aborted = function()
    {
        //Parameter validation code removed here...
        return false;
    }

    this.get_responseAvailable = function()
    {
        //Parameter validation code removed here...
        return this._responseAvailable;
    }

    this.get_responseData = function()
    {
        //Parameter validation code removed here...
        return this._xmlHttpRequest.responseText;
    }

    this.get_started = function()
    {
        //Parameter validation code removed here...
        return this._started;
    }

    this.get_statusCode = function()
    {
        //Parameter validation code removed here...
        return this._xmlHttpRequest.status;
    }

    this.get_statusText = function()
    {
        //Parameter validation code removed here...
        return this._xmlHttpRequest.statusText;
    }

    this.get_xml = function()
    {
        //Code removed
    }

    this.executeRequest = function()
    {
        //Parameter validation code removed here...
        var webRequest = this.get_webRequest();

        if (webRequest === null)
        {
            throw Error.invalidOperation(Sys.Res.nullWebRequest);
        }

        var body = webRequest.get_body();
        var headers = webRequest.get_headers();
        var verb = webRequest.get_httpVerb();

        var xmlHttpRequest = new XMLHttpRequest();
        this._onReceiveHandler = Function.createCallback(this._onReadyStateChange, { sender:this });
        this._started = true;
        xmlHttpRequest.onreadystatechange = this._onReceiveHandler;
        xmlHttpRequest.open(verb, webRequest.getResolvedUrl(), false); // False to call Synchronously

        if (headers)
        {
            for (var header in headers)
            {
                var val = headers[header];

                if (typeof(val) !== "function")
                {
                    xmlHttpRequest.setRequestHeader(header, val);
                }
            }
        }

        if (verb.toLowerCase() === "post")
        {
            if ((headers === null) || !headers['Content-Type'])
            {
                xmlHttpRequest.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');
            }

            if (!body)
            {
                body = '';
            }
        }

        this._started = true;
        this._xmlHttpRequest = xmlHttpRequest;
        xmlHttpRequest.send(body);
    }

    this.getAllResponseHeaders = function()
    {
        //Parameter validation code removed here...
        return this._xmlHttpRequest.getAllResponseHeaders();
    }

    this.getResponseHeader = function(header)
    {
        //Parameter validation code removed here...
        return this._xmlHttpRequest.getResponseHeader(header);
    }

    this._onReadyStateChange = function(e)
    {
        var executor = e.sender;

        if (executor._xmlHttpRequest.readyState === 4)
        {
            //Validation code removed here...

            executor._responseAvailable = true;

            executor._xmlHttpRequest.onreadystatechange = Function.emptyMethod;
            executor._onReceiveHandler = null;

            executor._started = false;

            var webRequest = executor.get_webRequest();
            webRequest.completed(Sys.EventArgs.Empty);

            //Once the completed callback handler has processed the data it needs from the XML HTTP request we can clean up
            executor._xmlHttpRequest = null;
        }
    }

}

Sjax.XMLHttpSyncExecutor.registerClass('Sjax.XMLHttpSyncExecutor', Sys.Net.WebRequestExecutor);

So now we can create tests that test objects that make synchronous webservice calls by changing the default executor in the setup of the unit test like this:

new Test.Unit.Runner(
{
    setup: function()
    {
        Sys.Net.WebRequestManager.set_defaultExecutorType("Sjax.XMLHttpSyncExecutor");
    },

    testShouldLoadDataOnInit: function()
    {
        var manager = new TestManager();
        manager.onInit();

        this.assertNotEqual(manager._TestData, null);
    },
});

I think you'll agree that with the extensibility they have provided in the ASP.Net AJAX framework, what we have now is quite a neat way of ensuring that our tests run consistently and as fast as they can. My thanks go out to Microsoft and the Scriptaculous team! :)

Using ASP.Net AJAX WebServices (ScriptServices) in ExtJS

Eden Ridgway — Tue, 09 Oct 2007 05:19:34 GMT

I've only just started investigating ExtJS and what it can possibly provide over and above the ASP.Net AJAX framework. The first thing I wanted to do was use the ASP.Net AJAX JavaScript webservice proxies instead of the expected REST style services. Unfortunately I ran into several problems and ended up using a slightly different approach to what I found on the ExtJS forums. Let me quickly outline the problem. In ExtJS they use a store wrapper for their data which is assigned to a grid panel. This store object may also know how to fetch external data that it needs via a proxy, which is defined like so:

var ds = new Ext.data.Store({
    proxy: new Ext.data.HttpProxy(
         {
          url: '/WebServices/OrderService.asmx/GetOrders',
          params: { userId: getUserId(), day: new Date() }
         }),
    reader: new Ext.data.JsonReader(
        {
           id: 'WorkDoneId'
        },
    ...
});

Unfortunately, try as I might to get this to call my web service (and I tried quite a few different approaches), I would get errors returned by the service. It may have been that unlike all the examples I had seen on the web, like Daniel Larson's my service calls actually needed to send parameters through to the service. One was to add additional properties to the ScriptMethod attribute on the web service like so:

[ScriptMethod(ResponseFormat = ResponseFormat.Json, UseHttpGet = true, XmlSerializeString = false)]

I also tried using the ASPProxy class found on this forum post. This also didn't work properly and there were a few minor irritations with the code. It all seemed pretty silly since with the ScriptManager service reference I had a client side proxy that I knew would work properly. So I created the AspWebServiceProxy object that would allow you to simply pass in a reference to the client side web service proxy object and method and use that:

AspWebServiceProxy = function (conn)
           {
              AspWebServiceProxy.superclass.constructor.call(this);
              Ext.apply(this, conn);
           };

Ext.extend(AspWebServiceProxy, Ext.data.DataProxy,
{
     load : function (params, reader, callback, scope, arg)
            {
               var userContext = {
                                    callback: callback,
                                    reader: reader,
                                    arg: arg,
                                    scope: scope
                                 };

               var proxyWrapper = this;

               //Handles the response we get back from the web service call
               var webServiceCallback = function(response)
                                        {
                                            proxyWrapper.loadResponse(response, userContext);
                                        }

               var serviceParams = [];

               //Convert the params into an array of values so that they can be used in the call (note assumes that the properties on the object are in the correct order)
               for (var property in params)
               {
                  serviceParams.push(params[property]);
               }

               //Add the webservice callback handlers
               serviceParams.push(webServiceCallback);
               serviceParams.push(this.handleErrorResponse);

               //Make the actual ASP.Net web service call
               this.webServiceProxyMethod.apply(this.webServiceProxy, serviceParams);
            },

     handleErrorResponse : function(response, userContext, methodName)
                           {
                              alert("Error while calling web service method:" + methodName + "\n" + response.get_message());
                           },

     loadResponse : function (response, userContext, methodName)
                    {
                        var result = userContext.reader.readRecords(response);
                        userContext.callback.call(userContext.scope, result, userContext.arg, true);
                    }

});

So if you had an ASP.Net web service script reference on your page, like this:

<asp:ScriptManager ID="PageScriptManager" runat="server">
    <Services>
        <asp:ServiceReference Path="/WebServices/OrderService.asmx" InlineScript="false" />
    </Services>
</asp:ScriptManager>

to use the proxy class one would change the code at the start of the article to look like this:

var ds = new Ext.data.Store({
    proxy: new AspWebServiceProxy(
         {
          webServiceProxy: Example.OrderService,
          webServiceProxyMethod: Example.OrderService.GetOrders,
          params: {userId: getUserId(), day: new Date()}
         }),
    reader: new Ext.data.JsonReader(
        {
           id: 'WorkDoneId'
        },
    ...
});

I'm sure that I'll be tweaking the AspWebServiceProxy object over time as I discover its short comings, but at least it is a decent start and is hopefully useful to others as well. One of my concerns are the manner in which the attributes on the extraParams object must be specified in the same order as the web service method parameters. I did it this way to keep it consistent with all the other examples I had seen where they used an object as opposed to an array. I may change this, but I like the fact that with the current approach the parameter value pairs are obvious.

Update - 2 Nov 2007: Changed the service options of extraParams to params. So now one can use dataStore.load({params: {userId: 4, day: new Date()} }) to load data after the creation of the data source.

Dojo Based CruiseControl Statistics Graphs

Eden Ridgway — Sun, 22 Apr 2007 07:00:28 GMT

I have been making a lot of changes to the CruiseControl Statistics replacement I released a while ago. Most significantly I changed it to use the Dojo charting library and divided the information into tabs. It now looks like this:

Reasons for the Changes

The reasons for these changes were as follows:

The display was becoming cluttered with a lot of information that could be nicely separated into tabs.
I can now defer the rendering of the tables and historic graphs until the point where the user actually wants to view them. This should dramatically reduce the load time on projects that have a lot of statistics.
The dojo charting library is significantly more flexible than PlotKit and will allow for more freedom in future versions. I also expect that others may want graphs that use more than one plotter, like a combination bar and line chart.
I will be able to tweak the charting library to do things like rotate the x-axis tick labels so that more of them can be placed on the axis.

Purpose of the Graphs

In my previous posts I forgot to explain why I wanted to do all of this in the first place. While having good looking graphs instead of an ugly table is great, what I really wanted to get the following from this, is the following:

Determine whether or not the development teams were following our development process and making certain their code compiles before checking in. A red flag for me is a project that is frequently broken. I chat to the team and determine the root cause and rectify it.
Being able to easily spot projects that are taking long or progressively longer to build and see how I can ensure that it stays at acceptable levels so that the team continue to use CruiseControl effectively.
Determine if the teams are fixing their FxCop violations and writing and using unit tests on their projects.

Other future benefits I would like to get from this are:

The addition of line counting statistics so I can see how much projects are being worked on and get a sense of code churn. Of course line counting is fraught with problems, but at least it's something.
Track project complexity and ensure that we keep it under control.

Installation

The installation instructions are the same as the previous version.

Works well in the following browsers:
Has some rendering quirks in:
Have not tested:

Downloads / Updates

[Version 2.3 - 27 April 2007] - Changes:
1. Significant restructuring of the graph configuration logic (now lives in a separate file).
2. Removed the timeline logic of the graphs.
3. Added coverage graphs (assumes that the coverage information is stored in the Coverage element).
4. Added tooltips to the graphs (there are problems in IE).
5. Graph rendering is now queued with a "Loading..." message in the graph display area. This has helped improve the user experience on the History tab which can be quite slow when displaying a lot of data.
6. Miscellaneous other changes in tick generation etc.
7. Table cells of greater than 25 characters now wrap. The exception to this is when you have a word without any spaces that exceeds this and is obviously not wrapped. This sometimes happens in the "Build Error Message" column where the path to a file may be very long. I'm considering implementing a solution that splits these up based on a best guess separator (underscore, full stop, or slash).
8. The successful/failed builds are color coded in the detailed table now.
[Version 2.4 - 28 April 2007] - Changes:
1. Fixed a problem that was caused when some statistics had coverage values and others didn't.
2. Dramatically improved the performance of the summary logic and it now summarises data on demand. The performance problem here was due to the fact I was using the dojo ArrayList method in the distinct method. On a report with over 1.7MB of data this caused Internet Explorer (it doesn't happen in FF) to display the "A script on this page is causing Internet Explorer to run slowly" message. The performance tweaks have removed this problem, but I'm concerned that it may reappear when processing a much larger file. I created a test case with over 8.5MB of data and I didn't receive the message. However the rendering of the tables on such a report is a bit of a joke. I need to look at further improving the performance of the table rendering functions (I don't believe that doing it in the XSLT is going to render the benefit one would expect due to the amount of HTML the client ends up downloading).
3. Split the detailed and summary tables on to separate tabs and now obviously only render them when the user wants to view the respective tab. However this is still too slow.
4. Increased the time delay between the rendering of the graphs to 150ms.
[Version 2.5 - 28 April 2007] - Changes:
1. Fixed a bug in the tick mark generation that I introduced while making the changes above. Note that now sometimes the last tick mark on the y-axis does not render. I suspect that this is due to the wonders of JavaScript floating point maths (I still need to investigate this fully), but it is a relatively minor issue so I'm not that phased about it.
2. Added the ability to drill down into the detail statistics from the summary table.
3. Now only averages, counts, etc the successful builds (except the for the failed count of course :) ).
4. Various code refactoring.
[Version 2.6 - 18 May 2007] - Changes:
1. Fixed a bug caused by unescaped string sequences in the JavaScript statistics object literal definition (i.e. \u was causing a hexidecimal value expected error in IE). Thank you Mo for reporting this.
[Version 2.7 - 22 May 2007] - Changes:
1. Have improved the manner in which the graphs are configured thereby making it easier for others to add their own custom settings. The details of how this is setup can be found here.
2. Now can jump to the build from the detailed data table.
3. Added pre-configured complexity and sloc graphs.

Concerns

The Dojo getBorderBox method that the tab control uses is really slow. In some test cases it takes over 5 seconds in total.
When switching out of the statistics page Internet Explorer (not FF) struggles and it can take a few seconds just to go back to the listing page. I'm wondering if this is caused by the Dojo framework in some way or by the graphs etc.
Internet Explorer is significantly slower than FireFox and Opera. This is inline with the graph rendering performance results at http://www.ajaxperformance.com/?p=58.
The detailed table rendering performance for projects with lots of builds. A possible solution to this is to introduce paging.

To Do

Use a frameworks such as JsUnit and JSMock (maybe even J3Unit) to properly test the logic. The JavaScript Coverage Validator may also be of use.
Cut down the size of the dojo toolkit distributed with the graphs as it takes quite some time to deploy and there is a lot unnecessary files in the deployment package at the moment.. Basically I need to create a modified build of the Dojo Toolkit that only includes the pieces that I am using.
Find a solution to the single word greater than 25 characters problem in the details table which causes the columns to be very wide.
Find a solution to the Internet Explorer tick tooltip problem. At the moment only the ticks for the last series rendered display tooltips. I believe this is because of the way the Graphing library nest each series inside of a div and layer them on top of each other, thereby preventing the data points on the underlying series to receive the events required to display the ticks.
Possibly make the ticks a little more informative (requires modifications to the Dojo library). I could use the Dojo tooltip widget to display detailed build information or I could place the information to the right of the graph.
Implement filtering in the historic build page that allows graphs to be filtered by date range. I'm a bit concerned about the performance impact of having one master filter that then causes all the graphs to be redrawn.

A Simple Dojo Charting Example

Eden Ridgway — Fri, 13 Apr 2007 05:14:08 GMT

When evaluating the Dojo Charting library as an option for my CruiseControl statistics page replacement it was very evident that the Dojo team hadn't gotten around to providing documentation for this donated library. If you are unfamiliar with what the features the library offers you should take a look at this Ajaxian post. There is also a test page for the library which I could have sworn was down a day or two ago.

In the interest of possibly helping others save some time getting into the chrting library I wanted to post a basic introductory example. Please be aware that I do not know this library in depth, nor have I worked with the Dojo libraries extensively.

What is great about the Dojo charting implementation is that it is very versatile. This however comes at the cost of a slightly more involved structure which makes it somewhat more tedious to use than PlotKit. The composition of the various graphing objects is illustrated in the diagram below. As you can see from the relationships in the diagram they have allowed for features such as the ability to render multiple different graphs, using different plotters, in one plot area. They also have different implementations of the Axis, PlotArea and Plotter objects for VML and SVG rendering.

The Example

So to create a graph one would start off by including a reference to the dojo library and the required namespaces:

You then define or retrieve the graph data and store it in the collection store. This then will be fed into your graph as a series, defined like this:

Next come the axis definitions where you can specify the data display range, data source and the tick labels:

//Define the x-axis
var xAxis = new dojo.charting.Axis();

//Set the upper and lower data range values
xAxis.range = { lower: exampleData[0].time, upper: exampleData[exampleData.length-1].time };

xAxis.origin = "max";
xAxis.showTicks = true;
xAxis.label = "Example chart";

//Setup the x tick marks on the chart
xAxis.labels = [
{ label: 'First', value: 20 },
{ label: 'Second', value: 25 },
{ label: 'Third', value: 35 },
{ label: 'Fourth', value: 50 },
{ label: 'Fifth', value: 55 }
];

//Define the y-axis
var yAxis = new dojo.charting.Axis();
yAxis.range = { lower: 0, upper: 5000 };
yAxis.showLines = true;
yAxis.showTicks = true;
yAxis.label = "Time Taken";

//Setup the y tick marks on the chart
yAxis.labels = [
{ label: "0s", value: 0 },
{ label: "1s", value: 1000 },
{ label: "2s", value: 2000 },
{ label: "3s", value: 3000 },
{ label: "4s", value: 4000 },
{ label: "5s", value: 5000 }
];

You then define how the data will be plotted by defining a Plot and assigning the series with a plotter to render it:

var chartPlot = new dojo.charting.Plot(xAxis, yAxis);

chartPlot.addSeries({
data: timeSeries,
plotter: dojo.charting.Plotters.CurvedArea
});

This then needs to be rendered into a specific area, so one defines the PlotArea and adds the Plot to it:

var chartPlotArea = new dojo.charting.PlotArea();
chartPlotArea.size = { width: 380, height: 170 };
chartPlotArea.padding = { top: 20, right: 20, bottom: 30, left: 50 };

//Add the plot to the area
chartPlotArea.plots.push(chartPlot);

Finally one needs to create the chart and add the PlotArea to it. The chart also needs to have a container element, which one assigns to the chart.node.

var chart = new dojo.charting.Chart(null, "Example chart", "This is the example chart description");

//Add the plot area at an offset of 10 pixels from the top left
chart.addPlotArea({ x: 10, y: 10, plotArea: chartPlotArea });

//Setup the chart to be added to the DOM on load
dojo.addOnLoad(function()
{
chart.node = dojo.byId("GraphContainerArea");
chart.render();
});

And voila, you have a graph that looks like this:

To see the graph in action, take a look at this demo page.

CruiseControl.Net Statistics Graphs using PlotKit

Eden Ridgway — Tue, 27 Mar 2007 07:23:42 GMT

[Update: This does not apply to the Dojo Toolkit vesion of the Statistics Plugin. For information on how to customize that version go to this page.]

As promised in my previous post, CruiseControl Statistics Graphs, here is a high level description of the implementation that allowed me to display graphs in CruiseControl.Net. I've also included a quick guide on how to extend the report to include your own custom statistics that you may have published as part of your build.

After I had decided to create a graphical view of the CruiseControl statistics I assessed various options that were available. My immediate thoughts were to use XSLT to create either SVG or VML graphs. The problem is that I wanted something that was quick and easy to implement and quite frankly the effort required to do all of this using xslt was just too great. My attentions then turned to doing the majority of the processing in JavaScript. The Dojo Charting library was the first solution that sprung to mind, but on further investigation is appeared to be undergoing some restructuring and the documentation is almost non-existent. It was then that I came across PlotKit, a cross browser charting framework built on top of MochiKit. It was simple and had documentation so it was a clear winner.

So the Statistics Graph solution ended up being structured as follows:

StatisticsGraphs.xsl - creates JSON representation of the detailed stats data and the general page structure
StatisticsGraphs.js - contains the logic to summarise the detailed stats data and to create the plotkit graphs and tables
MochiKit_lite_packed.js, PlotKit_Packed.js - packed versions of the libraries that provide the graphing functionality (from PlotKit)
excanvas.js - provides canvas emulation for Internet Explorer (from PlotKit)

If one wanted to create their own custom summary graph of new statistics on the page (for more info on including custom statistics see Grant Drake's blog post), you would open up StatisticsGraphs.js and do the following:

Edit the generateDailySummaries function and add your summarised statistic to daySummary

var daySummary =
{
day : day,
successfulBuildCount : count(currentStatistics, "Status", function(item) { return (item["Status"] == "Success"); }),
failedBuildCount : count(currentStatistics, "Status", function(item) { return (item["Status"] == "Failure"); }),
lastBuildLabel : getLastValue(currentStatistics, "BuildLabel"),
testCount : testCount,
testsPassed : testCount - testFailures - testsIgnored,
testFailures : testFailures,
testsIgnored : testsIgnored,
fxCopWarnings : average(currentStatistics, "FxCop Warnings"),
fxCopErrors : average(currentStatistics, "FxCop Errors"),
averageBuildDuration: average(currentStatistics, "Duration"),
minBuildDuration: min(currentStatistics, "Duration"),
maxBuildDuration: max(currentStatistics, "Duration"),

//Insert new summarised statistics here
averageCodeCoverage : average(currentStatistics, "CodeCoverage")
};
Create a new function to create the graph. Note that the createGraph function creates the title, graph and legend for you. eg.

function createCodeCoverageGraph()
{
createGraph({
graphName : "Average Daily Code Coverage",
series : [
{
name : "Average Code Coverage",
attributeName: "averageCodeCoverage",
color : Color.greenColor()
}
]
})
}
Then call your new function in the setupGraphs

function setupGraphs()
{
createBuildReportGraph();
createBuildDurationGraph();
createTestGraph();
createFxCopGraph();

//Insert calls to functions that create new graphs here
createCodeCoverageGraph();
}

So if you are keen to give it a go take a look at my previous post for the download.

CruiseControl Statistics Graphs

Eden Ridgway — Fri, 23 Mar 2007 05:04:17 GMT

[Update: Please note that this approach has been replaced by a Dojo based solution available from http://www.ridgway.co.za/archive/2007/04/22/Dojo-Based-CruiseControl-Statistics-Graphs.aspx]

Lets face it, the new CruiseControl Statistics report is damn ugly and is not as useful as it could be. So I decided to remedy this by replacing the standard statistics xslt transform with one that generates both graphs and tables. The results are graphs in CruiseControl that look something like the images below. So what my new CruiseControl Statistics report gives you at the moment is as follows:

Build Report Graph - the number of failed/successful builds per day and the last build of each day
Build Duration Graph - the average and maximum build duration per day
Test Summary Graph - the average number of tests that passed, failed or were ignored
FxCop Summary Graph - the average number of warnings and errors per day
Build Summary Statistics Table - a table containing the data that was used to generate the daily stats
Build Detailed Statistics Table - contains the original detailed statistics data as would have been displayed in the old statistics report

Installation Instructions

I'll get on to the actual implementation in another post, but for now here is how to get it working on your CruiseControl.Net build server:

Download the latest stable release: CCNetStatisticsGraphs-1.3.zip . Updates are also present at the end of this post if you feel adventurous. [Note: I have a new version available that uses the Dojo library]
Unzip the webdashboard folder over the current CruiseControl webdashboard folder (usually found at C:\Program Files\CruiseControl.NET)
Open the dashboard.config file in the webdashboard folder and change the the projectStatisticsPlugin setting to point to the new StatisticsGraphs.xsl stylesheet, like this:

<projectStatisticsPlugin xslFileName="xsl\StatisticsGraphs.xsl" />

If it everything is in order you should see the following type of report when viewing your project statics:

Future improvements to this will largely depend on the interest generated by this post. If enough people find it useful I'll add new features. Of course if you extend it, please let me know about it. Some potential enhancements are:

The ability to filter the report data
Additional reports such as code coverage and complexity reports (based on whether or not the stats are present of course)
Static table headers and sorting

Notes

Using PlotKit as the graph rendering engine unfortunately also results in the following issues in the graphs:

Graphs where you would only expect integer values, display decimals. There does not seem to be any easy solution to this apart from modifying PlotKit code or ignoring the auto-scaled y-axis figures and writing one's own routine to determine the values and specify the y-axis tick values. [Integer y-axis values were implemented in version 1.4]
There are problems with the labels being truncated on both the vertical and horizontal axis. One can alleviate the problem somewhat using different margins but at some point you run into trouble. The space required really needs to be determined dynamically (by the graphing engine). [Problem mostly removed in version 1.5]
With the FxCop report it is difficult to predict whether or not the errors will be greater than warnings and vice-a-versa and hence you cannot use a filled line graph because it may cover up the one set of values. What would be nice is if you could have the one set of values peek through the other using alpha blending. Canvas does provide the ability to do this to some degree through the globalCompositeOperation property on the canvas, however this does not work in the IECanvas emulation layer.

Given the problems mentioned above I may very well switch to using another graphing approach or roll a modified version of Plotkit out.

Some of the suggested enhancements I have received are:

Removal of outliers - Severe spikes completely mess up the scaling of the graphs. This one requires some thought because obviously it could also have the affect of making people believe that the graphs are incorrect when in fact the outliers have been removed.
Horizontal scaling - Add in zero values to the graphs for missing days from the start of recorded builds. [Implemented in version 1.4]
Removal of graphs if there no values - This one I had considered before and since someone else has asked for it, I'm going to implement it it. It's most obvious when you are not using FxCop and end up with a horrible empty grey block in your report. [Implemented in version 1.4]

Updates

[Version 1.2 - 8 April 2007] - Fixed an issue caused by apostrophes in the generated statistics (new version 1.2 released) (reported by Grant Drake).
[Version 1.3 - 8 April 2007] - Fixed an issue in the duration calculation where the duration had an int + string value which of course resulted in a string concatenation :) (reported by Grant Drake).
[Version 1.4 - 9 April 2007] - Implemented the following features:
1. Graphs are not displayed when there are no values for them
2. The x-series values are now a timeline not just all the builds next to each other. So if there was a build on the 1st of a month and the next build was only on the 4th, zero values are filled in for the 2nd and 3rd of the month. This has a down side though because at the moment it is displaying weekends for which most projects will have no activity and if a project is worked on sporadically you have large gaps in the graph. I do believe however there is value in seeing the activity over time in this way and maybe it it worth having some sort of filtering option that allows one to remove weekends and/or days for which there are no statistics.
3. The y-axis no longer displays values (I manually determined the y-ticks and added them to the graph).
Only download this release if you are curious about the new functionality. I will be writing a series of WatiN tests to ensure that everything is working as expected as the mini-project evolves. My biggest concern for this version is with locales specifically in relation to dates. To fill in missing days I've used JavaScript date functions and there is an assumption that the browser's locale is the same as that of the server. Unfortunately this is due to the fact that the CruiseControl month in the statistics xml file is a short name. Furthermore I cannot use the StartTime value because it is possible that the format of that information is not consistent: one of my example statistics files has StartTime values in both yyyy/MM/dd and dd/MM/yyyy format.
I found out on the 22 April 2007 that this version suffers from a daylight savings time bug that affected projects that started when daylight savings was in effect. I have fixed this release.
[Version 1.5 - 10 April 2007] - Modified the PlotKit canvas to allow rendering of HTML labels (i.e. it won't escape your tags) and now draws the graph border and tick marks in the same colour as the axis label colour (I may remove this modification and simply create a new custom canvas). This has allowed me to reduce the font size of the x-axis labels and put the date on a separate line to the build label. Unfortunately it has also meant that I'm now distributing the uncompressed version of the PlotKit library (I may use a JavaScript compressor on them later). I found out on the 22 April 2007 that this version suffers from a daylight savings time bug that affected projects that started when daylight savings was in effect. I have fixed this release.
[Version 2.1] - This is available from this post.