CruiseControl

Adding Custom Graphs to the CruiseControl.Net Statistics Replacement

Eden Ridgway — Mon, 21 May 2007 06:01:34 GMT

In version 2.7 of CCNet graphs I made some further improvements to the manner in which graphs are configured. Now any customisations that need to be done are housed in the GraphConfiguration.js file (in the webdashboard\javascript folder). The logic in the graph generation is now also resilient to problems such as blank or non-numeric data in the custom nodes.

If you want to include custom data in your report you should read the Statistics Publisher Wiki page to get the it included in the report.xml file (which you will find in your artifacts folder for each project).

Configuration Setup

There are 3 areas of configuration, the:

Recent Graphs Tab - the details of which are stored in the _recentGraphConfigurations array of configuration/option objects. The datasource of each refers to the _recentStatistics object array, that contains up to the last 20 build statistics.
Manner in which Summary Data is calculated - this is stored in the _summaryConfiguration object that contains functions that accept successful and failed build arrays for a day and return an numeric value for each.
Historic Builds Tab - is configured in the same way as the Recent Graphs Tab, except is defined in the _historicGraphConfigurations array and each configuration object uses the _summarisedStatistics datasource. This datasource contains, the item index, DurationInSeconds, TestsPassed (calculated properties not present in the report.xml file) and the summary data properties defined in _summaryConfiguration.

Example Customisation

Lets say that one wanted to add a complexity graph to the recent and historic tabs, the changes required would be this:

One would edit the _recentGraphConfigurations array and add the following (assuming that the xml element node [in the report.xml file] that contains the data is called AverageComplexity):

var _recentGraphConfigurations =
    [
        //... Other configuration objects excluded for brevity
        {
            graphName: "Complexity",
            dataSource: _recentStatistics,
            numXTicks: _numberRecentGraphXTicks,
            series: [
                            { name: "Average Complexity", attributeName: "AverageComplexity", color: "blue" }
                          ]
        }
    ];
Then the manner in which you want to summarise the data on a daily basis needs to be defined in summaryConfiguration. Note that all the standard summary functions are contained in the QueryFunctions.js file and include methods such as (getLastValue, select, distinct, sum, average, count, min and max). Say we wanted to display the average complexity value for the day, the configuration would be defined like this:

var _summaryConfiguration =
    {
        //Other attributes...
        averageComplexity: function(successfulBuilds, failedBuilds) { return average(successfulBuilds, "AverageComplexity") }
    };
Lastly the summarised data collected by the function above for each day must be configured for the history tab, like this:

var _historicGraphConfigurations =
    [
        //Other configuration objects...
        {
            graphName: "Complexity",
            dataSource: _summarisedStatistics,
            numXTicks: _numberHistoricGraphXTicks,
            series: [
                       { name: "Average Complexity", attributeName: "averageComplexity", color: "blue" }
                          ]
        }
    ];

Note how the attribute name here corresponds to the attribute name defined in _summaryConfiguration and not the original statistic configuration element.

Hopefully the rest is self explanatory. If something does not make sense or I need to explain the configuration approach in more detail, please leave a message on this post.

Update: 7 July 2007 - Clarified some points of confusion around the source of some of the data.

Dojo Based CruiseControl Statistics Graphs

Eden Ridgway — Sun, 22 Apr 2007 07:00:28 GMT

I have been making a lot of changes to the CruiseControl Statistics replacement I released a while ago. Most significantly I changed it to use the Dojo charting library and divided the information into tabs. It now looks like this:

Reasons for the Changes

The reasons for these changes were as follows:

The display was becoming cluttered with a lot of information that could be nicely separated into tabs.
I can now defer the rendering of the tables and historic graphs until the point where the user actually wants to view them. This should dramatically reduce the load time on projects that have a lot of statistics.
The dojo charting library is significantly more flexible than PlotKit and will allow for more freedom in future versions. I also expect that others may want graphs that use more than one plotter, like a combination bar and line chart.
I will be able to tweak the charting library to do things like rotate the x-axis tick labels so that more of them can be placed on the axis.

Purpose of the Graphs

In my previous posts I forgot to explain why I wanted to do all of this in the first place. While having good looking graphs instead of an ugly table is great, what I really wanted to get the following from this, is the following:

Determine whether or not the development teams were following our development process and making certain their code compiles before checking in. A red flag for me is a project that is frequently broken. I chat to the team and determine the root cause and rectify it.
Being able to easily spot projects that are taking long or progressively longer to build and see how I can ensure that it stays at acceptable levels so that the team continue to use CruiseControl effectively.
Determine if the teams are fixing their FxCop violations and writing and using unit tests on their projects.

Other future benefits I would like to get from this are:

The addition of line counting statistics so I can see how much projects are being worked on and get a sense of code churn. Of course line counting is fraught with problems, but at least it's something.
Track project complexity and ensure that we keep it under control.

Installation

The installation instructions are the same as the previous version.

Works well in the following browsers:
Has some rendering quirks in:
Have not tested:

Downloads / Updates

[Version 2.3 - 27 April 2007] - Changes:
1. Significant restructuring of the graph configuration logic (now lives in a separate file).
2. Removed the timeline logic of the graphs.
3. Added coverage graphs (assumes that the coverage information is stored in the Coverage element).
4. Added tooltips to the graphs (there are problems in IE).
5. Graph rendering is now queued with a "Loading..." message in the graph display area. This has helped improve the user experience on the History tab which can be quite slow when displaying a lot of data.
6. Miscellaneous other changes in tick generation etc.
7. Table cells of greater than 25 characters now wrap. The exception to this is when you have a word without any spaces that exceeds this and is obviously not wrapped. This sometimes happens in the "Build Error Message" column where the path to a file may be very long. I'm considering implementing a solution that splits these up based on a best guess separator (underscore, full stop, or slash).
8. The successful/failed builds are color coded in the detailed table now.
[Version 2.4 - 28 April 2007] - Changes:
1. Fixed a problem that was caused when some statistics had coverage values and others didn't.
2. Dramatically improved the performance of the summary logic and it now summarises data on demand. The performance problem here was due to the fact I was using the dojo ArrayList method in the distinct method. On a report with over 1.7MB of data this caused Internet Explorer (it doesn't happen in FF) to display the "A script on this page is causing Internet Explorer to run slowly" message. The performance tweaks have removed this problem, but I'm concerned that it may reappear when processing a much larger file. I created a test case with over 8.5MB of data and I didn't receive the message. However the rendering of the tables on such a report is a bit of a joke. I need to look at further improving the performance of the table rendering functions (I don't believe that doing it in the XSLT is going to render the benefit one would expect due to the amount of HTML the client ends up downloading).
3. Split the detailed and summary tables on to separate tabs and now obviously only render them when the user wants to view the respective tab. However this is still too slow.
4. Increased the time delay between the rendering of the graphs to 150ms.
[Version 2.5 - 28 April 2007] - Changes:
1. Fixed a bug in the tick mark generation that I introduced while making the changes above. Note that now sometimes the last tick mark on the y-axis does not render. I suspect that this is due to the wonders of JavaScript floating point maths (I still need to investigate this fully), but it is a relatively minor issue so I'm not that phased about it.
2. Added the ability to drill down into the detail statistics from the summary table.
3. Now only averages, counts, etc the successful builds (except the for the failed count of course :) ).
4. Various code refactoring.
[Version 2.6 - 18 May 2007] - Changes:
1. Fixed a bug caused by unescaped string sequences in the JavaScript statistics object literal definition (i.e. \u was causing a hexidecimal value expected error in IE). Thank you Mo for reporting this.
[Version 2.7 - 22 May 2007] - Changes:
1. Have improved the manner in which the graphs are configured thereby making it easier for others to add their own custom settings. The details of how this is setup can be found here.
2. Now can jump to the build from the detailed data table.
3. Added pre-configured complexity and sloc graphs.

Concerns

The Dojo getBorderBox method that the tab control uses is really slow. In some test cases it takes over 5 seconds in total.
When switching out of the statistics page Internet Explorer (not FF) struggles and it can take a few seconds just to go back to the listing page. I'm wondering if this is caused by the Dojo framework in some way or by the graphs etc.
Internet Explorer is significantly slower than FireFox and Opera. This is inline with the graph rendering performance results at http://www.ajaxperformance.com/?p=58.
The detailed table rendering performance for projects with lots of builds. A possible solution to this is to introduce paging.

To Do

Use a frameworks such as JsUnit and JSMock (maybe even J3Unit) to properly test the logic. The JavaScript Coverage Validator may also be of use.
Cut down the size of the dojo toolkit distributed with the graphs as it takes quite some time to deploy and there is a lot unnecessary files in the deployment package at the moment.. Basically I need to create a modified build of the Dojo Toolkit that only includes the pieces that I am using.
Find a solution to the single word greater than 25 characters problem in the details table which causes the columns to be very wide.
Find a solution to the Internet Explorer tick tooltip problem. At the moment only the ticks for the last series rendered display tooltips. I believe this is because of the way the Graphing library nest each series inside of a div and layer them on top of each other, thereby preventing the data points on the underlying series to receive the events required to display the ticks.
Possibly make the ticks a little more informative (requires modifications to the Dojo library). I could use the Dojo tooltip widget to display detailed build information or I could place the information to the right of the graph.
Implement filtering in the historic build page that allows graphs to be filtered by date range. I'm a bit concerned about the performance impact of having one master filter that then causes all the graphs to be redrawn.

A Simple Dojo Charting Example

Eden Ridgway — Fri, 13 Apr 2007 05:14:08 GMT

When evaluating the Dojo Charting library as an option for my CruiseControl statistics page replacement it was very evident that the Dojo team hadn't gotten around to providing documentation for this donated library. If you are unfamiliar with what the features the library offers you should take a look at this Ajaxian post. There is also a test page for the library which I could have sworn was down a day or two ago.

In the interest of possibly helping others save some time getting into the chrting library I wanted to post a basic introductory example. Please be aware that I do not know this library in depth, nor have I worked with the Dojo libraries extensively.

What is great about the Dojo charting implementation is that it is very versatile. This however comes at the cost of a slightly more involved structure which makes it somewhat more tedious to use than PlotKit. The composition of the various graphing objects is illustrated in the diagram below. As you can see from the relationships in the diagram they have allowed for features such as the ability to render multiple different graphs, using different plotters, in one plot area. They also have different implementations of the Axis, PlotArea and Plotter objects for VML and SVG rendering.

The Example

So to create a graph one would start off by including a reference to the dojo library and the required namespaces:

You then define or retrieve the graph data and store it in the collection store. This then will be fed into your graph as a series, defined like this:

Next come the axis definitions where you can specify the data display range, data source and the tick labels:

//Define the x-axis
var xAxis = new dojo.charting.Axis();

//Set the upper and lower data range values
xAxis.range = { lower: exampleData[0].time, upper: exampleData[exampleData.length-1].time };

xAxis.origin = "max";
xAxis.showTicks = true;
xAxis.label = "Example chart";

//Setup the x tick marks on the chart
xAxis.labels = [
{ label: 'First', value: 20 },
{ label: 'Second', value: 25 },
{ label: 'Third', value: 35 },
{ label: 'Fourth', value: 50 },
{ label: 'Fifth', value: 55 }
];

//Define the y-axis
var yAxis = new dojo.charting.Axis();
yAxis.range = { lower: 0, upper: 5000 };
yAxis.showLines = true;
yAxis.showTicks = true;
yAxis.label = "Time Taken";

//Setup the y tick marks on the chart
yAxis.labels = [
{ label: "0s", value: 0 },
{ label: "1s", value: 1000 },
{ label: "2s", value: 2000 },
{ label: "3s", value: 3000 },
{ label: "4s", value: 4000 },
{ label: "5s", value: 5000 }
];

You then define how the data will be plotted by defining a Plot and assigning the series with a plotter to render it:

var chartPlot = new dojo.charting.Plot(xAxis, yAxis);

chartPlot.addSeries({
data: timeSeries,
plotter: dojo.charting.Plotters.CurvedArea
});

This then needs to be rendered into a specific area, so one defines the PlotArea and adds the Plot to it:

var chartPlotArea = new dojo.charting.PlotArea();
chartPlotArea.size = { width: 380, height: 170 };
chartPlotArea.padding = { top: 20, right: 20, bottom: 30, left: 50 };

//Add the plot to the area
chartPlotArea.plots.push(chartPlot);

Finally one needs to create the chart and add the PlotArea to it. The chart also needs to have a container element, which one assigns to the chart.node.

var chart = new dojo.charting.Chart(null, "Example chart", "This is the example chart description");

//Add the plot area at an offset of 10 pixels from the top left
chart.addPlotArea({ x: 10, y: 10, plotArea: chartPlotArea });

//Setup the chart to be added to the DOM on load
dojo.addOnLoad(function()
{
chart.node = dojo.byId("GraphContainerArea");
chart.render();
});

And voila, you have a graph that looks like this:

To see the graph in action, take a look at this demo page.

CruiseControl.Net Statistics Graphs using PlotKit

Eden Ridgway — Tue, 27 Mar 2007 07:23:42 GMT

[Update: This does not apply to the Dojo Toolkit vesion of the Statistics Plugin. For information on how to customize that version go to this page.]

As promised in my previous post, CruiseControl Statistics Graphs, here is a high level description of the implementation that allowed me to display graphs in CruiseControl.Net. I've also included a quick guide on how to extend the report to include your own custom statistics that you may have published as part of your build.

After I had decided to create a graphical view of the CruiseControl statistics I assessed various options that were available. My immediate thoughts were to use XSLT to create either SVG or VML graphs. The problem is that I wanted something that was quick and easy to implement and quite frankly the effort required to do all of this using xslt was just too great. My attentions then turned to doing the majority of the processing in JavaScript. The Dojo Charting library was the first solution that sprung to mind, but on further investigation is appeared to be undergoing some restructuring and the documentation is almost non-existent. It was then that I came across PlotKit, a cross browser charting framework built on top of MochiKit. It was simple and had documentation so it was a clear winner.

So the Statistics Graph solution ended up being structured as follows:

StatisticsGraphs.xsl - creates JSON representation of the detailed stats data and the general page structure
StatisticsGraphs.js - contains the logic to summarise the detailed stats data and to create the plotkit graphs and tables
MochiKit_lite_packed.js, PlotKit_Packed.js - packed versions of the libraries that provide the graphing functionality (from PlotKit)
excanvas.js - provides canvas emulation for Internet Explorer (from PlotKit)

If one wanted to create their own custom summary graph of new statistics on the page (for more info on including custom statistics see Grant Drake's blog post), you would open up StatisticsGraphs.js and do the following:

Edit the generateDailySummaries function and add your summarised statistic to daySummary

var daySummary =
{
day : day,
successfulBuildCount : count(currentStatistics, "Status", function(item) { return (item["Status"] == "Success"); }),
failedBuildCount : count(currentStatistics, "Status", function(item) { return (item["Status"] == "Failure"); }),
lastBuildLabel : getLastValue(currentStatistics, "BuildLabel"),
testCount : testCount,
testsPassed : testCount - testFailures - testsIgnored,
testFailures : testFailures,
testsIgnored : testsIgnored,
fxCopWarnings : average(currentStatistics, "FxCop Warnings"),
fxCopErrors : average(currentStatistics, "FxCop Errors"),
averageBuildDuration: average(currentStatistics, "Duration"),
minBuildDuration: min(currentStatistics, "Duration"),
maxBuildDuration: max(currentStatistics, "Duration"),

//Insert new summarised statistics here
averageCodeCoverage : average(currentStatistics, "CodeCoverage")
};
Create a new function to create the graph. Note that the createGraph function creates the title, graph and legend for you. eg.

function createCodeCoverageGraph()
{
createGraph({
graphName : "Average Daily Code Coverage",
series : [
{
name : "Average Code Coverage",
attributeName: "averageCodeCoverage",
color : Color.greenColor()
}
]
})
}
Then call your new function in the setupGraphs

function setupGraphs()
{
createBuildReportGraph();
createBuildDurationGraph();
createTestGraph();
createFxCopGraph();

//Insert calls to functions that create new graphs here
createCodeCoverageGraph();
}

So if you are keen to give it a go take a look at my previous post for the download.

CruiseControl Statistics Graphs

Eden Ridgway — Fri, 23 Mar 2007 05:04:17 GMT

[Update: Please note that this approach has been replaced by a Dojo based solution available from http://www.ridgway.co.za/archive/2007/04/22/Dojo-Based-CruiseControl-Statistics-Graphs.aspx]

Lets face it, the new CruiseControl Statistics report is damn ugly and is not as useful as it could be. So I decided to remedy this by replacing the standard statistics xslt transform with one that generates both graphs and tables. The results are graphs in CruiseControl that look something like the images below. So what my new CruiseControl Statistics report gives you at the moment is as follows:

Build Report Graph - the number of failed/successful builds per day and the last build of each day
Build Duration Graph - the average and maximum build duration per day
Test Summary Graph - the average number of tests that passed, failed or were ignored
FxCop Summary Graph - the average number of warnings and errors per day
Build Summary Statistics Table - a table containing the data that was used to generate the daily stats
Build Detailed Statistics Table - contains the original detailed statistics data as would have been displayed in the old statistics report

Installation Instructions

I'll get on to the actual implementation in another post, but for now here is how to get it working on your CruiseControl.Net build server:

Download the latest stable release: CCNetStatisticsGraphs-1.3.zip . Updates are also present at the end of this post if you feel adventurous. [Note: I have a new version available that uses the Dojo library]
Unzip the webdashboard folder over the current CruiseControl webdashboard folder (usually found at C:\Program Files\CruiseControl.NET)
Open the dashboard.config file in the webdashboard folder and change the the projectStatisticsPlugin setting to point to the new StatisticsGraphs.xsl stylesheet, like this:

<projectStatisticsPlugin xslFileName="xsl\StatisticsGraphs.xsl" />

If it everything is in order you should see the following type of report when viewing your project statics:

Future improvements to this will largely depend on the interest generated by this post. If enough people find it useful I'll add new features. Of course if you extend it, please let me know about it. Some potential enhancements are:

The ability to filter the report data
Additional reports such as code coverage and complexity reports (based on whether or not the stats are present of course)
Static table headers and sorting

Notes

Using PlotKit as the graph rendering engine unfortunately also results in the following issues in the graphs:

Graphs where you would only expect integer values, display decimals. There does not seem to be any easy solution to this apart from modifying PlotKit code or ignoring the auto-scaled y-axis figures and writing one's own routine to determine the values and specify the y-axis tick values. [Integer y-axis values were implemented in version 1.4]
There are problems with the labels being truncated on both the vertical and horizontal axis. One can alleviate the problem somewhat using different margins but at some point you run into trouble. The space required really needs to be determined dynamically (by the graphing engine). [Problem mostly removed in version 1.5]
With the FxCop report it is difficult to predict whether or not the errors will be greater than warnings and vice-a-versa and hence you cannot use a filled line graph because it may cover up the one set of values. What would be nice is if you could have the one set of values peek through the other using alpha blending. Canvas does provide the ability to do this to some degree through the globalCompositeOperation property on the canvas, however this does not work in the IECanvas emulation layer.

Given the problems mentioned above I may very well switch to using another graphing approach or roll a modified version of Plotkit out.

Some of the suggested enhancements I have received are:

Removal of outliers - Severe spikes completely mess up the scaling of the graphs. This one requires some thought because obviously it could also have the affect of making people believe that the graphs are incorrect when in fact the outliers have been removed.
Horizontal scaling - Add in zero values to the graphs for missing days from the start of recorded builds. [Implemented in version 1.4]
Removal of graphs if there no values - This one I had considered before and since someone else has asked for it, I'm going to implement it it. It's most obvious when you are not using FxCop and end up with a horrible empty grey block in your report. [Implemented in version 1.4]

Updates

[Version 1.2 - 8 April 2007] - Fixed an issue caused by apostrophes in the generated statistics (new version 1.2 released) (reported by Grant Drake).
[Version 1.3 - 8 April 2007] - Fixed an issue in the duration calculation where the duration had an int + string value which of course resulted in a string concatenation :) (reported by Grant Drake).
[Version 1.4 - 9 April 2007] - Implemented the following features:
1. Graphs are not displayed when there are no values for them
2. The x-series values are now a timeline not just all the builds next to each other. So if there was a build on the 1st of a month and the next build was only on the 4th, zero values are filled in for the 2nd and 3rd of the month. This has a down side though because at the moment it is displaying weekends for which most projects will have no activity and if a project is worked on sporadically you have large gaps in the graph. I do believe however there is value in seeing the activity over time in this way and maybe it it worth having some sort of filtering option that allows one to remove weekends and/or days for which there are no statistics.
3. The y-axis no longer displays values (I manually determined the y-ticks and added them to the graph).
Only download this release if you are curious about the new functionality. I will be writing a series of WatiN tests to ensure that everything is working as expected as the mini-project evolves. My biggest concern for this version is with locales specifically in relation to dates. To fill in missing days I've used JavaScript date functions and there is an assumption that the browser's locale is the same as that of the server. Unfortunately this is due to the fact that the CruiseControl month in the statistics xml file is a short name. Furthermore I cannot use the StartTime value because it is possible that the format of that information is not consistent: one of my example statistics files has StartTime values in both yyyy/MM/dd and dd/MM/yyyy format.
I found out on the 22 April 2007 that this version suffers from a daylight savings time bug that affected projects that started when daylight savings was in effect. I have fixed this release.
[Version 1.5 - 10 April 2007] - Modified the PlotKit canvas to allow rendering of HTML labels (i.e. it won't escape your tags) and now draws the graph border and tick marks in the same colour as the axis label colour (I may remove this modification and simply create a new custom canvas). This has allowed me to reduce the font size of the x-axis labels and put the date on a separate line to the build label. Unfortunately it has also meant that I'm now distributing the uncompressed version of the PlotKit library (I may use a JavaScript compressor on them later). I found out on the 22 April 2007 that this version suffers from a daylight savings time bug that affected projects that started when daylight savings was in effect. I have fixed this release.
[Version 2.1] - This is available from this post.