Database

.Net Developer's Guide to Getting Started with Cassandra

Eden Ridgway — Thu, 05 Nov 2009 22:32:51 GMT

I've been quite curious about the various BigTable like implementations as they could have some bearing on what I am involved with at work. After reading various write ups about the competing approaches (e.g. HyperTable, HBase, Dynomite, Voldemort, Mongodb) I decided that Cassandra looked like the best option for what I'm after. What I didn't realise was that it wasn't going to be as straight forward as it appears to set it all up as there is no end-to-end guide of getting started with it. So the objective of this post is not to explain Cassandra but save you several hours of jumping all over the web trying to work out what you should do next and how to generate the necessary C# Cassandra Thrift client code. If parts of the post don't apply, e.g. VM setup, simply skip ahead to the parts that do. If you want more information about Cassandra you should check out the following posts:

So to get Cassandra up and running in a Virtual Machine (or physical if you choose) you should follow the following steps (valid as at 5 Nov 2010):

Create a virtual machine to host Ubuntu 9.10 x86 Desktop. I use VirtualBox as my VM host on windows so I will refer to this here but feel free to substitute it with you favourite VM software. The VM I created has 384MB allocated to it with an 8GB virtual hard drive using a bridged network adapter (so it can be accessed from the Windows XP host). Once you've gone through the install process install the client tools so that you change the screen resolution, share folders, etc. To do this with virtual box go to Devices > Install Guest Additions and then open up a terminal window and execute

sudo /media/cdrom/VBoxLinuxAdditions-x86.run

Once this is complete you will need to reboot your VM.
Now it is time to install Cassandra/Thrift prerequisites and software that we will to set everything up:

sudo apt-get install libboost-dev automake libtool flex bison pkg-config g++ -y
sudo apt-get install sun-java6-jdk -y
sudo apt-get apt-get install ant -y
Once that is done we can install other useful Ubuntu software that we will use:

sudo apt-get install rapidsvn -y
sudo apt-get install monodevelop -y

I also like to install the following:

sudo apt-get install nautilus-open-terminal -y
sudo apt-get install samba smbfs smbclient winbind -y
sudo apt-get install nautilus-share -y
You may discover that there is an issue with your Java runtime home directory. If that occurs, select Sun's Java Runtime as the default Java version by running the following command from the terminal window and selecting the third option:

sudo update-alternatives --config java
Now open up Nautilus by going to Places > Home Folder. Right click and create a Development folder. Within the Development folder create another two folders: Cassandra and Thrift.
We can now get the source for Cassandra and Thrift. Run RapidSVN by going to Applications > Programming > RapidSVN. Check out the source from SVN by going to the RapidSVN menu option Repository > Checkout. In the URL enter http://svn.apache.org/repos/asf/incubator/thrift/trunk and for the Destination Directory browse to the Thrift development folder. So it should look like this:

Now get the Cassandra code by doing the same thing but entering http://svn.apache.org/repos/asf/incubator/cassandra/trunk as the URL and selecting the Cassandra directory.
Once we have all the code we are ready to compile Cassandra, so let's do that by executing the following commands from the terminal window:

cd ~/Development/Cassandra
ant
Start the Cassandra server by opening a new terminal window and running the following commands:

sudo ~/Development/Cassandra/bin/cassandra -f

The running server will look like the screenshot below. You should take note of the thrift binding address of localhost/127.0.0.1:9160.
Now we want to get communicating with the server, so we will need to compile Thrift so that we can use it to generate our C# client:

cd ~/Development/Thrift
./bootstrap.sh
./configure
make
sudo make install
Now we can finally generate a C# client for the Thrift calls to Cassandra:

cd ~/Development/Cassandra/interface/
~/Development/Thrift/compiler/cpp/thrift -gen csharp cassandra.thrift

This will create the following directories inside of the Cassandra/interface folder: gen-csharp/Apache/Cassandra. These files along with the Thrift.dll will be copied into a new demo project we will create using MonoDevelop in the Development folder. So again in the terminal window execute:

mkdir ~/Development/CassandraDemo
cp ~/Development/Cassandra/interface/gen-csharp ~/Development/CassandraDemo -R
cp ~/Development/Thrift/lib/csharp/Thrift.dll ~/Development/CassandraDemo
Open up MonoDevelop and create a new console application by going to File > New > Solution. Now do the following:
1. Select Console project
2. Enter CassandraDemo as the name
3. For the location browse to the Development folder that we created in your home directory.
4. Uncheck the 'Create separate Solution directory' check box.
5. Click the Forward button
6. Don't make any changes on the next screen and just click Ok.
7. Right click on the CassandraDemo project in the left tree view and select Add > Add Files.... Navigate to ~/Development/CassandraDemo/gen-csharp/Apache/Cassandra and select all the files in the folder. You will now have a project that looks like this:
8. Add a reference to the Thrift assembly by right clicking on References and selecting Edit References. Change to the .Net Assembly tab and double click on the Thrift.dll and click OK.
9. Now we can add the sample code to test it all out. So double click on Main.cs to ensure that is the class in focus in the editor. Replace the contents of the file with this:
  
  namespace CassandraDemo
  {
      using System;
      using System.Collections.Generic;
      using System.Diagnostics;
  
      using Apache.Cassandra;
      using Thrift.Protocol;
      using Thrift.Transport;
  
      class Program
      {
          static void Main(string[] args)
          {
              TTransport transport = new TSocket("localhost", 9160);
              TProtocol protocol = new TBinaryProtocol(transport);
              Cassandra.Client client = new Cassandra.Client(protocol);
  
              Console.WriteLine("Opening connection");
              transport.Open();
  
              System.Text.Encoding utf8Encoding = System.Text.Encoding.UTF8;
  
              long timeStamp = DateTime.Now.Millisecond;
              ColumnPath nameColumnPath = new ColumnPath()
                                          {
                                              Column_family = "Standard1",
                                              Column = utf8Encoding.GetBytes("name")
                                          };
  
              Console.WriteLine("Inserting name columns");
  
              //Insert the data into the column 'name'
              client.insert("Keyspace1",
                            "1",
                            nameColumnPath,
                            utf8Encoding.GetBytes("Joe Bloggs"),
                            timeStamp,
                            ConsistencyLevel.ONE);
  
              client.insert("Keyspace1",
                            "2",
                            nameColumnPath,
                            utf8Encoding.GetBytes("Joe Soap"),
                            timeStamp,
                            ConsistencyLevel.ONE);
  
              //Simple single value get using the key to get column 'name'
              ColumnOrSuperColumn returnedColumn = client.get("Keyspace1", "1", nameColumnPath, ConsistencyLevel.ONE);
              Console.WriteLine("Column Data in Keyspace1/Standard1: name: {0}, value: {1}",
                                utf8Encoding.GetString(returnedColumn.Column.Name),
                                utf8Encoding.GetString(returnedColumn.Column.Value));
  
              Console.WriteLine("Getting splice range");
  
              //Read an entire row
              SlicePredicate predicate = new SlicePredicate()
                                         {
                                            Slice_range = new SliceRange()
                                                          {
                                                              //Start and Finish cannot be null
                                                              Start = new byte[0],
                                                              Finish = new byte[0],
                                                              Count = 10,
                                                              Reversed = false
                                                          }
                                         };
  
              ColumnParent parent = new ColumnParent() { Column_family = "Standard1" };
              Dictionary<string , List<ColumnOrSuperColumn>> results = client.multiget_slice("Keyspace1",
                                                                        new List<string>() { "1", "2"},
                                                                        parent,
                                                                        predicate,
                                                                        ConsistencyLevel.ONE);
  
              foreach (KeyValuePair<string, List<ColumnOrSuperColumn>> resultPair in results)
              {
                  Console.WriteLine("Key: {0}", resultPair.Key);
  
                  foreach (ColumnOrSuperColumn resultColumn in resultPair.Value)
                  {
                      Column column = resultColumn.Column;
                      Console.WriteLine("name: {0}, value: {1}", utf8Encoding.GetString(column.Name), utf8Encoding.GetString(column.Value));
                  }
              }
  
              Console.WriteLine("Closing connection");
              transport.Close();
          }
      }
  }
10. Hit F5 and watch the Application Output window. This is what you should see:

Are Database Migration Frameworks Worth the Effort?

Eden Ridgway — Sat, 03 Jan 2009 11:52:03 GMT

I’ve always used SQL scripts to manage database creation and migration. At my current company we have a database project with patch scripts that conditionally execute based on the database version. These scripts are simply run in sequence by an installer which stores the current database version as a database extended property. The approach relies on version based naming of SQL scripts and developers to remember to update the patch script whenever they make a structure or stored procedure change. Of course one could try to use a tool such as Redgate SQL compare to generate diff patch scripts but that won’t always suffice as that won’t capture refactoring such as column name changes or data migration (for more read The Problem with Database Diffs).

The use of SQL scripts is not without it’s problems. The most annoying of these is that a patch script may break as SQL will parse and validate your script against the existing table structure even if the statements are not going to run due to version check condition. This forces one to use nasty EXEC/sp_execute alter/create statements with double escaped strings (Yuck!). I find the following particularly problematic with this approach:

The string based SQL statements will always pass validation even though when they are executed they may be completely screwed
It is impractical to generate these by hand and the best way to create them is to use SQL Management Studio.
If your stored procedures are not wrapped in strings, to update the patch script requires you to script out the scripts from the database. Basically this increases development friction and discourages updating of the patch script.
It becomes difficult to determine when a structural change was made as one has to trawl through huge SQL scripts and it is not easy to use a regex search to find a change.

Hence I long for something a little more strongly typed, which is why the growing number of .Net based Database Migration frameworks have caught my eye. I want to be sure that there isn’t something obvious that I was missing out by not using one. Fortunately Ben Scheirman has done a fair amount of research already and wrote a fairly good round up of available .Net migration tools. Looking at the tools there are clearly two different types of frameworks available:

Structured SQL script execution – These tools provide a framework under which SQL scripts will be executed to create and upgrade a database to a specific version. Examples of these are: Tarantino and LiquiBase.
Rails like Active Record Migration – Commands to upgrade or downgrade the database schema and data can be coded in a .Net language, with the possibility of executing SQL statements if required. Tools that fall into this category are: RikMigrations, SubSonic Migrations, Machine.Migrations and Migrator.Net.

The obvious question that springs to mind when looking at these frameworks is whether or not the additional development friction (additional tasks that must be performed during the course of development) is worth the benefit. To be honest I don’t see a massive difference between the structured SQL script execution approach and what I’ve been doing all along. However the likes of LiquiBase do provide a functionality that nicely captures database refactorings and diffs, so maybe it could be used as more of a general toolset. However the Active Migration approach would be a more radical departure so I’d like to explore that more.

Here are my thoughts on Active Record Migration like frameworks:

Best paired with an ORM framework, such as NHibernate or Subsonic, where one is not writing a lot of custom SQL and hence the the upgrade classes can be kept clean and focus on structural and data migration. That is not to say that there will not be benefit to having them execute a lot of SQL statements to add/update stored procedures, it is just that it may get a little messy and possibly tedious.
Using C# code to do database change would provide an opportunity to use something like log4net to dump out progress information. I've always longed for a decent way of doing this that doesn't involve a custom proc that inserts into a table. I really like the idea of having a degree of compile time checking for the database changes. Of course with SQL scripts you could always run them on each build and if they break, you fail the build.
The whole idea of downgrading a database scares me because if someone accidently points an old version of an application (with automatic migration logic) to the database (think xcopy deployment) you could end up losing tables and data. Of course one should always backup a database before doing a deployment, but I’m talking about a situation where one doesn’t know it is pointing to the wrong database until it is too late. I hope these tools backup the database before applying changes. In my mind a downgrade should only be performed if a rollout fails and one has to revert to a backup taken before the upgrade was undertaken.
I really like the idea of using anonymous objects to insert data into the database (RikiMigration supports this) and avoiding the need to write insert statements.
Do you not run into potential development concurrency and structuring issues? So for example during a development cycle does one dump all changes into one class or into several?
The database vendor abstraction that is can provide is only attractive to a small number of projects. Furthermore one invariably will need to execute vendor specific statements and is best suited to a multi-vendor ORM such as NHibernate.
The approach may not fly in an organization where a DBA must review all SQL changes. Although I don’t see any reason why the frameworks couldn’t generate these for you.

I think if I was currently working on a project that used an ORM like NHibernate or LINQ to Entities, the migration frameworks would be a lot more appealing. However with other types of projects I’m worried that the additional effort isn’t really going to be worth the benefit.

Am I missing any other potential benefits (or problems)? I would really like to hear if there are other compelling reasons to adopt the approach.

Using a CTE to Allocate Tickets to Timeslots

Eden Ridgway — Sun, 27 Jan 2008 09:25:41 GMT

SQL Server Common Table Expressions (CTEs) are most often used to solve problems such as paging and hierarchical queries, however in this post I want to demonstrate another use: booking of timeslots. If you were writing a system for an attraction with predefined timeslots, say to see the damper in the Taipei 101 building, you would need to be able to distribute the allocation of tickets over these timeslots. So if a group of 16 people arrived, depending on the space available in the timeslot they may be allocated over several timeslots and would have to patiently wait until their turn came around.

The most obvious way to solve this problem is to use a looping structure that moves through the timeslots, allocates the tickets and reduces the number of tickets available for that timeslot. Unfortunately to ensure that slots are not over booked while this is going on, one has to lock the tables affected by the allocation process. This can cause performance and concurrency issues as the number of requested tickets and therefore the lock duration increases. A more efficient and elegant solution is to use the inherent looping/recursive nature of a CTE to do this allocation in one step. So instead of doing a loop, insert into select, loop again cycle, the code can simply do a single insert into select statement to block out the allocated timeslots in one implicit transaction.

To demonstrate the approach, let us take a scenario where we want to allocate 16 tickets. The number of remaining places available per timeslot are as per the diagram below.

Our simplified database would therefore have a Timeslot table that contains the time and the running available capacity for each one, like this:

Time	Available Capacity
1 Jan 2008 08:00:00	0
1 Jan 2008 08:20:00	5
1 Jan 2008 08:40:00	0
1 Jan 2008 09:00:00	7
1 Jan 2008 09:20:00	2
1 Jan 2008 09:40:00	10

We want to allocate 16 tickets across those timeslot. So the CTE we create should operate in the following manner:

The CTE to do this looks like this:

DECLARE @SeatsRequested SMALLINT

SET @SeatsRequested = 16;

WITH TimeSlotAllocation (Time, SeatsAllocated, SeatsLeftToAllocate) as
(
    --Find the first time slot to use
    SELECT TOP 1
        Time,
        dbo.MinNumber(@SeatsRequested, AvailableCapacity) as SeatsAllocated,
        @SeatsRequested - dbo.MinNumber(@SeatsRequested, AvailableCapacity) as SeatsLeftToAllocate
    FROM
        Timeslot
    WHERE
        AvailableCapacity > 0
    ORDER BY
        Time

    UNION ALL

    --Sequentially find the other timeslots that can be used to fulfill the sale request
    SELECT
        TS.Time,
        dbo.MinNumber(TSA.SeatsLeftToAllocate, TS.AvailableCapacity),
        TSA.SeatsLeftToAllocate - dbo.MinNumber(TSA.SeatsLeftToAllocate, TS.AvailableCapacity)
    FROM
        Timeslot TS

        --Join on to the next timeslot
        INNER JOIN TimeSlotAllocation TSA ON
            TS.Time = DATEADD(MINUTE, 20, TSA.Time)
    WHERE
        SeatsLeftToAllocate > 0
)

You’ll notice the code relies on a user defined function, MinNumber which simply returns the minimum value of the two supplied parameters. As you can see the logic is pretty straight forward:

Find the first available timeslot.
Allocate the lesser of the number of SeatsLeftToAllocate or the available timeslot capacity.
Store the number of tickets remaining in SeatsLeftToAllocate
Move on to the next timeslot by adding the number of minutes between timeslots.
Repeat steps 2 to 4 until there are no SeatsLeftToAllocate.

Of course the final solution would include other logic, such as bounding the allocation of timeslots by day and checking that all tickets had been allocated, but this is merely an illustration of an approach.

If you want to play around with a working example, here is the code.

Using a Common Table Expression (CTE) to Generate a Date Range

Eden Ridgway — Fri, 23 Nov 2007 06:51:14 GMT

There are times when you date reliant data that has gaps in it but you need a query that includes days even where data is missing. The data source may be a sales table where there may be days that no sales where made but you want to generate a calendar like cross-tab query for every day in a period. When I was asked by a colleague to help him with a similar problem it struck me that we could leverage a SQL 2005 CTE in a user defined function to act as a date range source for the query on which we could left join and fill in the missing days.

The solution is quite simple, all you need is a CTE that starts and the start date and recursively joins on to itself using DATEADD plus a day until it reaches the end date. Here is the function that I wrote:

CREATE FUNCTION GetDateRange
(
    @StartDate DateTime,
    @EndDate DateTime
)
RETURNS @DateRangeList TABLE
(
    Date DateTime NOT NULL
)
AS
BEGIN
    IF @StartDate > @EndDate
    BEGIN
        --Unfortunately you cannot raise an error in a UDF so simply return no rows
        RETURN
    END;

    WITH DateRange(Date) AS
    (
        SELECT
            @StartDate Date
        UNION ALL
        SELECT
            DATEADD(day, 1, Date) Date
        FROM
            DateRange
        WHERE
            Date < @EndDate
    )

    INSERT @DateRangeList
    SELECT Date
    FROM DateRange
    --You could remove Maximum Recursion level constraint by specifying a MaxRecusion of zero
    OPTION (MaxRecursion 10000);

    RETURN

END;

The performance of the query is pretty good, but is almost identical to using a temp table/table variable and a loop to generate the same data. On my P4 2.8GHZ machine at home both approaches generate a month's worth of days in 16ms and a year's worth in 30ms. I have to say I was somewhat surprised to find this out since I expected it to be faster but I suppose the inserts into the table variable negate the possible speed up. Regardless it is an interesting approach that I thought I'd share with you.

Profiler for SQL Express

Eden Ridgway — Tue, 10 Jul 2007 04:45:00 GMT

For those of you who are doing things on the cheap and only using SQL Express, you'll be glad to know that there is an open source SQL Express profiler project. As one would expect it is pretty basic, but has all the important features, like the ability to filter and select the events and columns to trace. I had a look around the code and it is pretty simple (it could do with some more comments though ), so it should be easy enough to extend should you want to.

WatiN - A .Net IE Testing Framework

Eden Ridgway — Thu, 11 Jan 2007 04:20:30 GMT

Today I discovered WatiN which is a Watir inspired testing framework for .Net that automates Internet Explorer actions. Here are some of the more notable features of the framework:

Support for frames (cross domain) and iframes
Support for popup dialogs like alert, confirm, login etc.
Support for HTML dialogs (modal and modeless)
Supports AJAX website testing.
Works with Internet Explorer 6 and Internet Explorer 7

The WatiN site has pretty good documentation documentation to get you started. Furthermore Richard Griffin ported the WATiR WebRecorder++ to create WATiN WebRecorder++ so there is a quick way of creating initial test scripts. Richard also has some great advice on test case design which I'm going to get my team to read.

This may be just what I've been looking for to promote more automated web testing at my current company .

Using GraphViz to Generate ERDs for SQL Server

Eden Ridgway — Fri, 05 Jan 2007 05:16:56 GMT

When working on a new system you usually want to be able to quickly get a simple view of the database. Usually this involves using Enterprise Manager (or Management Studio) to create a diagram and auto-arrange the tables for you. However it usually does a pretty poor job of this and any diagram you print ends up spanning multiple pages. What I wanted is something that would do a decent job of arranging an ERD for me. I decided that a good candidate for this was GraphViz since it has a variety of layout engines and the DOT file format (that it uses to generate graphs from) is really simple.

So I put together a SQL script and a batch file and generated the following graphs for Northwind.

For those of you who don’t care about the implementation details, here are the files: GraphVizGenerator.zip. The batch file assumes that osql is in your path, that you have used the default install directory for GraphViz and that you use integrated authentication to connect to your server, so you will need to change this if this is not the case. To use the batch file, run the following command:

"Generate Diagram.bat" ServerName DatabaseName

For example:

"Generate Diagram.bat" (local) Northwind

Okay, so how does it all work? Well the SQL script uses union statements to return a row per line in our DOT file, like this:

SELECT
'digraph G {'

UNION ALL

SELECT
' "' + TABLE_NAME + '"[shape=box];'
FROM
INFORMATION_SCHEMA.TABLES
WHERE
TABLE_TYPE = 'BASE TABLE' AND
TABLE_NAME NOT IN ('dtproperties', 'sysdiagrams')

UNION ALL

SELECT
' "' + FKRefTable + '" -> "' + PKTable + '";'
FROM
#Dependencies

UNION ALL

SELECT
'}'

In a step before this I use the INFORMATION_SCHEMA constraints tables to populate a temporary table (#Dependencies) with the relationships (to keep the code simple). I dump these results into a text file using the following command:

osql -S %1 -d %2 -E -n -w 1000 -h-1 -i "GraphViz Generator.sql" -o "ERD.dot"

Note that I use the –h-1 parameter to ensure that the headers are not dumped into the file. The –w 1000 also ensures that no wrapping of the columns occur.

This generates a dot file that looks something like this:

digraph G {
"Categories"[shape=box];
"CustomerCustomerDemo"[shape=box];
"CustomerDemographics"[shape=box];
"Customers"[shape=box];
"Employees"[shape=box];
"EmployeeTerritories"[shape=box];
"Order Details"[shape=box];
"Orders"[shape=box];
"Products"[shape=box];
"Region"[shape=box];
"Shippers"[shape=box];
"Suppliers"[shape=box];
"Territories"[shape=box];
"CustomerCustomerDemo" -> "CustomerDemographics";
"CustomerCustomerDemo" -> "Customers";
"Employees" -> "Employees";
"EmployeeTerritories" -> "Employees";
"EmployeeTerritories" -> "Territories";
"Order Details" -> "Orders";
"Order Details" -> "Products";
"Orders" -> "Customers";
"Orders" -> "Employees";
"Orders" -> "Shippers";
"Products" -> "Categories";
"Products" -> "Suppliers";
"Territories" -> "Region";
}

This is then used to spit out five ERDs using different layouts so that you can choose the one that you find most useful.

A SQL 2005 Managed CSV Split UDF

Tue, 17 Jan 2006 03:02:03 GMT

This morning I decided to delve into the wonders of managed code running in SQL 2005. To me it was obvious that the best candidate for this technology was a function that split a comma separated list of values and returned a table. I've used a variety of different approaches to solve this problem, ranging from a TSQL UDF that split the values to a CHARINDEX and LIKE trick. I was really hoping that a managed UDF would be orders of magnitude faster than a TSQL implementation of this sort.

So I created a SQL Server Project in Visual Studio 2005 and created the following class:

using System;
using Microsoft.SqlServer.Server;
using System.Collections;

public partial class UserDefinedFunctions
{
[SqlFunction(FillRowMethodName = "FillVarcharRow", TableDefinition = "value nvarchar(200)")]
public static IEnumerable GetCsvVarcharTable(string commaSeparatedList)
{
return commaSeparatedList.Split(new char[] { ',' });
}

public static void FillVarcharRow(object row, out string value)
{
value = (string)row;
}

}

After deploying it I tried to execute the UDF only to find out that CLR extensions are disabled by defualt. The following script got me back on my way:

sp_configure 'clr enabled', 1
GO
RECONFIGURE
GO

I plan on taking a closer look at the performance implications of using the managed UDF and I will update the post as more information becomes available.